diff --git a/.ko.yaml b/.ko.yaml index e49180e73f..7dd90c96e7 100644 --- a/.ko.yaml +++ b/.ko.yaml @@ -19,7 +19,7 @@ defaultPlatforms: - linux/arm64 baseImageOverrides: - github.com/agent-substrate/substrate/demos/sandbox: alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b + github.com/agent-substrate/substrate/demos/sandbox: alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6 # atelet needs glibc to run the node's kubelet credential provider, which # some platforms ship dynamically linked (e.g. AKS, GKE Ubuntu). github.com/agent-substrate/substrate/cmd/atelet: gcr.io/distroless/base-debian13:latest@sha256:389cad21f73e4c37b94ffe5b13736d5a92bd5bd3c6c6b38c2be1c881e14ba2bd diff --git a/README.md b/README.md index 251188fed9..3f6d4b021b 100644 --- a/README.md +++ b/README.md @@ -105,6 +105,7 @@ goals in the near term. To quickly set up the complete environment: 1. Make sure you have [Go](https://go.dev/doc/install), [`kubectl`](https://kubernetes.io/docs/tasks/tools/), and [`docker`](https://www.docker.com/) installed and configured on your dev machine. We will automatically manage other dependencies via Go, including [`kind`](https://kind.sigs.k8s.io/). + A Linux host that will run more than about a thousand sandboxes also needs a larger kernel neighbor table (`net.ipv4.neigh.default.gc_thresh1/2/3` = 4096/16384/32768); `hack/create-kind-cluster.sh` warns when the host is at the default of 1024. 2. Run the following steps: ```shell @@ -141,15 +142,6 @@ curl -X POST \ -i http://localhost:8000/ ``` -Worker capacity is versioned: the dataplane (the atelet DaemonSet and the -worker pods) schedules only on nodes that carry the -`ate.dev/substrate-version` label, and the install stamps it on every node -that exists when it runs. A node added later hosts no workers until you label -it with the installed version -(`kubectl label node ate.dev/substrate-version=`). -`kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version` -prints the installed version, off the atelet DaemonSet the install created. - ### GKE Quickstart (Development) 1. Create and configure your environment file: @@ -193,11 +185,6 @@ prints the installed version, off the atelet DaemonSet the install created. grant each atespace the namespaces it may read; see [Enable it](docs/egress-credential-injection.md#enable-it). - Nodes that GKE adds later (autoscaling, auto-repair, node upgrades) are - born with the node pool's labels, so the pool needs - `ate.dev/substrate-version` too; see - [Node version labels](tools/setup-gcp/README.md). - 5. You can then deploy the sample applications. See [demos/counter/README.md](demos/counter/README.md) or [demos/sandbox/README.md](demos/sandbox/README.md) for detailed walkthroughs. ```bash ./hack/install-ate.sh --deploy-demo-counter diff --git a/benchmarking/README.md b/benchmarking/README.md index 015964bc58..c057b4e6f3 100644 --- a/benchmarking/README.md +++ b/benchmarking/README.md @@ -53,10 +53,23 @@ convenience: ./hack/install-ate.sh --delete-benchmarks ``` -The installer accepts `--benchmark-worker-count N` (default `1`). `--skip-build` and `--worker-memory` are only available when invoking `benchmarking/deploy_locust.sh` directly. +### Settings + +`ate-setup` reads these settings in addition to those in +[docs/operator-install.md](../docs/operator-install.md#settings), through the same +flag, environment variable, and configuration document channels. +`hack/install-ate.sh` forwards them as `--benchmark-worker-count`, +`--benchmark-sandbox-class`, and `--benchmark-actor-memory`. + +| Key | Environment variable | Flag | Default | Applies to | +|--------------------------|-------------------------------|----------------------------|----------|------------------------------------------| +| `benchmark.actorMemory` | `BENCHMARK_ACTOR_MEMORY` | `--benchmark-actor-memory` | — | (global) | +| `benchmark.sandboxClass` | `ATE_BENCHMARK_SANDBOX_CLASS` | `--sandbox-class` | `gvisor` | `deploy benchmarks`, `delete benchmarks` | +| `benchmark.workerCount` | `ATE_BENCHMARK_WORKER_COUNT` | `--worker-count` | `1` | `deploy benchmarks`, `delete benchmarks` | + ## Running Tests ### Locust Web UI @@ -88,7 +101,7 @@ not a local entry point. See [automation/README.md](automation/README.md). python3 runner.py -f tests/.py -t 1m -u 1 --name --dest /tmp/bench ``` -Three flags control the optional post-run measurements described in +Four flags control the optional post-run measurements described in [Benchmark output files](#benchmark-output-files): * `--cluster-facts` / `--no-cluster-facts`: read node capacity and worker pod @@ -99,6 +112,11 @@ Three flags control the optional post-run measurements described in [Optional: Prometheus + Grafana](#optional-prometheus--grafana). * `--atelet-lag-s`: how long to wait after the run before reading the atelet's snapshot metrics. Defaults to 70. +* `--phase-breakdown` / `--no-phase-breakdown`: read the atelet and worker pod + logs once the run ends and add the suspend/resume phase percentiles to + `stats.jsonl`. On by default; needs `pods/log` access in `ate-system` and + `benchmark-workloads` (granted in `automation/manifests/runner-job.yaml.tmpl`). + See [analysis/README.md](analysis/README.md#automated-runs). Test-specific flags are appended to the same command; see the sections below. @@ -305,6 +323,14 @@ wake, the script also shows what an always-on agent costs a system that suspends it between events: an internal timer cannot fire in a suspended sandbox, so each of those turns needs an external wake. +`personal-assistant-light` is the same day with the RAM arrays scaled +down to 28Mi and everything else unchanged (files, reads, CPU, dwell and +the 61 wakes), for runs that want the always-on wake pattern with a small +memory image: about 40 to 95 MB per pause under gVisor, and under micro-VM +about 100 MB of guest before the script writes anything, bounded by the +actor's memory limit. It fits a 256Mi actor. Run it with the same +fractional think scale. + To run a script of your own without rebuilding anything, hand it to the locust deploy: @@ -364,6 +390,124 @@ The web UI shows the same fields; `0` keeps the value boomer-worker started with The `actors_per_*` ratios in `trial_summary` are wrong for this test: they count users × `--actors-per-user`, not `--total-actors`. +### ResumePaused Benchmark + +The ResumePaused benchmark drives ate-api-server's cold resume path at a set +rate. A cold resume here is a ResumeActor of a paused actor: ateapi takes the +actor lease, binds a worker, and restores the actor from its snapshot. A +ResumeActor of a running actor does none of that. Each user owns one actor +from the `glutton` template. On its first iteration the user creates the actor +and makes it cold with a resume and a pause; every later iteration resumes the +actor and pauses it again, so every measured resume is cold. + +The warm-up runs in the step that added the user, unpaced, and is recorded +under its own names. + +It sends no requests to the actors themselves, so it runs on the fake data +plane, where it measures what a cold resume costs ate-api-server and Postgres +without the router or the restore in the way. The pauses run at the resume +rate, so a capacity it finds is one of resume and pause cycles, not of resumes +alone. + +Actors are named `rp-` and deleted when boomer exits. If the atespace +cannot be ensured, boomer exits at startup. An actor that fails to warm is +deleted, and its user creates another on its next iteration. When a step lowers +the user count, the actors of the stopped users stay paused, unused, until +boomer exits. + +#### ResumePaused Configuration Knobs + +* `--target-rps`: Cold resumes per second the whole boomer process paces to + (default `0`, unpaced). A `--ladder` step ending in `@target_rps`, as in + `200:3m@1000`, changes it as the step starts. Keep the users at or above the + rate times the p99 of one resume plus one pause in seconds, or the step falls + short of its rate. + +The web UI shows the same field. + +Trim the start of each ladder step. A rate change reaches boomer at its next +poll of the master, within 10 s, and a step that adds users carries their +warm-up until they have all joined at the spawn rate. + +#### ResumePaused Reported Metrics + +* `ResumeActorCold`: Latency of each measured resume. A resume that finds its + actor already running fails, as no cold resume happened. +* `PauseActor`: Latency of the pause that makes the actor cold again. +* `PauseActorRecovery`: After a failed cycle, the pause that makes the actor + cold for the next one. It is load on the server, so it is recorded, but + apart from `PauseActor`. A recovery pause that finds the actor already + paused counts as a success. +* `CreateActor`, `ResumeActorWarmup`, `PauseActorWarmup`: A user's warm-up of + its actor, in the step that added the user. + +### Isolate Mode + +A real cold resume is limited by the node's restore capacity, which hides the +limits of ate-api-server and Postgres. Isolate mode takes the nodes out by +installing substrate with a fake data plane: + +```bash +hack/install-ate.sh --deploy-ate-system --fake-data-plane +./benchmarking/workloads/deploy.sh --deploy --worker-count 200 \ + --fake-atelet-env FAKE_ATELET_DELAY=500ms +``` + +`deploy.sh` deploys the usual WorkerPool, but the install serves it with two +fakes from `cmd/benchmarking/isolate/` instead of worker pods: + +* `fake-workersync` stands in for ate-controller's WorkerPool controller and + worker syncer. It honors each pool's `spec.replicas` with fake Workers, none + backed by a pod, round robin on the nodes that run fake-atelet, and scales them + up and down with the pool, draining a Worker before deleting it. Each Worker + reports what a real worker would: cpu and memory from the pool's + `template.resources.limits`, or the node's allocatable without one, and 1000 + actors, ateom's default, unless the pool's `ate.dev/fake-max-actors` + annotation says otherwise. It writes the pool's status, so + `kubectl get workerpool` and `kubectl scale workerpool` work. A node keeps + its Workers for `--node-grace`, five minutes by default, after its + fake-atelet pod goes, so a DaemonSet rollout does not move them; past that + they move to a node that still runs fake-atelet. +* `fake-atelet` takes atelet's place in its DaemonSet, on every node atelet + would run on, and answers every + lifecycle call with success after its `--delay`, without running or + saving any workload. It writes one placeholder object per snapshot, because + ate-api-server refuses to tag an empty one. Like ateom, it mints the actor's + certificate from ate-api-server on every run and restore that names an + egress gateway, which the default install does; the mint is ate-api-server's + time, so it adds to `--delay`, the data plane's share of the call. + It also relays fake-workersync's capacity reports, as atelet relays ateom's: + ate-api-server accepts a report only from the atelet on the Worker's node. + The relay admits only fake-workersync's pod identity. + +The install also scales ate-controller to zero, whose worker syncer would +delete the fake Workers. Its other controllers stop with it: no NetworkPolicy +is made for a WorkerPool, and the egress MITM trust bundle is not derived. +Fake actors need neither. fake-atelet takes the storage settings of atelet's +objectstore-plugin sidecar, whose writes it takes over, so its snapshot +placeholders go to the storage the install points atelet at. The install +refuses a cluster whose WorkerPools still have worker pods, which would +outlive ate-controller. A normal `hack/install-ate.sh --deploy-ate-system` +puts the real data plane back, and `--delete-ate-system` removes the fakes +either way. Never use the mode on a +cluster that serves real actors: actors placed on fake Workers do not exist. + +The installer only chooses the fake data plane; the benchmark tunes it. +fake-atelet reads each tuning flag from a `FAKE_ATELET_` variable in the +optional `fake-atelet-tuning` ConfigMap, which no install creates or resets: +`FAKE_ATELET_DELAY` sets `--delay`, `FAKE_ATELET_DELAY_RESTORE` overrides it +for Restore alone, and `FAKE_ATELET_MINT_ACTOR_CERTIFICATE=false` skips the +certificate mint. `deploy.sh --fake-atelet-env NAME=VALUE`, repeatable, writes +that ConfigMap to exactly the given entries and restarts fake-atelet when it +changes, before the WorkerPool is applied. + +Nothing answers behind the router, so the mode suits only a user class that +sends the actors no requests, such as `ate_api.py`, which creates, gets, +resumes and suspends actors. Every boomer user class pings its actors through +the router. In `tests.yaml`, an `isolate` block selects the mode, and its +keys become `deploy.sh --fake-atelet-env` entries: `delay` sets +`FAKE_ATELET_DELAY`, and so on. + ### Viewing Traces You must have enabled otel tracing for your cluster to view traces. @@ -382,7 +526,10 @@ them are checked into the repository. * `logs.txt`, `traces.txt`: the runner log, and the trace IDs seen during the run. * `stats.jsonl`: one JSON object per line, one per metric. Every row carries the same five keys: `timestamp`, `tag`, `test_name`, `metric`, and a flat - `measurements` map holding that metric's numbers. + `measurements` map holding that metric's numbers. The `phase_*` rows and + the `phase_breakdown_summary` row are the suspend/resume phase percentiles + read from the node logs (see + [analysis/README.md](analysis/README.md#automated-runs)). * `server_summary.json`: server-side telemetry harvested from Prometheus, including the per-sample bin-packing timeseries. diff --git a/benchmarking/analysis/README.md b/benchmarking/analysis/README.md index 957b0f9894..e5c0ca043c 100644 --- a/benchmarking/analysis/README.md +++ b/benchmarking/analysis/README.md @@ -12,8 +12,8 @@ Two joinable JSON log records feed it, each written by both layers: | record (`msg`) | emitter | keys | |---|---|---| -| `Restore timing breakdown` | atelet and ateom-microvm | `ate.actor.restore.duration.` / `ateom.actor.restore.duration.` | -| `Checkpoint timing breakdown` | atelet and ateom-microvm | `ate.actor.checkpoint.duration.` / `ateom.actor.checkpoint.duration.` | +| `Restore timing breakdown` | atelet, ateom-microvm and ateom-gvisor | `ate.actor.restore.duration.` / `ateom.actor.restore.duration.` | +| `Checkpoint timing breakdown` | atelet, ateom-microvm and ateom-gvisor | `ate.actor.checkpoint.duration.` / `ateom.actor.checkpoint.duration.` | Every record carries the full actor identity (`ate.actor.uid`, name, atespace, template) and the snapshot scope, which the histograms are barred @@ -74,13 +74,18 @@ A failed atelet operation still writes its record, marked with `error.type` **Phase percentiles.** Per layer, operation, sandbox class, snapshot kind, scope and phase: count, p50, p90, p95 and max. A `golden` restore downloads the golden image and a `latest` one the actor's own, so they are separate -rows, as are gVisor and micro-VM checkpoints. The atelet rows split a checkpoint between -`sandbox_assets`, `ateom_checkpoint` and `persist`, and a restore between -`volume_mount`, `manifest_fetch`, `sandbox_assets`, `download`, `oci_unpack` -and `ateom_restore`. The ateom rows split the `ateom_*` phase further: +rows, as are gVisor and micro-VM operations. The atelet rows split a +checkpoint between `sandbox_assets`, `ateom_checkpoint` and `persist`, and a +restore between `volume_mount`, `manifest_fetch`, `sandbox_assets`, +`download`, `oci_unpack` and `ateom_restore`. The ateom rows split the +`ateom_*` phase further with the runtime's own phases: for micro-VM `prep` / `pause` / `snapshot` / `durable_dir` / `rootfs_upper` / `teardown` -for a checkpoint, `prep` / `bundles` / `upper_join` / `lowers` / `tap` / -`vmm_launch` / `vm_restore` / `resume` / `wakeup_probe` for a restore. +on a checkpoint and `prep` / … / `vm_restore` / `wakeup_probe` on a restore; +for gVisor `pause` / `checkpoint` / `durable_dir` / `resume` / `teardown` and +`prep` / `net_setup` / … / `pause_restore` / `app_restore` / `wakeup_probe` / +`activate` (the full lists are in `cmd/ateom-*/phaselog.go`). The ateom +records carry no sandbox class or kind of their own; a paired one takes both +from its atelet record, so both layers split into the same rows. Concurrency matters when reading them: the atelet restore phases overlap (the download runs alongside the asset fetch and OCI unpack), the three @@ -107,6 +112,27 @@ every time are systematic; different phases each time are environmental. The parser is deliberately tolerant: it scans any line for a JSON object and matches on `msg`, so raw `kubectl logs` dumps (even with prefixes) work. +## Automated runs + +The locust runner (`benchmarking/locust/runner.py`) does the collection +itself at the end of a headless run: `phase_breakdown.py` reads the atelet +and worker pod logs through the Kubernetes API while the pods still exist +(the orchestrator deletes them as soon as the runner exits) and appends the +percentiles to the run's `stats.jsonl`, one row per layer / operation / +class / kind / scope / phase (`metric: phase______`; +the measurements map carries those dimensions as fields next to `count`, +`p50_ms`, `p90_ms`, `p95_ms`, `max_ms`, all as strings like the runner's +other rows). A `phase_breakdown_summary` row is always written: pods read +and failed, the record count, and the atelet record counts next to locust's +`SuspendActor` / `ResumeActor` request and failure counts. More records than +requests is normal (boomer suspends its actors on shutdown); fewer records +than successful requests means records were lost, to pods that could not be +read or to a node log rotated during the run, and the runner's log says +which. `--no-phase-breakdown` +skips it. The runner's service account needs `list` on `pods` and `get` on +`pods/log` in `ate-system` and `benchmark-workloads` +(`automation/manifests/runner-job.yaml.tmpl`). + ## Tests ```bash diff --git a/benchmarking/analysis/phase_report.py b/benchmarking/analysis/phase_report.py index 962046f793..3d0e4d2f92 100755 --- a/benchmarking/analysis/phase_report.py +++ b/benchmarking/analysis/phase_report.py @@ -50,6 +50,7 @@ import statistics import sys from collections import defaultdict +from collections.abc import Iterable from dataclasses import dataclass, field from datetime import datetime from pathlib import Path @@ -73,9 +74,11 @@ TEMPLATE_KEY = "ate.template.name" ERROR_TYPE_KEY = "error.type" -# Only ateom-microvm writes the ateom records, and they carry no sandbox -# class of their own; the atelet record of the same operation does. -ATEOM_SANDBOX_CLASS = "microvm" +# The ateom records carry no sandbox class or snapshot kind of their own; +# both ateoms write the same two messages, and the atelet record of the same +# operation says which runtime it was. join_layers copies them across. +MICROVM = "microvm" +GVISOR = "gvisor" # The phase every record reports, and the one the report derives: the part of # the total no logged phase accounts for. @@ -88,11 +91,14 @@ # atelet restore "volume_mount", "manifest_fetch", "sandbox_assets", "download", "oci_unpack", "ateom_restore", - # ateom restore - "prep", "bundles", "upper_join", "lowers", "tap", "vmm_launch", - "vm_restore", "resume", "wakeup_probe", + # ateom restore: micro-VM (cmd/ateom-microvm/phaselog.go) and gVisor + # (cmd/ateom-gvisor/phaselog.go), each in its own sequential order + "prep", "egress_prepare", "net_setup", "bundles", "upper_join", "lowers", + "pause_rootfs", "pause_create", "pause_restore", + "app_rootfs", "app_create", "app_restore", + "tap", "vmm_launch", "vm_restore", "resume", "wakeup_probe", "activate", # atelet + ateom checkpoint - "pause", "snapshot", "durable_dir", "rootfs_upper", + "pause", "checkpoint", "snapshot", "durable_dir", "rootfs_upper", "ateom_checkpoint", "teardown", "persist", UNATTRIBUTED, TOTAL, ] @@ -101,8 +107,9 @@ # Which atelet phase wraps the ateom record. INNER_PHASE = {"restore": "ateom_restore", "checkpoint": "ateom_checkpoint"} -# The ateom checkpoint captures that run concurrently on the paused guest: the -# paused window costs their max, not their sum. +# The micro-VM checkpoint captures that run concurrently on the paused guest: +# the paused window costs their max, not their sum. gVisor's checkpoint phases +# are sequential. CONCURRENT_CAPTURES = ("snapshot", "durable_dir", "rootfs_upper") # Slack on the window in which the ateom record of an operation must land: @@ -155,24 +162,28 @@ def parse_time(s: str) -> float | None: return None -def unattributed(source: str, op: str, phases: dict[str, float]) -> float | None: +def unattributed(source: str, op: str, sandbox_class: str, phases: dict[str, float]) -> float | None: """The total minus what the logged phases account for, where that is - well-defined: the atelet checkpoint phases are sequential; the ateom - checkpoint is prep, pause, the concurrent captures (max), then teardown. - The atelet restore phases overlap by design (download runs alongside the - asset fetch and OCI unpack) and the ateom restore phases partition their - total by construction, so neither gets a residual. The phases are - sub-intervals of the total, so a negative result is a bug in the emitter - or in this formula; it is not clamped, so that it shows.""" + well-defined: the atelet checkpoint phases are sequential; the micro-VM + ateom checkpoint is prep, pause, the concurrent captures (max), then + teardown; the gVisor ateom checkpoint phases are all sequential. The + atelet restore phases overlap by design (download runs alongside the + asset fetch and OCI unpack) and both ateoms' restore phases partition + their total by construction, so neither gets a residual; nor does an + ateom checkpoint whose runtime is unknown (its atelet record is missing). + The phases are sub-intervals of the total, so a negative result is a bug + in the emitter or in this formula; it is not clamped, so that it shows.""" total = phases.get(TOTAL) if total is None: return None if (source, op) == ("atelet", "checkpoint"): spent = sum(phases.get(p, 0.0) for p in ("sandbox_assets", "ateom_checkpoint", "persist")) - elif (source, op) == ("ateom", "checkpoint"): + elif (source, op, sandbox_class) == ("ateom", "checkpoint", MICROVM): spent = (phases.get("prep", 0.0) + phases.get("pause", 0.0) + max(phases.get(p, 0.0) for p in CONCURRENT_CAPTURES) + phases.get("teardown", 0.0)) + elif (source, op, sandbox_class) == ("ateom", "checkpoint", GVISOR): + spent = sum(v for k, v in phases.items() if k != TOTAL) else: return None return total - spent @@ -200,9 +211,6 @@ def parse_line(obj: dict, out: Parsed) -> None: if not phases: continue time_s = obj.get("time", "") - residual = unattributed(source, op, phases) - if residual is not None: - phases[UNATTRIBUTED] = residual out.breakdowns.append(Breakdown( source=source, op=op, @@ -210,8 +218,7 @@ def parse_line(obj: dict, out: Parsed) -> None: ts=parse_time(time_s), actor_uid=obj.get(ACTOR_UID_KEY, ""), template=obj.get(TEMPLATE_KEY, ""), - sandbox_class=obj.get(SANDBOX_CLASS_KEY, "") - or (ATEOM_SANDBOX_CLASS if source == "ateom" else ""), + sandbox_class=obj.get(SANDBOX_CLASS_KEY, ""), scope=obj.get(SCOPE_KEY, ""), kind=obj.get(KIND_KEY, ""), phases=phases, @@ -221,6 +228,27 @@ def parse_line(obj: dict, out: Parsed) -> None: return +def parse_lines(lines: Iterable[str], out: Parsed | None = None) -> Parsed: + """Parse one pod log, line by line, into out (a new Parsed by default). + This is also the entry point for a caller that already holds the log + text, such as the locust runner reading pods through the Kubernetes API.""" + out = Parsed() if out is None else out + for line in lines: + # kubectl log dumps may prefix each line (pod name, timestamp); + # recover the JSON object from the first brace. + brace = line.find("{") + if brace < 0: + continue + out.lines_seen += 1 + try: + obj = json.loads(line[brace:]) + except json.JSONDecodeError: + continue + if isinstance(obj, dict): + parse_line(obj, out) + return out + + def parse_files(paths: list[str]) -> Parsed: out = Parsed() for path in paths: @@ -245,19 +273,7 @@ def parse_files(paths: list[str]) -> Parsed: out.lines_seen += 1 parse_line(obj, out) continue - for line in text.splitlines(): - # kubectl log dumps may prefix each line (pod name, timestamp); - # recover the JSON object from the first brace. - brace = line.find("{") - if brace < 0: - continue - out.lines_seen += 1 - try: - obj = json.loads(line[brace:]) - except json.JSONDecodeError: - continue - if isinstance(obj, dict): - parse_line(obj, out) + parse_lines(text.splitlines(), out) return out @@ -341,7 +357,8 @@ def pair_records(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: operation without a match prints without one. Matching in time order with a consumed set keeps rapid cycles of one actor from sharing or swapping records. A paired ateom record also inherits the snapshot kind - its layer does not log, so the two layers' percentiles split alike.""" + and sandbox class its layer does not log, so the two layers' percentiles + split alike and the residual knows which runtime's rule applies.""" by_actor: dict[tuple, list[Breakdown]] = defaultdict(list) for b in breakdowns: if b.source == "ateom" and b.ts is not None: @@ -367,6 +384,20 @@ def pair_records(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: pairs[id(op)] = inner if not inner.kind: inner.kind = op.kind + if not inner.sandbox_class: + inner.sandbox_class = op.sandbox_class + return pairs + + +def join_layers(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: + """Pair the layers' records, then derive each record's unattributed + residual, which for an ateom checkpoint depends on the runtime the pairing + just identified. Returns the pairs, as pair_records does.""" + pairs = pair_records(breakdowns) + for b in breakdowns: + residual = unattributed(b.source, b.op, b.sandbox_class, b.phases) + if residual is not None: + b.phases[UNATTRIBUTED] = residual return pairs @@ -376,7 +407,7 @@ def report_waterfalls(breakdowns: list[Breakdown], writer, slowest: int, ateom_* phase, and the gap between that phase and the ateom total (RPC and queueing between the layers).""" if pairs is None: - pairs = pair_records(breakdowns) + pairs = join_layers(breakdowns) atelet = [b for b in breakdowns if b.source == "atelet" and not b.failed] atelet.sort(key=lambda b: b.phases.get(TOTAL, 0), reverse=True) @@ -402,6 +433,30 @@ def report_waterfalls(breakdowns: list[Breakdown], writer, slowest: int, writer(f" (gap) {'rpc/queueing':13} {fmt_s(gap)}") +def stats_rows(rows: list[dict], timestamp: str, tag: str, test_name: str) -> list[dict]: + """report_phases() rows in the shape of the benchmark runner's stats.jsonl + (benchmarking/locust/runner.py): one object per metric with the run's + timestamp, tag and test name, and a flat, string-valued measurements map, + as the runner's other writers emit. The metric name carries the row's + dimensions so a dashboard can select a phase the way it selects a locust + request type; the dimensions are also fields of the map, since phase + names contain underscores and the name cannot be split back apart.""" + out = [] + for r in rows: + dims = {k: (r[k] if r[k] and r[k] != "-" else "none") + for k in ("layer", "op", "class", "kind", "scope", "phase")} + metric = "_".join(("phase", *dims.values())) + measurements = {**dims, **{k: r[k] for k in ("count", "p50_ms", "p90_ms", "p95_ms", "max_ms")}} + out.append({ + "timestamp": timestamp, + "tag": tag, + "test_name": test_name, + "metric": re.sub(r"[^a-z0-9_]+", "_", metric.lower()), + "measurements": {k: str(v) for k, v in measurements.items()}, + }) + return out + + def write_csv(dest: Path, name: str, rows: list[dict]) -> None: if not rows: return @@ -434,9 +489,10 @@ def main() -> int: print("no matching records; are these the atelet and worker pod logs?", file=sys.stderr) return 1 - # Pair before the percentiles: a paired ateom record takes its kind from - # the atelet record, so both layers' rows split the same way. - pairs = pair_records(parsed.breakdowns) + # Pair before the percentiles: a paired ateom record takes its kind and + # sandbox class from the atelet record, so both layers' rows split the + # same way and each checkpoint's residual uses its runtime's rule. + pairs = join_layers(parsed.breakdowns) print() phase_rows = report_phases(parsed.breakdowns, print) report_waterfalls(parsed.breakdowns, print, args.slowest, pairs) diff --git a/benchmarking/analysis/test_phase_report.py b/benchmarking/analysis/test_phase_report.py index 45e61169d6..0ad518f77b 100644 --- a/benchmarking/analysis/test_phase_report.py +++ b/benchmarking/analysis/test_phase_report.py @@ -162,11 +162,19 @@ def test_cloud_logging_export_file_is_a_json_array(self): out = phase_report.parse_files([path]) self.assertEqual((out.lines_seen, len(out.breakdowns)), (2, 2)) - def test_ateom_records_default_to_the_microvm_class(self): - out = parse(ATEOM_RESTORE, ATELET_RESTORE) - self.assertEqual(out.breakdowns[0].sandbox_class, "microvm") - self.assertEqual(out.breakdowns[1].sandbox_class, "") # not on this fixture - + def test_ateom_records_take_class_and_kind_from_the_paired_atelet_record(self): + atelet = dict(ATELET_RESTORE, **{"ate.sandbox.class": "gvisor"}) + out = parse(ATEOM_RESTORE, atelet) + self.assertEqual(out.breakdowns[0].sandbox_class, "") # not on the ateom record + phase_report.join_layers(out.breakdowns) + self.assertEqual((out.breakdowns[0].sandbox_class, out.breakdowns[0].kind), ("gvisor", "latest")) + + def test_an_unpaired_ateom_record_keeps_no_class(self): + out = parse(ATEOM_RESTORE) + phase_report.join_layers(out.breakdowns) + self.assertEqual(out.breakdowns[0].sandbox_class, "") + rows = phase_report.report_phases(out.breakdowns, lambda _: None) + self.assertEqual({r["class"] for r in rows}, {"-"}) def test_non_numeric_duration_is_skipped_not_fatal(self): rec = dict(ATELET_RESTORE, **{"ate.actor.restore.duration.download": None, "ate.actor.restore.duration.oci_unpack": "fast"}) @@ -201,6 +209,12 @@ def test_bracket_prefixed_dump_is_read_line_by_line(self): self.assertEqual(len(out.breakdowns), 2) self.assertIn("reading it line by line", err.getvalue()) + def test_parse_lines_feeds_an_existing_parse(self): + out = phase_report.parse_lines([json.dumps(ATELET_RESTORE), "not json", ""]) + self.assertEqual((out.lines_seen, len(out.breakdowns)), (1, 1)) + phase_report.parse_lines(["pod/x " + json.dumps(ATEOM_RESTORE)], out) + self.assertEqual(len(out.breakdowns), 2) + def test_prefixed_kubectl_lines_still_parse(self): with tempfile.TemporaryDirectory() as d: path = os.path.join(d, "pod.log") @@ -213,25 +227,58 @@ def test_prefixed_kubectl_lines_still_parse(self): class UnattributedTest(unittest.TestCase): def test_atelet_checkpoint_residual_is_total_minus_sequential_phases(self): - b = parse(ATELET_CHECKPOINT).breakdowns[0] - self.assertAlmostEqual(b.phases["unattributed"], 6.0 - (0.01 + 1.14 + 4.48)) - - def test_ateom_checkpoint_counts_the_concurrent_captures_once(self): - b = parse(ATEOM_CHECKPOINT).breakdowns[0] + out = parse(ATELET_CHECKPOINT) + phase_report.join_layers(out.breakdowns) + self.assertAlmostEqual(out.breakdowns[0].phases["unattributed"], 6.0 - (0.01 + 1.14 + 4.48)) + def test_microvm_ateom_checkpoint_counts_the_concurrent_captures_once(self): # prep + pause + max(snapshot, rootfs_upper) + teardown; the two # captures overlapped, so only the slower one is spent wall time. - self.assertAlmostEqual(b.phases["unattributed"], 1.2 - (0.04 + 0.01 + 0.9 + 0.2)) - + phases = {k[len("ateom.actor.checkpoint.duration."):]: v + for k, v in ATEOM_CHECKPOINT.items() if k.startswith("ateom.actor.checkpoint.duration.")} + got = phase_report.unattributed("ateom", "checkpoint", "microvm", phases) + self.assertAlmostEqual(got, 1.2 - (0.04 + 0.01 + 0.9 + 0.2)) + + def test_gvisor_ateom_checkpoint_phases_are_sequential(self): + phases = {"prep": 0.01, "pause": 0.02, "checkpoint": 0.15, "durable_dir": 0.03, + "resume": 0.02, "teardown": 0.05, "total": 0.30} + got = phase_report.unattributed("ateom", "checkpoint", "gvisor", phases) + self.assertAlmostEqual(got, 0.30 - (0.01 + 0.02 + 0.15 + 0.03 + 0.02 + 0.05)) + + def test_ateom_checkpoint_of_unknown_runtime_gets_no_residual(self): + out = parse(ATEOM_CHECKPOINT) # no atelet partner, so no class + phase_report.join_layers(out.breakdowns) + self.assertNotIn("unattributed", out.breakdowns[0].phases) + + def test_gvisor_pair_splits_both_layers_by_class(self): + atelet = dict(ATELET_CHECKPOINT, **{"ate.sandbox.class": "gvisor"}) + ateom = { + "time": "2026-09-23T10:00:55.000000000Z", "msg": "Checkpoint timing breakdown", + "ate.actor.uid": "uid-1", "ate.template.name": "swebench-astropy-7336", "ate.snapshot.scope": "full", + "ateom.actor.checkpoint.duration.pause": 0.02, + "ateom.actor.checkpoint.duration.checkpoint": 0.15, + "ateom.actor.checkpoint.duration.teardown": 0.05, + "ateom.actor.checkpoint.duration.total": 0.25, + } + out = parse(ateom, atelet) + phase_report.join_layers(out.breakdowns) + rows = phase_report.report_phases(out.breakdowns, lambda _: None) + ateom_rows = {r["phase"]: r for r in rows if r["layer"] == "ateom"} + self.assertEqual({r["class"] for r in ateom_rows.values()}, {"gvisor"}) + self.assertEqual({r["kind"] for r in ateom_rows.values()}, {"latest"}) + self.assertAlmostEqual(ateom_rows["unattributed"]["p50_ms"], (0.25 - 0.22) * 1000) + # gVisor's phases sort in their own order, before the shared tail. + order = [r["phase"] for r in rows if r["layer"] == "ateom"] + self.assertEqual(order, ["pause", "checkpoint", "teardown", "unattributed", "total"]) def test_restore_layers_get_no_residual(self): out = parse(ATELET_RESTORE, ATEOM_RESTORE) + phase_report.join_layers(out.breakdowns) for b in out.breakdowns: self.assertNotIn("unattributed", b.phases) - def test_negative_residual_is_not_clamped(self): # Cannot happen on a well-formed record; if it does, it must show. - rec = dict(ATELET_CHECKPOINT, **{"ate.actor.checkpoint.duration.total": 1.0}) - self.assertAlmostEqual(parse(rec).breakdowns[0].phases["unattributed"], 1.0 - (0.01 + 1.14 + 4.48)) - + phases = {"sandbox_assets": 0.01, "ateom_checkpoint": 1.14, "persist": 4.48, "total": 1.0} + self.assertAlmostEqual(phase_report.unattributed("atelet", "checkpoint", "microvm", phases), + 1.0 - (0.01 + 1.14 + 4.48)) class ReportTest(unittest.TestCase): def test_failed_records_are_flagged_and_excluded_from_percentiles(self): @@ -285,6 +332,19 @@ def test_waterfall_ignores_an_ateom_record_from_another_cycle(self): self.assertNotIn("ateom vm_restore", text) self.assertNotIn("(gap)", text) + def test_stats_rows_take_the_runner_jsonl_shape(self): + rows = phase_report.report_phases(parse(ATELET_RESTORE).breakdowns, lambda _: None) + entries = phase_report.stats_rows(rows, "2026-09-23T10:00:00Z", "abc123", "glutton_mem_1gi") + download = next(e for e in entries if e["metric"].endswith("_download")) + self.assertEqual(download["metric"], "phase_atelet_restore_none_latest_full_download") + self.assertEqual((download["timestamp"], download["tag"], download["test_name"]), + ("2026-09-23T10:00:00Z", "abc123", "glutton_mem_1gi")) + # String values and the dimensions as fields, like the runner's rows. + self.assertEqual(download["measurements"], { + "layer": "atelet", "op": "restore", "class": "none", "kind": "latest", + "scope": "full", "phase": "download", + "count": "1", "p50_ms": "2400.0", "p90_ms": "2400.0", "p95_ms": "2400.0", "max_ms": "2400.0"}) + def test_waterfall_nests_ateom_and_gap_under_atelet(self): out = parse(ATEOM_RESTORE, ATELET_RESTORE) text = render(phase_report.report_waterfalls, out.breakdowns, slowest=1) diff --git a/benchmarking/automation/manifests/runner-job.yaml.tmpl b/benchmarking/automation/manifests/runner-job.yaml.tmpl index 4ec02c7bb0..9267de1219 100644 --- a/benchmarking/automation/manifests/runner-job.yaml.tmpl +++ b/benchmarking/automation/manifests/runner-job.yaml.tmpl @@ -72,7 +72,40 @@ metadata: rules: - apiGroups: [""] resources: ["pods"] - verbs: ["list"] + # get as well as list: the runner reads the worker pod logs after a test + # for the suspend/resume phase breakdown (phase_breakdown.py). + verbs: ["get", "list"] +- apiGroups: [""] + resources: ["pods/log"] + verbs: ["get"] +--- +# The atelet pod logs hold the other half of the phase breakdown. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: benchmark-runner-atelet-logs + namespace: ate-system +rules: +- apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list"] +- apiGroups: [""] + resources: ["pods/log"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: benchmark-runner-atelet-logs + namespace: ate-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: benchmark-runner-atelet-logs +subjects: +- kind: ServiceAccount + name: benchmark-runner + namespace: benchmarking --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding diff --git a/benchmarking/automation/orchestrator.py b/benchmarking/automation/orchestrator.py index 9ae216539a..2539b2e33a 100644 --- a/benchmarking/automation/orchestrator.py +++ b/benchmarking/automation/orchestrator.py @@ -53,7 +53,14 @@ import yaml from testtypes import TYPES -from util import parse_duration_seconds, run, run_no_check +from util import ( + isolate_block, + isolate_install_args, + isolate_workload_args, + parse_duration_seconds, + run, + run_no_check, +) SUBSTRATE_DIR = "/workspace/substrate" @@ -319,6 +326,15 @@ def validate_and_normalize_tests(tests: list[dict[str, Any]]) -> None: f"(want one of {list(TEST_TYPES)})" ) TYPES[ttype].validate(t) + isolate_block(t) + + +def substrate_args(test: dict[str, Any]) -> list[str]: + """Return the hack/install-ate.sh --deploy-ate-system arguments for a + tests.yaml entry: its ateArgs, then the fake data plane flag its isolate + block asks for. The teardown needs no matching flag: --delete-ate-system + removes the fake data plane whether or not it was installed.""" + return [*test.get("ateArgs", []), *isolate_install_args(test)] def deploy_substrate(ate_args: Iterable[str] = ()) -> None: @@ -349,6 +365,7 @@ def deploy_workloads( actor_memory: str = "", wait_timeout_secs: int | str = "", worker_memory: str = "", + fake_atelet_args: Iterable[str] = (), ) -> None: cmd = [ "benchmarking/workloads/deploy.sh", @@ -370,6 +387,8 @@ def deploy_workloads( # (whole seconds). if wait_timeout_secs != "": cmd += ["--wait-timeout", str(wait_timeout_secs)] + # An isolate block's tuning of fake-atelet, on the fake data plane. + cmd += list(fake_atelet_args) # deploy.sh itself blocks until every template's golden snapshot exists. run(cmd) @@ -534,7 +553,7 @@ def main() -> None: failure_msg = None start_time = time.time() try: - deploy_substrate(test.get("ateArgs", [])) + deploy_substrate(substrate_args(test)) TYPES[ttype].pre_test(test) # install-microvm-deps needs the CRDs from deploy_substrate; # deploy_workloads needs the microvm SandboxConfig. @@ -546,6 +565,7 @@ def main() -> None: test.get("actorMemory", ""), test.get("workerWaitTimeout", ""), test.get("workerMemory", ""), + isolate_workload_args(test), ) try: status = run_test( diff --git a/benchmarking/automation/test_orchestrator.py b/benchmarking/automation/test_orchestrator.py index 952685a82c..d5fea10c9f 100644 --- a/benchmarking/automation/test_orchestrator.py +++ b/benchmarking/automation/test_orchestrator.py @@ -24,6 +24,41 @@ from testtypes import locust +class DeployWorkloadsFakeAteletTest(unittest.TestCase): + @mock.patch("orchestrator.run") + def test_fake_atelet_tuning_goes_to_deploy_sh(self, run): + orchestrator.deploy_workloads( + fake_atelet_args=["--fake-atelet-env", "FAKE_ATELET_DELAY=500ms"], + ) + cmd = run.call_args.args[0] + i = cmd.index("--fake-atelet-env") + self.assertEqual(cmd[i + 1], "FAKE_ATELET_DELAY=500ms") + + +class SubstrateArgsTest(unittest.TestCase): + def test_ate_args_only(self): + self.assertEqual( + orchestrator.substrate_args({"ateArgs": ["--cluster-size=size10"]}), + ["--cluster-size=size10"], + ) + + def test_isolate_installs_the_fake_data_plane(self): + self.assertEqual( + orchestrator.substrate_args({ + "ateArgs": ["--cluster-size=size10"], + "isolate": {"delay": "500ms"}, + }), + # The delay tunes fake-atelet through deploy.sh, not the install. + ["--cluster-size=size10", "--fake-data-plane"], + ) + + def test_neither(self): + self.assertEqual(orchestrator.substrate_args({}), []) + + def test_bare_isolate_key(self): + self.assertEqual(orchestrator.substrate_args({"isolate": None}), ["--fake-data-plane"]) + + class DeployWorkloadsTest(unittest.TestCase): @mock.patch("orchestrator.run") def test_defaults_omit_optional_flags(self, run): diff --git a/benchmarking/automation/test_util.py b/benchmarking/automation/test_util.py index ed300a5aae..0b24f2fbe9 100644 --- a/benchmarking/automation/test_util.py +++ b/benchmarking/automation/test_util.py @@ -17,6 +17,8 @@ import unittest from unittest import mock +import yaml + import util @@ -51,5 +53,64 @@ def test_builds_amd64_and_pushes(self, run_mock): self.assertEqual(push_cmd, ["docker", "push", "gcr.io/p/repo/img:tag"]) +class IsolateTest(unittest.TestCase): + def test_absent(self): + test = {"name": "t"} + self.assertIsNone(util.isolate_block(test)) + self.assertEqual(util.isolate_install_args(test), []) + self.assertEqual(util.isolate_workload_args(test), []) + + def test_empty_block_selects_the_mode_with_defaults(self): + test = {"isolate": {}} + self.assertEqual(util.isolate_install_args(test), ["--fake-data-plane"]) + self.assertEqual(util.isolate_workload_args(test), []) + + def test_bare_key_selects_the_mode(self): + # `isolate:` with nothing after it, as YAML loads it. + test = yaml.safe_load("name: t\nisolate:\n") + self.assertEqual(util.isolate_install_args(test), ["--fake-data-plane"]) + self.assertEqual(util.isolate_workload_args(test), []) + + def test_tuning_goes_to_deploy_sh_not_the_install(self): + test = {"isolate": { + "delay": "500ms", + "delayRestore": "2s", + "mintActorCertificate": False, + }} + self.assertEqual(util.isolate_install_args(test), ["--fake-data-plane"]) + self.assertEqual( + util.isolate_workload_args(test), + [ + "--fake-atelet-env", "FAKE_ATELET_DELAY=500ms", + "--fake-atelet-env", "FAKE_ATELET_DELAY_RESTORE=2s", + "--fake-atelet-env", "FAKE_ATELET_MINT_ACTOR_CERTIFICATE=false", + ], + ) + + def test_durations(self): + for delay in ("0", "0s", "1.5s", "1m30s", "250ms", "100us"): + with self.subTest(delay=delay): + self.assertEqual( + util.isolate_workload_args({"isolate": {"delayCheckpoint": delay}}), + ["--fake-atelet-env", f"FAKE_ATELET_DELAY_CHECKPOINT={delay}"], + ) + + def test_invalid(self): + for block in ( + # Every fake-atelet node serves the pool; there is no node count. + {"nodes": 4}, + {"run": "nightly"}, + {"delay": "fast"}, + {"delay": "500"}, + {"delay": "-1s"}, + {"delay": 0}, + {"delayRun": "soon"}, + {"mintActorCertificate": "no"}, + ["nodes"], + ): + with self.subTest(block=block), self.assertRaises(ValueError): + util.isolate_block({"isolate": block}) + + if __name__ == "__main__": unittest.main() diff --git a/benchmarking/automation/tests.yaml b/benchmarking/automation/tests.yaml index 325d273cae..d254ed1223 100644 --- a/benchmarking/automation/tests.yaml +++ b/benchmarking/automation/tests.yaml @@ -54,6 +54,22 @@ # as one YAML string, '--credential-provider={"name":"k8s.io"}' # (no shell quoting: the args are passed without a shell), # or as ATE_CREDENTIAL_PROVIDER in .ate-dev-env.sh. +# isolate Optional. Isolate mode: installs substrate with the fake +# data plane (hack/install-ate.sh --fake-data-plane), which +# serves the WorkerPool without worker pods: fake-atelet +# takes atelet's place on every atelet node and answers each +# lifecycle call after a delay, and fake-workersync honors +# the pool with workerCount fake Workers on them, each +# reporting the capacity a real worker would. +# ate-api-server and Postgres can then be driven past what +# real nodes restore. The keys tune fake-atelet, through +# workloads/deploy.sh --fake-atelet-env: delay (default 0s), +# delayRun, delayRestore, delayCheckpoint, +# delayPromoteSnapshot, and delayTerminate, which +# override delay for one call, and mintActorCertificate +# (default true). A bare `isolate:` selects the mode with +# the defaults. Actors do not exist, so the entry's user +# class must send them no requests. # # ---- type: locust ----------------------------------------------------------- # @@ -85,6 +101,13 @@ # `--trace-probability` in `flags` is the rate before the first such step. # Thus a ladder with no probability in any step keeps the rate of the flag. # +# A step can end in `@target_rps` (e.g. `200:3m@4000`), which caps the +# requests per second of the whole boomer worker from the step's start; 0 +# removes the cap. Boomer users that pace themselves (resumepaused) honor it. +# Keep `users` at or above the rate times the p99 of one user iteration in +# seconds, or the step falls short of its rate. A resumepaused iteration is a +# resume plus a pause, so size it by the p99 of both together. +# # ---- type: nighthawk-ingress -------------------------------------------------------- # # Runs the Nighthawk adaptive router-capacity benchmark. `duration` is the diff --git a/benchmarking/automation/util.py b/benchmarking/automation/util.py index 52f0df5841..ad6a183906 100644 --- a/benchmarking/automation/util.py +++ b/benchmarking/automation/util.py @@ -65,3 +65,71 @@ def parse_duration_seconds(s: str) -> int: n = int(m.group(1)) unit = m.group(2) or "s" return n * {"s": 1, "m": 60, "h": 3600}[unit] + + +# The keys of a tests.yaml isolate block and the fake-atelet tuning +# variable each one sets, through deploy.sh --fake-atelet-env. +ISOLATE_TUNING = { + "delay": "FAKE_ATELET_DELAY", + "delayRun": "FAKE_ATELET_DELAY_RUN", + "delayRestore": "FAKE_ATELET_DELAY_RESTORE", + "delayCheckpoint": "FAKE_ATELET_DELAY_CHECKPOINT", + "delayPromoteSnapshot": "FAKE_ATELET_DELAY_PROMOTE_SNAPSHOT", + "delayTerminate": "FAKE_ATELET_DELAY_TERMINATE", + "mintActorCertificate": "FAKE_ATELET_MINT_ACTOR_CERTIFICATE", +} + + +# A non-negative Go duration, as fake-atelet parses its delays. +ISOLATE_DELAY_PATTERN = re.compile(r"0|([0-9]+(\.[0-9]+)?(ns|us|ms|s|m|h))+") + + +def isolate_block(test: dict) -> dict | None: + """Return a tests.yaml entry's isolate block, or None when the entry has + none. A bare `isolate:` selects the mode with every default. Raises + ValueError on an unknown key or a value fake-atelet would reject, so a bad + entry fails before any cluster work rather than in a crash-looping + fake-atelet.""" + if "isolate" not in test: + return None + # YAML loads a key with no value as None. + block = test["isolate"] + if block is None: + block = {} + if not isinstance(block, dict): + raise ValueError(f"isolate must be a mapping, got {block!r}") + unknown = sorted(set(block) - set(ISOLATE_TUNING)) + if unknown: + raise ValueError( + f"isolate has unknown keys {unknown} " + f"(want some of {sorted(ISOLATE_TUNING)})" + ) + for key, value in block.items(): + if key == "mintActorCertificate": + if not isinstance(value, bool): + raise ValueError(f"isolate.{key} must be true or false, got {value!r}") + elif not isinstance(value, str) or not ISOLATE_DELAY_PATTERN.fullmatch(value): + raise ValueError(f"isolate.{key} must be a duration string such as 500ms, got {value!r}") + return block + + +def isolate_install_args(test: dict) -> list[str]: + """Return the hack/install-ate.sh flags a tests.yaml entry's isolate block + installs the fake data plane with, or none without a block. The installer + only chooses the data plane; the block's tuning goes to deploy.sh.""" + return [] if isolate_block(test) is None else ["--fake-data-plane"] + + +def isolate_workload_args(test: dict) -> list[str]: + """Return the benchmarking/workloads/deploy.sh flags that tune fake-atelet + for a tests.yaml entry's isolate block, or none without a block.""" + block = isolate_block(test) or {} + args = [] + for key, env in ISOLATE_TUNING.items(): + if key not in block: + continue + value = block[key] + if isinstance(value, bool): + value = "true" if value else "false" + args += ["--fake-atelet-env", f"{env}={value}"] + return args diff --git a/benchmarking/locust/Dockerfile b/benchmarking/locust/Dockerfile index 2fadab5f80..67791e75fc 100644 --- a/benchmarking/locust/Dockerfile +++ b/benchmarking/locust/Dockerfile @@ -58,6 +58,8 @@ COPY benchmarking/locust/tests/ /app/tests/ COPY benchmarking/locust/runner.py /app/runner.py COPY benchmarking/locust/cluster_facts.py /app/cluster_facts.py COPY benchmarking/locust/server_telemetry.py /app/server_telemetry.py +COPY benchmarking/locust/phase_breakdown.py /app/phase_breakdown.py +COPY benchmarking/analysis/phase_report.py /app/phase_report.py ENV PYTHONPATH=/app:/app/deps ENV PYTHONUNBUFFERED=1 diff --git a/benchmarking/locust/common/boomer_config.py b/benchmarking/locust/common/boomer_config.py index 17902d78c9..e78c30ea60 100644 --- a/benchmarking/locust/common/boomer_config.py +++ b/benchmarking/locust/common/boomer_config.py @@ -24,6 +24,7 @@ * --mem-target / --mem-churn / --mem-read → common.memload_config.add_memload_arguments * --cpu-cores / --cpu-duty-cycle → common.cpuload_config.add_cpuload_arguments * --max-pings-per-wake → common.ping_config.add_ping_arguments + * --target-rps → common.rate_config.add_rate_arguments This module ties them together so boomer-Go workers can pick up the values the operator set in the web UI form: @@ -76,6 +77,7 @@ "--total-actors": int, "--spawn-concurrency": int, "--actor-deadline": float, + "--target-rps": float, } @@ -157,6 +159,7 @@ def init_boomer_config() -> None: from common.lifecycle_mode import add_lifecycle_mode_arguments from common.memload_config import add_memload_arguments from common.ping_config import add_ping_arguments + from common.rate_config import add_rate_arguments from common.resume_mode import add_resume_mode_arguments from common.spawn_config import add_spawn_arguments from common.sweperf_config import add_sweperf_arguments diff --git a/benchmarking/locust/common/rate_config.py b/benchmarking/locust/common/rate_config.py new file mode 100644 index 0000000000..47e28984dd --- /dev/null +++ b/benchmarking/locust/common/rate_config.py @@ -0,0 +1,36 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Request-rate runtime flag, for boomer user classes that pace +(resumepaused).""" + +from locust import events +from locust.argument_parser import LocustArgumentParser + + +@events.init_command_line_parser.add_listener +def add_rate_arguments(parser: LocustArgumentParser) -> None: + parser.add_argument( + "--target-rps", + type=float, + default=0.0, + env_var="LOCUST_TARGET_RPS", + help=( + "Requests per second the whole boomer worker paces to, for user " + "classes that pace (resumepaused). 0 removes the cap. " + "A --ladder step with @target_rps changes it while the run " + "continues" + ), + include_in_web_ui=True, + ) diff --git a/benchmarking/locust/deploy.sh b/benchmarking/locust/deploy.sh index 09b2f3fc96..393ec76d97 100755 --- a/benchmarking/locust/deploy.sh +++ b/benchmarking/locust/deploy.sh @@ -72,6 +72,9 @@ deploy() { # benchmarking/monitoring.yaml is otherwise optional. echo "Ensuring benchmarking namespace exists..." kubectl create namespace benchmarking --dry-run=client -o yaml | kubectl apply -f - + # The boomer workers call the Substrate API, which ate-system's + # NetworkPolicies admit only from namespaces carrying this label. + kubectl label namespace benchmarking ate.dev/api-access=true --overwrite if [[ -n "${AGENTSESSION_SCRIPT}" ]]; then echo "Validating agent-session script ${AGENTSESSION_SCRIPT}..." (cd "${SCRIPT_DIR}/../.." && go run ./cmd/benchmarking/boomer-worker --check-agentsession-script "${AGENTSESSION_SCRIPT}") diff --git a/benchmarking/locust/phase_breakdown.py b/benchmarking/locust/phase_breakdown.py new file mode 100644 index 0000000000..ea2953e67c --- /dev/null +++ b/benchmarking/locust/phase_breakdown.py @@ -0,0 +1,203 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Suspend/resume phase percentiles for stats.jsonl, read from the node logs. + +atelet and ateom-microvm write one `Checkpoint timing breakdown` / `Restore +timing breakdown` record per operation (see benchmarking/analysis/README.md). +The orchestrator deletes the worker and ate-system pods right after a test, +so the runner reads those pod logs itself, through the Kubernetes API it +already uses for cluster facts, while the pods still exist, and appends the +percentiles to stats.jsonl next to the SuspendActor / ResumeActor rows. + +Like the cluster facts this is additive: a failure here must never cost the +locust measurements. Pass --no-phase-breakdown to runner.py to skip it. +""" + +import csv +import json +import sys +from pathlib import Path +from typing import Any, TextIO + +from kubernetes import client + +from cluster_facts import ( + API_TIMEOUT_SECONDS, + WORKER_POOL_LABEL, + WORKER_POOL_NAMESPACE, + _load_kube_config, + _log, +) + +# In the repository phase_report.py lives in benchmarking/analysis/; the +# locust image copies it next to this file (benchmarking/locust/Dockerfile). +_ANALYSIS_DIR = Path(__file__).resolve().parent.parent / "analysis" +if _ANALYSIS_DIR.is_dir(): + sys.path.insert(0, str(_ANALYSIS_DIR)) +import phase_report # noqa: E402 + +ATELET_NAMESPACE = "ate-system" +ATELET_LABEL = "app=atelet" + +# A log body is a download, not a cached list; give it longer than the +# API_TIMEOUT_SECONDS the list calls use. +LOG_READ_TIMEOUT_SECONDS = 60 + +# The locust request names whose counts the atelet records must reach. More +# records than requests is normal: boomer suspends the actors it created when +# it shuts down, and those checkpoints are measured like any other. +LOCUST_OP_NAMES = {"checkpoint": "SuspendActor", "restore": "ResumeActor"} + + +class PodLogs: + """The lines read from one namespace's pods, and how the reads went.""" + + def __init__(self) -> None: + self.lines: list[str] = [] + self.pods_read = 0 + self.pods_failed = 0 + + +def _read_log(v1: client.CoreV1Api, name: str, namespace: str, + since_seconds: int, previous: bool = False) -> str: + # The raw response: with preloading, the client renders a text/plain body + # as the str() of bytes (one line of b'...'). + resp = v1.read_namespaced_pod_log( + name, namespace, since_seconds=since_seconds, previous=previous, + _request_timeout=LOG_READ_TIMEOUT_SECONDS, _preload_content=False, + ) + return resp.data.decode("utf-8", errors="replace") + + +def read_pod_logs(v1: client.CoreV1Api, namespace: str, label: str, + since_seconds: int, logs: TextIO | None = None) -> PodLogs: + """The log lines of every pod matching label in namespace, from + since_seconds ago. A container that restarted is read twice, its previous + log too, so a mid-run restart does not drop the records before it. A pod + that cannot be read is reported, counted and skipped.""" + out = PodLogs() + try: + pods = v1.list_namespaced_pod( + namespace=namespace, label_selector=label, + resource_version="0", _request_timeout=API_TIMEOUT_SECONDS, + ).items + except Exception as e: + _log(logs, f"Notice: could not list pods in {namespace} ({label}): " + f"{getattr(e, 'reason', e)}") + out.pods_failed += 1 + return out + for pod in pods: + name = pod.metadata.name + statuses = pod.status.container_statuses or [] + restarted = any(getattr(c, "restart_count", 0) for c in statuses) + try: + text = _read_log(v1, name, namespace, since_seconds) + except Exception as e: + _log(logs, f"Notice: could not read logs of {namespace}/{name}: " + f"{getattr(e, 'reason', e)}") + out.pods_failed += 1 + continue + if restarted: + # kubelet may have discarded the previous container's log already; + # that loses the records before the restart, not the current log. + try: + text += "\n" + _read_log(v1, name, namespace, since_seconds, previous=True) + except Exception as e: + _log(logs, f"Notice: no previous log for {namespace}/{name}: " + f"{getattr(e, 'reason', e)}") + out.pods_read += 1 + out.lines.extend(text.splitlines()) + return out + + +def locust_request_counts(stats_csv: Path) -> dict[str, tuple[int, int]]: + """SuspendActor / ResumeActor (requests, failures) from locust's stats.csv.""" + counts: dict[str, tuple[int, int]] = {} + with open(stats_csv) as f: + for row in csv.DictReader(f): + if row.get("Name") in LOCUST_OP_NAMES.values(): + counts[row["Name"]] = (int(row.get("Request Count") or 0), + int(row.get("Failure Count") or 0)) + return counts + + +def collect_phase_breakdown(v1: client.CoreV1Api, since_seconds: int, + logs: TextIO | None = None) -> tuple[list[dict], Any, PodLogs]: + """Reads the atelet and worker pod logs and aggregates the records. + Returns the percentile rows, the parse result and the read tallies.""" + parsed = phase_report.Parsed() + reads = PodLogs() + for namespace, label in ((ATELET_NAMESPACE, ATELET_LABEL), + (WORKER_POOL_NAMESPACE, WORKER_POOL_LABEL)): + got = read_pod_logs(v1, namespace, label, since_seconds, logs) + reads.pods_read += got.pods_read + reads.pods_failed += got.pods_failed + phase_report.parse_lines(got.lines, parsed) + phase_report.join_layers(parsed.breakdowns) + rows = phase_report.report_phases(parsed.breakdowns, lambda _: None) + return rows, parsed, reads + + +def append_phase_breakdown(jsonl_path: Path, stats_csv: Path, since_seconds: int, + timestamp: str, tag: str, test_name: str, + logs: TextIO | None = None) -> int: + """Appends the phase percentile rows to jsonl_path and returns how many. + + A phase_breakdown_summary row is always written once the API was + reachable, so a run with no records is distinguishable from a run where + the reads failed: it carries the pods read and failed, and the atelet + record counts next to locust's SuspendActor / ResumeActor request and + failure counts. A successful request must have reached atelet, so fewer + atelet records than successful requests means records were lost: to pods + that could not be read if there were any, else to a node log rotated + during the run. The log says which. + """ + if not _load_kube_config(logs): + return 0 + v1 = client.CoreV1Api() + rows, parsed, reads = collect_phase_breakdown(v1, since_seconds, logs) + + expected = locust_request_counts(stats_csv) if stats_csv.exists() else {} + summary: dict[str, Any] = {"records": parsed.lines_matched, + "pods_read": reads.pods_read, "pods_failed": reads.pods_failed} + for op, request_name in LOCUST_OP_NAMES.items(): + got = sum(1 for b in parsed.breakdowns if b.source == "atelet" and b.op == op) + summary[f"atelet_{op}_records"] = got + if request_name in expected: + requests, failures = expected[request_name] + summary[f"locust_{request_name}"] = requests + summary[f"locust_{request_name}_failures"] = failures + # A request that failed before reaching atelet leaves no record, + # so only the successful ones are owed one. + if got < requests - failures: + cause = (f"{reads.pods_failed} pod(s) could not be read" if reads.pods_failed + else "a node log was rotated during the run") + _log(logs, f"Warning: locust made {requests - failures} successful {request_name} " + f"requests but the atelet logs hold {got} {op} records; {cause}") + if not rows: + _log(logs, f"Warning: no timing breakdown records in the node logs " + f"({reads.pods_read} pods read, {reads.pods_failed} failed); " + f"check the runner's pods/log access if the run made requests") + + entries = phase_report.stats_rows(rows, timestamp, tag, test_name) + entries.append({"timestamp": timestamp, "tag": tag, "test_name": test_name, + "metric": "phase_breakdown_summary", + "measurements": {k: str(v) for k, v in summary.items()}}) + with open(jsonl_path, "a") as out: + for entry in entries: + out.write(json.dumps(entry) + "\n") + _log(logs, f"Appended {len(entries)} phase breakdown rows from " + f"{parsed.lines_matched} records") + return len(entries) diff --git a/benchmarking/locust/runner.py b/benchmarking/locust/runner.py index e43870ad07..9c722b5b1c 100644 --- a/benchmarking/locust/runner.py +++ b/benchmarking/locust/runner.py @@ -50,6 +50,7 @@ ) from common.boomer_config import build_config_json from server_telemetry import extract_and_record_server_telemetry +from phase_breakdown import append_phase_breakdown # Path inside the locust image to the boomer-worker binary baked in by # benchmarking/locust/Dockerfile. @@ -166,6 +167,17 @@ def parse_args() -> argparse.Namespace: "the default fits the OTel SDK's 60s default export and a 10s scrape" ), ) + p.add_argument( + "--phase-breakdown", + action=argparse.BooleanOptionalAction, + default=True, + help=( + "Read the atelet and worker pod logs after the run and append the " + "suspend/resume phase percentiles to stats.jsonl (see " + "benchmarking/analysis/README.md). Pass --no-phase-breakdown to " + "skip reading pod logs" + ), + ) args, extra = p.parse_known_args() args.locust_extra = extra return args @@ -583,6 +595,26 @@ def main() -> None: except Exception as e: tee(logs, f"Warning: Failed to harvest server telemetry: {e}") + # The phase breakdown is additive too. It reads the node logs now, + # while the pods still exist: the orchestrator deletes them as soon + # as this process exits. The window reaches back to this run's start + # with a margin for the clocks involved. + if stats_generated and args.phase_breakdown: + try: + append_phase_breakdown( + jsonl_path, + stats_csv, + int(time.time() - now.timestamp()) + 60, + data_ts, + args.tag, + args.name, + logs, + ) + except Exception as e: + tee(logs, f"Warning: Failed to record phase breakdown: {e}") + elif stats_generated: + tee(logs, "Skipping phase breakdown (--no-phase-breakdown)") + status_path.write_text( json.dumps( {"locust_exit_code": exit_code, "stats_generated": stats_generated} diff --git a/benchmarking/locust/shapes/ladder_shape.py b/benchmarking/locust/shapes/ladder_shape.py index 4e13014c9b..a3e33561a4 100644 --- a/benchmarking/locust/shapes/ladder_shape.py +++ b/benchmarking/locust/shapes/ladder_shape.py @@ -20,11 +20,12 @@ a set of cold caches, and a new baseline for the reader to align. `--ladder` makes the steps one run. Give the steps as a comma-separated list -of `users:duration[:trace_probability]`: +of `users:duration[:trace_probability][@target_rps]`: --ladder 5:3m,15:3m,30:3m the user sweep, S1 --ladder 15:3m:0,15:3m:0.1,15:3m:1 the sample-rate sweep, S2 --ladder 0:5m the idle floor, S0 + --ladder 200:3m@1000,200:3m@2000 a rate sweep at 200 users Name this file in the `file` of the test, after the test itself: @@ -45,6 +46,16 @@ the step change makes. boomer fetches that endpoint on each spawn message, and a step change is a spawn message. +A step that names a target rate works the same way: the rate applies as the +step starts and stays until another step changes it, and `--target-rps` gives +the rate before the first step that names one. Only a user class that paces +reads it (resumepaused), and it caps the requests per second of the whole +worker. The users must be enough to reach it, as each waits for its whole +iteration to return: for resumepaused, a resume plus a pause. +A step that holds the number of users makes no spawn message, thus boomer +reads the new rate at its next poll of /boomer-config, within 10 s; trim the +start of each step. `0` removes the cap. + Use a boomer user class for a ladder. Python and gRPC hold each other at a high number of users, and the latency of that condition is the latency of the load generator and not of substrate. @@ -58,17 +69,19 @@ logger = logging.getLogger(__name__) -# users:duration[:trace_probability] +# users:duration[:trace_probability][@target_rps] _STEP_RE = re.compile( r"^(?P\d+):(?P\d+)(?P[smh]?)" - r"(?::(?P[0-9.]+))?$" + r"(?::(?P[0-9.]+))?" + r"(?:@(?P[0-9.]+))?$" ) _UNIT_SECONDS = {"s": 1, "m": 60, "h": 3600} # A step of the ladder holds one number of users, for one period, at one -# sample rate. `probability` is None when the step does not change the rate. -Step = tuple[int, int, float | None] +# sample rate and one target request rate. `probability` and `rps` are None +# when the step does not change them. +Step = tuple[int, int, float | None, float | None] def parse_ladder(spec: str) -> list[Step]: @@ -84,27 +97,33 @@ def parse_ladder(spec: str) -> list[Step]: if not m: raise ValueError( f"unrecognized ladder step {item!r}; " - f"the form is users:duration[:trace_probability], " - f"as in 15:3m or 15:3m:0.1" + f"the form is users:duration[:trace_probability][@target_rps], " + f"as in 15:3m, 15:3m:0.1, or 200:3m@1000" ) seconds = int(m.group("duration")) * _UNIT_SECONDS[m.group("unit") or "s"] if seconds <= 0: raise ValueError(f"ladder step {item!r} has no duration") - probability = m.group("probability") - if probability is None: - steps.append((int(m.group("users")), seconds, None)) - continue - value = float(probability) - if not 0.0 <= value <= 1.0: + probability = _parse_float(item, m.group("probability"), "trace probability") + if probability is not None and not 0.0 <= probability <= 1.0: raise ValueError( f"ladder step {item!r} has a trace probability outside 0.0 to 1.0" ) - steps.append((int(m.group("users")), seconds, value)) + rps = _parse_float(item, m.group("rps"), "target rate") + steps.append((int(m.group("users")), seconds, probability, rps)) if not steps: raise ValueError("--ladder has no steps") return steps +def _parse_float(item: str, value: str | None, what: str) -> float | None: + if value is None: + return None + try: + return float(value) + except ValueError: + raise ValueError(f"ladder step {item!r} has a {what} that is not a number") + + @events.init_command_line_parser.add_listener def _(parser: LocustArgumentParser) -> None: parser.add_argument( @@ -114,8 +133,8 @@ def _(parser: LocustArgumentParser) -> None: env_var="LOCUST_LADDER", help=( "Run a ladder of steps in one test, as " - "users:duration[:trace_probability],... For example " - "5:3m,15:3m,30:3m" + "users:duration[:trace_probability][@target_rps],... For " + "example 5:3m,15:3m,30:3m" ), include_in_web_ui=True, ) @@ -142,6 +161,11 @@ def _set_trace_probability(environment, probability: float) -> None: set_trace_probability(probability) +def _set_target_rps(environment, rps: float) -> None: + """Apply `rps` to the parsed options, which /boomer-config serves.""" + environment.parsed_options.target_rps = rps + + class LadderShape(LoadTestShape): """Holds each step of `--ladder` in turn, then stops the run. @@ -165,7 +189,7 @@ def _load(self) -> bool: self._steps = parse_ladder(spec) if spec else [] elapsed = 0.0 self._boundaries = [] - for _, seconds, _ in self._steps: + for _, seconds, _, _ in self._steps: elapsed += seconds self._boundaries.append(elapsed) if self._steps: @@ -190,19 +214,22 @@ def tick(self) -> tuple[int, float] | None: logger.info("Ladder complete after %.0fs", run_time) return None - users, _, probability = self._steps[index] + users, _, probability, rps = self._steps[index] spawn_rate = self.runner.environment.parsed_options.ladder_spawn_rate if index != self._current: self._current = index if probability is not None: _set_trace_probability(self.runner.environment, probability) + if rps is not None: + _set_target_rps(self.runner.environment, rps) logger.info( - "Ladder step %d/%d: users=%d trace_probability=%s", + "Ladder step %d/%d: users=%d trace_probability=%s target_rps=%s", index + 1, len(self._steps), users, "unchanged" if probability is None else probability, + "unchanged" if rps is None else rps, ) return (users, spawn_rate) diff --git a/benchmarking/locust/tests/resumepaused.py b/benchmarking/locust/tests/resumepaused.py new file mode 100644 index 0000000000..431ced1e83 --- /dev/null +++ b/benchmarking/locust/tests/resumepaused.py @@ -0,0 +1,49 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Stub ResumePausedUser declaration. + +The real load implementation lives in the boomer-Go worker at +internal/benchmarking/boomer/resumepaused/ (registered by the boomer-worker +binary in cmd/benchmarking/boomer-worker/); this Python class is declared +only so the master recognizes the name and attributes boomer's stats rows +to it. A Python worker container sets LOCUST_NO_RESUMEPAUSED_USER=1 to skip +loading this file, leaving boomer as the sole owner of ResumePausedUser load. + +Each user creates one actor and makes it cold on its first iteration, then +resumes and pauses it again on every later one, with no wait. Pace the cold +resumes with --target-rps, or with @target_rps steps of a --ladder. +""" + +import os + +if os.environ.get("LOCUST_NO_RESUMEPAUSED_USER") != "1": + from locust import User, task + from common.boomer_config import init_boomer_config + + # Master serves /boomer-config so the boomer-worker workers can fetch + # runtime flag values (target rate, trace probability). No-op on + # workers without a web UI. + init_boomer_config() + + class ResumePausedUser(User): + host = "api.ate-system.svc.cluster.local:443" + + @task + def noop(self) -> None: + # Unreached under normal operation: the Python worker container + # does not load this file (LOCUST_NO_RESUMEPAUSED_USER=1). Body is + # required because locust validates that every User has at least + # one @task method. + pass diff --git a/benchmarking/locust/unit_tests/test_ladder_shape.py b/benchmarking/locust/unit_tests/test_ladder_shape.py new file mode 100644 index 0000000000..25034b1cd0 --- /dev/null +++ b/benchmarking/locust/unit_tests/test_ladder_shape.py @@ -0,0 +1,93 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for shapes/ladder_shape.py. + +Run via: python3 benchmarking/locust/unit_tests/test_ladder_shape.py +Needs locust: pip install -r benchmarking/locust/requirements.txt +""" + +import sys +import unittest +from pathlib import Path +from types import SimpleNamespace + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from shapes.ladder_shape import LadderShape, parse_ladder + + +class ParseLadderTest(unittest.TestCase): + def test_users_and_duration(self): + self.assertEqual( + parse_ladder("5:3m,15:90s,30:1h"), + [(5, 180, None, None), (15, 90, None, None), (30, 3600, None, None)], + ) + + def test_trace_probability(self): + self.assertEqual(parse_ladder("15:3m:0.1"), [(15, 180, 0.1, None)]) + + def test_target_rps(self): + self.assertEqual( + parse_ladder("200:3m@1000, 200:3m@2500.5"), + [(200, 180, None, 1000.0), (200, 180, None, 2500.5)], + ) + + def test_probability_and_target_rps(self): + self.assertEqual(parse_ladder("200:3m:0@0"), [(200, 180, 0.0, 0.0)]) + + def test_invalid(self): + for bad in ("", "200", "200:3m@", "200:3m@x", "200:0m@100", + "200:3m@1.2.3", "200:3m:2@100", "200:3m@100:0.1"): + with self.assertRaises(ValueError, msg=repr(bad)): + parse_ladder(bad) + + +class TickTest(unittest.TestCase): + def shape(self, ladder, run_time): + options = SimpleNamespace( + ladder=ladder, ladder_spawn_rate=5.0, target_rps=0.0 + ) + shape = LadderShape() + shape.runner = SimpleNamespace( + environment=SimpleNamespace(parsed_options=options) + ) + shape.get_run_time = lambda: run_time[0] + return shape, options + + def test_step_sets_target_rps_and_keeps_it(self): + run_time = [0.0] + shape, options = self.shape("200:60s@1000,100:60s,200:60s@0", run_time) + + self.assertEqual(shape.tick(), (200, 5.0)) + self.assertEqual(options.target_rps, 1000.0) + + run_time[0] = 61.0 + self.assertEqual(shape.tick(), (100, 5.0)) + self.assertEqual(options.target_rps, 1000.0) + + run_time[0] = 121.0 + self.assertEqual(shape.tick(), (200, 5.0)) + self.assertEqual(options.target_rps, 0.0) + + run_time[0] = 181.0 + self.assertIsNone(shape.tick()) + + def test_no_ladder_is_inert(self): + shape, _ = self.shape("", [0.0]) + self.assertIsNone(shape.tick()) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarking/locust/unit_tests/test_phase_breakdown.py b/benchmarking/locust/unit_tests/test_phase_breakdown.py new file mode 100644 index 0000000000..52063ac0d2 --- /dev/null +++ b/benchmarking/locust/unit_tests/test_phase_breakdown.py @@ -0,0 +1,169 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for phase_breakdown.py. + +Run via: python3 benchmarking/locust/unit_tests/test_phase_breakdown.py +Never contacts a cluster: pod logs come from a mocked CoreV1Api. Needs the +kubernetes client: pip install -r benchmarking/locust/requirements.txt +""" + +import contextlib +import io +import json +import sys +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + +from kubernetes.client.rest import ApiException + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +import phase_breakdown + +ATELET = json.dumps({ + "time": "2026-09-23T10:00:05.500000000Z", "msg": "Restore timing breakdown", + "ate.actor.uid": "uid-1", "ate.template.name": "glutton", "ate.sandbox.class": "microvm", + "ate.snapshot.scope": "full", "ate.snapshot.kind": "latest", + "ate.actor.restore.duration.download": 2.4, + "ate.actor.restore.duration.ateom_restore": 1.1, + "ate.actor.restore.duration.total": 3.9, +}) +ATEOM = json.dumps({ + "time": "2026-09-23T10:00:05.400000000Z", "msg": "Restore timing breakdown", + "ate.actor.uid": "uid-1", "ate.template.name": "glutton", "ate.snapshot.scope": "full", + "ateom.actor.restore.duration.vm_restore": 0.8, + "ateom.actor.restore.duration.total": 1.05, +}) +STATS_CSV = ("Type,Name,Request Count,Failure Count\n" + "grpc,SuspendActor,0,0\ngrpc,ResumeActor,1,0\n,Aggregated,1,0\n") + + +def fake_api(logs_by_namespace, broken_pod=None, restarted=False, previous_logs=None, + previous_missing=False): + """Stand-in CoreV1Api: one pod per namespace, its log from the mapping.""" + api = mock.Mock() + api.list_namespaced_pod.side_effect = lambda namespace, **_: SimpleNamespace( + items=[SimpleNamespace( + metadata=SimpleNamespace(name=f"{namespace}-pod"), + status=SimpleNamespace(container_statuses=[SimpleNamespace(restart_count=1 if restarted else 0)]))]) + + def read_log(name, namespace, previous=False, **_): + if name == broken_pod: + raise ApiException(status=400, reason="container is being created") + if previous and previous_missing: + raise ApiException(status=400, reason="previous terminated container not found") + source = (previous_logs or {}) if previous else logs_by_namespace + # The raw (un-preloaded) response the code asks for: bytes in .data. + return SimpleNamespace(data=source.get(namespace, "").encode()) + api.read_namespaced_pod_log.side_effect = read_log + return api + + +def append(api, stats=STATS_CSV): + with tempfile.TemporaryDirectory() as d: + jsonl, stats_csv = Path(d) / "stats.jsonl", Path(d) / "stats.csv" + jsonl.write_text('{"metric": "grpc_ResumeActor"}\n') + stats_csv.write_text(stats) + out = io.StringIO() + with mock.patch.object(phase_breakdown, "_load_kube_config", return_value=True), \ + mock.patch.object(phase_breakdown.client, "CoreV1Api", return_value=api), \ + contextlib.redirect_stdout(out): + n = phase_breakdown.append_phase_breakdown( + jsonl, stats_csv, 600, "2026-09-23T10:01:00Z", "abc123", "unit-run") + return n, [json.loads(line) for line in jsonl.read_text().splitlines()], out.getvalue() + + +class PhaseBreakdownTest(unittest.TestCase): + def test_rows_are_appended_after_the_locust_rows(self): + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}) + n, entries, _ = append(api) + self.assertEqual(entries[0]["metric"], "grpc_ResumeActor") # untouched + metrics = {e["metric"] for e in entries[1:]} + self.assertIn("phase_atelet_restore_microvm_latest_full_download", metrics) + # The ateom row took its kind from the paired atelet record. + self.assertIn("phase_ateom_restore_microvm_latest_full_vm_restore", metrics) + self.assertEqual(n, len(entries) - 1) + for e in entries[1:]: + self.assertEqual((e["timestamp"], e["tag"], e["test_name"]), + ("2026-09-23T10:01:00Z", "abc123", "unit-run")) + summary = next(e for e in entries if e["metric"] == "phase_breakdown_summary") + self.assertEqual(summary["measurements"]["atelet_restore_records"], "1") + self.assertEqual(summary["measurements"]["locust_ResumeActor"], "1") + self.assertEqual((summary["measurements"]["pods_read"], summary["measurements"]["pods_failed"]), ("2", "0")) + + def test_fewer_records_than_successful_requests_is_warned(self): + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}) + _, entries, output = append(api, stats=STATS_CSV.replace("ResumeActor,1,0", "ResumeActor,5,0")) + self.assertIn("Warning: locust made 5 successful ResumeActor requests but the atelet logs hold 1", output) + self.assertIn("a node log was rotated", output) + summary = next(e for e in entries if e["metric"] == "phase_breakdown_summary") + self.assertEqual(summary["measurements"]["locust_ResumeActor_failures"], "0") + + def test_requests_that_never_reached_atelet_are_not_owed_a_record(self): + # 5 requests, 4 failed client-side: only 1 successful one reached atelet. + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}) + _, _, output = append(api, stats=STATS_CSV.replace("ResumeActor,1,0", "ResumeActor,5,4")) + self.assertNotIn("Warning: locust made", output) + + def test_missing_records_blame_unreadable_pods_when_there_are_any(self): + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}, broken_pod="ate-system-pod") + _, _, output = append(api) + self.assertIn("1 pod(s) could not be read", output) + self.assertNotIn("rotated", output) + + def test_a_missing_previous_log_keeps_the_current_one(self): + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}, restarted=True, previous_missing=True) + _, entries, output = append(api) + self.assertIn("no previous log for", output) + self.assertTrue(any(e["metric"].startswith("phase_atelet_restore") for e in entries)) + summary = next(e for e in entries if e["metric"] == "phase_breakdown_summary") + self.assertEqual((summary["measurements"]["pods_read"], summary["measurements"]["pods_failed"]), ("2", "0")) + + def test_an_unreadable_pod_is_skipped_not_fatal(self): + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}, + broken_pod="benchmark-workloads-pod") + n, entries, output = append(api) + self.assertIn("could not read logs of benchmark-workloads/benchmark-workloads-pod", output) + self.assertTrue(any(e["metric"].startswith("phase_atelet_") for e in entries)) + self.assertFalse(any(e["metric"].startswith("phase_ateom_") for e in entries)) + + def test_no_records_still_writes_the_summary_row(self): + # A failed configuration must not look like an idle run. + n, entries, output = append(fake_api({})) + self.assertEqual((n, len(entries)), (1, 2)) + self.assertEqual(entries[1]["metric"], "phase_breakdown_summary") + self.assertEqual(entries[1]["measurements"]["records"], "0") + self.assertIn("Warning: no timing breakdown records", output) + + def test_a_restarted_container_is_read_twice(self): + api = fake_api({"ate-system": "", "benchmark-workloads": ATEOM}, restarted=True, + previous_logs={"ate-system": ATELET}) + _, entries, _ = append(api) + self.assertTrue(any(e["metric"].startswith("phase_atelet_restore") for e in entries)) + kwargs = [c.kwargs for c in api.read_namespaced_pod_log.call_args_list] + self.assertEqual(sum(1 for k in kwargs if k.get("previous")), 2) + + def test_failed_requests_do_not_trigger_the_rotation_warning(self): + failed = ATELET.replace('"ate.snapshot.kind": "latest",', '"ate.snapshot.kind": "latest", "error.type": "DeadlineExceeded",') + api = fake_api({"ate-system": failed, "benchmark-workloads": ATEOM}) + _, _, output = append(api) + self.assertNotIn("Warning: locust made", output) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarking/monitoring.yaml b/benchmarking/monitoring.yaml index f815c7b22a..66c13fdf22 100644 --- a/benchmarking/monitoring.yaml +++ b/benchmarking/monitoring.yaml @@ -16,6 +16,10 @@ apiVersion: v1 kind: Namespace metadata: name: benchmarking + labels: + # Admits the benchmark clients to the Substrate API through + # manifests/ate-install/ate-system-networkpolicies.yaml. + ate.dev/api-access: "true" --- apiVersion: v1 kind: ServiceAccount diff --git a/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml b/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml index 799ad103bc..4e0f223d6d 100644 --- a/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml +++ b/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml @@ -110,7 +110,7 @@ spec: containers: - name: envoy # Same envoyproxy/envoy digest as cmd/dataplane/envoy/Dockerfile. - image: envoyproxy/envoy:v1.39.1@sha256:57e14a549d7bd43c8d3f6d03e8cfa653e037d4b38e133acd9b54f38c524401b4 + image: envoyproxy/envoy:v1.39.4@sha256:74bd4eb5629708d3fadca1a5e6b85d208f338d7bb54e06b69565905c5294cc01 command: - /usr/local/bin/envoy args: diff --git a/benchmarking/telemetry/meter.yaml b/benchmarking/telemetry/meter.yaml index 27adc08243..cc9f140658 100644 --- a/benchmarking/telemetry/meter.yaml +++ b/benchmarking/telemetry/meter.yaml @@ -32,6 +32,10 @@ apiVersion: v1 kind: Namespace metadata: name: benchmarking + labels: + # Admits the benchmark clients to the Substrate API through + # manifests/ate-install/ate-system-networkpolicies.yaml. + ate.dev/api-access: "true" --- apiVersion: v1 kind: ConfigMap diff --git a/benchmarking/workloads/deploy.sh b/benchmarking/workloads/deploy.sh index ff5623c98f..e524156d7a 100755 --- a/benchmarking/workloads/deploy.sh +++ b/benchmarking/workloads/deploy.sh @@ -65,6 +65,9 @@ OTLP_ENDPOINT="" # The timeout, in whole seconds, for waiting for the ateom worker pods to be # ready. WAIT_TIMEOUT_SECS=300 +# fake-atelet's tuning on the fake data plane, as FAKE_ATELET_=VALUE +# entries for the fake-atelet-tuning ConfigMap; --fake-atelet-env adds one. +FAKE_ATELET_ENV=() usage() { echo "Usage: $0 [options]" @@ -85,6 +88,10 @@ usage() { echo " sends telemetry (default: the endpoint in the" echo " ate-otel-config ConfigMap)" echo " --wait-timeout SECONDS The timeout in seconds for waiting for the ateom workers to be ready (default: 300)" + echo " --fake-atelet-env NAME=VALUE" + echo " On a substrate installed with --fake-data-plane, set a fake-atelet flag:" + echo " FAKE_ATELET_DELAY=500ms sets --delay, FAKE_ATELET_DELAY_RESTORE=2s sets" + echo " --delay-restore. Repeatable; unset flags keep fake-atelet's defaults" echo " -h, --help Show this help message" } @@ -201,15 +208,66 @@ wait_templates_ready() { done } +# fake_data_plane reports whether substrate was installed with +# --fake-data-plane, which serves the WorkerPool with fake Workers instead of +# worker pods. +fake_data_plane() { + [[ -n "$(kubectl get deployment/fake-workersync --namespace=ate-system -o name --ignore-not-found)" ]] +} + +# tune_fake_atelet writes FAKE_ATELET_ENV, and nothing else, to the +# fake-atelet-tuning ConfigMap fake-atelet loads its flags from, and restarts +# fake-atelet when that changes the ConfigMap. No install creates or resets +# it: the installer chooses the fake data plane, and the benchmark tunes it. +tune_fake_atelet() { + local before after + before="$(kubectl get configmap fake-atelet-tuning --namespace=ate-system -o jsonpath='{.data}' --ignore-not-found)" + local literals=() + local entry + for entry in "${FAKE_ATELET_ENV[@]}"; do + literals+=("--from-literal=${entry}") + done + kubectl create configmap fake-atelet-tuning --namespace=ate-system "${literals[@]}" \ + --dry-run=client -o yaml | kubectl apply -f - + after="$(kubectl get configmap fake-atelet-tuning --namespace=ate-system -o jsonpath='{.data}')" + if [[ "${before}" == "${after}" ]]; then + return 0 + fi + echo "Restarting fake-atelet with its new tuning (${FAKE_ATELET_ENV[*]:-defaults})..." + # The DaemonSet name carries the version, so it is addressed by label. + local ds + for ds in $(kubectl get daemonset --namespace=ate-system -l app=atelet -o name); do + kubectl rollout restart --namespace=ate-system "${ds}" + kubectl rollout status --namespace=ate-system "${ds}" --timeout="${WAIT_TIMEOUT_SECS}s" + done +} + deploy() { resolve_otlp_endpoint + # Before the pool, so its fake Workers are served by the tuned fake-atelet. + if fake_data_plane; then + tune_fake_atelet + elif ((${#FAKE_ATELET_ENV[@]} > 0)); then + echo "Error: --fake-atelet-env needs substrate installed with --fake-data-plane" >&2 + exit 1 + fi echo "Deploying workloads (worker_count=${WORKER_COUNT}, actor_memory=${ACTOR_MEMORY}, worker_memory=${WORKER_MEMORY:-unset}, storage_class=${STORAGE_CLASS_NAME:-none}, otlp_endpoint=${OTLP_ENDPOINT})..." substitute "${POOL_MANIFEST}" | hack/run-tool.sh ko apply -f - echo "Waiting for worker pool to be ready (timeout: ${WAIT_TIMEOUT_SECS}s)..." - kubectl wait --for=create deployment/benchmark-ateom \ - --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" - kubectl rollout status deployment/benchmark-ateom \ - --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" + if fake_data_plane; then + # No Deployment is made for the pool; fake-workersync counts a fake Worker + # ready once its capacity is reported, and writes that to the pool's + # status. + if ((WORKER_COUNT > 0)); then + kubectl wait --for=jsonpath='{.status.readyReplicas}'="${WORKER_COUNT}" workerpool/benchmark-ateom \ + --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" + fi + else + kubectl wait --for=create deployment/benchmark-ateom \ + --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" + kubectl rollout status deployment/benchmark-ateom \ + --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" + fi # The store enforces that a template's atespace exists at create time. run_kubectl_ate create atespace benchmark-workloads >/dev/null 2>&1 \ @@ -311,6 +369,13 @@ while [[ "$#" -gt 0 ]]; do --wait-timeout=*) WAIT_TIMEOUT_SECS="${1#*=}" ;; + --fake-atelet-env) + shift + FAKE_ATELET_ENV+=("$1") + ;; + --fake-atelet-env=*) + FAKE_ATELET_ENV+=("${1#*=}") + ;; -h|--help) usage exit 0 @@ -350,6 +415,13 @@ if [[ "${action}" == "deploy" && -z "${STORAGE_CLASS_NAME}" && " ${TEMPLATES[*]} exit 1 fi +for entry in "${FAKE_ATELET_ENV[@]}"; do + if ! [[ "${entry}" =~ ^FAKE_ATELET_[A-Z_]+=.+$ ]]; then + echo "Error: --fake-atelet-env must be FAKE_ATELET_=VALUE, such as FAKE_ATELET_DELAY=500ms, got '${entry}'" >&2 + exit 1 + fi +done + if [[ "${action}" == "deploy" ]]; then build_kubectl_ate deploy diff --git a/charts/substrate/README.md b/charts/substrate/README.md index 6f4c1e416d..d996c673e4 100644 --- a/charts/substrate/README.md +++ b/charts/substrate/README.md @@ -115,6 +115,20 @@ The default gVisor config now uses the 2026-10-10 nightly (`v3`) for split filesystem checkpoints. Upgrade the sandbox config together with ateom-gvisor; custom configs must pin a release with the same checkpoint support. +The chart now denies ingress to the control-plane namespace by default and +allows the component, router, egress, storage, and monitoring traffic it needs. +Label namespaces containing API clients with `ate.dev/api-access=true`; worker +pods remain excluded from that client allowance. Custom workloads sharing the +control-plane namespace need their own ingress policies. Custom health probes +for the API server, atelet, and credential provider must use port 9091; +port 9090 serves metrics only. + +Update API clients for pagination under `ListOptions`, the renamed +`MintActorJwt` RPC and TitleCase protobuf type names, `WakeupProbe`, and the +`*_time` timestamp fields. Pausing now transitions through `SUSPENDING` to +`SUSPENDED`; use snapshot locality to distinguish a paused actor. Upgrade +clients and components together. + ## PostgreSQL setup The chart does not deploy or initialize PostgreSQL. Prepare the database, diff --git a/charts/substrate/templates/ate-api-server.yaml b/charts/substrate/templates/ate-api-server.yaml index 00966f8183..af72b6d50e 100644 --- a/charts/substrate/templates/ate-api-server.yaml +++ b/charts/substrate/templates/ate-api-server.yaml @@ -129,6 +129,9 @@ spec: - "--actor-id-ca-pool=/run/actor-id-ca-pool/pool.json" - "--default-egress-gateway-address={{ include "substrate.fullname" (list "atenet-egress" .) }}.{{ .Release.Namespace }}.svc:443" - '--atelet-service-account={{ include "substrate.fullname" (list "atelet" .) }}' + - '--controller-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "ate-controller" .) }}' + - '--egress-gateway-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "atenet-egress" .) }}' + - '--ingress-router-spiffe-id=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "atenet-router" .) }}' - "--atelet-client-cred-bundle=/run/podidentity.podcert.ate.dev/credential-bundle.pem" - "--pod-identity-ca-certs=/run/podidentity.podcert.ate.dev/trust-bundle.pem" - "--drain-delay=13s" @@ -183,17 +186,25 @@ spec: - containerPort: 443 - name: prometheus containerPort: 9090 + - name: health + containerPort: 9091 readinessProbe: httpGet: path: /readyz - port: 9090 + port: health initialDelaySeconds: 5 periodSeconds: 2 failureThreshold: 3 + startupProbe: + httpGet: + path: /healthz + port: health + periodSeconds: 5 + failureThreshold: 24 livenessProbe: httpGet: path: /healthz - port: 9090 + port: health initialDelaySeconds: 10 periodSeconds: 10 volumes: diff --git a/charts/substrate/templates/ate-system-networkpolicies.yaml b/charts/substrate/templates/ate-system-networkpolicies.yaml new file mode 100644 index 0000000000..197d28f66e --- /dev/null +++ b/charts/substrate/templates/ate-system-networkpolicies.yaml @@ -0,0 +1,219 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Ingress NetworkPolicies for the ate-system control plane. +# +# Every pod in ate-system is denied ingress by default; the policies below +# admit only the flows the control plane needs. NetworkPolicies are additive, +# so a port is reachable exactly when some policy here (or another one that +# selects the pod, such as k8s-credential-provider's) admits it. Ports that no +# policy admits -- the router's /statusz port (4040), the bundled postgres from +# anything but ate-api-server, atelet's AteomHerder from anything but +# ate-api-server -- are reachable only through `kubectl port-forward`, which +# NetworkPolicy does not filter. +# +# Policies only take effect on a cluster whose network plugin enforces them +# (GKE Dataplane V2, Calico, Cilium, kindnet, ...). +# +# Several workloads are applied by the installer outside this bundle (postgres, +# atenet-egress, k8s-credential-provider); their policies live here anyway so +# the default deny never lands without the allow rules that go with it. A +# policy whose pods do not exist selects nothing. +# +# Ports are named rather than numbered where the workloads name them, so the +# policy follows the ports exposed by the chart workloads. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "default-deny-ingress" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: {} + policyTypes: + - Ingress +--- +# The Substrate API. Admits control plane components and pods in namespaces +# labeled ate.dev/api-access=true (e.g. benchmarks, dashboards). Worker pods +# (ate.dev/worker-pool) are excluded. Requests are still authenticated and +# authorized by ate-api-server; this restricts network-level ingress. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "ate-api-server" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + app: ate-api-server + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchExpressions: + - key: app + operator: In + values: + - ate-controller + - atelet + - atenet-egress + - atenet-router + - namespaceSelector: + matchLabels: + ate.dev/api-access: "true" + podSelector: + matchExpressions: + - key: ate.dev/worker-pool + operator: DoesNotExist + ports: + - protocol: TCP + port: 443 +--- +# The bundled postgres is reached only by ate-api-server. The installer runs +# psql through `kubectl exec`, and the probes are exec probes, so neither needs +# network access. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "postgres" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + app: postgres + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + app: ate-api-server + ports: + - protocol: TCP + port: postgres +--- +# atelet's gRPC port is called only by ate-api-server, which dials the atelet +# pod IP. Worker pods talk to atelet over its node-local Unix socket, not this +# port. The hostPort mapping lands on the pod, so this policy covers it too. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "atelet" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + app: atelet + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + app: ate-api-server + ports: + - protocol: TCP + port: grpc +--- +# The router's data ports carry every request to an actor, from pods in any +# namespace and from load balancers and Gateways outside the cluster, so they +# admit any source. What can actually reach them is decided by the Service +# type (ClusterIP by default), any Gateway or LoadBalancer in front of the +# router, and the cluster's firewall rules; the router itself checks the target +# actor of every request. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "atenet-router" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + app: atenet-router + policyTypes: + - Ingress + ingress: + - ports: + - protocol: TCP + port: http + - protocol: TCP + port: https + - protocol: TCP + port: connect + - protocol: TCP + port: connect-tls +--- +# The egress gateway is reached by atunnel in worker pods, which the +# WorkerPool controller labels ate.dev/worker-pool, in whatever namespace the +# WorkerPool lives in. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "atenet-egress" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + app: atenet-egress + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: {} + podSelector: + matchExpressions: + - key: ate.dev/worker-pool + operator: Exists + ports: + - protocol: TCP + port: https +--- +# Metrics and health endpoints. Kubelet probes come from the node and scrapers +# run in whatever namespace the monitoring stack uses (GKE Managed Prometheus, +# otel-system, a user's Prometheus), so these admit any source. They serve +# metrics and readiness only; the router's /statusz is not among them. +# +# k8s-credential-provider is left out: its own policy admits only the egress +# gateway, and it stays that way (docs/egress-credential-injection.md). Its +# kubelet probes come from the node, which NetworkPolicy does not block. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "ate-system-observability" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchExpressions: + - key: app + operator: NotIn + values: + - k8s-credential-provider + policyTypes: + - Ingress + ingress: + - ports: + - protocol: TCP + port: metrics + - protocol: TCP + port: prometheus + - protocol: TCP + port: gw-metrics + - protocol: TCP + port: health + - protocol: TCP + port: healthz + - protocol: TCP + port: stats + - protocol: TCP + port: readiness diff --git a/charts/substrate/templates/atelet.yaml b/charts/substrate/templates/atelet.yaml index cdc6958c6d..733ec6533d 100644 --- a/charts/substrate/templates/atelet.yaml +++ b/charts/substrate/templates/atelet.yaml @@ -59,8 +59,9 @@ rules: verbs: ["get", "watch", "list"] # SandboxConfigs are watched to pre-download sandbox assets into the node's # cache before the first actor needs them (see cmd/atelet/sandbox_prewarm.go). +# WorkerPools identify the SandboxConfig for each local worker. - apiGroups: ["ate.dev"] - resources: ["sandboxconfigs"] + resources: ["sandboxconfigs", "workerpools"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 @@ -228,8 +229,23 @@ spec: hostPort: 8085 - name: prometheus containerPort: 9090 - hostPort: 9090 protocol: TCP + - name: health + containerPort: 9091 + protocol: TCP + readinessProbe: + httpGet: + path: /readyz + port: health + initialDelaySeconds: 5 + periodSeconds: 2 + failureThreshold: 3 + livenessProbe: + httpGet: + path: /healthz + port: health + initialDelaySeconds: 10 + periodSeconds: 10 volumeMounts: - name: objectstore-plugin mountPath: /run/objectstore-plugin diff --git a/charts/substrate/templates/k8s-credential-provider.yaml b/charts/substrate/templates/k8s-credential-provider.yaml index 71b0e9202c..f96f38a870 100644 --- a/charts/substrate/templates/k8s-credential-provider.yaml +++ b/charts/substrate/templates/k8s-credential-provider.yaml @@ -81,6 +81,7 @@ spec: args: - "--listen-address=:50051" - "--metrics-address=:9090" + - "--health-address=:9091" - "--status-address=:4040" # Use this Service's DNS certificate; only the egress injector may call. - "--server-cred-bundle=/run/servicedns.podcert.ate.dev/credential-bundle.pem" @@ -94,12 +95,14 @@ spec: containerPort: 50051 - name: metrics containerPort: 9090 + - name: health + containerPort: 9091 - name: statusz containerPort: 4040 readinessProbe: httpGet: path: /readyz - port: metrics + port: health periodSeconds: 10 securityContext: allowPrivilegeEscalation: false diff --git a/charts/substrate/templates/rustfs.yaml b/charts/substrate/templates/rustfs.yaml index 922a189114..ab04cb1a20 100644 --- a/charts/substrate/templates/rustfs.yaml +++ b/charts/substrate/templates/rustfs.yaml @@ -148,4 +148,31 @@ spec: done echo "timed out waiting for rustfs" exit 1 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "substrate.fullname" (list "rustfs" .) }} + namespace: {{ .Release.Namespace }} +spec: + podSelector: + matchLabels: + app: rustfs + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchExpressions: + - key: app + operator: In + values: + - ate-api-server + - atelet + - podSelector: + matchLabels: + batch.kubernetes.io/job-name: {{ include "substrate.fullname" (list "rustfs-bucket-init" .) }} + ports: + - protocol: TCP + port: api {{- end }} diff --git a/charts/substrate/tests/control_plane_test.yaml b/charts/substrate/tests/control_plane_test.yaml new file mode 100644 index 0000000000..44c6938fa5 --- /dev/null +++ b/charts/substrate/tests/control_plane_test.yaml @@ -0,0 +1,163 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +suite: Control plane integration +templates: +- ate-api-server.yaml +- atelet.yaml +- k8s-credential-provider.yaml +- ate-system-networkpolicies.yaml +- rustfs.yaml +release: + name: custom + namespace: custom-system +tests: +- it: probes the API health listener and resolves component identities + template: ate-api-server.yaml + documentSelector: + path: kind + value: Deployment + asserts: + - contains: + path: spec.template.spec.containers[0].ports + content: + name: health + containerPort: 9091 + - equal: + path: spec.template.spec.containers[0].readinessProbe.httpGet.port + value: health + - equal: + path: spec.template.spec.containers[0].startupProbe.httpGet.port + value: health + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.port + value: health + - contains: + path: spec.template.spec.containers[0].args + content: --controller-spiffe-id=spiffe://cluster.local/ns/custom-system/sa/custom-ate-controller + - contains: + path: spec.template.spec.containers[0].args + content: --egress-gateway-spiffe-id=spiffe://cluster.local/ns/custom-system/sa/custom-atenet-egress + - contains: + path: spec.template.spec.containers[0].args + content: --ingress-router-spiffe-id=spiffe://cluster.local/ns/custom-system/sa/custom-atenet-router +- it: lets atelet watch the configs and pools of its workers + template: atelet.yaml + documentSelector: + path: kind + value: ClusterRole + asserts: + - contains: + path: rules + content: + apiGroups: [ate.dev] + resources: [sandboxconfigs, workerpools] + verbs: [get, list, watch] +- it: probes atelet health without exposing metrics on the host + template: atelet.yaml + documentSelector: + path: kind + value: DaemonSet + asserts: + - contains: + path: spec.template.spec.containers[0].ports + content: + name: prometheus + containerPort: 9090 + protocol: TCP + - contains: + path: spec.template.spec.containers[0].ports + content: + name: health + containerPort: 9091 + protocol: TCP + - equal: + path: spec.template.spec.containers[0].readinessProbe.httpGet.port + value: health + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.port + value: health +- it: probes the credential provider on its health listener + template: k8s-credential-provider.yaml + documentSelector: + path: kind + value: Deployment + asserts: + - contains: + path: spec.template.spec.containers[0].ports + content: + name: health + containerPort: 9091 + - equal: + path: spec.template.spec.containers[0].readinessProbe.httpGet.port + value: health +- it: denies ingress by default in the release namespace + template: ate-system-networkpolicies.yaml + documentSelector: + path: metadata.name + value: custom-default-deny-ingress + asserts: + - equal: + path: metadata.namespace + value: custom-system + - equal: + path: spec.podSelector + value: {} + - equal: + path: spec.policyTypes + value: [Ingress] + - notExists: + path: spec.ingress +- it: permits agentgateway metrics but keeps credential provider status private + template: ate-system-networkpolicies.yaml + documentSelector: + path: metadata.name + value: custom-ate-system-observability + asserts: + - contains: + path: spec.ingress[0].ports + content: + protocol: TCP + port: gw-metrics + - contains: + path: spec.podSelector.matchExpressions + content: + key: app + operator: NotIn + values: [k8s-credential-provider] + - notContains: + path: spec.ingress[0].ports + content: + protocol: TCP + port: statusz +- it: allows the named bucket initialization job to reach RustFS + template: rustfs.yaml + documentSelector: + path: kind + value: NetworkPolicy + asserts: + - equal: + path: metadata.namespace + value: custom-system + - contains: + path: spec.ingress[0].from + content: + podSelector: + matchLabels: + batch.kubernetes.io/job-name: custom-rustfs-bucket-init + - equal: + path: spec.ingress[0].ports + value: + - protocol: TCP + port: api diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index 0b40317b16..1c88b0832b 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -30,6 +30,7 @@ a pre-scan pass, so they may appear anywhere on its command line. | `--cluster-size size0\|size10` | `--cluster-size size0\|size10` | Footprint profile (default `size0`). `size10` assumes a dedicated PostgreSQL node: it resizes the bundled StatefulSet and its `postgresql.conf`, pins the apiserver's connection pool, and raises the podcertificate-controller's API rate limits. `ATE_INSTALL_CLUSTER_SIZE` when the flag is absent | | `--ateapi-postgres-storage-class NAME` | `ATE_API_POSTGRES_STORAGE_CLASS=NAME` | StorageClass for the bundled PostgreSQL volume instead of the cluster default, such as `dynamic-rwo` on GKE so C4 and N4 nodes can attach it. An existing install must delete the postgres StatefulSet and its claim first | | `--cordon-control-plane` | `--cordon-control-plane` | Keep the control plane off the worker nodes. Assumes a small shared pool labeled and tainted `ate.dev/workloadType=ate-control-plane:NoSchedule`, across which each workload's replicas are spread, and a one-node pool labeled and tainted `ate.dev/workloadType=ate-postgres:NoSchedule` for postgres alone. `ATE_INSTALL_CORDON_CONTROL_PLANE=true` when the flag is absent | +| `--fake-data-plane` | `--fake-data-plane` | Install the fake data plane for control-plane benchmarks: fake-atelet takes atelet's place in its DaemonSet, taking its objectstore-plugin's storage settings, ate-controller is scaled to zero, and fake-workersync honors every WorkerPool with fake Workers on the fake-atelet nodes. Reaches `deploy ate-system`, `deploy atelet`, and `deploy ate-controller`; only `deploy ate-system` switches a cluster between the real and fake data planes, and the other two refuse a data plane other than the installed one. `deploy ate-system --fake-data-plane` refuses a cluster whose WorkerPools have worker pods. fake-atelet runs with its defaults; a benchmark tunes it through the optional `fake-atelet-tuning` ConfigMap, whose `FAKE_ATELET_*` variables set its flags, and which no install creates or resets. Actors do not exist; never use it on a cluster that serves real actors. `ATE_INSTALL_FAKE_DATA_PLANE=true` when the flag is absent | | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | External processor authorization filter | | `--credential-provider JSON` | `--credential-provider JSON`, or `ATE_CREDENTIAL_PROVIDER` | Required by `deploy ate-system` and `deploy atenet`. A JSON object: `{"name":"k8s.io"}` deploys and uses the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off; `{"name":"","address":":"}` uses a provider you deploy yourself. See [`docs/egress-credential-injection.md`](../../docs/egress-credential-injection.md) | | `--otlp-endpoint URL` | `--otlp-endpoint URL`, or `ATE_OTLP_ENDPOINT=URL` | Send control plane telemetry to `URL` instead of the cluster default (see [`benchmarking/telemetry/README.md`](../../benchmarking/telemetry/README.md)) | diff --git a/cmd/ate-setup/differences.md b/cmd/ate-setup/differences.md index 6e92de84d5..832ca52fea 100644 --- a/cmd/ate-setup/differences.md +++ b/cmd/ate-setup/differences.md @@ -127,13 +127,7 @@ unpinned reference is rejected by admission and every demo fails to deploy. Pinning also gives a pre-built install the property a `ko` install had for free, that a tag moving afterwards cannot change what is running. -That tag also becomes the substrate version, which names the atelet DaemonSet -and sets the node label partitioning nodes across coexisting versions. It has -to: that label must describe the atelet actually running, which came from the -image, not the checkout `git describe` happens to be sitting on. `VERSION` -overrides it, as it always has. - -The rewrite is textual, one token at a time, matching `SubstituteVersion` above. +The rewrite is textual, one token at a time. A reference is replaced wherever it appears, so the code needs no list of which fields hold images — `workerImage:` in a CRD spec is rewritten like any other — and the applied manifest differs from the checked-in YAML only in the image diff --git a/cmd/ate-setup/internal/cmd/deploy.go b/cmd/ate-setup/internal/cmd/deploy.go index 574b07e6db..8ca21b06d3 100644 --- a/cmd/ate-setup/internal/cmd/deploy.go +++ b/cmd/ate-setup/internal/cmd/deploy.go @@ -59,8 +59,8 @@ The bundled PostgreSQL StatefulSet is skipped when ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING or the ATE_API_POSTGRES_CLOUDSQL_* variables select an external database. -Shape the install with the global --atenet-dataplane, --cluster-size, and ---cordon-control-plane flags.`, +Shape the install with the global --atenet-dataplane, --cluster-size, +--cordon-control-plane, and --fake-data-plane flags.`, // Args runs before the root command loads the configuration, so only the // flag is readable here. Checking it keeps an unusable value from costing // a credential fetch, which is the common case; a value arriving from the diff --git a/cmd/ate-setup/internal/cmd/root.go b/cmd/ate-setup/internal/cmd/root.go index 2b1fd8995f..33c23026e2 100644 --- a/cmd/ate-setup/internal/cmd/root.go +++ b/cmd/ate-setup/internal/cmd/root.go @@ -164,9 +164,7 @@ func recordRun(cmd *cobra.Command, runErr error) { // failed before it had an environment to ask. var version string if env != nil { - if v, _, err := env.SubstrateVersion(); err == nil { - version = v - } + version = env.SubstrateVersion() } var path string diff --git a/cmd/ate-setup/internal/config/config.go b/cmd/ate-setup/internal/config/config.go index 0b415b25c3..249f61bea1 100644 --- a/cmd/ate-setup/internal/config/config.go +++ b/cmd/ate-setup/internal/config/config.go @@ -194,6 +194,11 @@ type Config struct { // ate.dev/workloadType=ate-postgres:NoSchedule for postgres alone. CordonControlPlane bool + // FakeDataPlane installs fake-atelet in place of atelet and + // fake-workersync in place of ate-controller + // (ATE_INSTALL_FAKE_DATA_PLANE), for control-plane benchmarks. + FakeDataPlane bool + // AdditionalEgressExtprocService is the optional NS/SVC:PORT external processor filter. AdditionalEgressExtprocService string @@ -432,6 +437,10 @@ func (c *Config) ScriptEnv() []string { if c.CordonControlPlane { merged["ATE_INSTALL_CORDON_CONTROL_PLANE"] = "true" } + delete(merged, "ATE_INSTALL_FAKE_DATA_PLANE") + if c.FakeDataPlane { + merged["ATE_INSTALL_FAKE_DATA_PLANE"] = "true" + } if c.AdditionalEgressExtprocService != "" { merged["ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE"] = c.AdditionalEgressExtprocService } diff --git a/cmd/ate-setup/internal/config/config_test.go b/cmd/ate-setup/internal/config/config_test.go index 919181a4e9..b9b786299c 100644 --- a/cmd/ate-setup/internal/config/config_test.go +++ b/cmd/ate-setup/internal/config/config_test.go @@ -58,6 +58,7 @@ func loadEnv(t *testing.T) { "ATE_IMAGE_TAG", "ATE_INSTALL_CLUSTER_SIZE", "ATE_INSTALL_CORDON_CONTROL_PLANE", + "ATE_INSTALL_FAKE_DATA_PLANE", "ATE_INSTALL_KIND", "ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER", "ATE_INSTALL_ROLLOUT_TIMEOUT", @@ -208,6 +209,34 @@ func TestLoadCordonControlPlane(t *testing.T) { } } +func TestLoadFakeDataPlane(t *testing.T) { + for _, tc := range []struct { + name string + opts Options + env string + want bool + }{ + {name: "off by default"}, + {name: "flag", opts: Options{FakeDataPlane: true}, want: true}, + {name: "environment", env: "true", want: true}, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + t.Setenv("ATE_INSTALL_FAKE_DATA_PLANE", tc.env) + cfg, err := Load(tc.opts) + if err != nil { + t.Fatalf("Load() error = %v", err) + } + if cfg.FakeDataPlane != tc.want { + t.Errorf("FakeDataPlane = %v, want %v", cfg.FakeDataPlane, tc.want) + } + if _, exported := scriptEnvMap(t, cfg)["ATE_INSTALL_FAKE_DATA_PLANE"]; exported != tc.want { + t.Errorf("ScriptEnv() exports ATE_INSTALL_FAKE_DATA_PLANE = %v, want %v", exported, tc.want) + } + }) + } +} + func TestLoadDockerBuildFlags(t *testing.T) { loadEnv(t) t.Setenv("DOCKER_BUILD_FLAGS", " --cache-from type=gha --cache-to type=gha,mode=max ") diff --git a/cmd/ate-setup/internal/config/load.go b/cmd/ate-setup/internal/config/load.go index 5acc702b56..47a945be29 100644 --- a/cmd/ate-setup/internal/config/load.go +++ b/cmd/ate-setup/internal/config/load.go @@ -259,6 +259,7 @@ func buildConfig(root string, env map[string]string, r *Resolved) (*Config, erro EnvoyConcurrency: envoyConcurrency, ClusterSize: r.String("clusterSize"), CordonControlPlane: r.Bool("cordonControlPlane"), + FakeDataPlane: r.Bool("fakeDataPlane.enabled"), AdditionalEgressExtprocService: r.String("atenet.egress.additionalExtprocService"), CredentialProviderJSON: r.String("atenet.egress.credentialProvider"), AnthropicAPIKey: r.String("demo.anthropicAPIKey"), diff --git a/cmd/ate-setup/internal/config/options_test.go b/cmd/ate-setup/internal/config/options_test.go index 61621ed224..3e8fba4ffc 100644 --- a/cmd/ate-setup/internal/config/options_test.go +++ b/cmd/ate-setup/internal/config/options_test.go @@ -39,6 +39,7 @@ type Options struct { EnvoyConcurrency int ClusterSize string CordonControlPlane bool + FakeDataPlane bool AdditionalEgressExtprocService string CredentialProvider string OtlpEndpoint string @@ -103,6 +104,7 @@ func optionsFlagSet(opts Options) (*pflag.FlagSet, error) { }{ {"kind", opts.Kind}, {"cordon-control-plane", opts.CordonControlPlane}, + {"fake-data-plane", opts.FakeDataPlane}, } { if kv.val { if err := fs.Set(kv.flag, "true"); err != nil { diff --git a/cmd/ate-setup/internal/config/setting.go b/cmd/ate-setup/internal/config/setting.go index b2527db22b..2dcfc6c894 100644 --- a/cmd/ate-setup/internal/config/setting.go +++ b/cmd/ate-setup/internal/config/setting.go @@ -272,6 +272,13 @@ var Registry = []Setting{ "and tainted ate.dev/workloadType=ate-control-plane:NoSchedule, and a one-node pool " + "labeled and tainted ate.dev/workloadType=ate-postgres:NoSchedule for postgres alone", }, + { + Key: "fakeDataPlane.enabled", Env: "ATE_INSTALL_FAKE_DATA_PLANE", Flag: "fake-data-plane", + Kind: KindBool, Default: "false", + Usage: "Install the fake data plane for control-plane benchmarks: fake-atelet in place of " + + "atelet, and fake-workersync in place of ate-controller. Actors do not exist; never " + + "use it on a cluster that serves real actors", + }, { Key: "otlpEndpoint", Env: "ATE_OTLP_ENDPOINT", Flag: "otlp-endpoint", Kind: KindString, Usage: "Send control plane telemetry to this OTLP collector instead of the cluster default", diff --git a/cmd/ate-setup/internal/demos/demotest/demotest.go b/cmd/ate-setup/internal/demos/demotest/demotest.go index f320794be0..66395ed54e 100644 --- a/cmd/ate-setup/internal/demos/demotest/demotest.go +++ b/cmd/ate-setup/internal/demos/demotest/demotest.go @@ -32,9 +32,6 @@ import ( // configuration, so they can be exercised without a kubeconfig. func Env(t *testing.T) *steps.Env { t.Helper() - // Pin VERSION so rendering does not depend on whatever tags (e.g. - // archive/*) happen to be in the local git checkout. - t.Setenv("VERSION", "v0.0.0-test") root, err := config.RepoRoot() if err != nil { t.Fatalf("RepoRoot: %v", err) diff --git a/cmd/ate-setup/internal/demos/render.go b/cmd/ate-setup/internal/demos/render.go index 970ecab412..d5061fd43e 100644 --- a/cmd/ate-setup/internal/demos/render.go +++ b/cmd/ate-setup/internal/demos/render.go @@ -33,16 +33,10 @@ var ExternalVolumePlaceholders = []string{ "EXTERNAL_VOLUMES", } -// Render expands a demo template with the configured bucket name and the -// build version (pool templates pin worker pods to version-labeled nodes). +// Render expands a demo template with the configured bucket name. func Render(e *steps.Env, relPath string, extraValues map[string]string, drop []string) ([]byte, error) { - version, _, err := e.SubstrateVersion() - if err != nil { - return nil, err - } values := map[string]string{ bucketNamePlaceholder: e.Cfg.BucketName, - "SUBSTRATE_VERSION": version, } for k, v := range extraValues { values[k] = v diff --git a/cmd/ate-setup/internal/images/images.go b/cmd/ate-setup/internal/images/images.go index 73902271e6..fb5b392620 100644 --- a/cmd/ate-setup/internal/images/images.go +++ b/cmd/ate-setup/internal/images/images.go @@ -46,6 +46,8 @@ var Components = []string{ "cmd/atenet", "cmd/ateom-gvisor", "cmd/ateom-microvm", + "cmd/benchmarking/isolate/fake-atelet", + "cmd/benchmarking/isolate/fake-workersync", "cmd/credential-provider/kubernetes-secrets", "cmd/objectstore-plugin", "cmd/podcertcontroller", diff --git a/cmd/ate-setup/internal/images/prebuilt.go b/cmd/ate-setup/internal/images/prebuilt.go index 5ffdb39b45..9f3a5403bb 100644 --- a/cmd/ate-setup/internal/images/prebuilt.go +++ b/cmd/ate-setup/internal/images/prebuilt.go @@ -123,11 +123,10 @@ func (p *Prebuilt) Pin(ctx context.Context, image string) (string, error) { // rewrite replaces every ko:// reference with the image it maps to. // -// Substitution is textual, the same way SubstituteVersion fills -// ${SUBSTRATE_VERSION}. References appear in CRD fields as well as pod specs (a -// WorkerPool's spec.workerImage, an ActorTemplate's image), so a schema walk -// would need to know every such field, and rewriting the bytes leaves -// everything else -- the multi-kilobyte Envoy configuration blocks in +// Substitution is textual. References appear in CRD fields as well as pod +// specs (a WorkerPool's spec.workerImage, an ActorTemplate's image), so a +// schema walk would need to know every such field, and rewriting the bytes +// leaves everything else -- the multi-kilobyte Envoy configuration blocks in // particular -- exactly as committed. func (p *Prebuilt) rewrite(ctx context.Context, manifest []byte) ([]byte, error) { // Collect the failures rather than stopping at the first. Diagnosing them diff --git a/cmd/ate-setup/internal/kube/client.go b/cmd/ate-setup/internal/kube/client.go index e97c54209e..5d89ea2253 100644 --- a/cmd/ate-setup/internal/kube/client.go +++ b/cmd/ate-setup/internal/kube/client.go @@ -69,6 +69,12 @@ func New(kubeconfig, kubeContext string) (*Client, error) { return nil, fmt.Errorf("while creating the discovery client: %w", err) } + return NewFromClients(restCfg, typed, dyn, disco), nil +} + +// NewFromClients builds a Client on clients already made, mapping resources +// through a cache of disco's discovery data. +func NewFromClients(restCfg *rest.Config, typed kubernetes.Interface, dyn dynamic.Interface, disco discovery.DiscoveryInterface) *Client { cached := memory.NewMemCacheClient(disco) return &Client{ Config: restCfg, @@ -76,7 +82,7 @@ func New(kubeconfig, kubeContext string) (*Client, error) { Dynamic: dyn, discovery: cached, mapper: restmapper.NewDeferredDiscoveryRESTMapper(cached), - }, nil + } } // InvalidateDiscovery drops the cached discovery data so newly installed CRDs diff --git a/cmd/ate-setup/internal/kube/resources.go b/cmd/ate-setup/internal/kube/resources.go index 8319b6349e..7f310bbbb8 100644 --- a/cmd/ate-setup/internal/kube/resources.go +++ b/cmd/ate-setup/internal/kube/resources.go @@ -18,6 +18,7 @@ import ( "context" "encoding/json" "fmt" + "time" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" @@ -132,6 +133,25 @@ func (c *Client) GetDeployment(ctx context.Context, namespace, name string) (*ap return dep, nil } +// DeleteDeploymentAndWait deletes a Deployment in the foreground and waits +// until it is gone, which is once its pods are. A missing Deployment is not an +// error. +func (c *Client) DeleteDeploymentAndWait(ctx context.Context, namespace, name string, timeout time.Duration) error { + foreground := metav1.DeletePropagationForeground + err := c.Typed.AppsV1().Deployments(namespace).Delete(ctx, name, metav1.DeleteOptions{PropagationPolicy: &foreground}) + if err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("while deleting deployment %s/%s: %w", namespace, name, err) + } + err = poll(ctx, timeout, func(ctx context.Context) (bool, error) { + dep, err := c.GetDeployment(ctx, namespace, name) + return dep == nil, err + }) + if err != nil { + return fmt.Errorf("waiting for deployment %s/%s to be deleted: %w", namespace, name, err) + } + return nil +} + // DeploymentExists reports whether a Deployment is present. delete_demo_actors // used this to decide whether the control plane is still up before trying to // talk to it. @@ -155,21 +175,6 @@ func (c *Client) PatchDeployment(ctx context.Context, namespace, name string, pa return nil } -// DaemonSetNames lists the DaemonSets in a namespace matching a label -// selector. The atelet DaemonSet name carries a substrate version suffix, so -// the installed versions can only be found by label. -func (c *Client) DaemonSetNames(ctx context.Context, namespace, selector string) ([]string, error) { - list, err := c.Typed.AppsV1().DaemonSets(namespace).List(ctx, metav1.ListOptions{LabelSelector: selector}) - if err != nil { - return nil, fmt.Errorf("while listing daemonsets in %s matching %q: %w", namespace, selector, err) - } - names := make([]string, 0, len(list.Items)) - for _, ds := range list.Items { - names = append(names, ds.Name) - } - return names, nil -} - // SetServiceAccountAnnotation adds or removes one annotation on a // ServiceAccount; an empty value removes it. A missing ServiceAccount is not // an error, matching the `|| true` the shell installer removed one under. diff --git a/cmd/ate-setup/internal/kube/resources_test.go b/cmd/ate-setup/internal/kube/resources_test.go new file mode 100644 index 0000000000..2062d4f613 --- /dev/null +++ b/cmd/ate-setup/internal/kube/resources_test.go @@ -0,0 +1,72 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package kube + +import ( + "context" + "testing" + "time" + + appsv1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + kubefake "k8s.io/client-go/kubernetes/fake" + k8stesting "k8s.io/client-go/testing" +) + +// The delete is foreground, so the Deployment outlives its pods and its +// absence means they are gone too: the wait lasts until it is. +func TestDeleteDeploymentAndWait(t *testing.T) { + shortPollInterval(t) + dep := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: "ate-system", Name: "fake-workersync"}} + cs := kubefake.NewClientset(dep) + // The Deployment lingers, as while the garbage collector deletes its + // pods, for the first few reads after the delete. + lingering, gets := 3, 0 + cs.PrependReactor("get", "deployments", func(k8stesting.Action) (bool, runtime.Object, error) { + gets++ + if gets > lingering { + return false, nil, nil + } + terminating := dep.DeepCopy() + now := metav1.Now() + terminating.DeletionTimestamp = &now + return true, terminating, nil + }) + c := &Client{Typed: cs} + + if err := c.DeleteDeploymentAndWait(context.Background(), "ate-system", "fake-workersync", time.Second); err != nil { + t.Fatalf("DeleteDeploymentAndWait() error = %v", err) + } + if gets <= lingering { + t.Errorf("DeleteDeploymentAndWait read the Deployment %d times, returning while it still existed", gets) + } + if dep, err := c.GetDeployment(context.Background(), "ate-system", "fake-workersync"); err != nil || dep != nil { + t.Errorf("after DeleteDeploymentAndWait, GetDeployment() = %v, %v, want nil, nil", dep, err) + } + var policy *metav1.DeletionPropagation + for _, action := range cs.Actions() { + if del, ok := action.(k8stesting.DeleteAction); ok { + policy = del.GetDeleteOptions().PropagationPolicy + } + } + if policy == nil || *policy != metav1.DeletePropagationForeground { + t.Errorf("delete propagation = %v, want %s", policy, metav1.DeletePropagationForeground) + } + + if err := c.DeleteDeploymentAndWait(context.Background(), "ate-system", "fake-workersync", time.Second); err != nil { + t.Errorf("DeleteDeploymentAndWait() of a missing Deployment = %v, want nil", err) + } +} diff --git a/cmd/ate-setup/internal/steps/actors.go b/cmd/ate-setup/internal/steps/actors.go index 7c23d298d1..a8d5658c9a 100644 --- a/cmd/ate-setup/internal/steps/actors.go +++ b/cmd/ate-setup/internal/steps/actors.go @@ -39,8 +39,7 @@ func listAllActors(ctx context.Context, client *ateclient.Client) ([]*ateapipb.A for { // An empty Atespace lists across all atespaces. resp, err := client.ListActors(ctx, &ateapipb.ListActorsRequest{ - PageSize: 1000, - PageToken: pageToken, + Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: pageToken}, }) if err != nil { return nil, err diff --git a/cmd/ate-setup/internal/steps/apienv.go b/cmd/ate-setup/internal/steps/apienv.go index 43e2e26dea..da988eb528 100644 --- a/cmd/ate-setup/internal/steps/apienv.go +++ b/cmd/ate-setup/internal/steps/apienv.go @@ -81,12 +81,18 @@ func (e *Env) CreateAPIServerEnvVars(ctx context.Context) error { return err } } else { + if err := e.refuseBundledOverExternal(ctx); err != nil { + return err + } if readWriteDSN, ownerDSN, err = e.postgresReadWriteConnectionStrings(); err != nil { return err } } } if ownerDSN == "" { + if err := e.refuseDroppingOwnerLogin(ctx); err != nil { + return err + } ownerDSN = readWriteDSN } schema := e.Cfg.PostgresSchemaName() @@ -156,8 +162,40 @@ func buildAPIServerEnvVars(readWriteDSN, ownerDSN, schema string) map[string]str } } -// recordedConnectionStrings reads the credentials paired with an adopted Cloud -// SQL instance. +// refuseBundledOverExternal stops a run that names no database from moving an +// install off its external PostgreSQL, since records hold no connection strings. +func (e *Env) refuseBundledOverExternal(ctx context.Context) error { + // An explicitly empty Cloud SQL instance asks for the bundled database. + if e.Cfg.CloudSQL.InstanceSet { + return nil + } + recorded, _, err := e.recordedConnectionStrings(ctx) + if err != nil { + return err + } + if recorded == "" || usesBundledPostgres(recorded) { + return nil + } + return fmt.Errorf("ate-api-server uses an external PostgreSQL, but ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING is not set") +} + +// refuseDroppingOwnerLogin stops a run that names no owner connection string +// from replacing a separate owner login with the read-write one. The bundled +// database's owner login is ate-setup's own, so it does not count. +func (e *Env) refuseDroppingOwnerLogin(ctx context.Context) error { + recordedReadWrite, recordedOwner, err := e.recordedConnectionStrings(ctx) + if err != nil { + return err + } + if recordedOwner == recordedReadWrite || usesBundledPostgres(recordedOwner) { + return nil + } + return fmt.Errorf("ate-api-server uses a separate PostgreSQL owner login, but ATE_API_POSTGRES_OWNER_CONNECTION_STRING is not set; " + + "set it to the read-write connection string to use one login") +} + +// recordedConnectionStrings reads the connection strings in the cluster's +// Secret. func (e *Env) recordedConnectionStrings(ctx context.Context) (string, string, error) { secret, err := e.Kube.GetSecret(ctx, e.Namespace(), SecretAPIEnvVars) if err != nil || secret == nil { diff --git a/cmd/ate-setup/internal/steps/apienv_test.go b/cmd/ate-setup/internal/steps/apienv_test.go index 16b28d78c3..903aeeef17 100644 --- a/cmd/ate-setup/internal/steps/apienv_test.go +++ b/cmd/ate-setup/internal/steps/apienv_test.go @@ -151,6 +151,114 @@ func TestCreateAPIServerEnvVarsRejectsCustomBundledIdentity(t *testing.T) { } } +// A run without a connection string must not point an install that uses an +// external PostgreSQL at the bundled one: records hold no connection strings, +// so a forgotten export is the common way to get here. +func TestCreateAPIServerEnvVarsKeepsExternalPostgres(t *testing.T) { + for _, tc := range []struct { + name string + cloudSQL config.CloudSQLConfig + recorded string + wantErr bool + }{ + {name: "fresh install", recorded: ""}, + {name: "bundled install", recorded: bundledPostgresDSN(postgressetup.ReadWriteUser)}, + {name: "external install", recorded: "postgres://runtime@database/atepg", wantErr: true}, + { + // An explicitly empty instance is how an operator leaves Cloud SQL. + name: "leaving Cloud SQL", + cloudSQL: config.CloudSQLConfig{InstanceSet: true}, + recorded: "user=svc@p.iam host=127.0.0.1 port=5432 dbname=atepg sslmode=disable", + }, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := config.Config{ + PostgresReadWriteRole: config.DefaultPostgresReadWriteRole, PostgresOwnerRole: config.DefaultPostgresOwnerRole, + CloudSQL: tc.cloudSQL, + } + e := &Env{Cfg: &cfg, Kube: fakeKube(t, + &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}}, + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: SecretAPIEnvVars, Namespace: NamespaceAteSystem}, + Data: map[string][]byte{"ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING": []byte(tc.recorded)}, + }, + )} + err := e.CreateAPIServerEnvVars(t.Context()) + if !tc.wantErr { + if err != nil { + t.Fatal(err) + } + return + } + if err == nil || !strings.Contains(err.Error(), "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING is not set") { + t.Fatalf("CreateAPIServerEnvVars() error = %v, want the missing connection string named", err) + } + secret, err := e.Kube.GetSecret(t.Context(), NamespaceAteSystem, SecretAPIEnvVars) + if err != nil { + t.Fatal(err) + } + if got := string(secret.Data["ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"]); got != tc.recorded || len(secret.StringData) != 0 { + t.Fatalf("Secret rewritten: Data %q, StringData %v", got, secret.StringData) + } + }) + } +} + +// The owner login is no more recorded than the read-write one, so a run that +// gives only the read-write connection string must not replace a separate owner. +func TestCreateAPIServerEnvVarsKeepsOwnerLogin(t *testing.T) { + const readWrite = "postgres://runtime@database/atepg" + for _, tc := range []struct { + name string + owner string + recordedRW string + recordedOwner string + wantErr bool + }{ + {name: "separate owner not given", recordedRW: readWrite, recordedOwner: "postgres://owner@database/atepg", wantErr: true}, + {name: "separate owner given", owner: "postgres://owner@database/atepg", recordedRW: readWrite, recordedOwner: "postgres://owner@database/atepg"}, + {name: "one login", recordedRW: readWrite, recordedOwner: readWrite}, + { + name: "moving off the bundled database", + recordedRW: bundledPostgresDSN(postgressetup.ReadWriteUser), + recordedOwner: bundledPostgresDSN(postgressetup.OwnerUser), + }, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := config.Config{PostgresReadWriteConnectionString: readWrite, PostgresOwnerConnectionString: tc.owner} + e := &Env{Cfg: &cfg, Kube: fakeKube(t, + &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigMapAPIEnvVars, Namespace: NamespaceAteSystem}}, + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: SecretAPIEnvVars, Namespace: NamespaceAteSystem}, + Data: map[string][]byte{ + "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING": []byte(tc.recordedRW), + "ATE_API_POSTGRES_OWNER_CONNECTION_STRING": []byte(tc.recordedOwner), + }, + }, + )} + err := e.CreateAPIServerEnvVars(t.Context()) + if !tc.wantErr { + if err != nil { + t.Fatal(err) + } + return + } + if err == nil || !strings.Contains(err.Error(), "ATE_API_POSTGRES_OWNER_CONNECTION_STRING is not set") { + t.Fatalf("CreateAPIServerEnvVars() error = %v, want the missing owner connection string named", err) + } + secret, err := e.Kube.GetSecret(t.Context(), NamespaceAteSystem, SecretAPIEnvVars) + if err != nil { + t.Fatal(err) + } + if got := string(secret.Data["ATE_API_POSTGRES_OWNER_CONNECTION_STRING"]); got != tc.recordedOwner || len(secret.StringData) != 0 { + t.Fatalf("Secret rewritten: owner %q, StringData %v", got, secret.StringData) + } + }) + } +} + func TestCreateAPIServerEnvVarsPoolSize(t *testing.T) { cfg := config.Config{ PostgresReadWriteConnectionString: "postgres://runtime@postgres/atepg", diff --git a/cmd/ate-setup/internal/steps/cloudsql.go b/cmd/ate-setup/internal/steps/cloudsql.go index 651858633a..d2b0c9892d 100644 --- a/cmd/ate-setup/internal/steps/cloudsql.go +++ b/cmd/ate-setup/internal/steps/cloudsql.go @@ -180,7 +180,8 @@ func cloudSQLDSN(s cloudSQLSettings) (string, error) { // sidecar's configuration, and nothing else. It is empty without Cloud SQL, // which prunes the keys a previous Cloud SQL install left behind. // -// The health checks listen on 9801 because ateapi's metrics own 9090. +// The health checks listen on 9801 because ateapi's metrics and health servers +// own 9090 and 9091. func cloudSQLEnvVars(s cloudSQLSettings) map[string]string { if s.Instance == "" { return map[string]string{} diff --git a/cmd/ate-setup/internal/steps/csi.go b/cmd/ate-setup/internal/steps/csi.go index 4dcf0914a8..129bdd4b5b 100644 --- a/cmd/ate-setup/internal/steps/csi.go +++ b/cmd/ate-setup/internal/steps/csi.go @@ -20,6 +20,7 @@ import ( "os" "os/exec" "strings" + "time" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -290,8 +291,8 @@ spec: // The image cache directories under ateomHostDir were just wiped, so // atelet has to recreate them. - log.Infof("Restarting the atelet DaemonSets (if present)...") - return e.RestartAteletDaemonSets(ctx) + log.Infof("Restarting the atelet DaemonSet (if present)...") + return e.Kube.RolloutRestart(ctx, e.Namespace(), "atelet", time.Now()) } func (e *Env) setupCSINFS(ctx context.Context) error { @@ -422,8 +423,8 @@ spec: return err } - log.Infof("Restarting the atelet DaemonSets (if present)...") - return e.RestartAteletDaemonSets(ctx) + log.Infof("Restarting the atelet DaemonSet (if present)...") + return e.Kube.RolloutRestart(ctx, e.Namespace(), "atelet", time.Now()) } // checkNFSDSupport verifies the host kernel can serve NFS. The in-cluster NFS diff --git a/cmd/ate-setup/internal/steps/delete.go b/cmd/ate-setup/internal/steps/delete.go index 74568bf8b4..6ba85d1516 100644 --- a/cmd/ate-setup/internal/steps/delete.go +++ b/cmd/ate-setup/internal/steps/delete.go @@ -33,13 +33,14 @@ import ( // DeleteAteSystem removes the control plane. // -// PostgreSQL, the agentgateway ConfigMap, the bundled credential provider, and -// the CRDs are deleted explicitly because they are not part of every rendered -// bundle: which of them the install created depends on the router and -// credential provider that were selected, and teardown must not depend on -// remembering that. The provider goes first so that a later failure cannot -// leave its ClusterRole and binding behind. The cluster-scoped trust bundles -// go last, once the controllers that publish them are gone. +// PostgreSQL, the agentgateway ConfigMap, the bundled credential provider, +// fake-workersync, and the CRDs are deleted explicitly because they are not +// part of every rendered bundle: which of them the install created depends on +// the router, credential provider, and data plane that were selected, and +// teardown must not depend on remembering that. The provider goes first so +// that a later failure cannot leave its ClusterRole and binding behind. The +// cluster-scoped trust bundles go last, once the controllers that publish +// them are gone. func (e *Env) DeleteAteSystem(ctx context.Context) error { log.Step("delete_ate_system") @@ -64,14 +65,9 @@ func (e *Env) DeleteAteSystem(ctx context.Context) error { return err } - // atelet DaemonSet names carry a version suffix. - if err := e.Kube.Typed.AppsV1().DaemonSets(e.Namespace()).DeleteCollection(ctx, - metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: "app=atelet"}); err != nil { - return fmt.Errorf("while deleting atelet daemonsets: %w", err) - } - for _, path := range [][]string{ {"components", "agentgateway", "configmap.yaml"}, + {"components", "fake-data-plane", "fake-workersync.yaml"}, {"postgres", "postgres.yaml"}, {"generated"}, } { @@ -94,10 +90,7 @@ func (e *Env) DeleteAteSystem(ctx context.Context) error { if err := e.Kube.WaitDeleted(ctx, namespace, "", NamespacePodCert, e.Cfg.WaitTimeout(BootstrapTimeout)); err != nil { return err } - if err := e.DeleteTrustBundles(ctx); err != nil { - return err - } - return e.UnlabelNodesSubstrateVersion(ctx) + return e.DeleteTrustBundles(ctx) } // releaseSandboxConfigs removes the WorkerPool protection finalizer from diff --git a/cmd/ate-setup/internal/steps/deploy.go b/cmd/ate-setup/internal/steps/deploy.go index 77a9f98a2e..4cf56d1208 100644 --- a/cmd/ate-setup/internal/steps/deploy.go +++ b/cmd/ate-setup/internal/steps/deploy.go @@ -17,9 +17,13 @@ package steps import ( "context" "fmt" + "slices" "strconv" + "strings" "time" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime/schema" @@ -27,19 +31,13 @@ import ( "github.com/agent-substrate/substrate/cmd/ate-setup/internal/log" ) -// crdGVK identifies the CRDs whose presence EnsureCRDs checks for. +// crdGVK is the kind of a CustomResourceDefinition. var crdGVK = schema.GroupVersionKind{ Group: "apiextensions.k8s.io", Version: "v1", Kind: "CustomResourceDefinition", } -// ateCRDs are the custom resources the control plane and demos depend on. -var ateCRDs = []string{ - "workerpools.ate.dev", - "sandboxconfigs.ate.dev", -} - // DeployOptions carries the per-invocation choices for DeployAteSystem. type DeployOptions struct { // SetupCSI additionally installs the CSI driver (nfs, hostpath, both, none). @@ -64,11 +62,8 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { if err := e.RequireCanonicalNamespace("deploy ate-system"); err != nil { return err } - // Fail fast on an unusable build version or credential provider selection - // before touching the cluster. - if _, _, err := e.SubstrateVersion(); err != nil { - return err - } + // Fail fast on an unusable credential provider selection before touching + // the cluster. provider, err := e.Cfg.CredentialProvider() if err != nil { return err @@ -79,15 +74,12 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { if err := e.CheckCSIDriver(opts.SetupCSI); err != nil { return err } - - // The namespace has to exist before RBAC or CRDs are applied. - if err := e.EnsureAteSystemNamespace(ctx); err != nil { + if err := e.requireNoWorkerPods(ctx); err != nil { return err } - // Before the bundle: the atelet DaemonSet applied below and the demo - // WorkerPools' version-pinned pods schedule only to version-labeled nodes. - if err := e.LabelNodesSubstrateVersion(ctx); err != nil { + // The namespace has to exist before RBAC or CRDs are applied. + if err := e.EnsureAteSystemNamespace(ctx); err != nil { return err } @@ -139,10 +131,8 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { if err != nil { return err } - // The atelet DaemonSet in the bundle is version-keyed; fill its - // placeholders after the render (kustomize and ko pass them through). - manifests, err = e.SubstituteVersion(manifests) - if err != nil { + // Before the bundle, which starts ate-controller on the real data plane. + if err := e.removeFakeWorkersync(ctx); err != nil { return err } if err := e.applyWithAPIServerEnvHash(ctx, manifests); err != nil { @@ -172,10 +162,6 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { return err } - ateletName, err := e.AteletDaemonSetName() - if err != nil { - return err - } log.Step("Waiting for ATE system components to be ready...") type rollout struct{ kind, name string } var waits []rollout @@ -184,7 +170,7 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { rollout{kube.KindDeployment, "ate-controller"}, rollout{kube.KindDeployment, "atenet-router"}, rollout{kube.KindDeployment, "atenet-egress"}, - rollout{kube.KindDaemonSet, ateletName}, + rollout{kube.KindDaemonSet, "atelet"}, ) if provider.Kubernetes() { waits = append(waits, rollout{kube.KindDeployment, k8sCredentialProviderDeployment}) @@ -194,6 +180,9 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { return err } } + if err := e.waitFakeWorkersync(ctx); err != nil { + return err + } return e.applyOtelEndpointOverride(ctx) } @@ -329,6 +318,9 @@ func (e *Env) DeployAteAPIServer(ctx context.Context) error { func (e *Env) DeployAteController(ctx context.Context) error { log.Step("deploy_ate_controller") + if err := e.requireInstalledDataPlane(ctx, "deploy ate-controller"); err != nil { + return err + } if err := e.DeployCRDs(ctx); err != nil { return err } @@ -338,23 +330,126 @@ func (e *Env) DeployAteController(ctx context.Context) error { if err := e.applyOtelConfig(ctx); err != nil { return err } - if err := e.renderResolveApply(ctx, e.Cfg.Manifest("ate-controller.yaml")); err != nil { + manifest, err := e.renderDataPlane(e.Cfg.Manifest("ate-controller.yaml")) + if err != nil { + return err + } + if err := e.ResolveAndApplyBytes(ctx, manifest); err != nil { + return err + } + if err := e.Kube.RolloutStatus(ctx, kube.KindDeployment, e.Namespace(), "ate-controller", e.Cfg.RolloutTimeout); err != nil { + return err + } + return e.waitFakeWorkersync(ctx) +} + +// fakeWorkersyncDeployment is the Deployment fakeDataPlaneComponent adds. +const fakeWorkersyncDeployment = "fake-workersync" + +// fakeWorkersyncPath is fakeDataPlaneComponent's fake-workersync manifest. +func (e *Env) fakeWorkersyncPath() string { + return e.Cfg.Manifest("components", "fake-data-plane", "fake-workersync.yaml") +} + +// requireInstalledDataPlane refuses a single-component redeploy that asks for +// a data plane other than the installed one. atelet and ate-controller switch +// together: fake-atelet beside a running ate-controller would have its fake +// Workers deleted, and the real atelet beside a stopped one would have no +// WorkerPool served. deploy ate-system switches both. +func (e *Env) requireInstalledDataPlane(ctx context.Context, step string) error { + installed, err := e.Kube.DeploymentExists(ctx, e.Namespace(), fakeWorkersyncDeployment) + if err != nil { + return err + } + switch { + case e.Cfg.FakeDataPlane && !installed: + return fmt.Errorf("%s --fake-data-plane needs the fake data plane installed; install it with deploy ate-system --fake-data-plane", step) + case !e.Cfg.FakeDataPlane && installed: + return fmt.Errorf("the cluster has the fake data plane, which %s alone would half replace; pass --fake-data-plane, or switch back with deploy ate-system", step) + } + return nil +} + +// workerPoolLabel is the label ate-controller puts on each WorkerPool's +// worker pods, naming the pool. +const workerPoolLabel = "ate.dev/worker-pool" + +// requireNoWorkerPods refuses to install the fake data plane while worker +// pods exist, terminating ones included. Their WorkerPools own them, so +// stopping ate-controller would leave them running, or leave the Workers of +// those terminating unregistered, taking actors fake-workersync knows nothing +// of, beside a fake atelet. A pod that has finished, as an evicted one has, +// holds no Worker, so a pool scaled to zero does not stand in the way. +func (e *Env) requireNoWorkerPods(ctx context.Context) error { + if !e.Cfg.FakeDataPlane { + return nil + } + list, err := e.Kube.Typed.CoreV1().Pods(metav1.NamespaceAll).List(ctx, metav1.ListOptions{LabelSelector: workerPoolLabel}) + if err != nil { + return fmt.Errorf("while listing worker pods: %w", err) + } + names := make([]string, 0, len(list.Items)) + for _, p := range list.Items { + if p.Status.Phase == corev1.PodSucceeded || p.Status.Phase == corev1.PodFailed { + continue + } + names = append(names, p.Namespace+"/"+p.Labels[workerPoolLabel]) + } + if len(names) == 0 { + return nil + } + slices.Sort(names) + return fmt.Errorf("--fake-data-plane cannot replace the real data plane while worker pods exist; delete these WorkerPools and wait for their pods to go: %s", strings.Join(slices.Compact(names), ", ")) +} + +// stopFakeWorkersync deletes fake-workersync's Deployment, if any, and waits +// until its pod is gone. +func (e *Env) stopFakeWorkersync(ctx context.Context) error { + running, err := e.Kube.DeploymentExists(ctx, e.Namespace(), fakeWorkersyncDeployment) + if err != nil || !running { + return err + } + log.Step("remove_fake_workersync") + return e.Kube.DeleteDeploymentAndWait(ctx, e.Namespace(), fakeWorkersyncDeployment, e.Cfg.RolloutTimeout) +} + +// removeFakeWorkersync removes what an earlier fake install left of +// fake-workersync, on an install of the real data plane. It runs before the +// bundle, so fake-workersync is gone before ate-controller starts; +// ate-controller's startup sweep then deletes the fake Workers, which no pod +// backs. The rest of its manifest is deleted on every real install, as +// missing objects are skipped, so a run that failed partway leaves nothing +// behind for long. +func (e *Env) removeFakeWorkersync(ctx context.Context) error { + if e.Cfg.FakeDataPlane { + return nil + } + if err := e.stopFakeWorkersync(ctx); err != nil { return err } - return e.Kube.RolloutStatus(ctx, kube.KindDeployment, e.Namespace(), "ate-controller", e.Cfg.RolloutTimeout) + return e.Kube.DeletePath(ctx, e.fakeWorkersyncPath()) +} + +// waitFakeWorkersync waits for fake-workersync's rollout under +// --fake-data-plane, after every apply that carries atelet or ate-controller. +func (e *Env) waitFakeWorkersync(ctx context.Context) error { + if !e.Cfg.FakeDataPlane { + return nil + } + return e.Kube.RolloutStatus(ctx, kube.KindDeployment, e.Namespace(), fakeWorkersyncDeployment, e.Cfg.RolloutTimeout) } // DeployAtelet redeploys only the atelet DaemonSet. func (e *Env) DeployAtelet(ctx context.Context) error { log.Step("deploy_atelet") - if err := e.EnsureCRDs(ctx); err != nil { + if err := e.requireInstalledDataPlane(ctx, "deploy atelet"); err != nil { return err } - if err := e.EnsureAteSystemNamespace(ctx); err != nil { + if err := e.EnsureCRDs(ctx); err != nil { return err } - if err := e.LabelNodesSubstrateVersion(ctx); err != nil { + if err := e.EnsureAteSystemNamespace(ctx); err != nil { return err } if err := e.applyOtelConfig(ctx); err != nil { @@ -364,29 +459,26 @@ func (e *Env) DeployAtelet(ctx context.Context) error { return err } - var manifest []byte - var err error + path := e.Cfg.Manifest("atelet.yaml") if e.Cfg.Kind { // The kind overlay patches the DaemonSet for the local node layout. - manifest, err = e.KustomizeResolve(ctx, installDir+"/kind/atelet") - } else { - manifest, err = e.ResolveManifest(ctx, e.Cfg.Manifest("atelet.yaml")) + path = e.Cfg.Path(installDir + "/kind/atelet") } + manifest, err := e.renderDataPlane(path) if err != nil { return err } - manifest, err = e.SubstituteVersion(manifest) + manifest, err = e.ResolveManifestBytes(ctx, manifest) if err != nil { return err } if err := e.Kube.ApplyBytes(ctx, manifest); err != nil { return err } - ateletName, err := e.AteletDaemonSetName() - if err != nil { + if err := e.Kube.RolloutStatus(ctx, kube.KindDaemonSet, e.Namespace(), "atelet", e.Cfg.RolloutTimeout); err != nil { return err } - return e.Kube.RolloutStatus(ctx, kube.KindDaemonSet, e.Namespace(), ateletName, e.Cfg.RolloutTimeout) + return e.waitFakeWorkersync(ctx) } // DeployAtenet redeploys the atenet dataplane: router and egress. @@ -466,26 +558,58 @@ func (e *Env) DeploySandboxConfig(ctx context.Context) error { return e.Kube.ApplyPath(ctx, e.Cfg.Manifest("sandboxconfig-gvisor.yaml")) } -// EnsureCRDs installs the CRDs only if they are missing. Component redeploys -// use this so they do not pay for a full CRD apply on every run. +// EnsureCRDs creates the generated CRDs that are missing. Existing CRDs and +// RBAC change only through DeployCRDs, so a component redeploy cannot move them. func (e *Env) EnsureCRDs(ctx context.Context) error { log.Step("ensure_crds") - allPresent := true - for _, name := range ateCRDs { - present, err := e.Kube.Exists(ctx, crdGVK, "", name) + crds, err := e.generatedCRDs() + if err != nil { + return err + } + missing, err := missingCRDs(crds, func(name string) (bool, error) { + return e.Kube.Exists(ctx, crdGVK, "", name) + }) + if err != nil { + return err + } + if len(missing) == 0 { + return nil + } + if err := e.Kube.Apply(ctx, missing); err != nil { + return err + } + return e.waitCRDsEstablished(ctx, missing) +} + +// generatedCRDs loads the generated CRDs, leaving out the RBAC beside them. +func (e *Env) generatedCRDs() ([]*unstructured.Unstructured, error) { + objs, err := kube.LoadPath(e.Cfg.Manifest("generated")) + if err != nil { + return nil, err + } + var crds []*unstructured.Unstructured + for _, obj := range objs { + if obj.GroupVersionKind() == crdGVK { + crds = append(crds, obj) + } + } + return crds, nil +} + +// missingCRDs returns the CRDs that exists reports absent. +func missingCRDs(crds []*unstructured.Unstructured, exists func(name string) (bool, error)) ([]*unstructured.Unstructured, error) { + var missing []*unstructured.Unstructured + for _, crd := range crds { + present, err := exists(crd.GetName()) if err != nil { - return err + return nil, err } if !present { - allPresent = false - break + missing = append(missing, crd) } } - if allPresent { - return nil - } - return e.DeployCRDs(ctx) + return missing, nil } // DeployCRDs applies the generated CRDs and RBAC, waiting for them to reach Established condition. @@ -494,8 +618,17 @@ func (e *Env) DeployCRDs(ctx context.Context) error { if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("generated")); err != nil { return err } - for _, name := range ateCRDs { - if err := e.Kube.WaitCondition(ctx, crdGVK, "", name, "Established", 30*time.Second); err != nil { + crds, err := e.generatedCRDs() + if err != nil { + return err + } + return e.waitCRDsEstablished(ctx, crds) +} + +// waitCRDsEstablished waits until the CRDs are served. +func (e *Env) waitCRDsEstablished(ctx context.Context, crds []*unstructured.Unstructured) error { + for _, crd := range crds { + if err := e.Kube.WaitCondition(ctx, crdGVK, "", crd.GetName(), "Established", 30*time.Second); err != nil { return err } } diff --git a/cmd/ate-setup/internal/steps/deploy_test.go b/cmd/ate-setup/internal/steps/deploy_test.go index e4c57d0f62..48fb56ab11 100644 --- a/cmd/ate-setup/internal/steps/deploy_test.go +++ b/cmd/ate-setup/internal/steps/deploy_test.go @@ -15,6 +15,7 @@ package steps import ( + "path/filepath" "testing" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" @@ -110,3 +111,49 @@ func TestSetPodcertWorkersPerSignerRejectsMissingDeployment(t *testing.T) { t.Error("setPodcertWorkersPerSigner succeeded without a Deployment, want an error") } } + +// Runs over the real generated manifests: EnsureCRDs and DeployCRDs act on what +// generatedCRDs returns, so it has to hold every CRD file and no RBAC. +func TestGeneratedCRDsLeavesOutRBAC(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t)}} + crds, err := e.generatedCRDs() + if err != nil { + t.Fatalf("loading the generated CRDs: %v", err) + } + for _, crd := range crds { + if crd.GetKind() != "CustomResourceDefinition" { + t.Errorf("generatedCRDs returned %s/%s", crd.GetKind(), crd.GetName()) + } + } + files, err := filepath.Glob(filepath.Join(e.Cfg.Manifest("generated"), "ate.dev_*.yaml")) + if err != nil { + t.Fatal(err) + } + if len(crds) != len(files) { + t.Errorf("generatedCRDs returned %d CRDs, want one per CRD file: %v", len(crds), files) + } +} + +// A component redeploy creates only the CRDs the cluster lacks. One that is +// already there, even at an older schema, is left to DeployCRDs. +func TestMissingCRDsLeavesPresentOnesAlone(t *testing.T) { + crd := func(name string) *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(crdGVK) + obj.SetName(name) + return obj + } + crds := []*unstructured.Unstructured{crd("workerpools.ate.dev"), crd("sandboxconfigs.ate.dev"), crd("csidriverconfigs.ate.dev")} + present := map[string]bool{"workerpools.ate.dev": true, "sandboxconfigs.ate.dev": true} + missing, err := missingCRDs(crds, func(name string) (bool, error) { return present[name], nil }) + if err != nil { + t.Fatal(err) + } + if len(missing) != 1 || missing[0].GetName() != "csidriverconfigs.ate.dev" { + var names []string + for _, m := range missing { + names = append(names, m.GetName()) + } + t.Errorf("missingCRDs = %v, want only csidriverconfigs.ate.dev", names) + } +} diff --git a/cmd/ate-setup/internal/steps/env.go b/cmd/ate-setup/internal/steps/env.go index 1e22e5b889..af7f4a1a40 100644 --- a/cmd/ate-setup/internal/steps/env.go +++ b/cmd/ate-setup/internal/steps/env.go @@ -81,11 +81,6 @@ type Env struct { // work on a machine with no container registry credentials configured, and // the build-from-source resolver fails at construction without them. resolver imageResolver - - // substrateVersion caches the derived build version and its object-name - // suffix; see SubstrateVersion. - substrateVersion string - substrateVersionSuffix string } // Namespace is the namespace the control plane is installed into. It is @@ -232,16 +227,6 @@ func (e *Env) Kustomize(overlay string) ([]byte, error) { return kustomize.Build(e.Cfg.Path(overlay)) } -// KustomizeResolve renders an overlay and resolves its images, the -// `kubectl kustomize ... | run_ko resolve -f -` pipeline. -func (e *Env) KustomizeResolve(ctx context.Context, overlay string) ([]byte, error) { - built, err := e.Kustomize(overlay) - if err != nil { - return nil, err - } - return e.ResolveManifestBytes(ctx, built) -} - // EnsureAteSystemNamespace creates the control-plane namespace and waits for it // to go Active. Every deploy path starts here so that RBAC, ConfigMaps, and // workloads have somewhere to land. diff --git a/cmd/ate-setup/internal/steps/overlay.go b/cmd/ate-setup/internal/steps/overlay.go index 9f884d5ee6..fdf671e100 100644 --- a/cmd/ate-setup/internal/steps/overlay.go +++ b/cmd/ate-setup/internal/steps/overlay.go @@ -43,6 +43,11 @@ const ( // apply under --cordon-control-plane. const cordonControlPlaneComponent = installDir + "/components/cordon-control-plane" +// fakeDataPlaneComponent is the kustomize component that puts fake-atelet in +// atelet's DaemonSet and fake-workersync in place of ate-controller, layered +// over the atelet and ate-controller applies under --fake-data-plane. +const fakeDataPlaneComponent = installDir + "/components/fake-data-plane" + // SystemOverlay picks the kustomization for a full control plane install. // // The choice is a product of two switches: kind vs GKE, and the atenet router @@ -66,15 +71,19 @@ func SystemOverlay(cfg *config.Config) string { } // render emits the manifests at path, an absolute manifest file or -// kustomization directory, before image resolution. Under -// --cordon-control-plane it composes path with the cordon-control-plane -// component, so the same node pinning reaches every control plane workload -// whichever apply path delivers it. The component's patch has a name-regex -// target and kustomize leaves a stream alone when nothing in it matches, so -// wrapping a manifest that carries none of those workloads is harmless. -func (e *Env) render(path string) ([]byte, error) { +// kustomization directory, before image resolution, composed with the given +// component directories. Under --cordon-control-plane it adds the +// cordon-control-plane component, so the same node pinning reaches every +// control plane workload whichever apply path delivers it. The component's +// patch has a name-regex target and kustomize leaves a stream alone when +// nothing in it matches, so wrapping a manifest that carries none of those +// workloads is harmless. +func (e *Env) render(path string, components ...string) ([]byte, error) { if e.Cfg.CordonControlPlane { - return kustomize.Compose(path, e.Cfg.Path(cordonControlPlaneComponent)) + components = append(components, e.Cfg.Path(cordonControlPlaneComponent)) + } + if len(components) > 0 { + return kustomize.Compose(path, components...) } info, err := os.Stat(path) if err != nil { @@ -98,6 +107,17 @@ func (e *Env) renderBytes(manifest []byte) ([]byte, error) { return manifest, nil } +// renderDataPlane is render for the manifests that carry atelet or +// ate-controller. Under --fake-data-plane it composes them with the +// fake-data-plane component, whose patches leave a manifest without their +// target alone. +func (e *Env) renderDataPlane(path string) ([]byte, error) { + if !e.Cfg.FakeDataPlane { + return e.render(path) + } + return e.render(path, e.Cfg.Path(fakeDataPlaneComponent)) +} + // renderResolve renders path and resolves its image references. func (e *Env) renderResolve(ctx context.Context, path string) ([]byte, error) { manifest, err := e.render(path) @@ -119,7 +139,11 @@ func (e *Env) renderResolveApply(ctx context.Context, path string) error { // renderSystemManifests produces the full control plane manifest with all // image references resolved. func (e *Env) renderSystemManifests(ctx context.Context) ([]byte, error) { - return e.renderResolve(ctx, e.Cfg.Path(SystemOverlay(e.Cfg))) + manifest, err := e.renderDataPlane(e.Cfg.Path(SystemOverlay(e.Cfg))) + if err != nil { + return nil, err + } + return e.ResolveManifestBytes(ctx, manifest) } // renderAtenetRouterManifest produces the atenet router manifest for the @@ -529,16 +553,5 @@ func (e *Env) applyOtelEndpointOverride(ctx context.Context) error { return err } } - // atelet DaemonSet names carry a version suffix; restart whichever - // versions are installed. - daemonSets, err := e.Kube.DaemonSetNames(ctx, e.Namespace(), "app=atelet") - if err != nil { - return err - } - for _, name := range daemonSets { - if err := e.Kube.RolloutRestart(ctx, e.Namespace(), name, now); err != nil { - return err - } - } - return nil + return e.Kube.RolloutRestart(ctx, e.Namespace(), "atelet", now) } diff --git a/cmd/ate-setup/internal/steps/overlay_test.go b/cmd/ate-setup/internal/steps/overlay_test.go index b213fbfa96..b40979a843 100644 --- a/cmd/ate-setup/internal/steps/overlay_test.go +++ b/cmd/ate-setup/internal/steps/overlay_test.go @@ -22,15 +22,24 @@ import ( "slices" "strings" "testing" + "time" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + dynamicfake "k8s.io/client-go/dynamic/fake" + "k8s.io/client-go/kubernetes/fake" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" kyaml "sigs.k8s.io/kustomize/kyaml/yaml" "sigs.k8s.io/yaml" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" ) // Splicing the provider selection into the real egress manifest replaces the @@ -542,6 +551,230 @@ func TestRenderWithoutCordonLeavesManifestsAlone(t *testing.T) { } } +// dataPlaneWorkloads returns the DaemonSets and Deployments in a rendered +// manifest, by name. +func dataPlaneWorkloads(t *testing.T, manifest []byte) (daemonSets map[string]appsv1.DaemonSet, deployments map[string]appsv1.Deployment) { + t.Helper() + daemonSets, deployments = map[string]appsv1.DaemonSet{}, map[string]appsv1.Deployment{} + for _, doc := range strings.Split(string(manifest), "\n---\n") { + var meta metav1.TypeMeta + if err := yaml.Unmarshal([]byte(doc), &meta); err != nil { + t.Fatalf("rendered document is not valid YAML: %v", err) + } + switch meta.Kind { + case "DaemonSet": + var ds appsv1.DaemonSet + if err := yaml.Unmarshal([]byte(doc), &ds); err != nil { + t.Fatalf("decoding DaemonSet: %v", err) + } + daemonSets[ds.Name] = ds + case "Deployment": + var d appsv1.Deployment + if err := yaml.Unmarshal([]byte(doc), &d); err != nil { + t.Fatalf("decoding Deployment: %v", err) + } + deployments[d.Name] = d + } + } + return daemonSets, deployments +} + +// Under --fake-data-plane every path that applies atelet or ate-controller +// carries the fake data plane: fake-atelet in atelet's DaemonSet, with the +// storage settings the overlay gives atelet's objectstore-plugin, whose +// writes it takes over, ate-controller at zero replicas, and fake-workersync +// beside them. +func TestRenderFakeDataPlane(t *testing.T) { + root := repoRoot(t) + const ateletDS = "atelet" + for _, tc := range []struct { + name string + cfg config.Config + path func(e *Env) string + wantAtelet bool + wantController bool + // wantStorage is the ATE_STORAGE_BACKEND the overlay gives + // atelet's objectstore-plugin. + wantStorage string + }{ + { + name: "base bundle", + cfg: config.Config{Router: config.RouterEnvoy}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + wantAtelet: true, wantController: true, wantStorage: "gcs", + }, + { + name: "kind bundle", + cfg: config.Config{Router: config.RouterEnvoy, Kind: true}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + wantAtelet: true, wantController: true, wantStorage: "s3", + }, + { + name: "agentgateway bundle", + cfg: config.Config{Router: config.RouterAgentgateway}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + wantAtelet: true, wantController: true, wantStorage: "gcs", + }, + { + name: "kind agentgateway bundle", + cfg: config.Config{Router: config.RouterAgentgateway, Kind: true}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + wantAtelet: true, wantController: true, wantStorage: "s3", + }, + { + name: "atelet file", + path: func(e *Env) string { return e.Cfg.Manifest("atelet.yaml") }, + wantAtelet: true, wantStorage: "gcs", + }, + { + name: "kind atelet overlay", + cfg: config.Config{Kind: true}, + path: func(e *Env) string { return e.Cfg.Path(installDir + "/kind/atelet") }, + wantAtelet: true, wantStorage: "s3", + }, + { + name: "controller file", + path: func(e *Env) string { return e.Cfg.Manifest("ate-controller.yaml") }, + wantController: true, + }, + { + name: "controller file under cordon", + cfg: config.Config{CordonControlPlane: true}, + path: func(e *Env) string { return e.Cfg.Manifest("ate-controller.yaml") }, + wantController: true, + }, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := tc.cfg + cfg.Root = root + cfg.FakeDataPlane = true + e := &Env{Cfg: &cfg} + + rendered, err := e.renderDataPlane(tc.path(e)) + if err != nil { + t.Fatalf("renderDataPlane: %v", err) + } + daemonSets, deployments := dataPlaneWorkloads(t, rendered) + + if d, ok := deployments[fakeWorkersyncDeployment]; !ok { + t.Errorf("rendered manifest has no Deployment %s", fakeWorkersyncDeployment) + } else if d.Spec.Strategy.Type != appsv1.RecreateDeploymentStrategyType { + // A rolling update would run two at once. + t.Errorf("%s strategy = %q, want %s", fakeWorkersyncDeployment, d.Spec.Strategy.Type, appsv1.RecreateDeploymentStrategyType) + } + + if ds, ok := daemonSets[ateletDS]; ok != tc.wantAtelet { + t.Errorf("DaemonSet %s present = %v, want %v", ateletDS, ok, tc.wantAtelet) + } else if ok { + pod := ds.Spec.Template + if pod.Labels["ate.dev/fake-atelet"] != "true" { + t.Errorf("fake-atelet pods carry labels %v, want ate.dev/fake-atelet=true", pod.Labels) + } + if len(pod.Spec.InitContainers) != 0 { + t.Errorf("fake-atelet pods keep atelet's init containers %v", pod.Spec.InitContainers) + } + for _, v := range pod.Spec.Volumes { + if v.HostPath != nil { + t.Errorf("fake-atelet pods keep atelet's hostPath volume %s", v.Name) + } + } + if _, ok := pod.Annotations["prometheus.io/scrape"]; ok { + t.Error("fake-atelet pods keep atelet's scrape annotation, but fake-atelet serves no metrics") + } + if pod.Spec.ServiceAccountName != "atelet" { + t.Errorf("fake-atelet runs as %q, want atelet's service account", pod.Spec.ServiceAccountName) + } + c := pod.Spec.Containers[0] + if want := images.KoReference("cmd/benchmarking/isolate/fake-atelet"); c.Image != want { + t.Errorf("atelet container image = %q, want %q", c.Image, want) + } + // fake-atelet runs with its defaults, tuned only through the + // ConfigMap a benchmark writes, beside atelet's own envFrom. + if len(c.Args) != 0 { + t.Errorf("fake-atelet args = %v, want none", c.Args) + } + var tuning *corev1.ConfigMapEnvSource + for _, src := range c.EnvFrom { + if src.ConfigMapRef != nil && src.ConfigMapRef.Name == "fake-atelet-tuning" { + tuning = src.ConfigMapRef + } + } + if tuning == nil || tuning.Optional == nil || !*tuning.Optional { + t.Errorf("fake-atelet envFrom = %+v, want the optional fake-atelet-tuning ConfigMap", c.EnvFrom) + } + if !slices.ContainsFunc(c.EnvFrom, func(src corev1.EnvFromSource) bool { + return src.ConfigMapRef != nil && src.ConfigMapRef.Name == otelConfigMap + }) { + t.Errorf("fake-atelet envFrom = %+v, lost atelet's %s", c.EnvFrom, otelConfigMap) + } + for _, p := range c.Ports { + if p.HostPort != 0 { + t.Errorf("fake-atelet port %s keeps hostPort %d", p.Name, p.HostPort) + } + } + // The probes are atelet's, kept by name; the ports + // replacement must still declare the ones they target. + for _, probe := range []*corev1.Probe{c.ReadinessProbe, c.LivenessProbe} { + if probe == nil || probe.HTTPGet == nil || probe.HTTPGet.Port.StrVal == "" { + continue + } + if !slices.ContainsFunc(c.Ports, func(p corev1.ContainerPort) bool { return p.Name == probe.HTTPGet.Port.StrVal }) { + t.Errorf("fake-atelet probe %s targets port %q, which the container does not declare", probe.HTTPGet.Path, probe.HTTPGet.Port.StrVal) + } + } + env := map[string]string{} + for _, v := range c.Env { + env[v.Name] = v.Value + } + if env["ATE_STORAGE_BACKEND"] != tc.wantStorage { + t.Errorf("fake-atelet ATE_STORAGE_BACKEND = %q, want objectstore-plugin's %q", env["ATE_STORAGE_BACKEND"], tc.wantStorage) + } + if tc.wantStorage == "s3" && env["AWS_ENDPOINT_URL"] == "" { + t.Error("fake-atelet lacks objectstore-plugin's AWS_ENDPOINT_URL") + } + } + + if d, ok := deployments["ate-controller"]; ok != tc.wantController { + t.Errorf("Deployment ate-controller present = %v, want %v", ok, tc.wantController) + } else if ok && (d.Spec.Replicas == nil || *d.Spec.Replicas != 0) { + t.Errorf("ate-controller replicas = %v, want 0", d.Spec.Replicas) + } + + // Both components compose: the cordon pinning survives the fake + // one, and reaches fake-workersync, which stands in for + // ate-controller. + if cfg.CordonControlPlane { + pinned, _ := pinnedWorkloads(t, rendered) + for _, name := range []string{"ate-controller", fakeWorkersyncDeployment} { + if pinned[name] != "ate-control-plane" { + t.Errorf("under --cordon-control-plane %s is pinned to %q, want ate-control-plane", name, pinned[name]) + } + } + } + }) + } +} + +// Without --fake-data-plane the data plane renders as it always has. +func TestRenderDataPlaneWithoutFakeIsRender(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t), Router: config.RouterEnvoy, Kind: true}} + path := e.Cfg.Path(SystemOverlay(e.Cfg)) + got, err := e.renderDataPlane(path) + if err != nil { + t.Fatalf("renderDataPlane: %v", err) + } + want, err := e.render(path) + if err != nil { + t.Fatalf("render: %v", err) + } + if string(got) != string(want) { + t.Error("renderDataPlane without --fake-data-plane differs from render") + } + if _, deployments := dataPlaneWorkloads(t, got); deployments[fakeWorkersyncDeployment].Name != "" { + t.Errorf("render without --fake-data-plane has a %s Deployment", fakeWorkersyncDeployment) + } +} + // The agentgateway egress overlay mounts the CA pool Secret // EnsureEgressMITMCAPoolSecret generates; without it atenet-egress waits on a // Secret nobody creates. @@ -772,13 +1005,10 @@ func TestApplyOtelEndpointOverride(t *testing.T) { }) t.Run("patches and restarts the consumers", func(t *testing.T) { - // The atelet DaemonSet name carries a substrate version suffix, so it - // can only be found by label. atelet := &appsv1.DaemonSet{ ObjectMeta: metav1.ObjectMeta{ Namespace: NamespaceAteSystem, - Name: "atelet-v1-2-3", - Labels: map[string]string{"app": "atelet"}, + Name: "atelet", }, } e := &Env{ @@ -799,7 +1029,7 @@ func TestApplyOtelEndpointOverride(t *testing.T) { if !restartedAt(t, e, "deployment", "ate-api-server") { t.Error("ate-api-server was not restarted") } - if !restartedAt(t, e, "daemonset", "atelet-v1-2-3") { + if !restartedAt(t, e, "daemonset", "atelet") { t.Error("the atelet DaemonSet was not restarted") } }) @@ -945,3 +1175,218 @@ func TestRenderAtenetEgressManifestEnvoyConcurrency(t *testing.T) { } }) } + +// Without --fake-data-plane there is no fake-workersync rollout to wait for. +func TestWaitFakeWorkersyncWithoutTheFlagIsANoOp(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t)}, Kube: fakeKube(t)} + if err := e.waitFakeWorkersync(context.Background()); err != nil { + t.Errorf("waitFakeWorkersync() without --fake-data-plane = %v, want nil", err) + } +} + +// Switching to the real data plane stops fake-workersync, so it is gone +// before ate-controller starts; with none running there is nothing to stop. +func TestStopFakeWorkersync(t *testing.T) { + fakeWorkersync := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: fakeWorkersyncDeployment}} + for _, tc := range []struct { + name string + objects []runtime.Object + }{ + {name: "running", objects: []runtime.Object{fakeWorkersync}}, + {name: "absent"}, + } { + t.Run(tc.name, func(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t), RolloutTimeout: time.Second}, Kube: fakeKube(t, tc.objects...)} + if err := e.stopFakeWorkersync(context.Background()); err != nil { + t.Fatalf("stopFakeWorkersync() = %v, want nil", err) + } + if running, err := e.Kube.DeploymentExists(context.Background(), NamespaceAteSystem, fakeWorkersyncDeployment); err != nil || running { + t.Errorf("after stopFakeWorkersync, DeploymentExists() = %v, %v, want false, nil", running, err) + } + }) + } +} + +// The fake data plane does not go in beside real worker pods, running or +// terminating, which would keep their Workers registered with no real atelet +// left to serve them. A pool scaled to zero has no pods and is no obstacle. +func TestRequireNoWorkerPods(t *testing.T) { + workerPod := func(name string, terminating bool) *corev1.Pod { + p := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{ + Namespace: "benchmark-workloads", Name: name, + Labels: map[string]string{workerPoolLabel: "benchmark-ateom"}, + }, Status: corev1.PodStatus{Phase: corev1.PodRunning}} + if terminating { + now := metav1.Now() + p.DeletionTimestamp = &now + p.Finalizers = []string{"test"} + } + return p + } + scaledToZero := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{ + Namespace: "benchmark-workloads", Name: "idle", + Labels: map[string]string{workerPoolLabel: "idle"}, + }} + evicted := workerPod("w-evicted", false) + evicted.Status = corev1.PodStatus{Phase: corev1.PodFailed, Reason: "Evicted"} + other := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: "ate-api-server-0", Labels: map[string]string{"app": "ate-api-server"}}} + for _, tc := range []struct { + name string + fake bool + objects []runtime.Object + wantErr bool + }{ + {name: "real install with worker pods", objects: []runtime.Object{workerPod("w-0", false), other}}, + {name: "fake install without worker pods", fake: true, objects: []runtime.Object{other}}, + {name: "fake install beside a pool scaled to zero", fake: true, objects: []runtime.Object{scaledToZero, other}}, + {name: "fake install beside evicted worker pods", fake: true, objects: []runtime.Object{evicted, other}}, + {name: "fake install with worker pods", fake: true, objects: []runtime.Object{workerPod("w-0", false), workerPod("w-1", false), other}, wantErr: true}, + {name: "fake install with terminating worker pods", fake: true, objects: []runtime.Object{workerPod("w-0", true), other}, wantErr: true}, + } { + t.Run(tc.name, func(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t), FakeDataPlane: tc.fake}, Kube: fakeKube(t, tc.objects...)} + err := e.requireNoWorkerPods(context.Background()) + if (err != nil) != tc.wantErr { + t.Fatalf("requireNoWorkerPods() = %v, want error %v", err, tc.wantErr) + } + if err != nil && strings.Count(err.Error(), "benchmark-workloads/benchmark-ateom") != 1 { + t.Errorf("error %q does not name the WorkerPool exactly once", err) + } + }) + } +} + +// What a real install removes and delete ate-system deletes is the component's +// own fake-workersync manifest, cluster-scoped RBAC included. +func TestFakeWorkersyncPathIsTheComponentsManifest(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t)}} + objs, err := kube.LoadPath(e.fakeWorkersyncPath()) + if err != nil { + t.Fatalf("LoadPath(%s): %v", e.fakeWorkersyncPath(), err) + } + var kinds []string + for _, obj := range objs { + kinds = append(kinds, obj.GetKind()+"/"+obj.GetName()) + } + for _, want := range []string{"Deployment/" + fakeWorkersyncDeployment, "ClusterRole/fake-workersync", "ClusterRoleBinding/fake-workersync"} { + if !slices.Contains(kinds, want) { + t.Errorf("%s has no %s (found %v)", e.fakeWorkersyncPath(), want, kinds) + } + } + component, err := os.ReadFile(e.Cfg.Path(fakeDataPlaneComponent, "kustomization.yaml")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(component), "- fake-workersync.yaml") { + t.Errorf("the fake-data-plane component does not list fake-workersync.yaml as a resource") + } +} + +// A lone atelet or controller redeploy keeps the installed data plane: it +// cannot switch half of it. +func TestRequireInstalledDataPlane(t *testing.T) { + fakeWorkersync := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: fakeWorkersyncDeployment}} + for _, tc := range []struct { + name string + fake bool + installed bool + wantErr bool + }{ + {name: "real on real"}, + {name: "fake on fake", fake: true, installed: true}, + {name: "fake on real", fake: true, wantErr: true}, + {name: "real on fake", installed: true, wantErr: true}, + } { + t.Run(tc.name, func(t *testing.T) { + var objects []runtime.Object + if tc.installed { + objects = append(objects, fakeWorkersync) + } + e := &Env{Cfg: &config.Config{Root: repoRoot(t), FakeDataPlane: tc.fake}, Kube: fakeKube(t, objects...)} + err := e.requireInstalledDataPlane(context.Background(), "deploy atelet") + if (err != nil) != tc.wantErr { + t.Errorf("requireInstalledDataPlane() = %v, want error %v", err, tc.wantErr) + } + }) + } +} + +// fakeDeleteKube is fakeKube with a dynamic client and discovery for the +// built-in kinds in fake-workersync's manifest, so DeletePath runs. typed +// seeds the typed client, which the Deployment checks read; dyn seeds the +// dynamic client, which DeletePath deletes through. +func fakeDeleteKube(t *testing.T, typed []runtime.Object, dyn []runtime.Object) (*kube.Client, *dynamicfake.FakeDynamicClient) { + t.Helper() + cs := fake.NewClientset(typed...) + cs.Fake.Resources = []*metav1.APIResourceList{ + {GroupVersion: "v1", APIResources: []metav1.APIResource{{Name: "serviceaccounts", Kind: "ServiceAccount", Namespaced: true}}}, + {GroupVersion: "apps/v1", APIResources: []metav1.APIResource{{Name: "deployments", Kind: "Deployment", Namespaced: true}}}, + {GroupVersion: "networking.k8s.io/v1", APIResources: []metav1.APIResource{{Name: "networkpolicies", Kind: "NetworkPolicy", Namespaced: true}}}, + {GroupVersion: "rbac.authorization.k8s.io/v1", APIResources: []metav1.APIResource{ + {Name: "clusterroles", Kind: "ClusterRole"}, + {Name: "clusterrolebindings", Kind: "ClusterRoleBinding"}, + {Name: "roles", Kind: "Role", Namespaced: true}, + {Name: "rolebindings", Kind: "RoleBinding", Namespaced: true}, + }}, + } + dc := dynamicfake.NewSimpleDynamicClient(clientgoscheme.Scheme, dyn...) + return kube.NewFromClients(nil, cs, dc, cs.Discovery()), dc +} + +// A real install deletes fake-workersync's whole manifest, whatever an +// earlier run left of it, after stopping it; a fake install leaves its own +// fake-workersync alone. +func TestRemoveFakeWorkersync(t *testing.T) { + clusterRoles := schema.GroupVersionResource{Group: "rbac.authorization.k8s.io", Version: "v1", Resource: "clusterroles"} + clusterRoleBindings := schema.GroupVersionResource{Group: "rbac.authorization.k8s.io", Version: "v1", Resource: "clusterrolebindings"} + deployment := func() *appsv1.Deployment { + return &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: fakeWorkersyncDeployment}} + } + rbac := func() []runtime.Object { + return []runtime.Object{ + &rbacv1.ClusterRole{ObjectMeta: metav1.ObjectMeta{Name: fakeWorkersyncDeployment}}, + &rbacv1.ClusterRoleBinding{ObjectMeta: metav1.ObjectMeta{Name: fakeWorkersyncDeployment}}, + } + } + for _, tc := range []struct { + name string + fake bool + // running seeds fake-workersync's Deployment; left seeds its RBAC. + running, left bool + wantGone bool + }{ + {name: "real install over a fake one", running: true, left: true, wantGone: true}, + {name: "real install over RBAC a failed run left", left: true, wantGone: true}, + {name: "real install with nothing left"}, + {name: "fake install keeps its own", fake: true, running: true, left: true}, + } { + t.Run(tc.name, func(t *testing.T) { + var typed, dyn []runtime.Object + if tc.running { + typed = append(typed, deployment()) + } + if tc.left { + dyn = rbac() + } + k, dc := fakeDeleteKube(t, typed, dyn) + e := &Env{Cfg: &config.Config{Root: repoRoot(t), FakeDataPlane: tc.fake, RolloutTimeout: time.Second}, Kube: k} + if err := e.removeFakeWorkersync(context.Background()); err != nil { + t.Fatalf("removeFakeWorkersync() = %v", err) + } + running, err := k.DeploymentExists(context.Background(), NamespaceAteSystem, fakeWorkersyncDeployment) + if err != nil { + t.Fatal(err) + } + if running != (tc.running && !tc.wantGone) { + t.Errorf("fake-workersync's Deployment exists = %v after removeFakeWorkersync", running) + } + for _, gvr := range []schema.GroupVersionResource{clusterRoles, clusterRoleBindings} { + _, err := dc.Resource(gvr).Get(context.Background(), fakeWorkersyncDeployment, metav1.GetOptions{}) + gone := apierrors.IsNotFound(err) + if tc.left && gone != tc.wantGone { + t.Errorf("%s/%s gone = %v, want %v (err %v)", gvr.Resource, fakeWorkersyncDeployment, gone, tc.wantGone, err) + } + } + }) + } +} diff --git a/cmd/ate-setup/internal/steps/postgres.go b/cmd/ate-setup/internal/steps/postgres.go index dea4883edf..5067d6a4c8 100644 --- a/cmd/ate-setup/internal/steps/postgres.go +++ b/cmd/ate-setup/internal/steps/postgres.go @@ -18,6 +18,7 @@ import ( "bytes" "context" "fmt" + "net/url" "os" "strings" @@ -31,10 +32,20 @@ import ( "github.com/agent-substrate/substrate/pkg/postgressetup" ) +// bundledPostgresHost is where ate-api-server reaches the bundled PostgreSQL. +const bundledPostgresHost = "postgres.ate-system.svc:5432" + // Each bundled connection presents a certificate for its own login user. func bundledPostgresDSN(user string) string { bundle := "/run/postgres.podcert.ate.dev/" + user + ".pem" - return fmt.Sprintf("postgresql://%s@postgres.ate-system.svc:5432/atepg?sslmode=verify-full&sslrootcert=/run/servicedns.podcert.ate.dev/trust-bundle.pem&sslcert=%s&sslkey=%s", user, bundle, bundle) + return fmt.Sprintf("postgresql://%s@%s/atepg?sslmode=verify-full&sslrootcert=/run/servicedns.podcert.ate.dev/trust-bundle.pem&sslcert=%s&sslkey=%s", user, bundledPostgresHost, bundle, bundle) +} + +// usesBundledPostgres reports whether a connection string names the bundled +// PostgreSQL. +func usesBundledPostgres(dsn string) bool { + u, err := url.Parse(dsn) + return err == nil && u.Host == bundledPostgresHost } func (e *Env) postgresReadWriteConnectionStrings() (string, string, error) { diff --git a/cmd/ate-setup/internal/steps/publish.go b/cmd/ate-setup/internal/steps/publish.go index e43f207a6f..d25ed6c493 100644 --- a/cmd/ate-setup/internal/steps/publish.go +++ b/cmd/ate-setup/internal/steps/publish.go @@ -34,10 +34,7 @@ var workerImages = []string{"ateom-gvisor", "ateom-microvm"} // output. A WorkerPool moves to this build by pointing its workerImage at // the ref. func (e *Env) PublishWorkerImages(ctx context.Context, w io.Writer) error { - version, _, err := e.SubstrateVersion() - if err != nil { - return err - } + version := e.SubstrateVersion() log.Stepf("publish_worker_images (%s)", version) runner, err := e.koRunner() if err != nil { @@ -71,10 +68,7 @@ func (e *Env) PublishReleaseImages(ctx context.Context, w io.Writer) error { if e.Cfg.KODockerRepo == "" { return fmt.Errorf("publishing release images needs a registry to push to; set KO_DOCKER_REPO or --ko-docker-repo") } - tag, _, err := e.SubstrateVersion() - if err != nil { - return err - } + tag := e.SubstrateVersion() log.Stepf("publish_release_images (%s)", tag) runner, err := e.koRunner() if err != nil { diff --git a/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go new file mode 100644 index 0000000000..48d414ed8a --- /dev/null +++ b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go @@ -0,0 +1,517 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package steps + +import ( + "maps" + "path/filepath" + "slices" + "testing" + + corev1 "k8s.io/api/core/v1" + networkingv1 "k8s.io/api/networking/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/runtime" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kustomize" +) + +// systemPod is one ate-system workload's pod template, as the NetworkPolicies +// see it. +type systemPod struct { + workload string // kind/name, for messages + labels map[string]string + ports []corev1.ContainerPort +} + +// systemInstall is everything an install of one configuration puts in +// ate-system: the system overlay plus the manifests the installer applies on +// their own (postgres, the egress gateway, the bundled credential provider). +type systemInstall struct { + pods []systemPod + policies []networkingv1.NetworkPolicy +} + +// unadmittedPorts are the container ports no NetworkPolicy may admit, keyed +// by the pod's app label and the port's name. Every other container port of an +// ate-system pod must be admitted by some policy, so a new port fails +// TestSystemNetworkPoliciesCoverEveryPort until someone decides which list it +// belongs on, rather than being silently dropped by the default deny. +var unadmittedPorts = map[string][]string{ + // /statusz exposes cross-tenant request history (#2275); reach it with a + // port-forward. + "atenet-router": {"status", "extproc"}, + // ext_proc is served to the envoy container on loopback. + "atenet-egress": {"extproc"}, + // The provider ships its own policy admitting only the egress gateway's + // gRPC, and docs/egress-credential-injection.md promises nothing else + // gets in; its probes come from the node. + "k8s-credential-provider": {"statusz", "metrics", "health"}, + // kind only: the rustfs web console. + "rustfs": {"console"}, +} + +// installConfigs are the configurations whose ate-system the policies must +// cover: both dataplanes, on GKE and on kind, each with the real and the fake +// data plane. +func installConfigs() []*config.Config { + var out []*config.Config + for _, kind := range []bool{false, true} { + for _, router := range []string{config.RouterEnvoy, config.RouterAgentgateway} { + for _, fake := range []bool{false, true} { + out = append(out, &config.Config{Kind: kind, Router: router, FakeDataPlane: fake}) + } + } + } + return out +} + +func installName(cfg *config.Config) string { + env := "gke" + if cfg.Kind { + env = "kind" + } + name := env + "-" + cfg.Router + if cfg.FakeDataPlane { + name += "-fake" + } + return name +} + +func loadSystemInstall(t *testing.T, cfg *config.Config) systemInstall { + t.Helper() + root := repoRoot(t) + cfg.Root = root + e := &Env{Cfg: cfg} + + var objs []*unstructured.Unstructured + add := func(what string, rendered []byte, err error) { + t.Helper() + if err != nil { + t.Fatalf("rendering %s: %v", what, err) + } + decoded, err := kube.DecodeManifestBytes(rendered) + if err != nil { + t.Fatalf("decoding %s: %v", what, err) + } + objs = append(objs, decoded...) + } + system := filepath.Join(root, SystemOverlay(cfg)) + rendered, err := e.renderDataPlane(system) + add(system, rendered, err) + rendered, err = e.render(e.postgresManifestPath()) + add(e.postgresManifestPath(), rendered, err) + if cfg.Router == config.RouterAgentgateway { + egress := filepath.Join(root, installDir, "agentgateway-egress") + rendered, err = kustomize.Build(egress) + add(egress, rendered, err) + } else { + loaded, err := kube.LoadPath(e.atenetEgressManifestPath()) + if err != nil { + t.Fatalf("loading %s: %v", e.atenetEgressManifestPath(), err) + } + objs = append(objs, loaded...) + } + provider, err := kube.LoadPath(e.k8sCredentialProviderPath(k8sCredentialProviderManifest)) + if err != nil { + t.Fatalf("loading the credential provider manifest: %v", err) + } + objs = append(objs, provider...) + + var install systemInstall + for _, obj := range objs { + if obj.GetNamespace() != NamespaceAteSystem { + continue + } + switch obj.GetKind() { + case "NetworkPolicy": + var np networkingv1.NetworkPolicy + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, &np); err != nil { + t.Fatalf("converting networkpolicy/%s: %v", obj.GetName(), err) + } + install.policies = append(install.policies, np) + case "Deployment", "DaemonSet", "StatefulSet", "Job": + raw, ok, _ := unstructured.NestedMap(obj.Object, "spec", "template") + if !ok { + t.Fatalf("%s/%s has no pod template", obj.GetKind(), obj.GetName()) + } + var tmpl corev1.PodTemplateSpec + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(raw, &tmpl); err != nil { + t.Fatalf("converting the pod template of %s/%s: %v", obj.GetKind(), obj.GetName(), err) + } + pod := systemPod{workload: obj.GetKind() + "/" + obj.GetName(), labels: tmpl.Labels} + if obj.GetKind() == "Job" { + // The Job controller adds this label to its pods. + pod.labels = map[string]string{"batch.kubernetes.io/job-name": obj.GetName()} + for k, v := range tmpl.Labels { + pod.labels[k] = v + } + } + for _, c := range slices.Concat(tmpl.Spec.InitContainers, tmpl.Spec.Containers) { + pod.ports = append(pod.ports, c.Ports...) + } + install.pods = append(install.pods, pod) + } + } + if len(install.pods) == 0 { + t.Fatal("found no ate-system workloads; the manifests changed shape and this test is checking nothing") + } + return install +} + +func mustSelector(t *testing.T, sel *metav1.LabelSelector) labels.Selector { + t.Helper() + s, err := metav1.LabelSelectorAsSelector(sel) + if err != nil { + t.Fatalf("invalid label selector %v: %v", sel, err) + } + return s +} + +// policiesFor returns the policies that select pod for ingress. +func (in systemInstall) policiesFor(t *testing.T, pod systemPod) []networkingv1.NetworkPolicy { + t.Helper() + var out []networkingv1.NetworkPolicy + for _, np := range in.policies { + if !slices.Contains(np.Spec.PolicyTypes, networkingv1.PolicyTypeIngress) { + continue + } + if mustSelector(t, &np.Spec.PodSelector).Matches(labels.Set(pod.labels)) { + out = append(out, np) + } + } + return out +} + +// portMatches reports whether a policy port admits the container port. +func portMatches(pp networkingv1.NetworkPolicyPort, cp corev1.ContainerPort) bool { + proto := corev1.ProtocolTCP + if pp.Protocol != nil { + proto = *pp.Protocol + } + cproto := cp.Protocol + if cproto == "" { + cproto = corev1.ProtocolTCP + } + if proto != cproto { + return false + } + if pp.Port == nil { + return true + } + if pp.Port.StrVal != "" { + return cp.Name != "" && pp.Port.StrVal == cp.Name + } + return pp.Port.IntVal == cp.ContainerPort +} + +// admittingRules returns the ingress rules, across every policy selecting pod, +// that admit the container port. +func (in systemInstall) admittingRules(t *testing.T, pod systemPod, cp corev1.ContainerPort) []networkingv1.NetworkPolicyIngressRule { + t.Helper() + var out []networkingv1.NetworkPolicyIngressRule + for _, np := range in.policiesFor(t, pod) { + for _, rule := range np.Spec.Ingress { + if len(rule.Ports) == 0 { + out = append(out, rule) + continue + } + for _, pp := range rule.Ports { + if portMatches(pp, cp) { + out = append(out, rule) + break + } + } + } + } + return out +} + +func (in systemInstall) pod(t *testing.T, app string) systemPod { + t.Helper() + for _, p := range in.pods { + if p.labels["app"] == app { + return p + } + } + t.Fatalf("no ate-system workload with app=%s", app) + return systemPod{} +} + +func namedPort(t *testing.T, pod systemPod, name string) corev1.ContainerPort { + t.Helper() + for _, p := range pod.ports { + if p.Name == name { + return p + } + } + t.Fatalf("%s has no container port named %q", pod.workload, name) + return corev1.ContainerPort{} +} + +func numberedPort(t *testing.T, pod systemPod, n int32) corev1.ContainerPort { + t.Helper() + for _, p := range pod.ports { + if p.ContainerPort == n { + return p + } + } + t.Fatalf("%s has no container port %d", pod.workload, n) + return corev1.ContainerPort{} +} + +// requireOnlyFromApps fails unless every rule admitting the port names only +// ate-system pods whose app label is one of apps. +func requireOnlyFromApps(t *testing.T, in systemInstall, pod systemPod, cp corev1.ContainerPort, apps ...string) { + t.Helper() + rules := in.admittingRules(t, pod, cp) + if len(rules) == 0 { + t.Errorf("%s port %d (%s) is not admitted from anywhere, want from app in %v", pod.workload, cp.ContainerPort, cp.Name, apps) + } + for _, rule := range rules { + if len(rule.From) == 0 { + t.Errorf("%s port %d (%s) is admitted from any source, want only app in %v", pod.workload, cp.ContainerPort, cp.Name, apps) + continue + } + for _, peer := range rule.From { + if peer.NamespaceSelector != nil || peer.IPBlock != nil || peer.PodSelector == nil { + t.Errorf("%s port %d (%s) admits peer %+v, want only ate-system pods with app in %v", pod.workload, cp.ContainerPort, cp.Name, peer, apps) + continue + } + sel := mustSelector(t, peer.PodSelector) + if sel.Empty() { + t.Errorf("%s port %d (%s) admits every ate-system pod, want only app in %v", pod.workload, cp.ContainerPort, cp.Name, apps) + continue + } + // The peer must not select a pod outside apps. + for _, other := range in.pods { + if sel.Matches(labels.Set(other.labels)) && !slices.Contains(apps, other.labels["app"]) { + t.Errorf("%s port %d (%s) admits %s, want only app in %v", pod.workload, cp.ContainerPort, cp.Name, other.workload, apps) + } + } + } + } +} + +// admitsFrom reports whether a rule admitting the port lets in the ate-system +// pod client. +func (in systemInstall) admitsFrom(t *testing.T, pod systemPod, cp corev1.ContainerPort, client systemPod) bool { + t.Helper() + for _, rule := range in.admittingRules(t, pod, cp) { + if len(rule.From) == 0 { + return true + } + for _, peer := range rule.From { + if peer.PodSelector != nil && peer.NamespaceSelector == nil && peer.IPBlock == nil && + mustSelector(t, peer.PodSelector).Matches(labels.Set(client.labels)) { + return true + } + } + } + return false +} + +// TestSystemNetworkPoliciesAdmitFakeDataPlane requires the fake data plane's +// own flows to get through the default deny: fake-workersync to +// ate-api-server, which it calls in ate-controller's place, and to the relay +// on each fake-atelet. +func TestSystemNetworkPoliciesAdmitFakeDataPlane(t *testing.T) { + for _, cfg := range installConfigs() { + if !cfg.FakeDataPlane { + continue + } + t.Run(installName(cfg), func(t *testing.T) { + in := loadSystemInstall(t, cfg) + workersync := in.pod(t, "fake-workersync") + api := in.pod(t, "ate-api-server") + if !in.admitsFrom(t, api, numberedPort(t, api, 443), workersync) { + t.Error("ate-api-server port 443 does not admit fake-workersync") + } + atelet := in.pod(t, "atelet") + if !in.admitsFrom(t, atelet, namedPort(t, atelet, "relay"), workersync) { + t.Error("fake-atelet's relay port does not admit fake-workersync") + } + }) + } +} + +// TestSystemNetworkPoliciesDefaultDeny requires every install to carry a +// policy that selects every ate-system pod for ingress and admits nothing. +func TestSystemNetworkPoliciesDefaultDeny(t *testing.T) { + for _, cfg := range installConfigs() { + t.Run(installName(cfg), func(t *testing.T) { + in := loadSystemInstall(t, cfg) + for _, np := range in.policies { + if len(np.Spec.PodSelector.MatchLabels) == 0 && len(np.Spec.PodSelector.MatchExpressions) == 0 && + slices.Equal(np.Spec.PolicyTypes, []networkingv1.PolicyType{networkingv1.PolicyTypeIngress}) && + len(np.Spec.Ingress) == 0 { + return + } + } + t.Error("no NetworkPolicy denies ingress to every ate-system pod by default") + }) + } +} + +// TestSystemNetworkPoliciesCoverEveryPort requires each container port of an +// ate-system pod to be either admitted by a policy or listed in +// unadmittedPorts, and the listed ones to be admitted by none. +func TestSystemNetworkPoliciesCoverEveryPort(t *testing.T) { + for _, cfg := range installConfigs() { + t.Run(installName(cfg), func(t *testing.T) { + in := loadSystemInstall(t, cfg) + for _, pod := range in.pods { + for _, cp := range pod.ports { + denied := slices.Contains(unadmittedPorts[pod.labels["app"]], cp.Name) + admitted := len(in.admittingRules(t, pod, cp)) > 0 + switch { + case denied && admitted: + t.Errorf("%s port %d (%s) must not be admitted by any NetworkPolicy", pod.workload, cp.ContainerPort, cp.Name) + case !denied && !admitted: + t.Errorf("%s port %d (%s) is admitted by no NetworkPolicy, so the default deny drops it; admit it in manifests/ate-install/ate-system-networkpolicies.yaml or add it to unadmittedPorts", pod.workload, cp.ContainerPort, cp.Name) + } + } + } + }) + } +} + +// TestSystemNetworkPoliciesNamedPortsResolve catches a policy port name that +// no selected pod declares, which Kubernetes accepts and silently matches +// nothing. Port names differ between the dataplanes (agentgateway has "stats", +// envoy "envoy-metrics"), so a name only has to resolve in one install. +func TestSystemNetworkPoliciesNamedPortsResolve(t *testing.T) { + type policyPort struct{ policy, port string } + referenced := map[policyPort]bool{} + resolved := map[policyPort]bool{} + for _, cfg := range installConfigs() { + in := loadSystemInstall(t, cfg) + for _, np := range in.policies { + sel := mustSelector(t, &np.Spec.PodSelector) + for _, rule := range np.Spec.Ingress { + for _, pp := range rule.Ports { + if pp.Port == nil || pp.Port.StrVal == "" { + continue + } + key := policyPort{np.Name, pp.Port.StrVal} + referenced[key] = true + for _, pod := range in.pods { + if !sel.Matches(labels.Set(pod.labels)) { + continue + } + for _, cp := range pod.ports { + if cp.Name == pp.Port.StrVal { + resolved[key] = true + } + } + } + } + } + } + } + if len(referenced) == 0 { + t.Fatal("no NetworkPolicy names a port; the manifests changed shape and this test is checking nothing") + } + for key := range referenced { + if !resolved[key] { + t.Errorf("networkpolicy/%s names port %q, which no pod it selects declares in any install", key.policy, key.port) + } + } +} + +// TestSystemNetworkPoliciesLeastPrivilege pins who may reach the sensitive +// ports. +func TestSystemNetworkPoliciesLeastPrivilege(t *testing.T) { + for _, cfg := range installConfigs() { + t.Run(installName(cfg), func(t *testing.T) { + in := loadSystemInstall(t, cfg) + + postgres := in.pod(t, "postgres") + requireOnlyFromApps(t, in, postgres, namedPort(t, postgres, "postgres"), "ate-api-server") + + atelet := in.pod(t, "atelet") + requireOnlyFromApps(t, in, atelet, namedPort(t, atelet, "grpc"), "ate-api-server") + if cfg.FakeDataPlane { + requireOnlyFromApps(t, in, atelet, namedPort(t, atelet, "relay"), "fake-workersync") + } + + // The API admits the control plane and opted-in namespaces only. + api := in.pod(t, "ate-api-server") + for _, rule := range in.admittingRules(t, api, numberedPort(t, api, 443)) { + if len(rule.From) == 0 { + t.Error("ate-api-server port 443 is admitted from any source") + } + for _, peer := range rule.From { + if peer.IPBlock != nil { + t.Errorf("ate-api-server port 443 admits ipBlock %v", peer.IPBlock) + } + if peer.NamespaceSelector != nil { + if want := map[string]string{"ate.dev/api-access": "true"}; !maps.Equal(peer.NamespaceSelector.MatchLabels, want) || len(peer.NamespaceSelector.MatchExpressions) != 0 { + t.Errorf("ate-api-server port 443 admits namespaces %v, want only those labeled %v", peer.NamespaceSelector, want) + } + // An opted-in namespace may host a WorkerPool; its worker + // pods never call the API and must stay out. + if peer.PodSelector == nil { + t.Errorf("ate-api-server port 443 admits every pod of an opted-in namespace, including worker pods") + continue + } + sel := mustSelector(t, peer.PodSelector) + if sel.Matches(labels.Set{workerPoolLabelForTest: "any"}) { + t.Errorf("ate-api-server port 443 admits worker pods (%s) of opted-in namespaces", workerPoolLabelForTest) + } + if !sel.Matches(labels.Set{"app": "dashboard"}) { + t.Errorf("ate-api-server port 443 namespace peer %v does not admit ordinary pods of opted-in namespaces", peer.PodSelector) + } + continue + } + if peer.PodSelector == nil || mustSelector(t, peer.PodSelector).Empty() { + t.Errorf("ate-api-server port 443 admits every ate-system pod (%+v)", peer) + } + } + } + + // The egress gateway admits worker pods only. + egress := in.pod(t, "atenet-egress") + rules := in.admittingRules(t, egress, namedPort(t, egress, "https")) + if len(rules) == 0 { + t.Error("atenet-egress https is admitted from nowhere; worker pods cannot reach the egress gateway") + } + for _, rule := range rules { + if len(rule.From) == 0 { + t.Error("atenet-egress https is admitted from any source, want worker pods only") + } + for _, peer := range rule.From { + // Workers live in their WorkerPool's namespace, so the peer + // has to reach across every namespace, not just ate-system. + if peer.NamespaceSelector == nil || !mustSelector(t, peer.NamespaceSelector).Empty() { + t.Errorf("atenet-egress https admits peer %+v, want worker pods from every namespace (namespaceSelector: {})", peer) + } + if peer.PodSelector == nil || !mustSelector(t, peer.PodSelector).Matches(labels.Set{workerPoolLabelForTest: "any"}) || + mustSelector(t, peer.PodSelector).Matches(labels.Set{"app": "probe"}) { + t.Errorf("atenet-egress https admits peer %+v, want pods labeled %s in any namespace", peer, workerPoolLabelForTest) + } + } + } + }) + } +} + +// workerPoolLabelForTest is the label the WorkerPool controller puts on +// worker pods (cmd/atecontroller/internal/controllers/workerpool_apply.go). +const workerPoolLabelForTest = "ate.dev/worker-pool" diff --git a/cmd/ate-setup/internal/steps/version.go b/cmd/ate-setup/internal/steps/version.go index 8ac5ac72b7..5fb94f21d9 100644 --- a/cmd/ate-setup/internal/steps/version.go +++ b/cmd/ate-setup/internal/steps/version.go @@ -15,139 +15,28 @@ package steps import ( - "context" - "fmt" "os" - "regexp" "strings" - "time" - - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/types" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/ko" - "github.com/agent-substrate/substrate/cmd/ate-setup/internal/log" - "github.com/agent-substrate/substrate/internal/versionlabel" ) -// SubstrateVersion returns the build version and the object-name suffix for atelet. -// The version is the same one ko stamps into the binaries. +// SubstrateVersion returns the build version, the same one ko stamps into the +// binaries. // // With --image-repo nothing is built, so `git describe` would report the -// checkout rather than the images being installed. The image tag is the version -// in that case. It has to be: the version names the atelet DaemonSet and sets -// the node label that partitions nodes across coexisting versions, so it must -// describe the atelet that is actually running, which came from the image. -// VERSION still wins over both, so a tag that is not a valid label value can be -// worked around the same way. -func (e *Env) SubstrateVersion() (version, suffix string, err error) { - if e.substrateVersion != "" { - return e.substrateVersion, e.substrateVersionSuffix, nil - } - var raw string +// checkout rather than the images being installed. The image tag is the +// version in that case. VERSION still wins over both. +func (e *Env) SubstrateVersion() string { if e.Cfg.Images.IsPrebuilt() { - raw = os.Getenv("VERSION") - if raw == "" { - // The tag may carry its digest (v1@sha256:...); the version - // is the tag alone. - raw, _, _ = strings.Cut(e.Cfg.Images.Tag, "@") - } - } else { - // ko.BuildVersion consults VERSION itself before `git describe`. - raw = ko.BuildVersion(e.Cfg.Root) - } - if v := versionlabel.Value(raw); v != raw { - return "", "", fmt.Errorf("build version %q is not a valid label value (it would sanitize to %q); pin a label-safe one with the VERSION env var", raw, v) - } - e.substrateVersion = raw - e.substrateVersionSuffix = versionlabel.NameSuffix(raw) - return e.substrateVersion, e.substrateVersionSuffix, nil -} - -func (e *Env) AteletDaemonSetName() (string, error) { - _, suffix, err := e.SubstrateVersion() - if err != nil { - return "", err - } - return "atelet-" + suffix, nil -} - -// unquotedVersionScalar matches a whole-scalar ${SUBSTRATE_VERSION} value -// kustomize re-emitted unquoted. It is re-quoted before substitution so an -// all-digit version lands as a YAML string, not an integer. -var unquotedVersionScalar = regexp.MustCompile(`(?m)^(\s*[^:\n]+): \$\{SUBSTRATE_VERSION\}$`) - -// SubstituteVersion fills the ${SUBSTRATE_VERSION} and -// ${SUBSTRATE_VERSION_SUFFIX} placeholders in a rendered manifest. -func (e *Env) SubstituteVersion(manifest []byte) ([]byte, error) { - version, suffix, err := e.SubstrateVersion() - if err != nil { - return nil, err - } - s := unquotedVersionScalar.ReplaceAllString(string(manifest), `$1: "$${SUBSTRATE_VERSION}"`) - s = strings.ReplaceAll(s, "${SUBSTRATE_VERSION}", version) - s = strings.ReplaceAll(s, "${SUBSTRATE_VERSION_SUFFIX}", suffix) - return []byte(s), nil -} - -// RestartAteletDaemonSets pod-restarts every atelet DaemonSet by label. -func (e *Env) RestartAteletDaemonSets(ctx context.Context) error { - list, err := e.Kube.Typed.AppsV1().DaemonSets(e.Namespace()).List(ctx, metav1.ListOptions{ - LabelSelector: "app=atelet", - }) - if err != nil { - return fmt.Errorf("while listing atelet daemonsets: %w", err) - } - for _, ds := range list.Items { - if err := e.Kube.RolloutRestart(ctx, e.Namespace(), ds.Name, time.Now()); err != nil { - return err - } - } - return nil -} - -// LabelNodesSubstrateVersion stamps ate.dev/substrate-version on every node -// that does not carry it yet. Nodes that already carry the label keep their -// value: during a rolling upgrade the operator owns the per-node value, and -// an install must not yank nodes across versions behind its back. -func (e *Env) LabelNodesSubstrateVersion(ctx context.Context) error { - version, _, err := e.SubstrateVersion() - if err != nil { - return err - } - log.Stepf("label_nodes_substrate_version (%s)", version) - nodes, err := e.Kube.Typed.CoreV1().Nodes().List(ctx, metav1.ListOptions{ - LabelSelector: "!" + versionlabel.Key, - }) - if err != nil { - return fmt.Errorf("while listing unlabeled nodes: %w", err) - } - patch := fmt.Sprintf(`{"metadata":{"labels":{%q:%q}}}`, versionlabel.Key, version) - for _, node := range nodes.Items { - if _, err := e.Kube.Typed.CoreV1().Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, []byte(patch), metav1.PatchOptions{}); err != nil { - return fmt.Errorf("while labeling node %s: %w", node.Name, err) - } - } - return nil -} - -// UnlabelNodesSubstrateVersion clears ate.dev/substrate-version from every -// node that carries it. Uninstall removes the label with the system: the -// install won't relabels a labeled node, so a leftover label would pin the -// next install to the old version world. -func (e *Env) UnlabelNodesSubstrateVersion(ctx context.Context) error { - log.Step("unlabel_nodes_substrate_version") - nodes, err := e.Kube.Typed.CoreV1().Nodes().List(ctx, metav1.ListOptions{ - LabelSelector: versionlabel.Key, - }) - if err != nil { - return fmt.Errorf("while listing labeled nodes: %w", err) - } - patch := fmt.Sprintf(`{"metadata":{"labels":{%q:null}}}`, versionlabel.Key) - for _, node := range nodes.Items { - if _, err := e.Kube.Typed.CoreV1().Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, []byte(patch), metav1.PatchOptions{}); err != nil { - return fmt.Errorf("while unlabeling node %s: %w", node.Name, err) + if v := os.Getenv("VERSION"); v != "" { + return v } + // The tag may carry its digest (v1@sha256:...); the version + // is the tag alone. + v, _, _ := strings.Cut(e.Cfg.Images.Tag, "@") + return v } - return nil + // ko.BuildVersion consults VERSION itself before `git describe`. + return ko.BuildVersion(e.Cfg.Root) } diff --git a/cmd/ate-setup/internal/steps/version_test.go b/cmd/ate-setup/internal/steps/version_test.go index d0772c1095..c1674e7eab 100644 --- a/cmd/ate-setup/internal/steps/version_test.go +++ b/cmd/ate-setup/internal/steps/version_test.go @@ -21,72 +21,36 @@ import ( "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" ) -// TestSubstrateVersionPrebuilt covers the version a prebuilt install stamps. +// TestSubstrateVersionPrebuilt covers the version a prebuilt install reports. // Root points at a directory that is not a checkout, so `git describe` has // nothing to say and only the image tag can supply it. func TestSubstrateVersionPrebuilt(t *testing.T) { t.Setenv("VERSION", "") cases := []struct { - name string - source images.Source - want string - wantSuffix string + name string + source images.Source + want string }{{ - name: "tag is the version", - source: images.Source{Repo: "example.com/substrate", Tag: "v0.0.0-503-4b3423c0"}, - want: "v0.0.0-503-4b3423c0", - wantSuffix: "v0-0-0-503-4b3423c0", + name: "tag is the version", + source: images.Source{Repo: "example.com/substrate", Tag: "v0.0.0-503-4b3423c0"}, + want: "v0.0.0-503-4b3423c0", }, { - name: "a tag carrying its digest is the version without it", - source: images.Source{Repo: "example.com/substrate", Tag: "v0.0.0@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, - want: "v0.0.0", - wantSuffix: "v0-0-0", + name: "a tag carrying its digest is the version without it", + source: images.Source{Repo: "example.com/substrate", Tag: "v0.0.0@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, + want: "v0.0.0", }, { - name: "built from source falls back to git", - source: images.Source{}, - want: "dev", - wantSuffix: "dev", + name: "built from source falls back to git", + source: images.Source{}, + want: "dev", }} for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { e := &Env{Cfg: &config.Config{Root: t.TempDir(), Images: tc.source}} - got, suffix, err := e.SubstrateVersion() - if err != nil { - t.Fatalf("SubstrateVersion: %v", err) - } - if got != tc.want || suffix != tc.wantSuffix { - t.Fatalf("SubstrateVersion = %q, %q; want %q, %q", got, suffix, tc.want, tc.wantSuffix) + if got := e.SubstrateVersion(); got != tc.want { + t.Fatalf("SubstrateVersion = %q, want %q", got, tc.want) } }) } } - -func TestSubstituteVersion(t *testing.T) { - e := &Env{substrateVersion: "v1.2.3", substrateVersionSuffix: "v1-2-3"} - in := "" + - "metadata:\n" + - " name: atelet-${SUBSTRATE_VERSION_SUFFIX}\n" + - " labels:\n" + - " ate.dev/substrate-version: ${SUBSTRATE_VERSION}\n" + - "nodeSelector:\n" + - " ate.dev/substrate-version: \"${SUBSTRATE_VERSION}\"\n" - // The unquoted scalar is re-quoted (an all-digit version must land as a - // YAML string); already-quoted values and name suffixes pass through. - want := "" + - "metadata:\n" + - " name: atelet-v1-2-3\n" + - " labels:\n" + - " ate.dev/substrate-version: \"v1.2.3\"\n" + - "nodeSelector:\n" + - " ate.dev/substrate-version: \"v1.2.3\"\n" - - got, err := e.SubstituteVersion([]byte(in)) - if err != nil { - t.Fatalf("SubstituteVersion: %v", err) - } - if string(got) != want { - t.Fatalf("SubstituteVersion mismatch:\ngot:\n%s\nwant:\n%s", got, want) - } -} diff --git a/cmd/ate-setup/shim_test.go b/cmd/ate-setup/shim_test.go index 5d9658c6f6..159a3e210a 100644 --- a/cmd/ate-setup/shim_test.go +++ b/cmd/ate-setup/shim_test.go @@ -214,6 +214,13 @@ func TestShimTranslatesFlags(t *testing.T) { name: "cluster profile flags are forwarded in either value form", args: []string{"--deploy-ate-system", "--cluster-size", "size10", "--cordon-control-plane"}, want: []string{"--cluster-size=size10 --cordon-control-plane deploy ate-system --setup-csi=none"}, + }, { + name: "--fake-data-plane is forwarded to every action", + args: []string{"--deploy-ate-system", "--fake-data-plane", "--deploy-atelet"}, + want: []string{ + "--fake-data-plane deploy ate-system --setup-csi=none", + "--fake-data-plane deploy atelet", + }, }, { name: "--cluster-size takes an attached value", args: []string{"--cluster-size=size10", "--deploy-ate-apiserver"}, @@ -482,33 +489,68 @@ func TestExampleConfigurationParses(t *testing.T) { } } -// The settings table is generated from the registry, so a setting added -// without documenting it is the drift this catches. -func TestOperatorDocListsEverySetting(t *testing.T) { - path := docPath(t, "docs/operator-install.md") - raw, err := os.ReadFile(path) +const operatorDoc = "docs/operator-install.md" + +// settingDocs maps a key prefix to the document that lists the settings under +// it. Demo, benchmark, and e2e settings are documented with what uses them, +// so the operator page lists only what an install uses. +var settingDocs = []struct{ prefix, doc string }{ + {"demo.counter.", "demos/counter/README.md"}, + {"demo.anthropicAPIKey", "demos/claude-code-multiplex/README.md"}, + {"benchmark.", "benchmarking/README.md"}, + {"e2e.", "internal/e2e/README.md"}, +} + +// settingDoc returns the document that lists the setting with this key. +func settingDoc(key string) string { + for _, d := range settingDocs { + if strings.HasPrefix(key, d.prefix) { + return d.doc + } + } + return operatorDoc +} + +// readDoc returns the contents of a repository file, by relative path. +func readDoc(t *testing.T, rel string) string { + t.Helper() + raw, err := os.ReadFile(docPath(t, rel)) if err != nil { t.Fatalf("ReadFile: %v", err) } - doc := string(raw) + return string(raw) +} + +// The settings tables are written by hand, so a setting added without +// documenting it is the drift this catches. A demo, benchmark, or e2e setting +// must also stay out of the operator page. +func TestDocsListEverySetting(t *testing.T) { + docs := map[string]string{operatorDoc: readDoc(t, operatorDoc)} + for _, d := range settingDocs { + docs[d.doc] = readDoc(t, d.doc) + } for _, s := range config.All() { t.Run(s.Key, func(t *testing.T) { + rel := settingDoc(s.Key) want := []string{"`" + s.Key + "`", "`" + s.Env + "`"} // A secret has no flag, so the docs have none to name. if s.Flag != "" { want = append(want, "`--"+s.Flag+"`") } for _, want := range want { - if !strings.Contains(doc, want) { - t.Errorf("docs/operator-install.md does not mention %s", want) + if !strings.Contains(docs[rel], want) { + t.Errorf("%s does not mention %s", rel, want) } } + if rel != operatorDoc && strings.Contains(docs[operatorDoc], "`"+s.Key+"`") { + t.Errorf("%s lists %s, which belongs in %s", operatorDoc, s.Key, rel) + } }) } } -// The operator documentation must not name a setting that no longer exists: +// The settings documentation must not name a setting that no longer exists: // following it would configure nothing and the install would silently use a // default. func TestDocsNameNoRemovedSetting(t *testing.T) { @@ -528,7 +570,11 @@ func TestDocsNameNoRemovedSetting(t *testing.T) { if err != nil { t.Fatalf("RepoRoot: %v", err) } - for _, rel := range []string{"docs/operator-install.md", "docs/examples/substrate-install.yaml"} { + files := []string{operatorDoc, "docs/examples/substrate-install.yaml"} + for _, d := range settingDocs { + files = append(files, d.doc) + } + for _, rel := range files { raw, err := os.ReadFile(filepath.Join(root, rel)) if err != nil { t.Fatalf("ReadFile(%s): %v", rel, err) diff --git a/cmd/ateapi/internal/apivalidation/access_policy_test.go b/cmd/ateapi/internal/apivalidation/access_policy_test.go index d229c84df2..b88d0add8c 100644 --- a/cmd/ateapi/internal/apivalidation/access_policy_test.go +++ b/cmd/ateapi/internal/apivalidation/access_policy_test.go @@ -115,6 +115,13 @@ func TestValidateCreateGlobalAccessPolicyRequest(t *testing.T) { p.Bindings[1].Role = "admin" })}, wantError: field.ErrorList{field.NotSupported[string](policyBindings.Index(1).Child("role"), "", nil)}, + }, { + // System component roles are granted only by ateapi's configuration. + name: "system component role", + req: &ateapipb.CreateGlobalAccessPolicyRequest{AccessPolicy: validAccessPolicy(func(p *ateapipb.AccessPolicy) { + p.Bindings[1].Role = authz.RoleIngressRouter + })}, + wantError: field.ErrorList{field.NotSupported[string](policyBindings.Index(1).Child("role"), "", nil)}, }, { name: "duplicate role", req: &ateapipb.CreateGlobalAccessPolicyRequest{AccessPolicy: validAccessPolicy(func(p *ateapipb.AccessPolicy) { @@ -285,6 +292,12 @@ func TestValidateCreateAtespaceAccessPolicyRequest(t *testing.T) { p.Bindings[0].Role = "admin" })}, wantError: field.ErrorList{field.NotSupported[string](policyBindings.Index(0).Child("role"), "", nil)}, + }, { + name: "system component role", + req: &ateapipb.CreateAtespaceAccessPolicyRequest{Atespace: atespace, AccessPolicy: validAccessPolicy(func(p *ateapipb.AccessPolicy) { + p.Bindings[0].Role = authz.RoleEgressGateway + })}, + wantError: field.ErrorList{field.NotSupported[string](policyBindings.Index(0).Child("role"), "", nil)}, }, { name: "invalid member", req: &ateapipb.CreateAtespaceAccessPolicyRequest{Atespace: atespace, AccessPolicy: validAccessPolicy(func(p *ateapipb.AccessPolicy) { diff --git a/cmd/ateapi/internal/apivalidation/actor.go b/cmd/ateapi/internal/apivalidation/actor.go index 77c0d27a5b..fc1b6895a7 100644 --- a/cmd/ateapi/internal/apivalidation/actor.go +++ b/cmd/ateapi/internal/apivalidation/actor.go @@ -16,12 +16,11 @@ package apivalidation import ( "context" - "strings" + "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/api/validate" - "k8s.io/apimachinery/pkg/util/validation" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -83,10 +82,10 @@ func ValidateRevertActorRequest(ctx context.Context, req *ateapipb.RevertActorRe return Validate_RevertActorRequest(ctx, op, nil, req, nil) } -func ValidateMintActorJWTRequest(ctx context.Context, req *ateapipb.MintActorJWTRequest) field.ErrorList { +func ValidateMintActorJWTRequest(ctx context.Context, req *ateapipb.MintActorJwtRequest) field.ErrorList { // Call the generated validation. op := operation.Operation{Type: operation.Create} - return Validate_MintActorJWTRequest(ctx, op, nil, req, nil) + return Validate_MintActorJwtRequest(ctx, op, nil, req, nil) } func ValidateMintActorCertificateRequest(ctx context.Context, req *ateapipb.MintActorCertificateRequest) field.ErrorList { @@ -134,34 +133,14 @@ func ValidateCustom_WorkerAssignment_WorkerPodIps(_ context.Context, _ operation return validateDualStackIPs(fldPath, value) } -// ValidateCustom_ExternalVolume_VolumeType checks that a volume type string is well-formed. -// It allows an optional "substrate.io/" prefix, followed by a valid DNS-1123 subdomain. func ValidateCustom_ExternalVolume_VolumeType(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - if value == nil || *value == "" { - return nil - } - var errs field.ErrorList - valToValidate := strings.TrimPrefix(*value, "substrate.io/") - for _, msg := range validation.IsDNS1123Subdomain(valToValidate) { - errs = append(errs, field.Invalid(fldPath, *value, msg)) - } - return errs + return resources.ValidateVolumeType(fldPath, *value) } -// ValidateCustom_ExternalVolume_StorageVolumeId checks that an external volume's storage ID does not -// contain control characters (U+0000-U+0008, U+000B, U+000C, U+000E-U+001F, U+007F-U+009F). func ValidateCustom_ExternalVolume_StorageVolumeId(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - if value == nil || *value == "" { - return nil - } - for _, r := range *value { - if (r >= 0x0000 && r <= 0x0008) || - r == 0x000B || - r == 0x000C || - (r >= 0x000E && r <= 0x001F) || - (r >= 0x007F && r <= 0x009F) { - return field.ErrorList{field.Invalid(fldPath, *value, "must not contain control characters (U+0000-U+0008, U+000B, U+000C, U+000E-U+001F, U+007F-U+009F)")} - } - } - return nil + return resources.ValidateStorageVolumeID(fldPath, *value) +} + +func ValidateCustom_ExternalVolume_VolumeContext(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ map[string]string) field.ErrorList { + return resources.ValidateCSIMap(fldPath, value) } diff --git a/cmd/ateapi/internal/apivalidation/actor_template.go b/cmd/ateapi/internal/apivalidation/actor_template.go index 4df9c84a41..483861fe91 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template.go +++ b/cmd/ateapi/internal/apivalidation/actor_template.go @@ -17,13 +17,8 @@ package apivalidation import ( "context" "fmt" - "regexp" - "strings" - - "github.com/distribution/reference" "github.com/agent-substrate/substrate/internal/resources" - "github.com/agent-substrate/substrate/internal/volumepath" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/api/resource" @@ -95,91 +90,40 @@ func ValidateCustom_CreateActorTemplateRequest_ActorTemplate(_ context.Context, return errs } -// httpGetPathRE constrains wakeup probe paths to RFC 3986 path-segment -// characters only, with well-formed percent-escapes, and no query string -// or fragment. -var httpGetPathRE = regexp.MustCompile(`^/([A-Za-z0-9\-._~!$&'()*+,;=:@/]|%[0-9A-Fa-f]{2})*$`) - -func ValidateCustom_HTTPGetAction_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - if !httpGetPathRE.MatchString(*value) { - return field.ErrorList{field.Invalid(fldPath, *value, "must be a URL path starting with '/', using only RFC 3986 path-segment characters, without query or fragment")} - } - return nil +func ValidateCustom_HttpGetAction_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateHTTPGetPath(fldPath, *value) } -// mountPathBadSegmentRE matches '.' or '..' path segments. -var mountPathBadSegmentRE = regexp.MustCompile(`(^|/)[.][.]?(/|$)`) - -// ValidateCustom_VolumeMount_MountPath requires a clean absolute Unix path -// that starts with '/', is not '/', and contains no ':', '.' or '..' -// segments, '//', trailing '/', or control characters. func ValidateCustom_VolumeMount_MountPath(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - p := *value - bad := !strings.HasPrefix(p, "/") || len(p) == 1 || - strings.HasSuffix(p, "/") || strings.Contains(p, "//") || - strings.Contains(p, ":") || mountPathBadSegmentRE.MatchString(p) - if !bad { - for _, r := range p { - if r < 0x20 || r == 0x7f { - bad = true - break - } - } - } - if bad { - return field.ErrorList{field.Invalid(fldPath, p, "must be a clean absolute Unix path: must start with '/', not be '/', and contain no ':', '..', '.', '//', trailing '/', or control characters")} - } - return nil + return resources.ValidateMountPath(fldPath, *value) } // ValidateCustom_Container_VolumeMounts rejects mounts that nest under one -// another. +// another. Mount-path uniqueness is enforced by the list key. func ValidateCustom_Container_VolumeMounts(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []*ateapipb.VolumeMount) field.ErrorList { - var errs field.ErrorList + paths := make([]string, len(value)) for i, m := range value { - path := m.GetMountPath() - if path == "" { - continue // required is enforced by tags - } - // Nested mounts are unsupported (volumes cannot mount onto - // other volumes). - for j := 0; j < i; j++ { - prior := value[j].GetMountPath() - if prior == "" || prior == path { - continue - } - if strings.HasPrefix(path, prior+"/") || strings.HasPrefix(prior, path+"/") { - errs = append(errs, field.Invalid(fldPath.Index(i).Child("mount_path"), path, - fmt.Sprintf("must not nest under or over another mount (%q)", prior))) - } - } - } - return errs -} - -// validateProjectedPath applies the projected-path rule shared with atelet, -// which re-checks it before writing to the host. -func validateProjectedPath(fldPath *field.Path, p string) field.ErrorList { - if err := volumepath.ValidateProjected(p); err != nil { - return field.ErrorList{field.Invalid(fldPath, p, err.Error())} + paths[i] = m.GetMountPath() } - return nil + return resources.ValidateNestedMountPaths(fldPath, paths) } func ValidateCustom_ActorMetadataItem_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - return validateProjectedPath(fldPath, *value) + return resources.ValidateProjectedPath(fldPath, *value) } func ValidateCustom_TrustBundleDataSource_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - return validateProjectedPath(fldPath, *value) + return resources.ValidateProjectedPath(fldPath, *value) } -// ValidateCustom_SystemInfoVolumeSource_DataSources requires every projected -// file path to be unique across all data sources: atelet writes them in -// order into one tree, so a repeated path silently clobbers the earlier file. +// ValidateCustom_SystemInfoVolumeSource_DataSources allows at most one +// actor_metadata entry and requires every projected file path to be unique +// across all data sources: atelet writes them in order into one tree, so a +// repeated path silently clobbers the earlier file. func ValidateCustom_SystemInfoVolumeSource_DataSources(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []*ateapipb.SystemInfoDataSource) field.ErrorList { var errs field.ErrorList seen := sets.New[string]() + sawMetadata := false for i, ds := range value { switch { case ds == nil: @@ -189,6 +133,10 @@ func ValidateCustom_SystemInfoVolumeSource_DataSources(_ context.Context, _ oper } seen.Insert(ds.TrustBundle.Path) case ds.ActorMetadata != nil: + if sawMetadata { + errs = append(errs, field.Forbidden(fldPath.Index(i).Child("actor_metadata"), "at most one actor_metadata entry may appear")) + } + sawMetadata = true for j, item := range ds.ActorMetadata.Items { if item == nil { continue @@ -203,31 +151,12 @@ func ValidateCustom_SystemInfoVolumeSource_DataSources(_ context.Context, _ oper return errs } -// validatePinnedImage requires a well-formed OCI image reference pinned by -// digest (e.g. "name@sha256:..."): changing the image content under a fixed -// reference invalidates snapshots. It parses with the same grammar the -// container runtimes use, so a malformed digest is rejected rather than -// treated as pinned. -func validatePinnedImage(fldPath *field.Path, value string) field.ErrorList { - if value == "" { - return nil // required is enforced by tags - } - ref, err := reference.ParseNormalizedNamed(value) - if err != nil { - return field.ErrorList{field.Invalid(fldPath, value, fmt.Sprintf("must be a well-formed image reference: %v", err))} - } - if _, ok := ref.(reference.Digested); !ok { - return field.ErrorList{field.Invalid(fldPath, value, "must be pinned by digest (changing the image invalidates snapshots)")} - } - return nil -} - func ValidateCustom_ImageVolumeSource_Reference(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - return validatePinnedImage(fldPath, *value) + return resources.ValidatePinnedImage(fldPath, *value) } func ValidateCustom_Container_Image(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - return validatePinnedImage(fldPath, *value) + return resources.ValidatePinnedImage(fldPath, *value) } func ValidateCustom_ExternalVolumeTemplate_Capacity(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { @@ -247,45 +176,14 @@ func ValidateCustom_SnapshotConfig_StorageLocation(_ context.Context, _ operatio return nil } -// envVarNameRE constrains env var names to any printable ASCII character -// except '='. -var envVarNameRE = regexp.MustCompile(`^[ -<>-~]+$`) - func ValidateCustom_EnvVar_Name(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - if !envVarNameRE.MatchString(*value) { - return field.ErrorList{field.Invalid(fldPath, *value, "may contain any printable ASCII character except '='")} - } - return nil -} - -// capabilityRE constrains Linux capability names: uppercase, without the -// "CAP_" prefix (which is added when the OCI spec is written; the prefixed -// spelling would silently grant nothing). -var capabilityRE = regexp.MustCompile(`^[A-Z][A-Z0-9_]*$`) - -func validateCapabilities(fldPath *field.Path, caps []string, allowAll bool) field.ErrorList { - var errs field.ErrorList - for i, c := range caps { - p := fldPath.Index(i) - switch { - case c == "ALL" && !allowAll: - errs = append(errs, field.Invalid(p, c, "add does not accept 'ALL'; name the individual capabilities the container needs")) - case c == "ALL": - case len(c) > 63: - errs = append(errs, field.TooLong(p, nil, 63)) - case strings.HasPrefix(c, "CAP_"): - errs = append(errs, field.Invalid(p, c, "must be named without the 'CAP_' prefix (e.g. 'NET_BIND_SERVICE')")) - case !capabilityRE.MatchString(c): - errs = append(errs, field.Invalid(p, c, "must be an uppercase capability name like 'NET_BIND_SERVICE'")) - } - } - return errs + return resources.ValidateEnvVarName(fldPath, *value) } func ValidateCustom_Capabilities_Add(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList { - return validateCapabilities(fldPath, value, false) + return resources.ValidateCapabilities(fldPath, value, false) } func ValidateCustom_Capabilities_Drop(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList { - return validateCapabilities(fldPath, value, true) + return resources.ValidateCapabilities(fldPath, value, true) } diff --git a/cmd/ateapi/internal/apivalidation/actor_template_test.go b/cmd/ateapi/internal/apivalidation/actor_template_test.go index 12d979784d..35d10bcf6f 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template_test.go +++ b/cmd/ateapi/internal/apivalidation/actor_template_test.go @@ -222,7 +222,7 @@ func TestValidateListActorTemplatesRequest(t *testing.T) { want field.ErrorList }{{ "valid", - &ateapipb.ListActorTemplatesRequest{PageSize: 10}, + &ateapipb.ListActorTemplatesRequest{Options: &ateapipb.ListOptions{PageSize: 10}}, nil, }, { "zero page size", @@ -238,8 +238,8 @@ func TestValidateListActorTemplatesRequest(t *testing.T) { field.ErrorList{field.Invalid(field.NewPath("atespace"), "NS_1", "").WithOrigin("format=k8s-short-name")}, }, { "negative page size", - &ateapipb.ListActorTemplatesRequest{PageSize: -1}, - field.ErrorList{field.Invalid(field.NewPath("page_size"), int32(-1), "").WithOrigin("minimum")}, + &ateapipb.ListActorTemplatesRequest{Options: &ateapipb.ListOptions{PageSize: -1}}, + field.ErrorList{field.Invalid(field.NewPath("options", "page_size"), int32(-1), "").WithOrigin("minimum")}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -447,40 +447,6 @@ func TestValidateActorTemplate(t *testing.T) { } }, want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("volume_mounts").Index(1).Child("mount_path"), nil, "")}, - }, { - name: "deeply nested mount path is rejected", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{ - {Name: "data", MountPath: "/data"}, - {Name: "deep", MountPath: "/data/a/b/c"}, - } - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("volume_mounts").Index(1).Child("mount_path"), nil, "")}, - }, { - name: "nesting is rejected regardless of listing order", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{ - {Name: "deep", MountPath: "/data/a/b/c"}, - {Name: "data", MountPath: "/data"}, - } - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("volume_mounts").Index(1).Child("mount_path"), nil, "")}, - }, { - name: "sibling subpaths under an unmounted ancestor are allowed", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{ - {Name: "data", MountPath: "/data/a"}, - {Name: "other", MountPath: "/data/b"}, - } - }, - }, { - name: "shared segment prefix below the root is allowed", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{ - {Name: "data", MountPath: "/data/a"}, - {Name: "other", MountPath: "/data/ab"}, - } - }, }, { name: "the same path in different containers is allowed", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -488,22 +454,6 @@ func TestValidateActorTemplate(t *testing.T) { tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{{Name: "data", MountPath: "/var/data"}} tmpl.Containers[1].VolumeMounts = []*ateapipb.VolumeMount{{Name: "data", MountPath: "/var/data"}} }, - }, { - name: "shared path prefix without nesting is allowed", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{ - {Name: "data", MountPath: "/data"}, - {Name: "other", MountPath: "/database"}, - } - }, - }, { - name: "two volumes at distinct paths are allowed", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{ - {Name: "data", MountPath: "/var/data"}, - {Name: "other", MountPath: "/mnt/other"}, - } - }, }, { name: "duplicate volume name", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -571,35 +521,11 @@ func TestValidateActorTemplate(t *testing.T) { field.Invalid(field.NewPath("containers").Index(0).Child("image"), nil, ""), field.TooLong(field.NewPath("containers").Index(0).Child("image"), nil, 512).WithOrigin("maxLength"), }, - }, { - name: "invalid image: bare repository without digest", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Image = "ubuntu" - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("image"), nil, "")}, }, { name: "valid image: pinned by digest", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Containers[0].Image = "example.com/app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" }, - }, { - name: "invalid image: uppercase repository", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Image = "example.com/App:v1" - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("image"), nil, "")}, - }, { - name: "invalid image: malformed digest", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Image = "example.com/app@sha256:abc" - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("image"), nil, "")}, - }, { - name: "invalid image: empty tag", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Image = "example.com/app:" - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("image"), nil, "")}, }, { name: "container image missing digest", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -623,23 +549,12 @@ func TestValidateActorTemplate(t *testing.T) { mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Containers[0].Env = []*ateapipb.EnvVar{{Name: "PORT", Value: "8080"}, {Name: "DEBUG"}} }, - }, { - name: "env name with unusual printable characters is allowed", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Env = []*ateapipb.EnvVar{{Name: "my.var-2 (test)!", Value: "v"}} - }, }, { name: "env name with an equals sign", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Containers[0].Env = []*ateapipb.EnvVar{{Name: "FOO=BAR"}} }, want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("env").Index(0).Child("name"), nil, "")}, - }, { - name: "env name with a control character", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Env = []*ateapipb.EnvVar{{Name: "FOO BAR"}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("env").Index(0).Child("name"), nil, "")}, }, { name: "env missing name", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -660,12 +575,6 @@ func TestValidateActorTemplate(t *testing.T) { tmpl.Containers[0].SecurityContext = &ateapipb.SecurityContext{Capabilities: &ateapipb.Capabilities{Add: []string{"ALL"}}} }, want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("security_context", "capabilities", "add").Index(0), nil, "")}, - }, { - name: "capability with CAP_ prefix", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].SecurityContext = &ateapipb.SecurityContext{Capabilities: &ateapipb.Capabilities{Add: []string{"CAP_NET_BIND_SERVICE"}}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("security_context", "capabilities", "add").Index(0), nil, "")}, }, { name: "lowercase capability", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -691,34 +600,34 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "valid wakeup probe", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/healthz", Port: 8080}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/healthz", Port: 8080}, TimeoutSeconds: 60, } }, }, { name: "wakeup probe missing http_get", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{TimeoutSeconds: 60} + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{TimeoutSeconds: 60} }, want: field.ErrorList{field.Required(field.NewPath("containers").Index(0).Child("wakeup_probe", "http_get"), "")}, }, { name: "missing wakeup probe timeout_seconds", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{HttpGet: &ateapipb.HTTPGetAction{Path: "/healthz", Port: 8080}} + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Path: "/healthz", Port: 8080}} }, want: field.ErrorList{field.Required(field.NewPath("containers").Index(0).Child("wakeup_probe", "timeout_seconds"), "")}, }, { name: "missing wakeup probe http_get.path", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{HttpGet: &ateapipb.HTTPGetAction{Port: 8080}, TimeoutSeconds: 60} + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 8080}, TimeoutSeconds: 60} }, want: field.ErrorList{field.Required(field.NewPath("containers").Index(0).Child("wakeup_probe", "http_get", "path"), "")}, }, { name: "wakeup probe timeout_seconds out of range", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/healthz", Port: 8080}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/healthz", Port: 8080}, TimeoutSeconds: 3601, } }, @@ -726,8 +635,8 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "negative wakeup probe timeout_seconds", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/healthz", Port: 8080}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/healthz", Port: 8080}, TimeoutSeconds: -1, } }, @@ -735,8 +644,8 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "wakeup probe missing port", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/healthz"}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/healthz"}, TimeoutSeconds: 60, } }, @@ -744,8 +653,8 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "negative wakeup probe port", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/healthz", Port: -1}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/healthz", Port: -1}, TimeoutSeconds: 60, } }, @@ -753,8 +662,8 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "wakeup probe port out of range", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/healthz", Port: 65536}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/healthz", Port: 65536}, TimeoutSeconds: 60, } }, @@ -762,8 +671,8 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "wakeup probe path with query string", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/readyz?verbose=1", Port: 8080}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/readyz?verbose=1", Port: 8080}, TimeoutSeconds: 60, } }, @@ -771,8 +680,8 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "wakeup probe path not starting with slash", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].WakeupProbe = &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "readyz", Port: 8080}, + tmpl.Containers[0].WakeupProbe = &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "readyz", Port: 8080}, TimeoutSeconds: 60, } }, @@ -806,24 +715,6 @@ func TestValidateActorTemplate(t *testing.T) { tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{{Name: "data", MountPath: "var/data"}} }, want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("volume_mounts").Index(0).Child("mount_path"), nil, "")}, - }, { - name: "root mount_path", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{{Name: "data", MountPath: "/"}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("volume_mounts").Index(0).Child("mount_path"), nil, "")}, - }, { - name: "mount_path with dot-dot segment", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{{Name: "data", MountPath: "/var/../etc"}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("volume_mounts").Index(0).Child("mount_path"), nil, "")}, - }, { - name: "mount_path with trailing slash", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].VolumeMounts = []*ateapipb.VolumeMount{{Name: "data", MountPath: "/var/data/"}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("volume_mounts").Index(0).Child("mount_path"), nil, "")}, }, { name: "valid durable_dir volume", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -882,18 +773,6 @@ func TestValidateActorTemplate(t *testing.T) { tmpl.Volumes = []*ateapipb.Volume{{Name: "tools", Image: &ateapipb.ImageVolumeSource{Reference: "example.com/app:v1"}}} }, want: field.ErrorList{field.Invalid(field.NewPath("volumes").Index(0).Child("image", "reference"), nil, "")}, - }, { - name: "image volume reference with malformed digest", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "tools", Image: &ateapipb.ImageVolumeSource{Reference: "example.com/app@sha256:abc"}}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("volumes").Index(0).Child("image", "reference"), nil, "")}, - }, { - name: "image volume reference not a reference at all", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Volumes = []*ateapipb.Volume{{Name: "tools", Image: &ateapipb.ImageVolumeSource{Reference: "@"}}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("volumes").Index(0).Child("image", "reference"), nil, "")}, }, { name: "valid external volume template", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -976,18 +855,6 @@ func TestValidateActorTemplate(t *testing.T) { tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "two gigs"}}} }, want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("quantity"), nil, "")}, - }, { - name: "zero limit quantity", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "0"}}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("quantity"), nil, "")}, - }, { - name: "cpu limit of 1000 cores", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Containers[0].Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1k"}}} - }, - want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "limits").Index(0).Child("quantity"), nil, "")}, }, { name: "too many limits", mutate: func(tmpl *ateapipb.ActorTemplate) { diff --git a/cmd/ateapi/internal/apivalidation/actor_test.go b/cmd/ateapi/internal/apivalidation/actor_test.go index 57d6e36892..b031dabb0e 100644 --- a/cmd/ateapi/internal/apivalidation/actor_test.go +++ b/cmd/ateapi/internal/apivalidation/actor_test.go @@ -306,7 +306,7 @@ func TestValidateActorUpdate(t *testing.T) { field.ErrorList{field.Required(field.NewPath("status", "state"), "")}, }, { "change actor.status.state", - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_PAUSED })), + validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_RUNNING })), validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_CRASHED })), nil, }, { @@ -716,20 +716,20 @@ func TestValidateListActorsRequest(t *testing.T) { field.ErrorList{field.Invalid(field.NewPath("atespace"), "NS1", "").WithOrigin("format=k8s-short-name")}, }, { "valid, positive page_size", - &ateapipb.ListActorsRequest{Atespace: "ns1", PageSize: 10}, + &ateapipb.ListActorsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageSize: 10}}, nil, }, { "negative page_size", - &ateapipb.ListActorsRequest{Atespace: "ns1", PageSize: -1}, - field.ErrorList{field.Invalid(field.NewPath("page_size"), int32(-1), "").WithOrigin("minimum")}, + &ateapipb.ListActorsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageSize: -1}}, + field.ErrorList{field.Invalid(field.NewPath("options", "page_size"), int32(-1), "").WithOrigin("minimum")}, }, { "valid page_token", - &ateapipb.ListActorsRequest{Atespace: "ns1", PageToken: strings.Repeat("x", 256)}, + &ateapipb.ListActorsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageToken: strings.Repeat("x", 256)}}, nil, }, { "too-large page_token", - &ateapipb.ListActorsRequest{Atespace: "ns1", PageToken: strings.Repeat("x", 257)}, - field.ErrorList{field.TooLongCharacters(field.NewPath("page_token"), "", 256).WithOrigin("maxLength")}, + &ateapipb.ListActorsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageToken: strings.Repeat("x", 257)}}, + field.ErrorList{field.TooLongCharacters(field.NewPath("options", "page_token"), "", 256).WithOrigin("maxLength")}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -1097,8 +1097,8 @@ func withActorWorkerAssignment(mods ...func(*ateapipb.WorkerAssignment)) func(*a } func TestValidateMintActorJWTRequest(t *testing.T) { - withExpiration := func(seconds int64) *ateapipb.MintActorJWTRequest { - return &ateapipb.MintActorJWTRequest{ + withExpiration := func(seconds int64) *ateapipb.MintActorJwtRequest { + return &ateapipb.MintActorJwtRequest{ Actor: &ateapipb.ObjectRef{Atespace: "ns1", Name: "id1"}, Audiences: []string{"https://example.com"}, ExpirationSeconds: seconds, @@ -1107,7 +1107,7 @@ func TestValidateMintActorJWTRequest(t *testing.T) { path := field.NewPath("expiration_seconds") tests := []struct { name string - req *ateapipb.MintActorJWTRequest + req *ateapipb.MintActorJwtRequest want field.ErrorList }{ {name: "minimum expiration", req: withExpiration(300)}, diff --git a/cmd/ateapi/internal/apivalidation/atespace_test.go b/cmd/ateapi/internal/apivalidation/atespace_test.go index 18f6482816..0f06d560b6 100644 --- a/cmd/ateapi/internal/apivalidation/atespace_test.go +++ b/cmd/ateapi/internal/apivalidation/atespace_test.go @@ -136,20 +136,24 @@ func TestValidateListAtespacesRequest(t *testing.T) { nil, }, { "valid, positive page_size", - validReq(func(r *ateapipb.ListAtespacesRequest) { r.PageSize = 10 }), + validReq(func(r *ateapipb.ListAtespacesRequest) { r.Options = &ateapipb.ListOptions{PageSize: 10} }), nil, }, { "negative page_size", - validReq(func(r *ateapipb.ListAtespacesRequest) { r.PageSize = -1 }), - field.ErrorList{field.Invalid(field.NewPath("page_size"), int32(-1), "").WithOrigin("minimum")}, + validReq(func(r *ateapipb.ListAtespacesRequest) { r.Options = &ateapipb.ListOptions{PageSize: -1} }), + field.ErrorList{field.Invalid(field.NewPath("options", "page_size"), int32(-1), "").WithOrigin("minimum")}, }, { "valid page_token", - validReq(func(r *ateapipb.ListAtespacesRequest) { r.PageToken = strings.Repeat("x", 256) }), + validReq(func(r *ateapipb.ListAtespacesRequest) { + r.Options = &ateapipb.ListOptions{PageToken: strings.Repeat("x", 256)} + }), nil, }, { "too-large page_token", - validReq(func(r *ateapipb.ListAtespacesRequest) { r.PageToken = strings.Repeat("x", 257) }), - field.ErrorList{field.TooLongCharacters(field.NewPath("page_token"), "", 256).WithOrigin("maxLength")}, + validReq(func(r *ateapipb.ListAtespacesRequest) { + r.Options = &ateapipb.ListOptions{PageToken: strings.Repeat("x", 257)} + }), + field.ErrorList{field.TooLongCharacters(field.NewPath("options", "page_token"), "", 256).WithOrigin("maxLength")}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { diff --git a/cmd/ateapi/internal/apivalidation/egress_policy.go b/cmd/ateapi/internal/apivalidation/egress_policy.go index 7a6bdabce7..c22f0a7c9e 100644 --- a/cmd/ateapi/internal/apivalidation/egress_policy.go +++ b/cmd/ateapi/internal/apivalidation/egress_policy.go @@ -160,15 +160,15 @@ func (k portKey) String() string { return fmt.Sprintf("port %d", k.number) } -func ValidateCustom_HTTPRule_Hostnames(_ context.Context, _ operation.Operation, p *field.Path, patterns, _ []string) field.ErrorList { +func ValidateCustom_HttpRule_Hostnames(_ context.Context, _ operation.Operation, p *field.Path, patterns, _ []string) field.ErrorList { return validateHostnamePatterns(patterns, p) } -func ValidateCustom_HTTPSRule_Hostnames(_ context.Context, _ operation.Operation, p *field.Path, patterns, _ []string) field.ErrorList { +func ValidateCustom_HttpsRule_Hostnames(_ context.Context, _ operation.Operation, p *field.Path, patterns, _ []string) field.ErrorList { return validateHostnamePatterns(patterns, p) } -func ValidateCustom_TLSPassthroughRule_Hostnames(_ context.Context, _ operation.Operation, p *field.Path, patterns, _ []string) field.ErrorList { +func ValidateCustom_TlsPassthroughRule_Hostnames(_ context.Context, _ operation.Operation, p *field.Path, patterns, _ []string) field.ErrorList { return validateHostnamePatterns(patterns, p) } diff --git a/cmd/ateapi/internal/apivalidation/egress_policy_test.go b/cmd/ateapi/internal/apivalidation/egress_policy_test.go index b45b0a1774..89829a165b 100644 --- a/cmd/ateapi/internal/apivalidation/egress_policy_test.go +++ b/cmd/ateapi/internal/apivalidation/egress_policy_test.go @@ -31,7 +31,7 @@ func validEgressPolicy() *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "default"}, Rules: []*ateapipb.EgressRule{{ - Http: &ateapipb.HTTPRule{ + Http: &ateapipb.HttpRule{ Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{ ReplaceHeaders: []*ateapipb.CredentialHeader{{ @@ -51,7 +51,7 @@ func actorJWTHeader(audiences ...string) *ateapipb.CredentialHeader { return &ateapipb.CredentialHeader{ Header: "Authorization", Prefix: "Bearer ", - ActorJwt: &ateapipb.ActorJWTSource{Audiences: audiences, ExpirationSeconds: 900}, + ActorJwt: &ateapipb.ActorJwtSource{Audiences: audiences, ExpirationSeconds: 900}, } } @@ -444,13 +444,13 @@ func TestValidateEgressPolicyRules(t *testing.T) { } withoutEffects := func(p *ateapipb.EgressPolicy) { p.Rules[0].Http.Effects = nil } trivialHTTPRule := func(hostname string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Http: &ateapipb.HTTPRule{Hostnames: []string{hostname}}} + return &ateapipb.EgressRule{Http: &ateapipb.HttpRule{Hostnames: []string{hostname}}} } httpsRule := func(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Https: &ateapipb.HTTPSRule{Hostnames: patterns, Ports: ports}} + return &ateapipb.EgressRule{Https: &ateapipb.HttpsRule{Hostnames: patterns, Ports: ports}} } passthroughRule := func(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: patterns, Ports: ports}} + return &ateapipb.EgressRule{TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: patterns, Ports: ports}} } portNumbers := func(numbers ...int32) *ateapipb.Ports { return &ateapipb.Ports{Numbers: numbers} } allPorts := func() *ateapipb.Ports { return &ateapipb.Ports{All: &ateapipb.AllPorts{}} } @@ -504,7 +504,7 @@ func TestValidateEgressPolicyRules(t *testing.T) { }, { name: "two protocols", mutate: func(p *ateapipb.EgressPolicy) { - p.Rules[0].Https = &ateapipb.HTTPSRule{Hostnames: []string{"api.example.com"}} + p.Rules[0].Https = &ateapipb.HttpsRule{Hostnames: []string{"api.example.com"}} }, want: field.ErrorList{ field.Invalid(rule, nil, "one of").WithOrigin("union"), @@ -737,7 +737,7 @@ func TestValidateEgressPolicyRules(t *testing.T) { }, { name: "same pattern on different ports", mutate: func(p *ateapipb.EgressPolicy) { - p.Rules = append(p.Rules, &ateapipb.EgressRule{Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}, Ports: portNumbers(8080)}}) + p.Rules = append(p.Rules, &ateapipb.EgressRule{Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}, Ports: portNumbers(8080)}}) }, }, { name: "same pattern on the default port ties", @@ -765,7 +765,7 @@ func TestValidateEgressPolicyRules(t *testing.T) { name: "http and tls passthrough on every port tie", mutate: func(p *ateapipb.EgressPolicy) { p.Rules = []*ateapipb.EgressRule{ - {Http: &ateapipb.HTTPRule{Hostnames: []string{"*"}, Ports: allPorts()}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"*"}, Ports: allPorts()}}, passthroughRule(allPorts(), "*"), } }, @@ -878,7 +878,7 @@ func TestValidateEgressPolicyRules(t *testing.T) { }, { name: "actor JWT on an https rule", mutate: func(p *ateapipb.EgressPolicy) { - p.Rules[0] = &ateapipb.EgressRule{Https: &ateapipb.HTTPSRule{ + p.Rules[0] = &ateapipb.EgressRule{Https: &ateapipb.HttpsRule{ Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{ReplaceHeaders: []*ateapipb.CredentialHeader{actorJWTHeader("https://api.example.com")}}, }} @@ -893,7 +893,7 @@ func TestValidateEgressPolicyRules(t *testing.T) { }, { name: "credential URI and actor JWT in one header", mutate: func(p *ateapipb.EgressPolicy) { - p.Rules[0].Http.Effects.ReplaceHeaders[0].ActorJwt = &ateapipb.ActorJWTSource{Audiences: []string{"https://api.example.com"}, ExpirationSeconds: 900} + p.Rules[0].Http.Effects.ReplaceHeaders[0].ActorJwt = &ateapipb.ActorJwtSource{Audiences: []string{"https://api.example.com"}, ExpirationSeconds: 900} }, want: field.ErrorList{ field.Invalid(staticHeader, nil, "one of").WithOrigin("union"), @@ -988,7 +988,7 @@ func TestValidateEgressPolicyRules(t *testing.T) { }, { name: "effects on an https rule", mutate: func(p *ateapipb.EgressPolicy) { - p.Rules[0] = &ateapipb.EgressRule{Https: &ateapipb.HTTPSRule{Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{}}} + p.Rules[0] = &ateapipb.EgressRule{Https: &ateapipb.HttpsRule{Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{}}} }, want: field.ErrorList{ field.Required(rule.Child("https", "effects"), "at least one effect must be specified"), diff --git a/cmd/ateapi/internal/apivalidation/tag_test.go b/cmd/ateapi/internal/apivalidation/tag_test.go index 23bbd43958..742e48263d 100644 --- a/cmd/ateapi/internal/apivalidation/tag_test.go +++ b/cmd/ateapi/internal/apivalidation/tag_test.go @@ -437,23 +437,23 @@ func TestValidateListTagsRequest(t *testing.T) { }, { name: "valid, positive page_size", - req: &ateapipb.ListTagsRequest{Atespace: "ns1", PageSize: 10}, + req: &ateapipb.ListTagsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageSize: 10}}, wantError: nil, }, { name: "negative page_size", - req: &ateapipb.ListTagsRequest{Atespace: "ns1", PageSize: -1}, - wantError: field.ErrorList{field.Invalid(field.NewPath("page_size"), nil, "").WithOrigin("minimum")}, + req: &ateapipb.ListTagsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageSize: -1}}, + wantError: field.ErrorList{field.Invalid(field.NewPath("options", "page_size"), nil, "").WithOrigin("minimum")}, }, { name: "valid page_token", - req: &ateapipb.ListTagsRequest{Atespace: "ns1", PageToken: strings.Repeat("x", 256)}, + req: &ateapipb.ListTagsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageToken: strings.Repeat("x", 256)}}, wantError: nil, }, { name: "too-large page_token", - req: &ateapipb.ListTagsRequest{Atespace: "ns1", PageToken: strings.Repeat("x", 257)}, - wantError: field.ErrorList{field.TooLongCharacters(field.NewPath("page_token"), "", 256).WithOrigin("maxLength")}, + req: &ateapipb.ListTagsRequest{Atespace: "ns1", Options: &ateapipb.ListOptions{PageToken: strings.Repeat("x", 257)}}, + wantError: field.ErrorList{field.TooLongCharacters(field.NewPath("options", "page_token"), "", 256).WithOrigin("maxLength")}, }, } for _, tt := range tests { diff --git a/cmd/ateapi/internal/apivalidation/validation_test.go b/cmd/ateapi/internal/apivalidation/validation_test.go index a8dd874f67..8f6b950c94 100644 --- a/cmd/ateapi/internal/apivalidation/validation_test.go +++ b/cmd/ateapi/internal/apivalidation/validation_test.go @@ -760,58 +760,12 @@ func TestValidateSystemInfoVolumeSource(t *testing.T) { s.DataSources[0].ActorMetadata.Items[0].Path = strings.Repeat("p", 256) }), want: field.ErrorList{field.TooLong(itemsPath.Index(0).Child("path"), nil, 255).WithOrigin("maxLength")}, - }, { - name: "valid: nested item path", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = "meta/actor-name" - }), }, { name: "item path traversal", obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { s.DataSources[0].ActorMetadata.Items[0].Path = "../../traversal-escape" }), want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, - }, { - name: "absolute item path", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = "/escaped" - }), - want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, - }, { - name: "item path with dot segment", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = "./actor-name" - }), - want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, - }, { - name: "item path with trailing slash", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = "actor-name/" - }), - want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, - }, { - name: "item path with empty segment", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = "meta//actor-name" - }), - want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, - }, { - name: "item path with NUL byte", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = "actor\x00name" - }), - want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, - }, { - name: "valid: unicode item path", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = "méta/имя" - }), - }, { - name: "item path with too many segments", - obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { - s.DataSources[0].ActorMetadata.Items[0].Path = strings.Repeat("d/", 16) + "actor-name" - }), - want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, }, { name: "trust bundle path traversal", obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { @@ -830,6 +784,19 @@ func TestValidateSystemInfoVolumeSource(t *testing.T) { s.DataSources[1].TrustBundle.Path = "actor-uid" }), want: field.ErrorList{field.Duplicate(dsPath.Index(1).Child("trust_bundle", "path"), nil)}, + }, { + name: "second actor_metadata entry", + obj: valid(func(s *ateapipb.SystemInfoVolumeSource) { + s.DataSources = append(s.DataSources, &ateapipb.SystemInfoDataSource{ + ActorMetadata: &ateapipb.ActorMetadataDataSource{ + Items: []*ateapipb.ActorMetadataItem{{ + Field: ateapipb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME, + Path: "actor-name-2", + }}, + }, + }) + }), + want: field.ErrorList{field.Forbidden(dsPath.Index(2).Child("actor_metadata"), "")}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -890,17 +857,10 @@ func TestValidateTrustBundleDataSource(t *testing.T) { name: "path too long", obj: valid(func(tb *ateapipb.TrustBundleDataSource) { tb.Path = strings.Repeat("p", 256) }), want: field.ErrorList{field.TooLong(field.NewPath("path"), nil, 255).WithOrigin("maxLength")}, - }, { - name: "valid: nested path", - obj: valid(func(tb *ateapipb.TrustBundleDataSource) { tb.Path = "certs/ca.pem" }), }, { name: "path traversal", obj: valid(func(tb *ateapipb.TrustBundleDataSource) { tb.Path = "../escape" }), want: field.ErrorList{field.Invalid(field.NewPath("path"), nil, "")}, - }, { - name: "absolute path", - obj: valid(func(tb *ateapipb.TrustBundleDataSource) { tb.Path = "/escaped" }), - want: field.ErrorList{field.Invalid(field.NewPath("path"), nil, "")}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -949,29 +909,10 @@ func TestValidateExternalVolume(t *testing.T) { name: "invalid storage volume id with null U+0000", obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol\x00id" }), want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "")}, - }, { - name: "invalid storage volume id with unit separator U+001F", - obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol\x1fid" }), - want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "")}, - }, { - name: "invalid storage volume id with DEL U+007F", - obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol\x7fid" }), - want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "")}, - }, { - name: "invalid storage volume id with C1 control U+0080", - obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol\u0080id" }), - want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "")}, - }, { - name: "invalid storage volume id with C1 control U+009F", - obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = "vol\u009fid" }), - want: field.ErrorList{field.Invalid(field.NewPath("storage_volume_id"), nil, "")}, }, { name: "storage volume id too long", obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = strings.Repeat("x", 257) }), want: field.ErrorList{field.TooLong(field.NewPath("storage_volume_id"), nil, 256).WithOrigin("maxLength")}, - }, { - name: "valid csi volume type", - obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "pd.csi.storage.gke.io" }), }, { name: "missing volume type", obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "" }), @@ -993,18 +934,6 @@ func TestValidateExternalVolume(t *testing.T) { }, { name: "valid volume with substrate.io prefixed volume type", obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "substrate.io/mock" }), - }, { - name: "invalid volume type with empty plugin after substrate.io prefix", - obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "substrate.io/" }), - want: field.ErrorList{field.Invalid(field.NewPath("volume_type"), nil, "")}, - }, { - name: "invalid volume type with invalid plugin name after substrate.io prefix", - obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "substrate.io/Mock_Plugin" }), - want: field.ErrorList{field.Invalid(field.NewPath("volume_type"), nil, "")}, - }, { - name: "invalid volume type with non-substrate prefix", - obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "other.io/mock" }), - want: field.ErrorList{field.Invalid(field.NewPath("volume_type"), nil, "")}, }, { name: "negative status", obj: valid(func(v *ateapipb.ExternalVolume) { v.Status = ateapipb.ExternalVolume_Status(-1) }), @@ -1017,25 +946,25 @@ func TestValidateExternalVolume(t *testing.T) { name: "storage volume id at the bound", obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = strings.Repeat("x", 256) }), }, { - name: "too many volume_context entries", + name: "volume_context with many entries, as a CSI driver may return", obj: valid(func(v *ateapipb.ExternalVolume) { - ctxMap := make(map[string]string, 33) - for i := 0; i < 33; i++ { - ctxMap[fmt.Sprintf("key-%d", i)] = "v" + ctxMap := make(map[string]string, 100) + for i := range 100 { + ctxMap[fmt.Sprintf("key-%03d", i)] = "v" } v.VolumeContext = ctxMap }), - want: field.ErrorList{field.TooMany(field.NewPath("volume_context"), 33, 32).WithOrigin("maxProperties")}, }, { - name: "volume_context key too long", - obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeContext = map[string]string{strings.Repeat("k", 129): "v"} }), - want: field.ErrorList{field.TooLong(field.NewPath("volume_context"), nil, 128).WithOrigin("maxLength")}, + name: "volume_context with one long value, as a CSI driver may return", + obj: valid(func(v *ateapipb.ExternalVolume) { + v.VolumeContext = map[string]string{"attachment": strings.Repeat("v", 4000)} + }), }, { - name: "volume_context value too long", + name: "volume_context over the size bound", obj: valid(func(v *ateapipb.ExternalVolume) { - v.VolumeContext = map[string]string{"attachment": strings.Repeat("v", 257)} + v.VolumeContext = map[string]string{"attachment": strings.Repeat("v", 40*1024)} }), - want: field.ErrorList{field.TooLong(field.NewPath("volume_context").Key("attachment"), nil, 256).WithOrigin("maxLength")}, + want: field.ErrorList{field.TooLong(field.NewPath("volume_context"), nil, 40*1024)}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -1774,18 +1703,18 @@ func TestValidateTagRequestPayloads(t *testing.T) { { name: "list: negative page_size", validate: func(ctx context.Context, op operation.Operation) field.ErrorList { - req := &ateapipb.ListTagsRequest{PageSize: -1} + req := &ateapipb.ListTagsRequest{Options: &ateapipb.ListOptions{PageSize: -1}} return Validate_ListTagsRequest(ctx, op, nil, req, nil) }, - want: field.ErrorList{field.Invalid(field.NewPath("page_size"), nil, "").WithOrigin("minimum")}, + want: field.ErrorList{field.Invalid(field.NewPath("options", "page_size"), nil, "").WithOrigin("minimum")}, }, { name: "list: over-long page_token", validate: func(ctx context.Context, op operation.Operation) field.ErrorList { - req := &ateapipb.ListTagsRequest{PageToken: strings.Repeat("t", 257)} + req := &ateapipb.ListTagsRequest{Options: &ateapipb.ListOptions{PageToken: strings.Repeat("t", 257)}} return Validate_ListTagsRequest(ctx, op, nil, req, nil) }, - want: field.ErrorList{field.TooLong(field.NewPath("page_token"), nil, 256).WithOrigin("maxLength")}, + want: field.ErrorList{field.TooLong(field.NewPath("options", "page_token"), nil, 256).WithOrigin("maxLength")}, }, } for _, tt := range tests { @@ -1798,8 +1727,8 @@ func TestValidateTagRequestPayloads(t *testing.T) { func validGoldenSnapshotStatus(mutate ...func(*ateapipb.GoldenSnapshotStatus)) *ateapipb.GoldenSnapshotStatus { s := &ateapipb.GoldenSnapshotStatus{ - GoldenTag: &ateapipb.ObjectRef{Atespace: "ate-golden", Name: "01234567-89ab-cdef-0123-456789abcdef"}, - TakeGoldenSnapshotAt: ×tamppb.Timestamp{Seconds: 867}, + GoldenTag: &ateapipb.ObjectRef{Atespace: "ate-golden", Name: "01234567-89ab-cdef-0123-456789abcdef"}, + SnapshotTime: ×tamppb.Timestamp{Seconds: 867}, } for _, m := range mutate { m(s) @@ -1827,8 +1756,8 @@ func TestValidateGoldenSnapshotStatus(t *testing.T) { { name: "valid failed", obj: &ateapipb.GoldenSnapshotStatus{ - TakeGoldenSnapshotAt: ×tamppb.Timestamp{Seconds: 867}, - ErrorMessage: "GoldenActorCrashed: golden actor crashed before its snapshot was taken", + SnapshotTime: ×tamppb.Timestamp{Seconds: 867}, + ErrorMessage: "GoldenActorCrashed: golden actor crashed before its snapshot was taken", }, }, { @@ -1877,14 +1806,14 @@ func TestValidateGoldenSnapshotStatusUpdate(t *testing.T) { { // The reconciler restarts the warmup clock on every resume, so the // deadline must stay mutable. - name: "take_golden_snapshot_at changed", + name: "snapshot_time changed", oldObj: valid(), newObj: valid(func(s *ateapipb.GoldenSnapshotStatus) { - s.TakeGoldenSnapshotAt = ×tamppb.Timestamp{Seconds: 5309} + s.SnapshotTime = ×tamppb.Timestamp{Seconds: 5309} }), }, { - name: "take_golden_snapshot_at set", + name: "snapshot_time set", oldObj: &ateapipb.GoldenSnapshotStatus{}, newObj: valid(), }, diff --git a/cmd/ateapi/internal/apivalidation/worker.go b/cmd/ateapi/internal/apivalidation/worker.go index 41206d29ed..13a1ae00b0 100644 --- a/cmd/ateapi/internal/apivalidation/worker.go +++ b/cmd/ateapi/internal/apivalidation/worker.go @@ -81,6 +81,11 @@ func ValidateRequestActorSuspendRequest(ctx context.Context, req *ateapipb.Reque return Validate_RequestActorSuspendRequest(ctx, op, nil, req, nil) } +func ValidateReportActorSnapshotStateRequest(ctx context.Context, req *ateapipb.ReportActorSnapshotStateRequest) field.ErrorList { + op := operation.Operation{Type: operation.Create} + return Validate_ReportActorSnapshotStateRequest(ctx, op, nil, req, nil) +} + // validateWorkerUpdate validates a Worker against the previous stored value. // It is what enforces the immutable fields, which need an old value to compare // against. diff --git a/cmd/ateapi/internal/apivalidation/worker_test.go b/cmd/ateapi/internal/apivalidation/worker_test.go index 70667446d1..02b7b58a09 100644 --- a/cmd/ateapi/internal/apivalidation/worker_test.go +++ b/cmd/ateapi/internal/apivalidation/worker_test.go @@ -22,6 +22,7 @@ import ( "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/timestamppb" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -624,6 +625,125 @@ func TestValidateRequestActorSuspendRequest(t *testing.T) { } } +func TestValidateReportActorSnapshotStateRequest(t *testing.T) { + report := func(gen int32) *ateapipb.SnapshotReport { + return &ateapipb.SnapshotReport{ + Uuid: fmt.Sprintf("snap-%d", gen), + Generation: gen, + Fidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateapipb.ObjectSnapshot{SnapshotUri: fmt.Sprintf("gs://bucket/actors/a/snap-%d", gen)}, + LocalCommitTime: timestamppb.Now(), + } + } + valid := func(mutate ...func(*ateapipb.ReportActorSnapshotStateRequest)) *ateapipb.ReportActorSnapshotStateRequest { + durable := report(1) + durable.DurableCompleteTime = timestamppb.Now() + r := &ateapipb.ReportActorSnapshotStateRequest{ + Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "actor-1"}, + ActorUid: apiOtherWorkerName, + State: &ateapipb.ActorSnapshotState{ + LatestLocal: report(2), + Uploading: report(2), + LatestDurable: durable, + UploadDueTime: timestamppb.Now(), + }, + } + for _, m := range mutate { + m(r) + } + return r + } + local := field.NewPath("state", "latest_local") + tests := []struct { + name string + req *ateapipb.ReportActorSnapshotStateRequest + want field.ErrorList + }{{ + name: "valid", + req: valid(), + }, { + name: "valid detach: no local snapshot", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { + r.State.LatestLocal = nil + r.State.Uploading = nil + r.State.UploadDueTime = nil + }), + }, { + name: "valid empty state", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State = &ateapipb.ActorSnapshotState{} }), + }, { + name: "missing actor", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.Actor = nil }), + want: field.ErrorList{field.Required(field.NewPath("actor"), "")}, + }, { + name: "missing actor.atespace", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.Actor.Atespace = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor", "atespace"), "")}, + }, { + name: "invalid actor.name", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.Actor.Name = "UPPER" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor", "name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing actor_uid", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.ActorUid = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_uid"), "")}, + }, { + name: "invalid actor_uid", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.ActorUid = "not-a-uuid" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "missing state", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State = nil }), + want: field.ErrorList{field.Required(field.NewPath("state"), "")}, + }, { + name: "missing uuid", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestLocal.Uuid = "" }), + want: field.ErrorList{field.Required(local.Child("uuid"), "")}, + }, { + name: "invalid uuid", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestLocal.Uuid = "../escape" }), + want: field.ErrorList{field.Invalid(local.Child("uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "generation below 1", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestLocal.Generation = -1 }), + want: field.ErrorList{field.Invalid(local.Child("generation"), nil, "").WithOrigin("minimum")}, + }, { + name: "missing fidelity", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { + r.State.LatestLocal.Fidelity = ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED + }), + want: field.ErrorList{field.Required(local.Child("fidelity"), "")}, + }, { + name: "unknown fidelity", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestLocal.Fidelity = 4 }), + want: field.ErrorList{field.Invalid(local.Child("fidelity"), nil, "").WithOrigin("maximum")}, + }, { + name: "missing object", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestLocal.Object = nil }), + want: field.ErrorList{field.Required(local.Child("object"), "")}, + }, { + name: "missing object.snapshot_uri", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestLocal.Object.SnapshotUri = "" }), + want: field.ErrorList{field.Required(local.Child("object", "snapshot_uri"), "")}, + }, { + name: "valid without local_commit_time", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestDurable.LocalCommitTime = nil }), + }, { + name: "uploading is validated", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.Uploading.Uuid = "" }), + want: field.ErrorList{field.Required(field.NewPath("state", "uploading", "uuid"), "")}, + }, { + name: "latest_durable is validated", + req: valid(func(r *ateapipb.ReportActorSnapshotStateRequest) { r.State.LatestDurable.Object = nil }), + want: field.ErrorList{field.Required(field.NewPath("state", "latest_durable", "object"), "")}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, ValidateReportActorSnapshotStateRequest(context.Background(), tt.req), tt.want) + }) + } +} + // validWorker returns a Worker in the shape CreateWorker accepts: named, with // its pod coordinates filled in and no status — status is output-only. func validWorker(name string, mods ...func(*ateapipb.Worker)) *ateapipb.Worker { diff --git a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go index a5fc763555..cee99621a5 100644 --- a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go +++ b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go @@ -396,13 +396,13 @@ func Validate_ActorCrash( return errs } -// Validate_ActorJWTSource validates an instance of ActorJWTSource according +// Validate_ActorJwtSource validates an instance of ActorJwtSource according // to declarative validation rules in the API schema. -func Validate_ActorJWTSource( +func Validate_ActorJwtSource( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.ActorJWTSource) (errs field.ErrorList) { + obj, oldObj *ateapipb.ActorJwtSource) (errs field.ErrorList) { - { // field ateapipb.ActorJWTSource.Audiences + { // field ateapipb.ActorJwtSource.Audiences fn := func( fldPath *field.Path, obj, oldObj []string, @@ -445,13 +445,13 @@ func Validate_ActorJWTSource( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ActorJWTSource) []string { + func(oldObj *ateapipb.ActorJwtSource) []string { return oldObj.Audiences }) errs = append(errs, fn(fldPath.Child("audiences"), obj.Audiences, oldVal, oldObj != nil)...) } - { // field ateapipb.ActorJWTSource.ExpirationSeconds + { // field ateapipb.ActorJwtSource.ExpirationSeconds fn := func( fldPath *field.Path, obj, oldObj *int64, @@ -480,7 +480,7 @@ func Validate_ActorJWTSource( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ActorJWTSource) *int64 { + func(oldObj *ateapipb.ActorJwtSource) *int64 { return &oldObj.ExpirationSeconds }) errs = append(errs, fn(fldPath.Child("expiration_seconds"), &obj.ExpirationSeconds, oldVal, oldObj != nil)...) @@ -631,6 +631,133 @@ func Validate_ActorMetadataItem( return errs } +// Validate_ActorSnapshotState validates an instance of ActorSnapshotState according +// to declarative validation rules in the API schema. +func Validate_ActorSnapshotState( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.ActorSnapshotState) (errs field.ErrorList) { + + { // field ateapipb.ActorSnapshotState.LatestLocal + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SnapshotReport, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SnapshotReport(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.ActorSnapshotState) *ateapipb.SnapshotReport { + return oldObj.LatestLocal + }) + errs = append(errs, fn(fldPath.Child("latest_local"), obj.LatestLocal, oldVal, oldObj != nil)...) + } + + { // field ateapipb.ActorSnapshotState.Uploading + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SnapshotReport, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SnapshotReport(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.ActorSnapshotState) *ateapipb.SnapshotReport { + return oldObj.Uploading + }) + errs = append(errs, fn(fldPath.Child("uploading"), obj.Uploading, oldVal, oldObj != nil)...) + } + + { // field ateapipb.ActorSnapshotState.LatestDurable + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SnapshotReport, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SnapshotReport(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.ActorSnapshotState) *ateapipb.SnapshotReport { + return oldObj.LatestDurable + }) + errs = append(errs, fn(fldPath.Child("latest_durable"), obj.LatestDurable, oldVal, oldObj != nil)...) + } + + { // field ateapipb.ActorSnapshotState.UploadDueTime + fn := func( + fldPath *field.Path, + obj, oldObj *timestamppb.Timestamp, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.ActorSnapshotState) *timestamppb.Timestamp { + return oldObj.UploadDueTime + }) + errs = append(errs, fn(fldPath.Child("upload_due_time"), obj.UploadDueTime, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_ActorStatus validates an instance of ActorStatus according // to declarative validation rules in the API schema. func Validate_ActorStatus( @@ -1752,7 +1879,7 @@ func Validate_Container( { // field ateapipb.Container.WakeupProbe fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.ContainerWakeupProbe, + obj, oldObj *ateapipb.WakeupProbe, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -1769,11 +1896,11 @@ func Validate_Container( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_ContainerWakeupProbe(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_WakeupProbe(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.Container) *ateapipb.ContainerWakeupProbe { + func(oldObj *ateapipb.Container) *ateapipb.WakeupProbe { return oldObj.WakeupProbe }) errs = append(errs, fn(fldPath.Child("wakeup_probe"), obj.WakeupProbe, oldVal, oldObj != nil)...) @@ -1892,81 +2019,6 @@ func Validate_Container( return errs } -// Validate_ContainerWakeupProbe validates an instance of ContainerWakeupProbe according -// to declarative validation rules in the API schema. -func Validate_ContainerWakeupProbe( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.ContainerWakeupProbe) (errs field.ErrorList) { - - { // field ateapipb.ContainerWakeupProbe.HttpGet - fn := func( - fldPath *field.Path, - obj, oldObj *ateapipb.HTTPGetAction, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - // call the type's validation function - errs = append(errs, Validate_HTTPGetAction(ctx, op, fldPath, obj, oldObj)...) - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ContainerWakeupProbe) *ateapipb.HTTPGetAction { - return oldObj.HttpGet - }) - errs = append(errs, fn(fldPath.Child("http_get"), obj.HttpGet, oldVal, oldObj != nil)...) - } - - { // field ateapipb.ContainerWakeupProbe.TimeoutSeconds - fn := func( - fldPath *field.Path, - obj, oldObj *int32, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3600); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ContainerWakeupProbe) *int32 { - return &oldObj.TimeoutSeconds - }) - errs = append(errs, fn(fldPath.Child("timeout_seconds"), &obj.TimeoutSeconds, oldVal, oldObj != nil)...) - } - - return errs -} - // Validate_CreateActorEgressPolicyRequest validates an instance of CreateActorEgressPolicyRequest according // to declarative validation rules in the API schema. func Validate_CreateActorEgressPolicyRequest( @@ -2522,7 +2574,7 @@ func Validate_CredentialHeader( { // field ateapipb.CredentialHeader.ActorJwt fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.ActorJWTSource, + obj, oldObj *ateapipb.ActorJwtSource, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -2539,11 +2591,11 @@ func Validate_CredentialHeader( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_ActorJWTSource(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_ActorJwtSource(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.CredentialHeader) *ateapipb.ActorJWTSource { + func(oldObj *ateapipb.CredentialHeader) *ateapipb.ActorJwtSource { return oldObj.ActorJwt }) errs = append(errs, fn(fldPath.Child("actor_jwt"), obj.ActorJwt, oldVal, oldObj != nil)...) @@ -3529,7 +3581,7 @@ func Validate_EgressRule( { // field ateapipb.EgressRule.Http fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.HTTPRule, + obj, oldObj *ateapipb.HttpRule, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -3546,11 +3598,11 @@ func Validate_EgressRule( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_HTTPRule(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_HttpRule(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.EgressRule) *ateapipb.HTTPRule { + func(oldObj *ateapipb.EgressRule) *ateapipb.HttpRule { return oldObj.Http }) errs = append(errs, fn(fldPath.Child("http"), obj.Http, oldVal, oldObj != nil)...) @@ -3559,7 +3611,7 @@ func Validate_EgressRule( { // field ateapipb.EgressRule.Https fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.HTTPSRule, + obj, oldObj *ateapipb.HttpsRule, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -3576,11 +3628,11 @@ func Validate_EgressRule( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_HTTPSRule(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_HttpsRule(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.EgressRule) *ateapipb.HTTPSRule { + func(oldObj *ateapipb.EgressRule) *ateapipb.HttpsRule { return oldObj.Https }) errs = append(errs, fn(fldPath.Child("https"), obj.Https, oldVal, oldObj != nil)...) @@ -3589,7 +3641,7 @@ func Validate_EgressRule( { // field ateapipb.EgressRule.TlsPassthrough fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.TLSPassthroughRule, + obj, oldObj *ateapipb.TlsPassthroughRule, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -3606,11 +3658,11 @@ func Validate_EgressRule( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_TLSPassthroughRule(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_TlsPassthroughRule(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.EgressRule) *ateapipb.TLSPassthroughRule { + func(oldObj *ateapipb.EgressRule) *ateapipb.TlsPassthroughRule { return oldObj.TlsPassthrough }) errs = append(errs, fn(fldPath.Child("tls_passthrough"), obj.TlsPassthrough, oldVal, oldObj != nil)...) @@ -3866,26 +3918,14 @@ func Validate_ExternalVolume( } // call field-attached validations earlyReturn := false - if e := validate.MaxProperties(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } if e := validate.OptionalMap(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.EachMapKey(ctx, op, fldPath, obj, oldObj, - func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { - return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 128) - }); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.EachMapVal(ctx, op, fldPath, obj, oldObj, validate.DirectEqual, - func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { - return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256) - }); len(e) != 0 { + // custom validation + if e := ValidateCustom_ExternalVolume_VolumeContext(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } return @@ -4586,7 +4626,7 @@ func Validate_GoldenSnapshotStatus( errs = append(errs, fn(fldPath.Child("golden_tag"), obj.GoldenTag, oldVal, oldObj != nil)...) } - { // field ateapipb.GoldenSnapshotStatus.TakeGoldenSnapshotAt + { // field ateapipb.GoldenSnapshotStatus.SnapshotTime fn := func( fldPath *field.Path, obj, oldObj *timestamppb.Timestamp, @@ -4609,9 +4649,9 @@ func Validate_GoldenSnapshotStatus( } oldVal := safe.Field(oldObj, func(oldObj *ateapipb.GoldenSnapshotStatus) *timestamppb.Timestamp { - return oldObj.TakeGoldenSnapshotAt + return oldObj.SnapshotTime }) - errs = append(errs, fn(fldPath.Child("take_golden_snapshot_at"), obj.TakeGoldenSnapshotAt, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("snapshot_time"), obj.SnapshotTime, oldVal, oldObj != nil)...) } { // field ateapipb.GoldenSnapshotStatus.ErrorMessage @@ -4648,13 +4688,13 @@ func Validate_GoldenSnapshotStatus( return errs } -// Validate_HTTPGetAction validates an instance of HTTPGetAction according +// Validate_HttpGetAction validates an instance of HttpGetAction according // to declarative validation rules in the API schema. -func Validate_HTTPGetAction( +func Validate_HttpGetAction( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.HTTPGetAction) (errs field.ErrorList) { + obj, oldObj *ateapipb.HttpGetAction) (errs field.ErrorList) { - { // field ateapipb.HTTPGetAction.Path + { // field ateapipb.HttpGetAction.Path fn := func( fldPath *field.Path, obj, oldObj *string, @@ -4675,7 +4715,7 @@ func Validate_HTTPGetAction( return // do not proceed } // custom validation - if e := ValidateCustom_HTTPGetAction_Path(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + if e := ValidateCustom_HttpGetAction_Path(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 1024); len(e) != 0 { @@ -4684,13 +4724,13 @@ func Validate_HTTPGetAction( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPGetAction) *string { + func(oldObj *ateapipb.HttpGetAction) *string { return &oldObj.Path }) errs = append(errs, fn(fldPath.Child("path"), &obj.Path, oldVal, oldObj != nil)...) } - { // field ateapipb.HTTPGetAction.Port + { // field ateapipb.HttpGetAction.Port fn := func( fldPath *field.Path, obj, oldObj *int32, @@ -4719,7 +4759,7 @@ func Validate_HTTPGetAction( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPGetAction) *int32 { + func(oldObj *ateapipb.HttpGetAction) *int32 { return &oldObj.Port }) errs = append(errs, fn(fldPath.Child("port"), &obj.Port, oldVal, oldObj != nil)...) @@ -4728,13 +4768,13 @@ func Validate_HTTPGetAction( return errs } -// Validate_HTTPRule validates an instance of HTTPRule according +// Validate_HttpRule validates an instance of HttpRule according // to declarative validation rules in the API schema. -func Validate_HTTPRule( +func Validate_HttpRule( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.HTTPRule) (errs field.ErrorList) { + obj, oldObj *ateapipb.HttpRule) (errs field.ErrorList) { - { // field ateapipb.HTTPRule.Hostnames + { // field ateapipb.HttpRule.Hostnames fn := func( fldPath *field.Path, obj, oldObj []string, @@ -4759,7 +4799,7 @@ func Validate_HTTPRule( return // do not proceed } // custom validation - if e := ValidateCustom_HTTPRule_Hostnames(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + if e := ValidateCustom_HttpRule_Hostnames(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } // lists with set semantics require unique values @@ -4769,13 +4809,13 @@ func Validate_HTTPRule( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPRule) []string { + func(oldObj *ateapipb.HttpRule) []string { return oldObj.Hostnames }) errs = append(errs, fn(fldPath.Child("hostnames"), obj.Hostnames, oldVal, oldObj != nil)...) } - { // field ateapipb.HTTPRule.Ports + { // field ateapipb.HttpRule.Ports fn := func( fldPath *field.Path, obj, oldObj *ateapipb.Ports, @@ -4799,13 +4839,13 @@ func Validate_HTTPRule( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPRule) *ateapipb.Ports { + func(oldObj *ateapipb.HttpRule) *ateapipb.Ports { return oldObj.Ports }) errs = append(errs, fn(fldPath.Child("ports"), obj.Ports, oldVal, oldObj != nil)...) } - { // field ateapipb.HTTPRule.Effects + { // field ateapipb.HttpRule.Effects fn := func( fldPath *field.Path, obj, oldObj *ateapipb.HttpRuleEffects, @@ -4829,7 +4869,7 @@ func Validate_HTTPRule( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPRule) *ateapipb.HttpRuleEffects { + func(oldObj *ateapipb.HttpRule) *ateapipb.HttpRuleEffects { return oldObj.Effects }) errs = append(errs, fn(fldPath.Child("effects"), obj.Effects, oldVal, oldObj != nil)...) @@ -4838,13 +4878,73 @@ func Validate_HTTPRule( return errs } -// Validate_HTTPSRule validates an instance of HTTPSRule according +// Validate_HttpRuleEffects validates an instance of HttpRuleEffects according +// to declarative validation rules in the API schema. +func Validate_HttpRuleEffects( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.HttpRuleEffects) (errs field.ErrorList) { + + // custom validation + if e := ValidateCustom_HttpRuleEffects(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + + { // field ateapipb.HttpRuleEffects.ReplaceHeaders + fn := func( + fldPath *field.Path, + obj, oldObj []*ateapipb.CredentialHeader, + oldValueCorrelated bool) (errs field.ErrorList) { + // Uniqueness validation is implemented via custom, handwritten validation + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateapipb.CredentialHeader](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 16).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_HttpRuleEffects_ReplaceHeaders(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateapipb.CredentialHeader, b *ateapipb.CredentialHeader) bool { return a.Header == b.Header }, ateDeepEqual, Validate_CredentialHeader); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.HttpRuleEffects) []*ateapipb.CredentialHeader { + return oldObj.ReplaceHeaders + }) + errs = append(errs, fn(fldPath.Child("replace_headers"), obj.ReplaceHeaders, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_HttpsRule validates an instance of HttpsRule according // to declarative validation rules in the API schema. -func Validate_HTTPSRule( +func Validate_HttpsRule( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.HTTPSRule) (errs field.ErrorList) { + obj, oldObj *ateapipb.HttpsRule) (errs field.ErrorList) { - { // field ateapipb.HTTPSRule.Hostnames + { // field ateapipb.HttpsRule.Hostnames fn := func( fldPath *field.Path, obj, oldObj []string, @@ -4869,7 +4969,7 @@ func Validate_HTTPSRule( return // do not proceed } // custom validation - if e := ValidateCustom_HTTPSRule_Hostnames(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + if e := ValidateCustom_HttpsRule_Hostnames(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } // lists with set semantics require unique values @@ -4879,13 +4979,13 @@ func Validate_HTTPSRule( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPSRule) []string { + func(oldObj *ateapipb.HttpsRule) []string { return oldObj.Hostnames }) errs = append(errs, fn(fldPath.Child("hostnames"), obj.Hostnames, oldVal, oldObj != nil)...) } - { // field ateapipb.HTTPSRule.Ports + { // field ateapipb.HttpsRule.Ports fn := func( fldPath *field.Path, obj, oldObj *ateapipb.Ports, @@ -4909,13 +5009,13 @@ func Validate_HTTPSRule( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPSRule) *ateapipb.Ports { + func(oldObj *ateapipb.HttpsRule) *ateapipb.Ports { return oldObj.Ports }) errs = append(errs, fn(fldPath.Child("ports"), obj.Ports, oldVal, oldObj != nil)...) } - { // field ateapipb.HTTPSRule.Effects + { // field ateapipb.HttpsRule.Effects fn := func( fldPath *field.Path, obj, oldObj *ateapipb.HttpRuleEffects, @@ -4939,7 +5039,7 @@ func Validate_HTTPSRule( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HTTPSRule) *ateapipb.HttpRuleEffects { + func(oldObj *ateapipb.HttpsRule) *ateapipb.HttpRuleEffects { return oldObj.Effects }) errs = append(errs, fn(fldPath.Child("effects"), obj.Effects, oldVal, oldObj != nil)...) @@ -4948,80 +5048,20 @@ func Validate_HTTPSRule( return errs } -// Validate_HttpRuleEffects validates an instance of HttpRuleEffects according +// Validate_ImageVolumeSource validates an instance of ImageVolumeSource according // to declarative validation rules in the API schema. -func Validate_HttpRuleEffects( +func Validate_ImageVolumeSource( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.HttpRuleEffects) (errs field.ErrorList) { - - // custom validation - if e := ValidateCustom_HttpRuleEffects(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) - } + obj, oldObj *ateapipb.ImageVolumeSource) (errs field.ErrorList) { - { // field ateapipb.HttpRuleEffects.ReplaceHeaders + { // field ateapipb.ImageVolumeSource.Reference fn := func( fldPath *field.Path, - obj, oldObj []*ateapipb.CredentialHeader, + obj, oldObj *string, oldValueCorrelated bool) (errs field.ErrorList) { - // Uniqueness validation is implemented via custom, handwritten validation // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.PtrSliceNoNils[ateapipb.CredentialHeader](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 16).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - // custom validation - if e := ValidateCustom_HttpRuleEffects_ReplaceHeaders(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) - } - // iterate the list and call the type's validation function - if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, - func(a *ateapipb.CredentialHeader, b *ateapipb.CredentialHeader) bool { return a.Header == b.Header }, ateDeepEqual, Validate_CredentialHeader); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.HttpRuleEffects) []*ateapipb.CredentialHeader { - return oldObj.ReplaceHeaders - }) - errs = append(errs, fn(fldPath.Child("replace_headers"), obj.ReplaceHeaders, oldVal, oldObj != nil)...) - } - - return errs -} - -// Validate_ImageVolumeSource validates an instance of ImageVolumeSource according -// to declarative validation rules in the API schema. -func Validate_ImageVolumeSource( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.ImageVolumeSource) (errs field.ErrorList) { - - { // field ateapipb.ImageVolumeSource.Reference - fn := func( - fldPath *field.Path, - obj, oldObj *string, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } @@ -5168,66 +5208,34 @@ func Validate_ListActorTemplatesRequest( errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) } - { // field ateapipb.ListActorTemplatesRequest.PageSize - fn := func( - fldPath *field.Path, - obj, oldObj *int32, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListActorTemplatesRequest) *int32 { - return &oldObj.PageSize - }) - errs = append(errs, fn(fldPath.Child("page_size"), &obj.PageSize, oldVal, oldObj != nil)...) - } - - { // field ateapipb.ListActorTemplatesRequest.PageToken + { // field ateapipb.ListActorTemplatesRequest.Options fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj *ateapipb.ListOptions, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ListOptions(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListActorTemplatesRequest) *string { - return &oldObj.PageToken + func(oldObj *ateapipb.ListActorTemplatesRequest) *ateapipb.ListOptions { + return oldObj.Options }) - errs = append(errs, fn(fldPath.Child("page_token"), &obj.PageToken, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("options"), obj.Options, oldVal, oldObj != nil)...) } return errs @@ -5270,78 +5278,85 @@ func Validate_ListActorsRequest( errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) } - { // field ateapipb.ListActorsRequest.PageSize + { // field ateapipb.ListActorsRequest.Options fn := func( fldPath *field.Path, - obj, oldObj *int32, + obj, oldObj *ateapipb.ListOptions, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ListOptions(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListActorsRequest) *int32 { - return &oldObj.PageSize + func(oldObj *ateapipb.ListActorsRequest) *ateapipb.ListOptions { + return oldObj.Options }) - errs = append(errs, fn(fldPath.Child("page_size"), &obj.PageSize, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("options"), obj.Options, oldVal, oldObj != nil)...) } - { // field ateapipb.ListActorsRequest.PageToken + return errs +} + +// Validate_ListAtespacesRequest validates an instance of ListAtespacesRequest according +// to declarative validation rules in the API schema. +func Validate_ListAtespacesRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.ListAtespacesRequest) (errs field.ErrorList) { + + { // field ateapipb.ListAtespacesRequest.Options fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj *ateapipb.ListOptions, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ListOptions(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListActorsRequest) *string { - return &oldObj.PageToken + func(oldObj *ateapipb.ListAtespacesRequest) *ateapipb.ListOptions { + return oldObj.Options }) - errs = append(errs, fn(fldPath.Child("page_token"), &obj.PageToken, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("options"), obj.Options, oldVal, oldObj != nil)...) } return errs } -// Validate_ListAtespacesRequest validates an instance of ListAtespacesRequest according +// Validate_ListOptions validates an instance of ListOptions according // to declarative validation rules in the API schema. -func Validate_ListAtespacesRequest( +func Validate_ListOptions( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.ListAtespacesRequest) (errs field.ErrorList) { + obj, oldObj *ateapipb.ListOptions) (errs field.ErrorList) { - { // field ateapipb.ListAtespacesRequest.PageSize + { // field ateapipb.ListOptions.PageSize fn := func( fldPath *field.Path, obj, oldObj *int32, @@ -5366,13 +5381,13 @@ func Validate_ListAtespacesRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListAtespacesRequest) *int32 { + func(oldObj *ateapipb.ListOptions) *int32 { return &oldObj.PageSize }) errs = append(errs, fn(fldPath.Child("page_size"), &obj.PageSize, oldVal, oldObj != nil)...) } - { // field ateapipb.ListAtespacesRequest.PageToken + { // field ateapipb.ListOptions.PageToken fn := func( fldPath *field.Path, obj, oldObj *string, @@ -5397,7 +5412,7 @@ func Validate_ListAtespacesRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListAtespacesRequest) *string { + func(oldObj *ateapipb.ListOptions) *string { return &oldObj.PageToken }) errs = append(errs, fn(fldPath.Child("page_token"), &obj.PageToken, oldVal, oldObj != nil)...) @@ -5443,66 +5458,34 @@ func Validate_ListTagsRequest( errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) } - { // field ateapipb.ListTagsRequest.PageSize - fn := func( - fldPath *field.Path, - obj, oldObj *int32, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListTagsRequest) *int32 { - return &oldObj.PageSize - }) - errs = append(errs, fn(fldPath.Child("page_size"), &obj.PageSize, oldVal, oldObj != nil)...) - } - - { // field ateapipb.ListTagsRequest.PageToken + { // field ateapipb.ListTagsRequest.Options fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj *ateapipb.ListOptions, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ListOptions(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListTagsRequest) *string { - return &oldObj.PageToken + func(oldObj *ateapipb.ListTagsRequest) *ateapipb.ListOptions { + return oldObj.Options }) - errs = append(errs, fn(fldPath.Child("page_token"), &obj.PageToken, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("options"), obj.Options, oldVal, oldObj != nil)...) } return errs @@ -5564,66 +5547,34 @@ func Validate_ListWorkerActorAssignmentsRequest( errs = append(errs, fn(fldPath.Child("worker"), obj.Worker, oldVal, oldObj != nil)...) } - { // field ateapipb.ListWorkerActorAssignmentsRequest.PageSize - fn := func( - fldPath *field.Path, - obj, oldObj *int32, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListWorkerActorAssignmentsRequest) *int32 { - return &oldObj.PageSize - }) - errs = append(errs, fn(fldPath.Child("page_size"), &obj.PageSize, oldVal, oldObj != nil)...) - } - - { // field ateapipb.ListWorkerActorAssignmentsRequest.PageToken + { // field ateapipb.ListWorkerActorAssignmentsRequest.Options fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj *ateapipb.ListOptions, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ListOptions(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListWorkerActorAssignmentsRequest) *string { - return &oldObj.PageToken + func(oldObj *ateapipb.ListWorkerActorAssignmentsRequest) *ateapipb.ListOptions { + return oldObj.Options }) - errs = append(errs, fn(fldPath.Child("page_token"), &obj.PageToken, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("options"), obj.Options, oldVal, oldObj != nil)...) } return errs @@ -5635,66 +5586,34 @@ func Validate_ListWorkersRequest( ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *ateapipb.ListWorkersRequest) (errs field.ErrorList) { - { // field ateapipb.ListWorkersRequest.PageSize + { // field ateapipb.ListWorkersRequest.Options fn := func( fldPath *field.Path, - obj, oldObj *int32, + obj, oldObj *ateapipb.ListOptions, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListWorkersRequest) *int32 { - return &oldObj.PageSize - }) - errs = append(errs, fn(fldPath.Child("page_size"), &obj.PageSize, oldVal, oldObj != nil)...) - } - - { // field ateapipb.ListWorkersRequest.PageToken - fn := func( - fldPath *field.Path, - obj, oldObj *string, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ListOptions(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ListWorkersRequest) *string { - return &oldObj.PageToken + func(oldObj *ateapipb.ListWorkersRequest) *ateapipb.ListOptions { + return oldObj.Options }) - errs = append(errs, fn(fldPath.Child("page_token"), &obj.PageToken, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("options"), obj.Options, oldVal, oldObj != nil)...) } return errs @@ -5804,13 +5723,13 @@ func Validate_MintActorCertificateRequest( return errs } -// Validate_MintActorJWTRequest validates an instance of MintActorJWTRequest according +// Validate_MintActorJwtRequest validates an instance of MintActorJwtRequest according // to declarative validation rules in the API schema. -func Validate_MintActorJWTRequest( +func Validate_MintActorJwtRequest( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.MintActorJWTRequest) (errs field.ErrorList) { + obj, oldObj *ateapipb.MintActorJwtRequest) (errs field.ErrorList) { - { // field ateapipb.MintActorJWTRequest.Actor + { // field ateapipb.MintActorJwtRequest.Actor fn := func( fldPath *field.Path, obj, oldObj *ateapipb.ObjectRef, @@ -5835,13 +5754,13 @@ func Validate_MintActorJWTRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintActorJWTRequest) *ateapipb.ObjectRef { + func(oldObj *ateapipb.MintActorJwtRequest) *ateapipb.ObjectRef { return oldObj.Actor }) errs = append(errs, fn(fldPath.Child("actor"), obj.Actor, oldVal, oldObj != nil)...) } - { // field ateapipb.MintActorJWTRequest.Audiences + { // field ateapipb.MintActorJwtRequest.Audiences fn := func( fldPath *field.Path, obj, oldObj []string, @@ -5878,13 +5797,13 @@ func Validate_MintActorJWTRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintActorJWTRequest) []string { + func(oldObj *ateapipb.MintActorJwtRequest) []string { return oldObj.Audiences }) errs = append(errs, fn(fldPath.Child("audiences"), obj.Audiences, oldVal, oldObj != nil)...) } - { // field ateapipb.MintActorJWTRequest.ExpirationSeconds + { // field ateapipb.MintActorJwtRequest.ExpirationSeconds fn := func( fldPath *field.Path, obj, oldObj *int64, @@ -5913,7 +5832,7 @@ func Validate_MintActorJWTRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintActorJWTRequest) *int64 { + func(oldObj *ateapipb.MintActorJwtRequest) *int64 { return &oldObj.ExpirationSeconds }) errs = append(errs, fn(fldPath.Child("expiration_seconds"), &obj.ExpirationSeconds, oldVal, oldObj != nil)...) @@ -6389,51 +6308,178 @@ func Validate_RegisterWorkerRequest( return // do not proceed } // call the type's validation function - errs = append(errs, Validate_WorkerResources(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_WorkerResources(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.RegisterWorkerRequest) *ateapipb.WorkerResources { + return oldObj.Capacity + }) + errs = append(errs, fn(fldPath.Child("capacity"), obj.Capacity, oldVal, oldObj != nil)...) + } + + { // field ateapipb.RegisterWorkerRequest.DefaultRuntime + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SandboxRuntime, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SandboxRuntime(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.RegisterWorkerRequest) *ateapipb.SandboxRuntime { + return oldObj.DefaultRuntime + }) + errs = append(errs, fn(fldPath.Child("default_runtime"), obj.DefaultRuntime, oldVal, oldObj != nil)...) + } + + { // field ateapipb.RegisterWorkerRequest.RestorableRuntimes + fn := func( + fldPath *field.Path, + obj, oldObj []*ateapipb.SandboxRuntime, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateapipb.SandboxRuntime](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, nil, nil, Validate_SandboxRuntime); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.RegisterWorkerRequest) []*ateapipb.SandboxRuntime { + return oldObj.RestorableRuntimes + }) + errs = append(errs, fn(fldPath.Child("restorable_runtimes"), obj.RestorableRuntimes, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_ReportActorSnapshotStateRequest validates an instance of ReportActorSnapshotStateRequest according +// to declarative validation rules in the API schema. +func Validate_ReportActorSnapshotStateRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.ReportActorSnapshotStateRequest) (errs field.ErrorList) { + + { // field ateapipb.ReportActorSnapshotStateRequest.Actor + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.ObjectRef, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + func() { // cohort = "atespace" + earlyReturn := false + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.RequiredValue).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + }() + // call the type's validation function + errs = append(errs, Validate_ObjectRef(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.RegisterWorkerRequest) *ateapipb.WorkerResources { - return oldObj.Capacity + func(oldObj *ateapipb.ReportActorSnapshotStateRequest) *ateapipb.ObjectRef { + return oldObj.Actor }) - errs = append(errs, fn(fldPath.Child("capacity"), obj.Capacity, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("actor"), obj.Actor, oldVal, oldObj != nil)...) } - { // field ateapipb.RegisterWorkerRequest.DefaultRuntime + { // field ateapipb.ReportActorSnapshotStateRequest.ActorUid fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.SandboxRuntime, + obj, oldObj *string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } - // call the type's validation function - errs = append(errs, Validate_SandboxRuntime(ctx, op, fldPath, obj, oldObj)...) + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.RegisterWorkerRequest) *ateapipb.SandboxRuntime { - return oldObj.DefaultRuntime + func(oldObj *ateapipb.ReportActorSnapshotStateRequest) *string { + return &oldObj.ActorUid }) - errs = append(errs, fn(fldPath.Child("default_runtime"), obj.DefaultRuntime, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) } - { // field ateapipb.RegisterWorkerRequest.RestorableRuntimes + { // field ateapipb.ReportActorSnapshotStateRequest.State fn := func( fldPath *field.Path, - obj, oldObj []*ateapipb.SandboxRuntime, + obj, oldObj *ateapipb.ActorSnapshotState, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -6443,31 +6489,22 @@ func Validate_RegisterWorkerRequest( } // call field-attached validations earlyReturn := false - if e := validate.PtrSliceNoNils[ateapipb.SandboxRuntime](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } - if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } if earlyReturn { return // do not proceed } - // iterate the list and call the type's validation function - if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, nil, nil, Validate_SandboxRuntime); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ActorSnapshotState(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.RegisterWorkerRequest) []*ateapipb.SandboxRuntime { - return oldObj.RestorableRuntimes + func(oldObj *ateapipb.ReportActorSnapshotStateRequest) *ateapipb.ActorSnapshotState { + return oldObj.State }) - errs = append(errs, fn(fldPath.Child("restorable_runtimes"), obj.RestorableRuntimes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("state"), obj.State, oldVal, oldObj != nil)...) } return errs @@ -7495,74 +7532,271 @@ func Validate_SnapshotConfig( oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_SnapshotConfig_StorageLocation(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 1024); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.SnapshotConfig) *string { + return &oldObj.StorageLocation + }) + errs = append(errs, fn(fldPath.Child("storage_location"), &obj.StorageLocation, oldVal, oldObj != nil)...) + } + + { // field ateapipb.SnapshotConfig.PreferredFidelity + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SnapshotFidelity, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.SnapshotConfig) *ateapipb.SnapshotFidelity { + return &oldObj.PreferredFidelity + }) + errs = append(errs, fn(fldPath.Child("preferred_fidelity"), &obj.PreferredFidelity, oldVal, oldObj != nil)...) + } + + { // field ateapipb.SnapshotConfig.GoldenSnapshotConfig + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.GoldenSnapshotConfig, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_GoldenSnapshotConfig(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.SnapshotConfig) *ateapipb.GoldenSnapshotConfig { + return oldObj.GoldenSnapshotConfig + }) + errs = append(errs, fn(fldPath.Child("golden_snapshot_config"), obj.GoldenSnapshotConfig, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_SnapshotReport validates an instance of SnapshotReport according +// to declarative validation rules in the API schema. +func Validate_SnapshotReport( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.SnapshotReport) (errs field.ErrorList) { + + { // field ateapipb.SnapshotReport.Uuid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.SnapshotReport) *string { + return &oldObj.Uuid + }) + errs = append(errs, fn(fldPath.Child("uuid"), &obj.Uuid, oldVal, oldObj != nil)...) + } + + { // field ateapipb.SnapshotReport.Generation + fn := func( + fldPath *field.Path, + obj, oldObj *int32, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.SnapshotReport) *int32 { + return &oldObj.Generation + }) + errs = append(errs, fn(fldPath.Child("generation"), &obj.Generation, oldVal, oldObj != nil)...) + } + + { // field ateapipb.SnapshotReport.Fidelity + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SnapshotFidelity, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.SnapshotReport) *ateapipb.SnapshotFidelity { + return &oldObj.Fidelity + }) + errs = append(errs, fn(fldPath.Child("fidelity"), &obj.Fidelity, oldVal, oldObj != nil)...) + } + + { // field ateapipb.SnapshotReport.Object + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.ObjectSnapshot, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } - // custom validation - if e := ValidateCustom_SnapshotConfig_StorageLocation(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 1024); len(e) != 0 { - errs = append(errs, e...) - } + // call the type's validation function + errs = append(errs, Validate_ObjectSnapshot(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.SnapshotConfig) *string { - return &oldObj.StorageLocation + func(oldObj *ateapipb.SnapshotReport) *ateapipb.ObjectSnapshot { + return oldObj.Object }) - errs = append(errs, fn(fldPath.Child("storage_location"), &obj.StorageLocation, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("object"), obj.Object, oldVal, oldObj != nil)...) } - { // field ateapipb.SnapshotConfig.PreferredFidelity + { // field ateapipb.SnapshotReport.LocalCommitTime fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.SnapshotFidelity, + obj, oldObj *timestamppb.Timestamp, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.SnapshotConfig) *ateapipb.SnapshotFidelity { - return &oldObj.PreferredFidelity + func(oldObj *ateapipb.SnapshotReport) *timestamppb.Timestamp { + return oldObj.LocalCommitTime }) - errs = append(errs, fn(fldPath.Child("preferred_fidelity"), &obj.PreferredFidelity, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("local_commit_time"), obj.LocalCommitTime, oldVal, oldObj != nil)...) } - { // field ateapipb.SnapshotConfig.GoldenSnapshotConfig + { // field ateapipb.SnapshotReport.DurableCompleteTime fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.GoldenSnapshotConfig, + obj, oldObj *timestamppb.Timestamp, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -7578,15 +7812,13 @@ func Validate_SnapshotConfig( if earlyReturn { return // do not proceed } - // call the type's validation function - errs = append(errs, Validate_GoldenSnapshotConfig(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.SnapshotConfig) *ateapipb.GoldenSnapshotConfig { - return oldObj.GoldenSnapshotConfig + func(oldObj *ateapipb.SnapshotReport) *timestamppb.Timestamp { + return oldObj.DurableCompleteTime }) - errs = append(errs, fn(fldPath.Child("golden_snapshot_config"), obj.GoldenSnapshotConfig, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("durable_complete_time"), obj.DurableCompleteTime, oldVal, oldObj != nil)...) } return errs @@ -7862,87 +8094,6 @@ func Validate_SystemInfoVolumeSource( return errs } -// Validate_TLSPassthroughRule validates an instance of TLSPassthroughRule according -// to declarative validation rules in the API schema. -func Validate_TLSPassthroughRule( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.TLSPassthroughRule) (errs field.ErrorList) { - - { // field ateapipb.TLSPassthroughRule.Hostnames - fn := func( - fldPath *field.Path, - obj, oldObj []string, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 256).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - // custom validation - if e := ValidateCustom_TLSPassthroughRule_Hostnames(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) - } - // lists with set semantics require unique values - if e := validate.ValSliceUnique(ctx, op, fldPath, obj, oldObj, validate.DirectEqual); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.TLSPassthroughRule) []string { - return oldObj.Hostnames - }) - errs = append(errs, fn(fldPath.Child("hostnames"), obj.Hostnames, oldVal, oldObj != nil)...) - } - - { // field ateapipb.TLSPassthroughRule.Ports - fn := func( - fldPath *field.Path, - obj, oldObj *ateapipb.Ports, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - // call the type's validation function - errs = append(errs, Validate_Ports(ctx, op, fldPath, obj, oldObj)...) - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.TLSPassthroughRule) *ateapipb.Ports { - return oldObj.Ports - }) - errs = append(errs, fn(fldPath.Child("ports"), obj.Ports, oldVal, oldObj != nil)...) - } - - return errs -} - // Validate_Tag validates an instance of Tag according // to declarative validation rules in the API schema. func Validate_Tag( @@ -8203,6 +8354,87 @@ func Validate_TagStatus( return errs } +// Validate_TlsPassthroughRule validates an instance of TlsPassthroughRule according +// to declarative validation rules in the API schema. +func Validate_TlsPassthroughRule( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.TlsPassthroughRule) (errs field.ErrorList) { + + { // field ateapipb.TlsPassthroughRule.Hostnames + fn := func( + fldPath *field.Path, + obj, oldObj []string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 256).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_TlsPassthroughRule_Hostnames(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + // lists with set semantics require unique values + if e := validate.ValSliceUnique(ctx, op, fldPath, obj, oldObj, validate.DirectEqual); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.TlsPassthroughRule) []string { + return oldObj.Hostnames + }) + errs = append(errs, fn(fldPath.Child("hostnames"), obj.Hostnames, oldVal, oldObj != nil)...) + } + + { // field ateapipb.TlsPassthroughRule.Ports + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.Ports, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_Ports(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.TlsPassthroughRule) *ateapipb.Ports { + return oldObj.Ports + }) + errs = append(errs, fn(fldPath.Child("ports"), obj.Ports, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_TrustBundleDataSource validates an instance of TrustBundleDataSource according // to declarative validation rules in the API schema. func Validate_TrustBundleDataSource( @@ -9037,6 +9269,81 @@ func Validate_VolumeMount( return errs } +// Validate_WakeupProbe validates an instance of WakeupProbe according +// to declarative validation rules in the API schema. +func Validate_WakeupProbe( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.WakeupProbe) (errs field.ErrorList) { + + { // field ateapipb.WakeupProbe.HttpGet + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.HttpGetAction, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_HttpGetAction(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.WakeupProbe) *ateapipb.HttpGetAction { + return oldObj.HttpGet + }) + errs = append(errs, fn(fldPath.Child("http_get"), obj.HttpGet, oldVal, oldObj != nil)...) + } + + { // field ateapipb.WakeupProbe.TimeoutSeconds + fn := func( + fldPath *field.Path, + obj, oldObj *int32, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3600); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.WakeupProbe) *int32 { + return &oldObj.TimeoutSeconds + }) + errs = append(errs, fn(fldPath.Child("timeout_seconds"), &obj.TimeoutSeconds, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_Worker validates an instance of Worker according // to declarative validation rules in the API schema. func Validate_Worker( diff --git a/cmd/ateapi/internal/authz/authorizer.go b/cmd/ateapi/internal/authz/authorizer.go index ff79c369c6..59fcaca465 100644 --- a/cmd/ateapi/internal/authz/authorizer.go +++ b/cmd/ateapi/internal/authz/authorizer.go @@ -37,12 +37,24 @@ type Authorizer struct { // AccessPolicy, cannot be revoked through the API, and lose access once // the server runs without them in its configuration. bootstrapOwners map[string]struct{} + + // systemPrincipals maps the OpenFGA user strings of Substrate's own + // components to the system roles they hold on global:root. Like + // bootstrapOwners they are contextual tuples, but they only apply to + // principals authenticated over mTLS. + // + // TODO: consider storing system grants as tuples, written from the + // configuration at startup, so they are visible through the API. That + // needs principal IDs qualified by authentication method, so a stored + // tuple can't match a JWT subject, and AccessPolicy reconciliation + // limited to its own relations, so it doesn't delete them. + systemPrincipals map[string][]string } // Check verifies that the principal in ctx has relation on object. // Structural hierarchy links (global:root as parent_global of every atespace // and worker, and each atespace as parent_atespace of its actors and actor templates) and -// the caller's bootstrap owner grant, if any, are injected as OpenFGA +// the caller's bootstrap owner and system role grants, if any, are injected as OpenFGA // ContextualTuples at evaluation time rather than persisted in the tuple table. func (a *Authorizer) Check(ctx context.Context, relation, object string) error { if IsBypassed(ctx) { @@ -56,7 +68,17 @@ func (a *Authorizer) Check(ctx context.Context, relation, object string) error { return apierror.Unauthenticated("unauthenticated: missing principal in context") } user := formatUser(p.ID) - allowed, err := a.checkRaw(ctx, user, relation, object) + var grants []*openfgav1.TupleKey + if p.Kind == principal.KindMTLS { + for _, role := range a.systemPrincipals[user] { + grants = append(grants, &openfgav1.TupleKey{ + User: user, + Relation: role, + Object: GlobalRootObject, + }) + } + } + allowed, err := a.checkRaw(ctx, user, relation, object, grants...) if err != nil { return err } @@ -66,8 +88,11 @@ func (a *Authorizer) Check(ctx context.Context, relation, object string) error { return nil } -func (a *Authorizer) checkRaw(ctx context.Context, user, relation, object string) (bool, error) { - tuples := contextualTuples(object) +// checkRaw reports whether user has relation on object. Besides the stored +// tuples, OpenFGA evaluates these as contextual tuples: object's structural +// parent links, user's bootstrap owner grant if configured, and grants. +func (a *Authorizer) checkRaw(ctx context.Context, user, relation, object string, grants ...*openfgav1.TupleKey) (bool, error) { + tuples := append(contextualTuples(object), grants...) // A Check only evaluates the caller, so only the caller's own bootstrap // grant can affect the result. if _, ok := a.bootstrapOwners[user]; ok { diff --git a/cmd/ateapi/internal/authz/authz_test.go b/cmd/ateapi/internal/authz/authz_test.go index f2c691a6ad..0a8a744049 100644 --- a/cmd/ateapi/internal/authz/authz_test.go +++ b/cmd/ateapi/internal/authz/authz_test.go @@ -439,7 +439,7 @@ func TestAuthorizerAndPolicyManager_RuntimeChecks(t *testing.T) { } t.Cleanup(fgaSrv.Close) - authorizer, policyManager, err := New(ctx, pool, fgaSrv, nil) + authorizer, policyManager, err := New(ctx, pool, fgaSrv, nil, testSystemPrincipals()) if err != nil { t.Fatalf("authz.New failed: %v", err) } diff --git a/cmd/ateapi/internal/authz/authz_util.go b/cmd/ateapi/internal/authz/authz_util.go index bb4d6c51cd..3882a11f5d 100644 --- a/cmd/ateapi/internal/authz/authz_util.go +++ b/cmd/ateapi/internal/authz/authz_util.go @@ -28,6 +28,7 @@ import ( "github.com/openfga/openfga/pkg/server" "github.com/openfga/openfga/pkg/storage/postgres" "github.com/openfga/openfga/pkg/storage/sqlcommon" + "github.com/spiffe/go-spiffe/v2/spiffeid" "google.golang.org/protobuf/proto" "google.golang.org/protobuf/types/known/wrapperspb" ) @@ -43,6 +44,10 @@ const ( RoleEditor = "editor" RoleViewer = "viewer" + RoleController = "controller" + RoleEgressGateway = "egress_gateway" + RoleIngressRouter = "ingress_router" + RelationCanCreateAtespace = "can_create_atespace" RelationCanListAtespaces = "can_list_atespaces" RelationCanCreateActor = "can_create_actor" @@ -212,25 +217,39 @@ func EnsureStoreAndModel(ctx context.Context, pool *pgxpool.Pool, fgaServer *ser return storeID, modelID, nil } +// SystemPrincipals holds the mTLS SPIFFE IDs of Substrate's own components. +// The Authorizer grants each one its system component role on global:root on +// every check, independent of any stored AccessPolicy. +type SystemPrincipals struct { + ControllerSPIFFEID string + EgressGatewaySPIFFEID string + IngressRouterSPIFFEID string +} + // New provisions the default OpenFGA store and authorization model via // EnsureStoreAndModel and returns the read-path Authorizer and write-path // PolicyManager. bootstrapOwners are principal IDs (with or without the // "user:" prefix) that the Authorizer treats as owners of global:root on every // check, independent of any stored AccessPolicy. -func New(ctx context.Context, pool *pgxpool.Pool, fgaServer *server.Server, bootstrapOwners []string) (*Authorizer, *PolicyManager, error) { +func New(ctx context.Context, pool *pgxpool.Pool, fgaServer *server.Server, bootstrapOwners []string, system SystemPrincipals) (*Authorizer, *PolicyManager, error) { owners, err := parseBootstrapOwners(bootstrapOwners) if err != nil { return nil, nil, err } + systemPrincipals, err := parseSystemPrincipals(system) + if err != nil { + return nil, nil, err + } storeID, modelID, err := EnsureStoreAndModel(ctx, pool, fgaServer) if err != nil { return nil, nil, err } authorizer := &Authorizer{ - fgaServer: fgaServer, - storeID: storeID, - modelID: modelID, - bootstrapOwners: owners, + fgaServer: fgaServer, + storeID: storeID, + modelID: modelID, + bootstrapOwners: owners, + systemPrincipals: systemPrincipals, } policyManager := &PolicyManager{ fgaServer: fgaServer, @@ -254,6 +273,31 @@ func parseBootstrapOwners(ids []string) (map[string]struct{}, error) { return owners, nil } +// parseSystemPrincipals validates system's SPIFFE IDs and maps each OpenFGA +// user string to the system component roles it holds. +func parseSystemPrincipals(system SystemPrincipals) (map[string][]string, error) { + bindings := []struct { + role string + spiffeID string + }{ + {role: RoleController, spiffeID: system.ControllerSPIFFEID}, + {role: RoleEgressGateway, spiffeID: system.EgressGatewaySPIFFEID}, + {role: RoleIngressRouter, spiffeID: system.IngressRouterSPIFFEID}, + } + out := make(map[string][]string, len(bindings)) + for _, b := range bindings { + if _, err := spiffeid.FromString(b.spiffeID); err != nil { + return nil, fmt.Errorf("invalid %s SPIFFE ID %q: %w", b.role, b.spiffeID, err) + } + user, err := FormatMember("user:" + b.spiffeID) + if err != nil { + return nil, fmt.Errorf("invalid %s SPIFFE ID %q: %w", b.role, b.spiffeID, err) + } + out[user] = append(out[user], b.role) + } + return out, nil +} + // ateFGAInitLockID is a 64-bit identifier ("atefga") for serializing // OpenFGA store provisioning across replicas. const ateFGAInitLockID = int64(0x6174656667610000) // "atefga\0\0" diff --git a/cmd/ateapi/internal/authz/authz_util_test.go b/cmd/ateapi/internal/authz/authz_util_test.go index e851f89f1b..40ad23d1f2 100644 --- a/cmd/ateapi/internal/authz/authz_util_test.go +++ b/cmd/ateapi/internal/authz/authz_util_test.go @@ -15,11 +15,22 @@ package authz import ( + "context" + "strings" "testing" + "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/google/go-cmp/cmp" ) +func testSystemPrincipals() SystemPrincipals { + return SystemPrincipals{ + ControllerSPIFFEID: installdefaults.ControllerSPIFFEID(installdefaults.SystemNamespace), + EgressGatewaySPIFFEID: installdefaults.EgressSPIFFEID(installdefaults.SystemNamespace), + IngressRouterSPIFFEID: installdefaults.RouterSPIFFEID(installdefaults.SystemNamespace), + } +} + func TestFormatMember(t *testing.T) { tests := []struct { name string @@ -78,3 +89,49 @@ func TestParseBootstrapOwners(t *testing.T) { } } } + +func TestParseSystemPrincipals(t *testing.T) { + const router = "spiffe://cluster.local/ns/ate-system/sa/atenet-router" + got, err := parseSystemPrincipals(SystemPrincipals{ + ControllerSPIFFEID: "spiffe://cluster.local/ns/ate-system/sa/ate-controller", + EgressGatewaySPIFFEID: router, + IngressRouterSPIFFEID: router, + }) + if err != nil { + t.Fatalf("parseSystemPrincipals failed: %v", err) + } + want := map[string][]string{ + "user:spiffe%3A//cluster.local/ns/ate-system/sa/ate-controller": {RoleController}, + "user:spiffe%3A//cluster.local/ns/ate-system/sa/atenet-router": {RoleEgressGateway, RoleIngressRouter}, + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("parseSystemPrincipals (-want +got):\n%s", diff) + } +} + +func TestNewRejectsInvalidSystemPrincipals(t *testing.T) { + defaults := testSystemPrincipals() + for _, tc := range []struct { + name string + mutate func(*SystemPrincipals) + wantErr string + }{ + {"empty controller SPIFFE ID", func(p *SystemPrincipals) { p.ControllerSPIFFEID = "" }, "invalid controller SPIFFE ID"}, + {"empty egress gateway SPIFFE ID", func(p *SystemPrincipals) { p.EgressGatewaySPIFFEID = "" }, "invalid egress_gateway SPIFFE ID"}, + {"empty ingress router SPIFFE ID", func(p *SystemPrincipals) { p.IngressRouterSPIFFEID = "" }, "invalid ingress_router SPIFFE ID"}, + {"bare ServiceAccount name", func(p *SystemPrincipals) { p.ControllerSPIFFEID = "ate-controller" }, "invalid controller SPIFFE ID"}, + {"non-spiffe URI", func(p *SystemPrincipals) { + p.EgressGatewaySPIFFEID = "https://cluster.local/ns/ate-system/sa/atenet-egress" + }, "invalid egress_gateway SPIFFE ID"}, + } { + t.Run(tc.name, func(t *testing.T) { + p := defaults + tc.mutate(&p) + // Principals are validated before the store is touched, so no + // database is needed. + if _, _, err := New(context.Background(), nil, nil, nil, p); err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("New(%+v) error = %v, want substring %q", p, err, tc.wantErr) + } + }) + } +} diff --git a/cmd/ateapi/internal/authz/interceptor_test.go b/cmd/ateapi/internal/authz/interceptor_test.go index de80415801..ae71b3ab23 100644 --- a/cmd/ateapi/internal/authz/interceptor_test.go +++ b/cmd/ateapi/internal/authz/interceptor_test.go @@ -37,7 +37,7 @@ func setupTestAuthorizer(t *testing.T) *Authorizer { } t.Cleanup(fgaServer.Close) - authorizer, policyManager, err := New(ctx, pool, fgaServer, nil) + authorizer, policyManager, err := New(ctx, pool, fgaServer, nil, testSystemPrincipals()) if err != nil { t.Fatalf("New failed: %v", err) } @@ -297,7 +297,7 @@ func TestUnaryServerInterceptor_ActorAndTemplateChecks(t *testing.T) { t.Cleanup(fgaServer.Close) // bootstrap-owner is a global owner through configuration only; it has no // stored tuples. - authorizer, policyManager, err := New(ctx, pool, fgaServer, []string{"bootstrap-owner"}) + authorizer, policyManager, err := New(ctx, pool, fgaServer, []string{"bootstrap-owner"}, testSystemPrincipals()) if err != nil { t.Fatalf("New failed: %v", err) } @@ -445,7 +445,7 @@ func TestUnaryServerInterceptor_LifecycleEgressAndWorkerChecks(t *testing.T) { t.Cleanup(fgaServer.Close) // bootstrap-owner is a global owner through configuration only; it has no // stored tuples. - authorizer, policyManager, err := New(ctx, pool, fgaServer, []string{"bootstrap-owner"}) + authorizer, policyManager, err := New(ctx, pool, fgaServer, []string{"bootstrap-owner"}, testSystemPrincipals()) if err != nil { t.Fatalf("New failed: %v", err) } @@ -554,3 +554,125 @@ func TestUnaryServerInterceptor_LifecycleEgressAndWorkerChecks(t *testing.T) { }) } } + +func TestUnaryServerInterceptor_SystemPrincipals(t *testing.T) { + ctx := context.Background() + pool := startPostgres(t) + fgaServer, err := NewOpenFGAServer(pool) + if err != nil { + t.Fatalf("NewOpenFGAServer failed: %v", err) + } + t.Cleanup(fgaServer.Close) + const ( + controllerID = "spiffe://cluster.local/ns/ate-system/sa/ate-controller" + egressID = "spiffe://cluster.local/ns/ate-system/sa/atenet-egress" + routerID = "spiffe://cluster.local/ns/ate-system/sa/atenet-router" + otherID = "spiffe://cluster.local/ns/ate-system/sa/other" + ) + authorizer, _, err := New(ctx, pool, fgaServer, nil, SystemPrincipals{ + ControllerSPIFFEID: controllerID, + EgressGatewaySPIFFEID: egressID, + IngressRouterSPIFFEID: routerID, + }) + if err != nil { + t.Fatalf("New failed: %v", err) + } + + type rpcCall struct { + fullMethod string + req any + } + actor := &ateapipb.ObjectRef{Atespace: "team-a", Name: "runner"} + worker := &ateapipb.ObjectRef{Name: "w-1"} + getActor := rpcCall{ateapipb.Control_GetActor_FullMethodName, &ateapipb.GetActorRequest{Actor: actor}} + deleteActor := rpcCall{ateapipb.Control_DeleteActor_FullMethodName, &ateapipb.DeleteActorRequest{Actor: actor}} + listAllActors := rpcCall{ateapipb.Control_ListActors_FullMethodName, &ateapipb.ListActorsRequest{}} + listAtespaces := rpcCall{ateapipb.Control_ListAtespaces_FullMethodName, &ateapipb.ListAtespacesRequest{}} + pause := rpcCall{ateapipb.Control_PauseActor_FullMethodName, &ateapipb.PauseActorRequest{Actor: actor}} + resume := rpcCall{ateapipb.Control_ResumeActor_FullMethodName, &ateapipb.ResumeActorRequest{Actor: actor}} + suspend := rpcCall{ateapipb.Control_SuspendActor_FullMethodName, &ateapipb.SuspendActorRequest{Actor: actor}} + getEgress := rpcCall{ateapipb.Control_GetActorEgressPolicy_FullMethodName, &ateapipb.GetActorEgressPolicyRequest{Actor: actor}} + updateEgress := rpcCall{ateapipb.Control_UpdateActorEgressPolicy_FullMethodName, &ateapipb.UpdateActorEgressPolicyRequest{Actor: actor}} + createWorker := rpcCall{ateapipb.Control_CreateWorker_FullMethodName, &ateapipb.CreateWorkerRequest{}} + listWorkers := rpcCall{ateapipb.Control_ListWorkers_FullMethodName, &ateapipb.ListWorkersRequest{}} + getWorker := rpcCall{ateapipb.Control_GetWorker_FullMethodName, &ateapipb.GetWorkerRequest{Worker: worker}} + updateWorker := rpcCall{ateapipb.Control_UpdateWorker_FullMethodName, &ateapipb.UpdateWorkerRequest{Worker: &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: worker.GetName()}, + }}} + deleteWorker := rpcCall{ateapipb.Control_DeleteWorker_FullMethodName, &ateapipb.DeleteWorkerRequest{Worker: worker}} + drainWorker := rpcCall{ateapipb.Control_DrainWorker_FullMethodName, &ateapipb.DrainWorkerRequest{Worker: worker}} + listAssignments := rpcCall{ateapipb.Control_ListWorkerActorAssignments_FullMethodName, &ateapipb.ListWorkerActorAssignmentsRequest{Worker: worker}} + + mtls := func(id string) principal.PrincipalInfo { + return principal.PrincipalInfo{ID: id, Kind: principal.KindMTLS} + } + jwt := func(id string) principal.PrincipalInfo { + return principal.PrincipalInfo{ID: id, Kind: principal.KindJWT} + } + + tests := []struct { + name string + principal principal.PrincipalInfo + call rpcCall + wantCode codes.Code + }{ + // ate-controller manages workers and nothing else. + {"controller creates worker", mtls(controllerID), createWorker, codes.OK}, + {"controller lists workers", mtls(controllerID), listWorkers, codes.OK}, + {"controller gets worker", mtls(controllerID), getWorker, codes.OK}, + {"controller updates worker", mtls(controllerID), updateWorker, codes.OK}, + {"controller deletes worker", mtls(controllerID), deleteWorker, codes.OK}, + {"controller drains worker", mtls(controllerID), drainWorker, codes.OK}, + {"controller cannot list worker assignments", mtls(controllerID), listAssignments, codes.PermissionDenied}, + {"controller cannot get actor", mtls(controllerID), getActor, codes.PermissionDenied}, + {"controller cannot resume actor", mtls(controllerID), resume, codes.PermissionDenied}, + {"controller cannot list atespaces", mtls(controllerID), listAtespaces, codes.PermissionDenied}, + + // atenet-egress reads actors and their egress policies. + {"egress gateway gets actor", mtls(egressID), getActor, codes.OK}, + {"egress gateway gets egress policy", mtls(egressID), getEgress, codes.OK}, + {"egress gateway cannot update egress policy", mtls(egressID), updateEgress, codes.PermissionDenied}, + {"egress gateway cannot delete actor", mtls(egressID), deleteActor, codes.PermissionDenied}, + {"egress gateway cannot resume actor", mtls(egressID), resume, codes.PermissionDenied}, + {"egress gateway lists actors", mtls(egressID), listAllActors, codes.OK}, + {"egress gateway cannot get worker", mtls(egressID), getWorker, codes.PermissionDenied}, + + // atenet-router resumes actors. Both it and atenet-egress list actors + // as their ateapi health check. + {"ingress router resumes actor", mtls(routerID), resume, codes.OK}, + {"ingress router lists actors", mtls(routerID), listAllActors, codes.OK}, + {"ingress router cannot get actor", mtls(routerID), getActor, codes.PermissionDenied}, + {"ingress router cannot pause actor", mtls(routerID), pause, codes.PermissionDenied}, + {"ingress router cannot suspend actor", mtls(routerID), suspend, codes.PermissionDenied}, + {"ingress router cannot delete actor", mtls(routerID), deleteActor, codes.PermissionDenied}, + {"ingress router cannot get egress policy", mtls(routerID), getEgress, codes.PermissionDenied}, + {"ingress router cannot create worker", mtls(routerID), createWorker, codes.PermissionDenied}, + + // A bearer token whose subject is a granted SPIFFE ID gets nothing. + {"jwt with controller ID cannot create worker", jwt(controllerID), createWorker, codes.PermissionDenied}, + {"jwt with egress ID cannot get actor", jwt(egressID), getActor, codes.PermissionDenied}, + {"jwt with router ID cannot resume actor", jwt(routerID), resume, codes.PermissionDenied}, + + // Other SPIFFE IDs get nothing. + {"other mTLS identity cannot resume actor", mtls(otherID), resume, codes.PermissionDenied}, + {"other mTLS identity cannot create worker", mtls(otherID), createWorker, codes.PermissionDenied}, + } + + interceptor := UnaryServerInterceptor(authorizer, true) + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + userCtx := principal.InjectContext(ctx, tc.principal) + handlerCalled := false + _, err := interceptor(userCtx, tc.call.req, &grpc.UnaryServerInfo{FullMethod: tc.call.fullMethod}, func(context.Context, any) (any, error) { + handlerCalled = true + return "ok", nil + }) + if apierror.Code(err) != tc.wantCode { + t.Fatalf("apierror.Code(err) = %v, want %v (err: %v)", apierror.Code(err), tc.wantCode, err) + } + if wantHandler := tc.wantCode == codes.OK; handlerCalled != wantHandler { + t.Fatalf("handlerCalled = %v, want %v", handlerCalled, wantHandler) + } + }) + } +} diff --git a/cmd/ateapi/internal/authz/model.fga b/cmd/ateapi/internal/authz/model.fga index deaea1e641..03e9fa70d3 100644 --- a/cmd/ateapi/internal/authz/model.fga +++ b/cmd/ateapi/internal/authz/model.fga @@ -31,11 +31,11 @@ type worker define parent_global: [global] define host_node: [node] - define can_get: viewer from parent_global + define can_get: viewer from parent_global or controller from parent_global define can_list_actor_assignments: viewer from parent_global - define can_update: owner from parent_global - define can_delete: owner from parent_global - define can_drain: owner from parent_global + define can_update: owner from parent_global or controller from parent_global + define can_delete: owner from parent_global or controller from parent_global + define can_drain: owner from parent_global or controller from parent_global # Global Scope (singleton: there will only ever be one instance, e.g. 'global:root') type global @@ -43,18 +43,24 @@ type global define owner: [user] define viewer: [user] or owner - # TODO: add system component roles (atenet-router, atenet-egress, - # ate-controller), granted from ateapi's configuration to their mTLS - # SPIFFE IDs, so they don't need global owner to call ateapi. + # --- System Component Roles --- + # Held only by Substrate's own components, through contextual tuples that + # ateapi adds for their mTLS SPIFFE IDs. AccessPolicy cannot grant them. + # Each grants only the permissions its component calls ateapi for. + define controller: [user] + define egress_gateway: [user] + define ingress_router: [user] define can_create_atespace: owner define can_list_atespaces: viewer - define can_create_worker: owner - define can_list_workers: viewer + define can_create_worker: owner or controller + define can_list_workers: viewer or controller define can_get: viewer # Listing across all atespaces (List RPCs with an empty atespace). - define can_list_actors: viewer + # atenet-router and atenet-egress, one binary, list actors as their ateapi + # health check. + define can_list_actors: viewer or ingress_router or egress_gateway define can_list_actor_templates: viewer # --- Policy Governance --- @@ -72,6 +78,10 @@ type atespace define editor: [user] or owner define viewer: [user] or editor or viewer from parent_global + # System component roles, inherited from global only. + define egress_gateway: egress_gateway from parent_global + define ingress_router: ingress_router from parent_global + # --- 2. Policy Governance --- define can_create_access_policy: owner define can_get_access_policy: viewer @@ -108,12 +118,13 @@ type actor define host_node: host_node from scheduled_worker # --- User-Facing Operations (Inherited from Atespace) --- - define can_get: viewer from parent_atespace + # atenet-egress reads the actor to resolve its egress policy. + define can_get: viewer from parent_atespace or egress_gateway from parent_atespace define can_update: editor from parent_atespace define can_delete: editor from parent_atespace # Egress policy: one per actor, so its permissions live on the actor. - define can_get_egress_policy: viewer from parent_atespace + define can_get_egress_policy: viewer from parent_atespace or egress_gateway from parent_atespace define can_create_egress_policy: editor from parent_atespace define can_update_egress_policy: editor from parent_atespace define can_delete_egress_policy: editor from parent_atespace @@ -124,7 +135,7 @@ type actor define can_mint_ateom_actor_credential: host_node # Data Plane Ingress: Routers (or editors) can resume - define can_resume: [user] or editor from parent_atespace + define can_resume: [user] or editor from parent_atespace or ingress_router from parent_atespace # Lifecycle Operations define can_pause: [user] or editor from parent_atespace diff --git a/cmd/ateapi/internal/authz/model_test.fga.yaml b/cmd/ateapi/internal/authz/model_test.fga.yaml index 191f65bbbb..1a9b70adcd 100644 --- a/cmd/ateapi/internal/authz/model_test.fga.yaml +++ b/cmd/ateapi/internal/authz/model_test.fga.yaml @@ -73,6 +73,18 @@ tuples: relation: can_revert object: actor:runner-1 + # System component roles (contextual tuples from ateapi's configuration at + # runtime) + - user: user:sys-controller + relation: controller + object: global:root + - user: user:sys-egress + relation: egress_gateway + object: global:root + - user: user:sys-router + relation: ingress_router + object: global:root + tests: - name: 1. Global Superadmin & Viewer Propagation check: @@ -359,3 +371,84 @@ tests: can_update: false can_delete: false can_drain: false + + - name: 7. System Component Roles (Granted on Global) + check: + - user: user:sys-controller # ate-controller manages workers only + object: global:root + assertions: + can_create_worker: true + can_list_workers: true + can_list_actors: false + can_list_atespaces: false + can_create_atespace: false + can_get: false + viewer: false + - user: user:sys-controller + object: worker:pod-1 + assertions: + can_get: true + can_update: true + can_delete: true + can_drain: true + can_list_actor_assignments: false + - user: user:sys-controller + object: actor:runner-1 + assertions: + can_get: false + can_resume: false + can_delete: false + - user: user:sys-egress # atenet-egress reads actors and egress policies + object: actor:runner-1 + assertions: + can_get: true + can_get_egress_policy: true + can_update: false + can_delete: false + can_create_egress_policy: false + can_update_egress_policy: false + can_delete_egress_policy: false + can_resume: false + can_pause: false + - user: user:sys-egress # lists actors as its ateapi health check + object: global:root + assertions: + can_list_actors: true + can_create_worker: false + - user: user:sys-egress + object: atespace:space-1 + assertions: + viewer: false + can_list_actors: false + can_get: false + - user: user:sys-router # atenet-router resumes actors and lists them + object: actor:runner-1 + assertions: + can_resume: true + can_get: false + can_pause: false + can_suspend: false + can_revert: false + can_delete: false + can_get_egress_policy: false + - user: user:sys-router + object: global:root + assertions: + can_list_actors: true + can_list_workers: false + can_list_atespaces: false + - user: user:sys-router + object: worker:pod-1 + assertions: + can_get: false + - user: user:alice # Atespace owner does not hold system roles + object: atespace:space-1 + assertions: + egress_gateway: false + ingress_router: false + - user: user:carol # Global owner does not hold system roles + object: global:root + assertions: + controller: false + egress_gateway: false + ingress_router: false diff --git a/cmd/ateapi/internal/authz/registry.go b/cmd/ateapi/internal/authz/registry.go index edc2a331f2..a6526b95b1 100644 --- a/cmd/ateapi/internal/authz/registry.go +++ b/cmd/ateapi/internal/authz/registry.go @@ -218,11 +218,6 @@ var defaultRPCPermissions = map[string]rpcRule{ return onActor(RelationCanDelete, r.GetActor(), field.NewPath("actor")) })), - // TODO: grant the system components that call the RPCs below - // (atenet-router: ResumeActor; atenet-egress: GetActor, - // GetActorEgressPolicy; ate-controller: worker RPCs) through system roles - // in model.fga. - // Actor lifecycle. ateapipb.Control_PauseActor_FullMethodName: rule(checksOf(func(r *ateapipb.PauseActorRequest) ([]check, field.ErrorList) { return onActor(RelationCanPause, r.GetActor(), field.NewPath("actor")) diff --git a/cmd/ateapi/internal/controlapi/access_policy_test.go b/cmd/ateapi/internal/controlapi/access_policy_test.go index 0ef92d41cf..77a2ea2a52 100644 --- a/cmd/ateapi/internal/controlapi/access_policy_test.go +++ b/cmd/ateapi/internal/controlapi/access_policy_test.go @@ -42,7 +42,7 @@ func TestAccessPolicy_GlobalAndAtespaceGovernance(t *testing.T) { // alice@example.com is a bootstrap owner: a global owner through server // configuration, never through a stored AccessPolicy. - authorizer, policyManager, err := authz.New(ctx, pool, fgaServer, []string{"alice@example.com"}) + authorizer, policyManager, err := authz.New(ctx, pool, fgaServer, []string{"alice@example.com"}, storetest.DefaultSystemPrincipals()) if err != nil { t.Fatalf("authz.New failed: %v", err) } @@ -196,7 +196,7 @@ func TestAccessPolicy_GlobalAndAtespaceGovernance(t *testing.T) { // 6. Removing Alice from the bootstrap owners (a restart with a different // configuration) leaves only her stored grant; once that is gone too she // is denied, while the new bootstrap owner is allowed. - restarted, _, err := authz.New(ctx, pool, fgaServer, []string{"charlie@example.com"}) + restarted, _, err := authz.New(ctx, pool, fgaServer, []string{"charlie@example.com"}, storetest.DefaultSystemPrincipals()) if err != nil { t.Fatalf("authz.New after restart failed: %v", err) } diff --git a/cmd/ateapi/internal/controlapi/actor.go b/cmd/ateapi/internal/controlapi/actor.go index 19740224df..52ba1ac464 100644 --- a/cmd/ateapi/internal/controlapi/actor.go +++ b/cmd/ateapi/internal/controlapi/actor.go @@ -219,7 +219,7 @@ func (s *RPCService) ListActors(ctx context.Context, req *ateapipb.ListActorsReq return nil, resources.ToAPIError(errs) } - page, err := s.impl.ListActors(ctx, req.GetAtespace(), store.ListOptions{PageSize: effectivePageSize(req.GetPageSize()), PageToken: req.GetPageToken()}) + page, err := s.impl.ListActors(ctx, req.GetAtespace(), toStoreListOptions(req.GetOptions())) if err != nil { return nil, mapListError(fmt.Errorf("while listing actors in db: %w", err)) } @@ -299,6 +299,12 @@ func (s *ServiceImpl) UpdateActor(ctx context.Context, actorRef resources.ActorR return apierror.FailedPrecondition( "actor must be %s to change its actor template (got: %s)", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, state) } + // A paused actor's latest snapshot is node-local and was captured + // from the old template's sandbox; suspend or revert it first. + if latestSnapshotIsLocalOnly(oldVal.GetStatus()) { + return apierror.FailedPrecondition( + "actor must hold no node-local snapshot to change its actor template: suspend or revert it first") + } newTemplate, err := resolveActorTemplate(ctx, s.store, newVal) if err != nil { return err @@ -429,8 +435,8 @@ func (s *RPCService) DeleteActor(ctx context.Context, req *ateapipb.DeleteActorR } start := time.Now() // Template dims only once the record resolved: the request names only the - // actor, so failures before the load carry none. No pool pair: delete only - // runs from SUSPENDED or CRASHED, which already released the worker. + // actor, so failures before the load carry none. No pool pair, even when + // any_state deletes an actor that still holds a worker. defer func() { var attrs []attribute.KeyValue if deleted != nil { @@ -542,7 +548,7 @@ func (s *RPCService) RevertActor(ctx context.Context, req *ateapipb.RevertActorR return &ateapipb.RevertActorResponse{Actor: actor}, nil } -func (s *RPCService) MintActorJWT(ctx context.Context, req *ateapipb.MintActorJWTRequest) (*ateapipb.MintActorJWTResponse, error) { +func (s *RPCService) MintActorJwt(ctx context.Context, req *ateapipb.MintActorJwtRequest) (*ateapipb.MintActorJwtResponse, error) { if errs := apivalidation.ValidateMintActorJWTRequest(ctx, req); len(errs) > 0 { return nil, apierror.InvalidArgument("%v", errs.ToAggregate()) } @@ -566,7 +572,7 @@ func (s *RPCService) MintActorJWT(ctx context.Context, req *ateapipb.MintActorJW return nil, fmt.Errorf("at least one audience must be requested") } - // JWT timestamps have one-second resolution; truncating keeps expires_at + // JWT timestamps have one-second resolution; truncating keeps expire_time // equal to the exp claim. now := time.Now().Truncate(time.Second) expiresAt := now.Add(time.Duration(req.GetExpirationSeconds()) * time.Second) @@ -591,9 +597,9 @@ func (s *RPCService) MintActorJWT(ctx context.Context, req *ateapipb.MintActorJW return nil, fmt.Errorf("while signing actor JWT: %w", err) } - return &ateapipb.MintActorJWTResponse{ - ActorJwt: actorJWT, - ExpiresAt: timestamppb.New(expiresAt), + return &ateapipb.MintActorJwtResponse{ + ActorJwt: actorJWT, + ExpireTime: timestamppb.New(expiresAt), }, nil } diff --git a/cmd/ateapi/internal/controlapi/actor_template.go b/cmd/ateapi/internal/controlapi/actor_template.go index 25ce835b6f..c625463e1b 100644 --- a/cmd/ateapi/internal/controlapi/actor_template.go +++ b/cmd/ateapi/internal/controlapi/actor_template.go @@ -105,7 +105,7 @@ func (s *RPCService) ListActorTemplates(ctx context.Context, req *ateapipb.ListA return nil, resources.ToAPIError(errs) } - page, err := s.impl.ListActorTemplates(ctx, req.GetAtespace(), store.ListOptions{PageSize: effectivePageSize(req.GetPageSize()), PageToken: req.GetPageToken()}) + page, err := s.impl.ListActorTemplates(ctx, req.GetAtespace(), toStoreListOptions(req.GetOptions())) if err != nil { return nil, mapListError(fmt.Errorf("while listing actor templates in db: %w", err)) } diff --git a/cmd/ateapi/internal/controlapi/atespace.go b/cmd/ateapi/internal/controlapi/atespace.go index 22e7a5a1af..f3f809a99d 100644 --- a/cmd/ateapi/internal/controlapi/atespace.go +++ b/cmd/ateapi/internal/controlapi/atespace.go @@ -85,7 +85,7 @@ func (s *RPCService) ListAtespaces(ctx context.Context, req *ateapipb.ListAtespa return nil, resources.ToAPIError(errs) } - page, err := s.impl.ListAtespaces(ctx, store.ListOptions{PageSize: req.PageSize, PageToken: req.PageToken}) + page, err := s.impl.ListAtespaces(ctx, toStoreListOptions(req.GetOptions())) if err != nil { return nil, err } @@ -96,7 +96,6 @@ func (s *RPCService) ListAtespaces(ctx context.Context, req *ateapipb.ListAtespa } func (s *ServiceImpl) ListAtespaces(ctx context.Context, opts store.ListOptions) (store.ListResponse[*ateapipb.Atespace], error) { - opts.PageSize = effectivePageSize(opts.PageSize) page, err := s.store.ListAtespaces(ctx, opts) if err != nil { return page, mapListError(fmt.Errorf("while listing atespaces in db: %w", err)) diff --git a/cmd/ateapi/internal/controlapi/atespace_test.go b/cmd/ateapi/internal/controlapi/atespace_test.go index 58b3ea15b6..4c8bf5613e 100644 --- a/cmd/ateapi/internal/controlapi/atespace_test.go +++ b/cmd/ateapi/internal/controlapi/atespace_test.go @@ -58,7 +58,7 @@ func TestAtespace_EndToEndOpenFGAScenarios(t *testing.T) { } t.Cleanup(fgaServer.Close) - authorizer, policyManager, err := authz.New(ctx, pool, fgaServer, nil) + authorizer, policyManager, err := authz.New(ctx, pool, fgaServer, nil, storetest.DefaultSystemPrincipals()) if err != nil { t.Fatalf("authz.New failed: %v", err) } diff --git a/cmd/ateapi/internal/controlapi/egress_policy_test.go b/cmd/ateapi/internal/controlapi/egress_policy_test.go index 6e782a3337..90f7777ade 100644 --- a/cmd/ateapi/internal/controlapi/egress_policy_test.go +++ b/cmd/ateapi/internal/controlapi/egress_policy_test.go @@ -65,7 +65,7 @@ func TestActorEgressPolicy(t *testing.T) { Uid: "ignored", Version: 99, }, Rules: []*ateapipb.EgressRule{{ - Http: &ateapipb.HTTPRule{ + Http: &ateapipb.HttpRule{ Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{ ReplaceHeaders: []*ateapipb.CredentialHeader{{ diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go index 697107cd24..8a5034ebf7 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go @@ -324,7 +324,7 @@ func TestActorLifecycleWithoutGoldenSnapshot(t *testing.T) { if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: ref}); err != nil { t.Fatal(err) } - if got := tc.fakeAtelet.lastRestoreRequest().GetExternalConfig().GetSnapshotUri(); got != snapshotURI { + if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetObject().GetSnapshotUri(); got != snapshotURI { t.Fatalf("restored snapshot = %q, want actor's snapshot %q", got, snapshotURI) } if err := assertNoGolden(ctx); err != nil { @@ -347,7 +347,7 @@ func TestListActorTemplates_InvalidPageToken(t *testing.T) { defer tc.cleanup() _, err := tc.client.ListActorTemplates(context.Background(), - &ateapipb.ListActorTemplatesRequest{PageToken: "%%%"}) + &ateapipb.ListActorTemplatesRequest{Options: &ateapipb.ListOptions{PageToken: "%%%"}}) assertGrpcError(t, err, codes.InvalidArgument, "invalid page_token") } diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go index 49f4f64b4d..e045588b6e 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go @@ -632,9 +632,8 @@ func TestListActors_Pagination(t *testing.T) { for { listResp, err := tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{ - Atespace: testAtespace, - PageSize: 2, - PageToken: pageToken, + Atespace: testAtespace, + Options: &ateapipb.ListOptions{PageSize: 2, PageToken: pageToken}, }) if err != nil { t.Fatalf("ListActors failed: %v", err) @@ -1319,6 +1318,56 @@ func TestDeleteActor_StateDeleting(t *testing.T) { } } +// TestDeleteActor_Paused deletes a paused actor without any_state: a paused +// actor is SUSPENDED (holding a node-local snapshot), so it is deletable by +// default, and the delete terminates it on its assigned node so the local +// snapshot is removed. +func TestDeleteActor_Paused(t *testing.T) { + ns := namespaceForTest("ns-delete-paused") + tc := setupTest(t, ns) + defer tc.cleanup() + + createTemplate(t, tc, ns) + createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1") + + actorRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"} + if _, err := tc.client.CreateActor(context.Background(), &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "id1"}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + }}); err != nil { + t.Fatalf("CreateActor failed: %v", err) + } + if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil { + t.Fatalf("ResumeActor failed: %v", err) + } + if _, err := tc.client.PauseActor(context.Background(), &ateapipb.PauseActorRequest{Actor: actorRef}); err != nil { + t.Fatalf("PauseActor failed: %v", err) + } + paused, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{Actor: actorRef}) + if err != nil { + t.Fatalf("GetActor failed: %v", err) + } + if got := paused.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Fatalf("actor state after pause = %v, want ACTOR_STATE_SUSPENDED", got) + } + if localSnapshot(paused.GetStatus()) == nil { + t.Fatal("paused actor has no local snapshot") + } + tc.fakeAtelet.Reset() + + if _, err := tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{Actor: actorRef}); err != nil { + t.Fatalf("DeleteActor of a paused actor without any_state failed: %v", err) + } + if !tc.fakeAtelet.TerminateCalled { + t.Error("expected atelet Terminate on the paused actor's node to remove its local snapshot") + } else if got, want := tc.fakeAtelet.TerminateRequest.GetActorUid(), paused.GetMetadata().GetUid(); got != want { + t.Errorf("Terminate actor_uid = %q, want %q", got, want) + } + if _, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{Actor: actorRef}); status.Code(err) != codes.NotFound { + t.Errorf("GetActor after DeleteActor err = %v, want NotFound", err) + } +} + func TestDeleteActor_WrongState(t *testing.T) { ns := namespaceForTest("ns-delete-wrong-status") tc := setupTest(t, ns) @@ -1557,17 +1606,17 @@ func TestValidation_Actor(t *testing.T) { }) t.Run("ListActors", func(t *testing.T) { - _, err := tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{PageSize: -1}) + _, err := tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{Options: &ateapipb.ListOptions{PageSize: -1}}) assertGrpcErrorRegex(t, err, codes.InvalidArgument, "page_size: Invalid value") }) t.Run("ListActors invalid token", func(t *testing.T) { - _, err := tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{PageToken: "%%%"}) + _, err := tc.client.ListActors(context.Background(), &ateapipb.ListActorsRequest{Options: &ateapipb.ListOptions{PageToken: "%%%"}}) assertGrpcError(t, err, codes.InvalidArgument, "invalid page_token") }) t.Run("ListTags invalid token", func(t *testing.T) { - _, err := tc.client.ListTags(context.Background(), &ateapipb.ListTagsRequest{PageToken: "%%%"}) + _, err := tc.client.ListTags(context.Background(), &ateapipb.ListTagsRequest{Options: &ateapipb.ListOptions{PageToken: "%%%"}}) assertGrpcError(t, err, codes.InvalidArgument, "invalid page_token") }) } @@ -1636,8 +1685,11 @@ func TestActorLifecycle_WithExternalVolumes(t *testing.T) { if err != nil { t.Fatalf("PauseActor failed: %v", err) } - if pauseResp.GetActor().GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSED { - t.Fatalf("expected state ACTOR_STATE_PAUSED after pause, got %v", pauseResp.GetActor().GetStatus().GetState()) + if pauseResp.GetActor().GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Fatalf("expected state ACTOR_STATE_SUSPENDED after pause, got %v", pauseResp.GetActor().GetStatus().GetState()) + } + if localSnapshot(pauseResp.GetActor().GetStatus()) == nil { + t.Fatal("expected a local snapshot on the paused actor") } waitForWorkerAvailable(t, tc, workerName) @@ -2634,14 +2686,14 @@ func TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState(t *testing.T) { // TestPauseActor_VolumeLifecycle_DetachAndResumeAttach tests scenario D: // external volume lifecycle across PauseActor and ResumeActor: // 1. When an actor is resumed to RUNNING, external volumes are attached to the worker node. -// 2. When the actor is paused, volumes are detached from the worker node and the actor transitions to ACTOR_STATE_PAUSED. -// 3. When the actor is resumed from PAUSED, volumes are re-attached to the worker node and the actor transitions to ACTOR_STATE_RUNNING. +// 2. When the actor is paused, volumes are detached from the worker node and the actor transitions to ACTOR_STATE_SUSPENDED with a local snapshot. +// 3. When the actor is resumed from pause, volumes are re-attached to the worker node and the actor transitions to ACTOR_STATE_RUNNING. // // Workflow: // 1. Creates a template with an external volume and creates a mock worker pod on node1. // 2. Creates and resumes an actor to ACTOR_STATE_RUNNING; verifies volume is attached to node1. -// 3. Calls PauseActor; verifies volume is detached from node1 and actor transitions to ACTOR_STATE_PAUSED. -// 4. Calls ResumeActor from PAUSED; verifies volume is re-attached to node1 and actor transitions to ACTOR_STATE_RUNNING. +// 3. Calls PauseActor; verifies volume is detached from node1 and actor transitions to ACTOR_STATE_SUSPENDED with a local snapshot. +// 4. Calls ResumeActor from pause; verifies volume is re-attached to node1 and actor transitions to ACTOR_STATE_RUNNING. // 5. Cleans up by suspending and deleting the actor. func TestPauseActor_VolumeLifecycle_DetachAndResumeAttach(t *testing.T) { ns := namespaceForTest("ns-pause-vol-lifecycle") @@ -2690,7 +2742,7 @@ func TestPauseActor_VolumeLifecycle_DetachAndResumeAttach(t *testing.T) { } plugin.mu.Unlock() - // 3. PauseActor: volume is detached from node1 and actor transitions to ACTOR_STATE_PAUSED. + // 3. PauseActor: volume is detached from node1 and actor transitions to ACTOR_STATE_SUSPENDED with a local snapshot. _, err = tc.client.PauseActor(context.Background(), &ateapipb.PauseActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "pause-vol-actor"}, }) @@ -2704,8 +2756,11 @@ func TestPauseActor_VolumeLifecycle_DetachAndResumeAttach(t *testing.T) { if err != nil { t.Fatalf("GetActor failed: %v", err) } - if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSED { - t.Errorf("actor state = %v, want ACTOR_STATE_PAUSED", getResp.GetStatus().GetState()) + if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("actor state = %v, want ACTOR_STATE_SUSPENDED", getResp.GetStatus().GetState()) + } + if localSnapshot(getResp.GetStatus()) == nil { + t.Error("expected a local snapshot on the paused actor") } plugin.mu.Lock() if diff := cmp.Diff([]string{"node1"}, plugin.detachedNodes); diff != "" { @@ -2714,12 +2769,12 @@ func TestPauseActor_VolumeLifecycle_DetachAndResumeAttach(t *testing.T) { plugin.mu.Unlock() waitForWorkerAvailable(t, tc, workerName) - // 4. ResumeActor from PAUSED: volume is re-attached to node1 and actor transitions to ACTOR_STATE_RUNNING. + // 4. ResumeActor from pause: volume is re-attached to node1 and actor transitions to ACTOR_STATE_RUNNING. _, err = tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "pause-vol-actor"}, }) if err != nil { - t.Fatalf("ResumeActor from PAUSED failed: %v", err) + t.Fatalf("ResumeActor from pause failed: %v", err) } getResp, err = tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{ @@ -2755,16 +2810,16 @@ func TestPauseActor_VolumeLifecycle_DetachAndResumeAttach(t *testing.T) { // TestPauseActor_VolumeDetachFailure_RetrySuccess tests scenario D: // when volume detachment fails during PauseActor: -// 1. PauseActor fails, leaving the actor in ACTOR_STATE_PAUSING. -// 2. A subsequent retry of PauseActor re-attempts volume detachment and succeeds, transitioning the actor to ACTOR_STATE_PAUSED. +// 1. PauseActor fails, leaving the actor in ACTOR_STATE_SUSPENDING. +// 2. A subsequent retry of PauseActor re-attempts volume detachment and succeeds, transitioning the actor to ACTOR_STATE_SUSPENDED. // // Workflow: // 1. Creates a template with an external volume and creates a mock worker pod on node1. // 2. Creates and resumes an actor to ACTOR_STATE_RUNNING. // 3. Calls PauseActor; plugin is configured to fail DetachVolume on attempt 1. -// 4. Verifies PauseActor returns an error and actor state is left in ACTOR_STATE_PAUSING. +// 4. Verifies PauseActor returns an error and actor state is left in ACTOR_STATE_SUSPENDING. // 5. Calls PauseActor a second time (retry); detachment succeeds. -// 6. Verifies actor state transitions to ACTOR_STATE_PAUSED. +// 6. Verifies actor state transitions to ACTOR_STATE_SUSPENDED with a local snapshot. // 7. Cleans up by deleting the actor with AnyState=true. func TestPauseActor_VolumeDetachFailure_RetrySuccess(t *testing.T) { ns := namespaceForTest("ns-pause-vol-detach-retry") @@ -2818,15 +2873,15 @@ func TestPauseActor_VolumeDetachFailure_RetrySuccess(t *testing.T) { t.Errorf("expected error containing simulated detach failure, got: %v", err) } - // 4. Verify actor state is ACTOR_STATE_PAUSING. + // 4. Verify actor state is ACTOR_STATE_SUSPENDING. getResp, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "pause-detach-retry-actor"}, }) if err != nil { t.Fatalf("GetActor after failed pause: %v", err) } - if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSING { - t.Errorf("actor state = %v, want ACTOR_STATE_PAUSING", getResp.GetStatus().GetState()) + if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { + t.Errorf("actor state = %v, want ACTOR_STATE_SUSPENDING", getResp.GetStatus().GetState()) } // 5. Attempt 2 (retry): PauseActor retry succeeds, detaching volume. @@ -2837,15 +2892,18 @@ func TestPauseActor_VolumeDetachFailure_RetrySuccess(t *testing.T) { t.Fatalf("expected retry PauseActor to succeed, got: %v", err) } - // 6. Verify actor state transitions to ACTOR_STATE_PAUSED. + // 6. Verify actor state transitions to ACTOR_STATE_SUSPENDED with a local snapshot. getResp, err = tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "pause-detach-retry-actor"}, }) if err != nil { t.Fatalf("GetActor after successful pause retry: %v", err) } - if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSED { - t.Errorf("actor state = %v, want ACTOR_STATE_PAUSED", getResp.GetStatus().GetState()) + if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("actor state = %v, want ACTOR_STATE_SUSPENDED", getResp.GetStatus().GetState()) + } + if localSnapshot(getResp.GetStatus()) == nil { + t.Error("expected a local snapshot on the paused actor") } // 7. Clean up by deleting the actor with AnyState=true. @@ -3134,10 +3192,10 @@ func TestResumeActor_DataSnapshotIgnoresGolden(t *testing.T) { if restoreReq == nil { t.Fatal("second resume sent no Restore request to atelet") } - if got := restoreReq.GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + if got := restoreReq.GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { t.Fatalf("restore scope = %v, want SNAPSHOT_FIDELITY_VOLUMES", got) } - if got := restoreReq.GetExternalConfig().GetSnapshotUri(); got != actorSnapshotURI { + if got := restoreReq.GetSnapshot().GetObject().GetSnapshotUri(); got != actorSnapshotURI { t.Errorf("restore config snapshot uri = %q, want the actor's data snapshot %q", got, actorSnapshotURI) } } @@ -3202,7 +3260,7 @@ func TestSuspendActor_ReplacedSnapshotReleaseFailure(t *testing.T) { if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil { t.Fatalf("ResumeActor after the release failure failed: %v", err) } - if got := tc.fakeAtelet.lastRestoreRequest().GetExternalConfig().GetSnapshotUri(); got != freshURI { + if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetObject().GetSnapshotUri(); got != freshURI { t.Errorf("restore snapshot uri = %q, want the second suspend's %q", got, freshURI) } @@ -3515,8 +3573,8 @@ func TestResumeActor_LocalRestoreFailureCrashesActor(t *testing.T) { if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{Actor: ref}); status.Code(err) != codes.Internal { t.Fatalf("ResumeActor status code = %v, want Internal (err: %v)", status.Code(err), err) } - if got := tc.fakeAtelet.RestoreRequest.GetType(); got != ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL { - t.Fatalf("restore type = %v, want LOCAL", got) + if got := tc.fakeAtelet.RestoreRequest.GetSnapshot(); got.GetSnapshotUuid() == "" || got.GetObject() != nil { + t.Fatalf("restore snapshot = %v, want local snapshot (uuid set, no object)", got) } actor, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{Actor: ref}) @@ -3657,7 +3715,7 @@ func TestSuspendActor(t *testing.T) { if diff := cmp.Diff(wantTag, stored, protocmp.Transform(), ignoreUID, ignoreVersion, ignoreTimestamps); diff != "" { t.Errorf("stored tag mismatch (-want +got):\n%s", diff) } - listed, err := tc.client.ListTags(context.Background(), &ateapipb.ListTagsRequest{Atespace: testAtespace, PageSize: 1}) + listed, err := tc.client.ListTags(context.Background(), &ateapipb.ListTagsRequest{Atespace: testAtespace, Options: &ateapipb.ListOptions{PageSize: 1}}) if err != nil || len(listed.GetTags()) != 1 { t.Fatalf("ListTags = (%v, %v), want one", listed, err) } @@ -3898,12 +3956,12 @@ func TestResumeActor_RepointTemplateBeforeResume(t *testing.T) { if got := restoreReq.GetActorTemplateName(); got != tt.wantTemplate { t.Errorf("restore request to atelet had actor template = %q, want %q", got, tt.wantTemplate) } - if got := restoreReq.GetFidelity(); got != tt.wantScope { + if got := restoreReq.GetSnapshot().GetFidelity(); got != tt.wantScope { t.Errorf("restore request to atelet had scope = %v, want %v", got, tt.wantScope) } // Either way the restore reads the snapshot the clone borrowed // from the tag, not the template's golden image. - if got := restoreReq.GetExternalConfig().GetSnapshotUri(); got != durableSnapshotURI(cloneActor.GetStatus()) { + if got := restoreReq.GetSnapshot().GetObject().GetSnapshotUri(); got != durableSnapshotURI(cloneActor.GetStatus()) { t.Errorf("restore request to atelet had snapshot uri = %q, want the clone's borrowed %q", got, durableSnapshotURI(cloneActor.GetStatus())) } }) @@ -3960,7 +4018,7 @@ func TestResumeActor_PausedAfterRepointUsesLocalProvenance(t *testing.T) { if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil { t.Fatalf("ResumeActor(v2 from v1 snapshot) failed: %v", err) } - if got := tc.fakeAtelet.lastRestoreRequest().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { t.Fatalf("first resume on v2 had scope = %v, want DATA", got) } if _, err := tc.client.PauseActor(ctx, &ateapipb.PauseActorRequest{Actor: actorRef}); err != nil { @@ -3968,16 +4026,16 @@ func TestResumeActor_PausedAfterRepointUsesLocalProvenance(t *testing.T) { } waitForWorkerAvailable(t, tc, worker) - // Resume from PAUSED: the local checkpoint was captured on v2 and the + // Resume from pause: the local checkpoint was captured on v2 and the // actor's template is v2, so the restore must be FULL. if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil { t.Fatalf("ResumeActor(from v2 pause) failed: %v", err) } restoreReq := tc.fakeAtelet.lastRestoreRequest() - if got := restoreReq.GetType(); got != ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL { - t.Errorf("restore request type = %v, want LOCAL", got) + if got := restoreReq.GetSnapshot(); got.GetSnapshotUuid() == "" || got.GetObject() != nil { + t.Errorf("restore request snapshot = %v, want local snapshot (uuid set, no object)", got) } - if got := restoreReq.GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { + if got := restoreReq.GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { t.Errorf("restore request scope = %v, want FULL (local checkpoint was captured on v2)", got) } } @@ -4044,7 +4102,7 @@ func TestPauseActor(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Name: name, Atespace: testAtespace}, ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_PAUSED, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, AssignedNode: "node1", LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{ @@ -4066,7 +4124,7 @@ func TestPauseActor(t *testing.T) { } // TestResumeActor_PausedLocalSnapshotMissing_Crashes tests that if the local checkpoint -// files on the worker node are missing (e.g. node reboot, /tmp wipe) when resuming a PAUSED actor, +// files on the worker node are missing (e.g. node reboot, /tmp wipe) when resuming a paused actor, // atelet returns a terminal file system error and ateapi marks the actor ACTOR_STATE_CRASHED // while releasing the assigned worker pod. func TestResumeActor_PausedLocalSnapshotMissing_Crashes(t *testing.T) { @@ -4106,8 +4164,8 @@ func TestResumeActor_PausedLocalSnapshotMissing_Crashes(t *testing.T) { if err != nil { t.Fatalf("GetActor failed: %v", err) } - if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSED { - t.Fatalf("actor state = %v, want ACTOR_STATE_PAUSED", getResp.GetStatus().GetState()) + if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Fatalf("actor state = %v, want ACTOR_STATE_SUSPENDED", getResp.GetStatus().GetState()) } if ls := localSnapshot(getResp.GetStatus()); ls == nil { t.Fatal("expected local snapshot to be present on paused actor") @@ -4680,7 +4738,7 @@ func TestSuspendActor_DanglingWorker(t *testing.T) { } } -// TestSuspendActor_FromPaused suspends a PAUSED actor end-to-end: instead of +// TestSuspendActor_FromPaused suspends a paused actor end-to-end: instead of // checkpointing a running workload, ateapi asks the atelet on the node // holding the pause snapshot to upload it, then finalizes the actor with a // durable ActorSnapshot and no node pinning left behind. @@ -4727,28 +4785,28 @@ func TestSuspendActor_FromPaused(t *testing.T) { t.Fatalf("SuspendActor failed: %v", err) } - if !tc.fakeAtelet.UploadCalled { - t.Fatal("expected atelet UploadPausedCheckpoint to be called") + if !tc.fakeAtelet.PromoteCalled { + t.Fatal("expected atelet PromoteSnapshot to be called") } if tc.fakeAtelet.CheckpointCalled { t.Error("atelet Checkpoint called for a paused actor; there is no workload to checkpoint") } - upload := tc.fakeAtelet.UploadRequest + promote := tc.fakeAtelet.PromoteRequest pausedLS := localSnapshot(paused.GetStatus()) if pausedLS == nil { t.Fatal("paused actor has no local snapshot") } - if got, want := upload.GetLocalSnapshotName(), pausedLS.GetUuid(); got != want { - t.Errorf("upload local_snapshot_name = %q, want the pause snapshot %q", got, want) + if got, want := promote.GetSnapshot().GetSnapshotUuid(), pausedLS.GetUuid(); got != want { + t.Errorf("promote snapshot_uuid = %q, want %q", got, want) } - if got, want := snapshotName(t, upload.GetDestinationSnapshotUri()), pausedLS.GetUuid(); got != want { - t.Errorf("upload destination snapshot name = %q, want the pause snapshot uuid %q", got, want) + if got, want := promote.GetSnapshot().GetFidelity(), ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY; got != want { + t.Errorf("promote fidelity = %v, want %v", got, want) } - if got, want := upload.GetAtespace(), testAtespace; got != want { - t.Errorf("upload atespace = %q, want %q", got, want) + if got, want := promote.GetAtespace(), testAtespace; got != want { + t.Errorf("promote atespace = %q, want %q", got, want) } - if got := upload.GetDesiredFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { - t.Errorf("upload desired_fidelity = %v, want MEMORY (template default)", got) + if got, want := promote.GetActorName(), name; got != want { + t.Errorf("promote actor_name = %q, want %q", got, want) } actor := suspended.GetActor() @@ -4758,8 +4816,11 @@ func TestSuspendActor_FromPaused(t *testing.T) { if ls := localSnapshot(actor.GetStatus()); ls != nil { t.Errorf("local snapshot = %v, want cleared (node pinning must not survive suspend)", ls) } - if got, want := durableSnapshotURI(actor.GetStatus()), upload.GetDestinationSnapshotUri(); got != want { - t.Errorf("snapshot URI = %q, want the upload destination %q", got, want) + if got, want := promote.GetSnapshot().GetObject().GetSnapshotUri(), durableSnapshotURI(actor.GetStatus()); got != want { + t.Errorf("promote snapshot_uri = %q, want %q", got, want) + } + if got, want := snapshotName(t, durableSnapshotURI(actor.GetStatus())), pausedLS.GetUuid(); got != want { + t.Errorf("snapshot name = %q, want the pause snapshot uuid %q", got, want) } // The suspend promotes the pause snapshot rather than taking a new one. durable := durableSnapshot(actor.GetStatus()) @@ -4802,7 +4863,7 @@ func TestSuspendActor_FromPaused_UploadFailureCrashes(t *testing.T) { } tc.fakeAtelet.Reset() - tc.fakeAtelet.FailUpload = status.Error(codes.Internal, "injected upload failure") + tc.fakeAtelet.FailPromote = status.Error(codes.Internal, "injected upload failure") _, err = tc.client.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}, }) @@ -4826,7 +4887,7 @@ func TestSuspendActor_FromPaused_UploadFailureCrashes(t *testing.T) { if crashed.GetStatus().GetWorkerAssignment() != nil { t.Errorf("expected worker assignment to be cleared, got %v", crashed.GetStatus().GetWorkerAssignment()) } - assertActorCrashStatus(t, tc, name, "suspend failed: atelet UploadPausedCheckpoint: injected upload failure") + assertActorCrashStatus(t, tc, name, "suspend failed: atelet PromoteSnapshot: injected upload failure") worker, err := tc.persistence.GetWorker(context.Background(), podUID) if err != nil { @@ -4838,7 +4899,7 @@ func TestSuspendActor_FromPaused_UploadFailureCrashes(t *testing.T) { // The crash is terminal: a healthy atelet does not make the actor suspendable. tc.fakeAtelet.Lock.Lock() - tc.fakeAtelet.FailUpload = nil + tc.fakeAtelet.FailPromote = nil tc.fakeAtelet.Lock.Unlock() _, err = tc.client.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}, @@ -4929,7 +4990,7 @@ func TestCheckpointFailureCrashes(t *testing.T) { } // TestResumeActor_RelocatesAfterSuspendFromPaused covers the capacity-recovery -// flow: a PAUSED actor is pinned to the node holding its local snapshot, so it +// flow: a paused actor is pinned to the node holding its local snapshot, so it // cannot resume while that node is full. Suspending it uploads the snapshot and // drops the node pinning, after which it is scheduled onto a worker on a different // node. @@ -5219,7 +5280,7 @@ func TestMintActorJWT_Success(t *testing.T) { if err != nil { t.Fatalf("CreateActor failed: %v", err) } - mintResp, err := tc.client.MintActorJWT(t.Context(), &ateapipb.MintActorJWTRequest{ + mintResp, err := tc.client.MintActorJwt(t.Context(), &ateapipb.MintActorJwtRequest{ Actor: &ateapipb.ObjectRef{ Atespace: createResp.GetMetadata().GetAtespace(), Name: createResp.GetMetadata().GetName(), @@ -5228,7 +5289,7 @@ func TestMintActorJWT_Success(t *testing.T) { ExpirationSeconds: 1800, }) if err != nil { - t.Fatalf("Error while calling MintActorJWT: %v", err) + t.Fatalf("Error while calling MintActorJwt: %v", err) } segments := strings.Split(mintResp.GetActorJwt(), ".") @@ -5256,12 +5317,12 @@ func TestMintActorJWT_Success(t *testing.T) { assertActorJWTLifetime(t, mintResp, claims, 30*time.Minute) } -// assertActorJWTLifetime checks that expires_at matches the exp claim and that +// assertActorJWTLifetime checks that expire_time matches the exp claim and that // the token is valid for want after it was issued. -func assertActorJWTLifetime(t *testing.T, resp *ateapipb.MintActorJWTResponse, claims actoridjwt.WireClaims, want time.Duration) { +func assertActorJWTLifetime(t *testing.T, resp *ateapipb.MintActorJwtResponse, claims actoridjwt.WireClaims, want time.Duration) { t.Helper() - if got, exp := resp.GetExpiresAt().AsTime(), time.Unix(int64(claims.Expiration), 0); !got.Equal(exp) { - t.Errorf("expires_at = %v, want the exp claim %v", got, exp) + if got, exp := resp.GetExpireTime().AsTime(), time.Unix(int64(claims.Expiration), 0); !got.Equal(exp) { + t.Errorf("expire_time = %v, want the exp claim %v", got, exp) } if got := time.Duration(claims.Expiration-claims.IssuedAt) * time.Second; got != want { t.Errorf("exp - iat = %v, want %v", got, want) diff --git a/cmd/ateapi/internal/controlapi/functionaltest/atespace_test.go b/cmd/ateapi/internal/controlapi/functionaltest/atespace_test.go index bd3e2bb74a..c79db5bcf0 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/atespace_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/atespace_test.go @@ -223,12 +223,12 @@ func TestValidation_Atespace(t *testing.T) { defer tc.cleanup() t.Run("ListAtespaces", func(t *testing.T) { - _, err := tc.client.ListAtespaces(context.Background(), &ateapipb.ListAtespacesRequest{PageSize: -1}) + _, err := tc.client.ListAtespaces(context.Background(), &ateapipb.ListAtespacesRequest{Options: &ateapipb.ListOptions{PageSize: -1}}) assertGrpcErrorRegex(t, err, codes.InvalidArgument, "page_size: Invalid value") }) t.Run("ListAtespaces invalid token", func(t *testing.T) { - _, err := tc.client.ListAtespaces(context.Background(), &ateapipb.ListAtespacesRequest{PageToken: "%%%"}) + _, err := tc.client.ListAtespaces(context.Background(), &ateapipb.ListAtespacesRequest{Options: &ateapipb.ListOptions{PageToken: "%%%"}}) assertGrpcError(t, err, codes.InvalidArgument, "invalid page_token") }) diff --git a/cmd/ateapi/internal/controlapi/functionaltest/egress_policy_test.go b/cmd/ateapi/internal/controlapi/functionaltest/egress_policy_test.go index 0ac4692738..65c5693b9f 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/egress_policy_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/egress_policy_test.go @@ -49,7 +49,7 @@ func functionalEgressPolicy() *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "default"}, Rules: []*ateapipb.EgressRule{{ - Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}}, + Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}}, }}, } } @@ -131,7 +131,7 @@ func TestUpdateActorEgressPolicy(t *testing.T) { tc, actor := setupEgressPolicyActor(t, "ns-update-egress-policy") created := createEgressPolicy(t, tc, actor) toUpdate := proto.Clone(created).(*ateapipb.EgressPolicy) - toUpdate.Rules = []*ateapipb.EgressRule{{TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}} + toUpdate.Rules = []*ateapipb.EgressRule{{TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}} updated, err := tc.client.UpdateActorEgressPolicy(context.Background(), &ateapipb.UpdateActorEgressPolicyRequest{ Actor: actor, diff --git a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go index 51df372a03..f622d4e8f9 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go @@ -128,9 +128,9 @@ type FakeAteletServer struct { FailRestore error RestoreDelay time.Duration - UploadCalled bool - UploadRequest *ateletpb.UploadPausedCheckpointRequest - FailUpload error + PromoteCalled bool + PromoteRequest *ateletpb.PromoteSnapshotRequest + FailPromote error TerminateCalled bool TerminateRequest *ateletpb.TerminateRequest @@ -180,9 +180,9 @@ func (f *FakeAteletServer) Reset() { f.FailRestore = nil f.RestoreDelay = 0 - f.UploadCalled = false - f.UploadRequest = nil - f.FailUpload = nil + f.PromoteCalled = false + f.PromoteRequest = nil + f.FailPromote = nil f.TerminateCalled = false f.TerminateRequest = nil @@ -191,19 +191,19 @@ func (f *FakeAteletServer) Reset() { f.objectStore = nil } -func (f *FakeAteletServer) UploadPausedCheckpoint(ctx context.Context, req *ateletpb.UploadPausedCheckpointRequest) (*ateletpb.UploadPausedCheckpointResponse, error) { +func (f *FakeAteletServer) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnapshotRequest) (*ateletpb.PromoteSnapshotResponse, error) { f.Lock.Lock() defer f.Lock.Unlock() - f.UploadCalled = true - f.UploadRequest = proto.Clone(req).(*ateletpb.UploadPausedCheckpointRequest) - if f.FailUpload != nil { - return nil, f.FailUpload + f.PromoteCalled = true + f.PromoteRequest = proto.Clone(req).(*ateletpb.PromoteSnapshotRequest) + if f.FailPromote != nil { + return nil, f.FailPromote } - if err := f.writeSnapshot(req.GetDestinationSnapshotUri()); err != nil { + if err := f.writeSnapshot(req.GetSnapshot().GetObject().GetSnapshotUri()); err != nil { return nil, err } - return &ateletpb.UploadPausedCheckpointResponse{}, nil + return &ateletpb.PromoteSnapshotResponse{}, nil } func (f *FakeAteletServer) Run(ctx context.Context, req *ateletpb.RunRequest) (*ateletpb.RunResponse, error) { @@ -229,8 +229,12 @@ func (f *FakeAteletServer) Checkpoint(ctx context.Context, req *ateletpb.Checkpo return nil, f.FailCheckpoint } - if err := f.writeSnapshot(req.GetExternalConfig().GetSnapshotUri()); err != nil { - return nil, err + // Only an immediate upload is in object storage when Checkpoint returns; + // atelet does not upload deferred snapshots yet. + if req.GetUploadConfig().GetMode() == ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE { + if err := f.writeSnapshot(req.GetSnapshot().GetObject().GetSnapshotUri()); err != nil { + return nil, err + } } return &ateletpb.CheckpointResponse{}, nil } diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go index 8b6a9fb483..4535709f02 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go @@ -315,12 +315,12 @@ func TestValidation_Worker(t *testing.T) { defer tc.cleanup() t.Run("ListWorkers", func(t *testing.T) { - _, err := tc.client.ListWorkers(context.Background(), &ateapipb.ListWorkersRequest{PageSize: -1}) + _, err := tc.client.ListWorkers(context.Background(), &ateapipb.ListWorkersRequest{Options: &ateapipb.ListOptions{PageSize: -1}}) assertGrpcErrorRegex(t, err, codes.InvalidArgument, "page_size: Invalid value") }) t.Run("ListWorkers invalid token", func(t *testing.T) { - _, err := tc.client.ListWorkers(context.Background(), &ateapipb.ListWorkersRequest{PageToken: "%%%"}) + _, err := tc.client.ListWorkers(context.Background(), &ateapipb.ListWorkersRequest{Options: &ateapipb.ListOptions{PageToken: "%%%"}}) assertGrpcError(t, err, codes.InvalidArgument, "invalid page_token") }) } diff --git a/cmd/ateapi/internal/controlapi/metrics.go b/cmd/ateapi/internal/controlapi/metrics.go index 8ace3106ea..e0c1a1405e 100644 --- a/cmd/ateapi/internal/controlapi/metrics.go +++ b/cmd/ateapi/internal/controlapi/metrics.go @@ -201,17 +201,20 @@ func (i *Instruments) recordLifecycleOp(ctx context.Context, op string, start ti i.lifecycleOpDuration.Record(ctx, time.Since(start).Seconds(), metric.WithAttributes(attrs...)) } -// lifecycleOpAttrs builds the resume/suspend/pause dimensions from workflow -// state. Nil-safe, and omits the pool, snapshot-kind and snapshot-fidelity labels -// until they are known so a failure before the assign/restore steps never emits -// an empty-string series. snapshotKind is empty for suspend/pause, which do not +// lifecycleOpAttrs builds the resume/suspend/pause/revert dimensions from +// workflow state. Nil-safe, and omits the template, pool, snapshot-kind and +// snapshot-fidelity labels until they are known so a failure before the +// load/assign/restore steps never emits an empty-string series. snapshotKind is empty for suspend/pause, which do not // restore; snapshotScope applies to all three and separates a full restore // from a data-only one of the same kind. // The pool keys are set together or not at all; see ateattr.WorkerPoolAttributes. func lifecycleOpAttrs(actor *ateapipb.Actor, template *ateapipb.ActorTemplate, snapshotKind, snapshotScope string) []attribute.KeyValue { - attrs := []attribute.KeyValue{ - ateattr.TemplateNameKey.String(actor.GetActorTemplate().GetName()), - ateattr.TemplateAtespaceKey.String(actor.GetActorTemplate().GetAtespace()), + var attrs []attribute.KeyValue + if actor != nil { + attrs = append(attrs, + ateattr.TemplateNameKey.String(actor.GetActorTemplate().GetName()), + ateattr.TemplateAtespaceKey.String(actor.GetActorTemplate().GetAtespace()), + ) } ass := actor.GetStatus().GetWorkerAssignment() attrs = append(attrs, ateattr.WorkerPoolAttributes(ass.GetWorkerNamespace(), ass.GetWorkerPool())...) diff --git a/cmd/ateapi/internal/controlapi/metrics_test.go b/cmd/ateapi/internal/controlapi/metrics_test.go index 8f4b1f6a72..c0b191e427 100644 --- a/cmd/ateapi/internal/controlapi/metrics_test.go +++ b/cmd/ateapi/internal/controlapi/metrics_test.go @@ -294,6 +294,16 @@ func TestLifecycleOpAttrsOmitsUnknownScope(t *testing.T) { } } +// TestLifecycleOpAttrsOmitsTemplateBeforeLoad guards a failure before or in the +// load step, such as lease contention or a missing ActorTemplate: the workflow +// holds no actor, so the template keys are omitted, as on a failed delete, +// rather than recorded as the empty string. +func TestLifecycleOpAttrsOmitsTemplateBeforeLoad(t *testing.T) { + if got := lifecycleOpAttrs(nil, nil, "", ""); len(got) != 0 { + t.Errorf("lifecycleOpAttrs(nil actor) = %v, want no attributes", got) + } +} + // TestRecordLifecycleOp_OutcomeClassification asserts success omits error.type and // each gRPC failure maps onto its status-code string; the absence of error.type is // the success signal, so there is no separate failure counter. diff --git a/cmd/ateapi/internal/controlapi/pagination.go b/cmd/ateapi/internal/controlapi/pagination.go index af9d4a9c44..f525f56cb9 100644 --- a/cmd/ateapi/internal/controlapi/pagination.go +++ b/cmd/ateapi/internal/controlapi/pagination.go @@ -19,6 +19,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) const maxPageSize = 1000 @@ -32,6 +33,10 @@ func effectivePageSize(requested int32) int32 { return requested } +func toStoreListOptions(opts *ateapipb.ListOptions) store.ListOptions { + return store.ListOptions{PageSize: effectivePageSize(opts.GetPageSize()), PageToken: opts.GetPageToken()} +} + func mapListError(err error) error { if errors.Is(err, store.ErrInvalidPageToken) { return apierror.InvalidArgument("invalid page_token") diff --git a/cmd/ateapi/internal/controlapi/pagination_test.go b/cmd/ateapi/internal/controlapi/pagination_test.go new file mode 100644 index 0000000000..c8414f454b --- /dev/null +++ b/cmd/ateapi/internal/controlapi/pagination_test.go @@ -0,0 +1,53 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package controlapi + +import ( + "testing" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +func TestToStoreListOptions(t *testing.T) { + tests := []struct { + name string + opts *ateapipb.ListOptions + want store.ListOptions + }{{ + name: "nil options default the page size", + opts: nil, + want: store.ListOptions{PageSize: 1000}, + }, { + name: "unset page size defaults", + opts: &ateapipb.ListOptions{PageToken: "tok"}, + want: store.ListOptions{PageSize: 1000, PageToken: "tok"}, + }, { + name: "page size within the cap is kept", + opts: &ateapipb.ListOptions{PageSize: 10, PageToken: "tok"}, + want: store.ListOptions{PageSize: 10, PageToken: "tok"}, + }, { + name: "page size above the cap is coerced", + opts: &ateapipb.ListOptions{PageSize: 5000}, + want: store.ListOptions{PageSize: 1000}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := toStoreListOptions(tt.opts); got != tt.want { + t.Errorf("toStoreListOptions(%v) = %+v, want %+v", tt.opts, got, tt.want) + } + }) + } +} diff --git a/cmd/ateapi/internal/controlapi/snapshot_helpers.go b/cmd/ateapi/internal/controlapi/snapshot_helpers.go index 362f08bede..2350e95bd0 100644 --- a/cmd/ateapi/internal/controlapi/snapshot_helpers.go +++ b/cmd/ateapi/internal/controlapi/snapshot_helpers.go @@ -67,6 +67,28 @@ func findLatestLocalSnapshot(status *ateapipb.ActorStatus) *ateapipb.Snapshot { }) } +// latestSnapshotIsLocalOnly reports whether the snapshot at the actor's latest +// generation is node-local with no durable copy: the shape a pause leaves +// behind. A paused actor is SUSPENDED with this shape, and resumes only on the +// node named by Locality; a suspend of such an actor uploads that snapshot +// instead of checkpointing a workload. A RUNNING actor resumed from pause +// keeps the local snapshot too, so callers that need "at rest" check the state +// as well. +func latestSnapshotIsLocalOnly(status *ateapipb.ActorStatus) bool { + snap := snapshotAtLatestGeneration(status) + return snap.GetLocality() != "" && snap.GetDurableSnapshot() == nil +} + +// latestSnapshotIsPendingLocal reports whether the snapshot at the actor's +// latest generation is node-local but not yet captured: it has no locality and +// no durable copy. This is the shape a pause in progress has, and what tells a +// SUSPENDING pause from a SUSPENDING suspend, whose latest snapshot has a +// pending durable copy. +func latestSnapshotIsPendingLocal(status *ateapipb.ActorStatus) bool { + snap := snapshotAtLatestGeneration(status) + return snap != nil && snap.GetLocality() == "" && snap.GetDurableSnapshot() == nil +} + // clearLocalSnapshots clears Locality across all Snapshots in status.Snapshots // without dropping PENDING durable snapshots (so DeleteActor can still collect // an in-flight durable snapshot after releasing the worker). diff --git a/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go b/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go index 1a0a38d959..c6eed381cc 100644 --- a/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go +++ b/cmd/ateapi/internal/controlapi/snapshot_helpers_test.go @@ -199,6 +199,79 @@ func TestSnapshotPruningHelpers(t *testing.T) { }) } +// TestLatestSnapshotIsLocalOnly pins the shape a pause leaves behind: the +// latest snapshot is node-local with no durable copy. A durable latest +// snapshot, a pending upload of the local one, or a local snapshot that is +// not captured yet do not count; a RUNNING actor that kept its local snapshot +// across a resume does. +func TestLatestSnapshotIsLocalOnly(t *testing.T) { + local := func(gen int32) *ateapipb.Snapshot { + return newLocalSnapshot(gen, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "node-1") + } + durable := func(gen int32, st ateapipb.SnapshotStorageStatus) *ateapipb.Snapshot { + return newDurableSnapshot(gen, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "durable", "gs://b/durable", st) + } + uploading := local(2) + uploading.DurableSnapshot = &ateapipb.SnapshotStorage{Status: ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING, Object: &ateapipb.ObjectSnapshot{SnapshotUri: "gs://b/up"}} + + tests := []struct { + name string + status *ateapipb.ActorStatus + want bool + }{ + {"nil status", nil, false}, + {"no snapshots", &ateapipb.ActorStatus{}, false}, + {"local only above a durable one", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{durable(1, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), local(2)}}, true}, + {"local only alone", &ateapipb.ActorStatus{LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{local(1)}}, true}, + {"durable only", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{durable(2, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED)}}, false}, + {"local with an upload pending", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{uploading}}, false}, + {"local not yet captured", &ateapipb.ActorStatus{LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "")}}, false}, + {"older local only under a pause in progress", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{local(1), newLocalSnapshot(2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "")}}, false}, + {"latest generation dropped by a revert", &ateapipb.ActorStatus{LastAssignedGeneration: 3, Snapshots: []*ateapipb.Snapshot{local(2)}}, false}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := latestSnapshotIsLocalOnly(tc.status); got != tc.want { + t.Errorf("latestSnapshotIsLocalOnly = %t, want %t", got, tc.want) + } + }) + } +} + +// TestLatestSnapshotIsPendingLocal pins what tells a SUSPENDING pause from a +// SUSPENDING suspend: the latest snapshot is node-local with no locality yet +// and no durable copy. A running-origin suspend has a pending durable copy, a +// paused-origin suspend has both locality and a pending durable copy, and an +// actor with no snapshot at all has nothing pending. +func TestLatestSnapshotIsPendingLocal(t *testing.T) { + pendingLocal := newLocalSnapshot(2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "") + capturedLocal := newLocalSnapshot(2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "node-1") + pendingDurable := newDurableSnapshot(2, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "durable", "gs://b/d", ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) + uploadingLocal := newLocalSnapshot(2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local", "node-1") + uploadingLocal.DurableSnapshot = &ateapipb.SnapshotStorage{Status: ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING, Object: &ateapipb.ObjectSnapshot{SnapshotUri: "gs://b/up"}} + + tests := []struct { + name string + status *ateapipb.ActorStatus + want bool + }{ + {"nil status", nil, false}, + {"no snapshots", &ateapipb.ActorStatus{}, false}, + {"local not yet captured", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{pendingLocal}}, true}, + {"durable upload pending", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{pendingDurable}}, false}, + {"local with an upload pending", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{uploadingLocal}}, false}, + {"local only", &ateapipb.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapipb.Snapshot{capturedLocal}}, false}, + {"latest generation dropped by a revert", &ateapipb.ActorStatus{LastAssignedGeneration: 3, Snapshots: []*ateapipb.Snapshot{pendingLocal}}, false}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := latestSnapshotIsPendingLocal(tc.status); got != tc.want { + t.Errorf("latestSnapshotIsPendingLocal = %t, want %t", got, tc.want) + } + }) + } +} + func TestTagDurableSnapshotURI(t *testing.T) { if got := tagDurableSnapshotURI(nil, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_UNSPECIFIED); got != "" { t.Errorf("tagDurableSnapshotURI(nil) = %q, want empty", got) diff --git a/cmd/ateapi/internal/controlapi/tag.go b/cmd/ateapi/internal/controlapi/tag.go index 6c8789e7b8..c356290af1 100644 --- a/cmd/ateapi/internal/controlapi/tag.go +++ b/cmd/ateapi/internal/controlapi/tag.go @@ -88,7 +88,7 @@ func (s *RPCService) ListTags(ctx context.Context, req *ateapipb.ListTagsRequest if errs := apivalidation.ValidateListTagsRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } - page, err := s.impl.ListTags(ctx, req.GetAtespace(), store.ListOptions{PageSize: effectivePageSize(req.GetPageSize()), PageToken: req.GetPageToken()}) + page, err := s.impl.ListTags(ctx, req.GetAtespace(), toStoreListOptions(req.GetOptions())) if err != nil { return nil, mapListError(fmt.Errorf("while listing tags: %w", err)) } diff --git a/cmd/ateapi/internal/controlapi/tag_test.go b/cmd/ateapi/internal/controlapi/tag_test.go index 73cca12212..7af624d22d 100644 --- a/cmd/ateapi/internal/controlapi/tag_test.go +++ b/cmd/ateapi/internal/controlapi/tag_test.go @@ -63,7 +63,7 @@ func TestListTags(t *testing.T) { if resp.GetNextPageToken() == "" { return got } - req.PageToken = resp.GetNextPageToken() + req.Options = &ateapipb.ListOptions{PageSize: req.GetOptions().GetPageSize(), PageToken: resp.GetNextPageToken()} } } @@ -84,7 +84,7 @@ func TestListTags(t *testing.T) { }, { name: "one tag per page", - req: &ateapipb.ListTagsRequest{PageSize: 1}, + req: &ateapipb.ListTagsRequest{Options: &ateapipb.ListOptions{PageSize: 1}}, want: []string{otherAtespace + "/v1", testAtespace + "/v1", testAtespace + "/v2"}, }, { @@ -101,7 +101,7 @@ func TestListTags(t *testing.T) { }) } - _, err := svc.ListTags(ctx, &ateapipb.ListTagsRequest{PageToken: "not-a-token"}) + _, err := svc.ListTags(ctx, &ateapipb.ListTagsRequest{Options: &ateapipb.ListOptions{PageToken: "not-a-token"}}) if code := apierror.Code(err); code != codes.InvalidArgument { t.Errorf("ListTags(bad page_token) error = %v (code %v), want code InvalidArgument", err, code) } diff --git a/cmd/ateapi/internal/controlapi/template_reconciler.go b/cmd/ateapi/internal/controlapi/template_reconciler.go index a1857d0f4d..2a00e3439b 100644 --- a/cmd/ateapi/internal/controlapi/template_reconciler.go +++ b/cmd/ateapi/internal/controlapi/template_reconciler.go @@ -254,7 +254,7 @@ func (r *ActorTemplateReconciler) reconcileOne(ctx context.Context, ref resource return 0, r.fail(ctx, tmpl, reasonGoldenActorCrashed, "golden actor crashed before its snapshot was taken") case ateapipb.ActorState_ACTOR_STATE_RUNNING: - takeAt := goldenSnapshotStatus.GetTakeGoldenSnapshotAt() + takeAt := goldenSnapshotStatus.GetSnapshotTime() if takeAt == nil { // Resumed, but the deadline write was lost (a replica died // after ResumeActor). The elapsed warmup is unknowable, so @@ -262,7 +262,7 @@ func (r *ActorTemplateReconciler) reconcileOne(ctx context.Context, ref resource slog.WarnContext(ctx, "Golden actor running without a snapshot deadline; restarting warmup", slog.String("ActorTemplate", ref.String())) deadline := time.Now().Add(goldenSnapshotWarmupFor(tmpl.GetContainers())) if tmpl, err = r.checkpoint(ctx, tmpl, func(snapshotStatus *ateapipb.GoldenSnapshotStatus) { - snapshotStatus.TakeGoldenSnapshotAt = timestamppb.New(deadline) + snapshotStatus.SnapshotTime = timestamppb.New(deadline) }); err != nil { return 0, err } @@ -303,12 +303,12 @@ func (r *ActorTemplateReconciler) reconcileOne(ctx context.Context, ref resource } deadline := time.Now().Add(goldenSnapshotWarmupFor(tmpl.GetContainers())) if tmpl, err = r.checkpoint(ctx, tmpl, func(snapshotStatus *ateapipb.GoldenSnapshotStatus) { - snapshotStatus.TakeGoldenSnapshotAt = timestamppb.New(deadline) + snapshotStatus.SnapshotTime = timestamppb.New(deadline) }); err != nil { return 0, err } - case ateapipb.ActorState_ACTOR_STATE_DELETING, ateapipb.ActorState_ACTOR_STATE_PAUSED, ateapipb.ActorState_ACTOR_STATE_PAUSING, ateapipb.ActorState_ACTOR_STATE_REVERTING: - // Nothing in the golden flow deletes, pauses, or reverts the actor before + case ateapipb.ActorState_ACTOR_STATE_DELETING, ateapipb.ActorState_ACTOR_STATE_REVERTING: + // Nothing in the golden flow deletes or reverts the actor before // the snapshot is taken; someone else interfered. return 0, r.fail(ctx, tmpl, reasonUnexpectedState, fmt.Sprintf("golden actor in unexpected state %v", state)) diff --git a/cmd/ateapi/internal/controlapi/template_reconciler_test.go b/cmd/ateapi/internal/controlapi/template_reconciler_test.go index 531248b099..d8a70e47f5 100644 --- a/cmd/ateapi/internal/controlapi/template_reconciler_test.go +++ b/cmd/ateapi/internal/controlapi/template_reconciler_test.go @@ -324,7 +324,7 @@ func testTemplate(opts ...func(*ateapipb.ActorTemplate)) *ateapipb.ActorTemplate tmpl := &ateapipb.ActorTemplate{ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: testTemplateName, Uid: testTemplateUID, Version: 1}, Containers: []*ateapipb.Container{ - {Name: "main", Image: "img", WakeupProbe: &ateapipb.ContainerWakeupProbe{}}, + {Name: "main", Image: "img", WakeupProbe: &ateapipb.WakeupProbe{}}, }, SnapshotConfig: &ateapipb.SnapshotConfig{ GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_ENABLED}, @@ -358,7 +358,7 @@ func seededGoldenStatus(tmpl *ateapipb.ActorTemplate) *ateapipb.GoldenSnapshotSt func withSnapshotDeadline(at time.Time) func(*ateapipb.ActorTemplate) { return func(tmpl *ateapipb.ActorTemplate) { - seededGoldenStatus(tmpl).TakeGoldenSnapshotAt = timestamppb.New(at) + seededGoldenStatus(tmpl).SnapshotTime = timestamppb.New(at) } } @@ -379,7 +379,7 @@ func newTestTemplateReconciler(persistence templateReconcilerStore, control gold } func TestGoldenSnapshotWarmupFor(t *testing.T) { - probe := &ateapipb.ContainerWakeupProbe{} + probe := &ateapipb.WakeupProbe{} tests := []struct { name string containers []*ateapipb.Container @@ -418,7 +418,7 @@ func TestReconcileOne(t *testing.T) { wantMessage string // wantTag indicates that the golden tag should be recorded. wantTag bool - // wantDeadline asserts whether take_golden_snapshot_at is set. + // wantDeadline asserts whether snapshot_time is set. wantDeadline bool wantCreates int wantResumes int @@ -632,8 +632,8 @@ func TestReconcileOne(t *testing.T) { if got := snapshotStatus.GetGoldenTag(); (got != nil) != tt.wantTag { t.Errorf("stored golden tag = %v, want tag %v", got, tt.wantTag) } - if tt.wantDeadline && snapshotStatus.GetTakeGoldenSnapshotAt() == nil { - t.Error("stored take_golden_snapshot_at is nil, want set") + if tt.wantDeadline && snapshotStatus.GetSnapshotTime() == nil { + t.Error("stored snapshot_time is nil, want set") } }) } @@ -675,8 +675,8 @@ func TestReconcileOne_GoldenActorRequests(t *testing.T) { if got := control.suspendReqs[0].GetActor().GetName(); got != testTemplateUID { t.Errorf("suspended actor = %q, want %q", got, testTemplateUID) } - if st.storedStatus(t, testTemplateRef).GetGoldenSnapshotStatus().GetTakeGoldenSnapshotAt() == nil { - t.Error("stored take_golden_snapshot_at is nil, want set") + if st.storedStatus(t, testTemplateRef).GetGoldenSnapshotStatus().GetSnapshotTime() == nil { + t.Error("stored snapshot_time is nil, want set") } } @@ -697,13 +697,13 @@ func TestCheckpoint_TerminalStateErrors(t *testing.T) { // Checkpoint against a template a concurrent writer already // drove to a terminal state. _, err := r.checkpoint(ctx, testTemplate(seed.opt), func(snapshotStatus *ateapipb.GoldenSnapshotStatus) { - snapshotStatus.TakeGoldenSnapshotAt = timestamppb.New(time.Now()) + snapshotStatus.SnapshotTime = timestamppb.New(time.Now()) }) if err == nil { t.Fatal("checkpoint succeeded, want error for terminal template") } - if st.storedStatus(t, testTemplateRef).GetGoldenSnapshotStatus().GetTakeGoldenSnapshotAt() != nil { - t.Error("take_golden_snapshot_at set, want store unchanged") + if st.storedStatus(t, testTemplateRef).GetGoldenSnapshotStatus().GetSnapshotTime() != nil { + t.Error("snapshot_time set, want store unchanged") } }) } @@ -806,13 +806,13 @@ func TestCheckpoint_StaleObservationConflicts(t *testing.T) { } _, err := r.checkpoint(ctx, testTemplate(), func(snapshotStatus *ateapipb.GoldenSnapshotStatus) { - snapshotStatus.TakeGoldenSnapshotAt = timestamppb.New(time.Now()) + snapshotStatus.SnapshotTime = timestamppb.New(time.Now()) }) if !errors.Is(err, store.ErrVersionConflict) { t.Fatalf("checkpoint error = %v, want ErrVersionConflict", err) } - if st.storedStatus(t, testTemplateRef).GetGoldenSnapshotStatus().GetTakeGoldenSnapshotAt() != nil { - t.Error("take_golden_snapshot_at set, want store unchanged") + if st.storedStatus(t, testTemplateRef).GetGoldenSnapshotStatus().GetSnapshotTime() != nil { + t.Error("snapshot_time set, want store unchanged") } } diff --git a/cmd/ateapi/internal/controlapi/worker.go b/cmd/ateapi/internal/controlapi/worker.go index 46d6ab0393..247376b4b5 100644 --- a/cmd/ateapi/internal/controlapi/worker.go +++ b/cmd/ateapi/internal/controlapi/worker.go @@ -48,8 +48,7 @@ func (s *RPCService) ListWorkerActorAssignments(ctx context.Context, req *ateapi return nil, fmt.Errorf("while fetching worker %s: %w", name, err) } - page, err := s.impl.ListWorkerAssignments(ctx, name, - store.ListOptions{PageSize: effectivePageSize(req.GetPageSize()), PageToken: req.GetPageToken()}) + page, err := s.impl.ListWorkerAssignments(ctx, name, toStoreListOptions(req.GetOptions())) if err != nil { return nil, mapListError(fmt.Errorf("while listing the assignments of worker %s: %w", name, err)) } @@ -64,7 +63,7 @@ func (s *RPCService) ListWorkers(ctx context.Context, req *ateapipb.ListWorkersR return nil, resources.ToAPIError(errs) } - page, err := s.impl.ListWorkers(ctx, store.ListOptions{PageSize: effectivePageSize(req.GetPageSize()), PageToken: req.GetPageToken()}) + page, err := s.impl.ListWorkers(ctx, toStoreListOptions(req.GetOptions())) if err != nil { return nil, mapListError(fmt.Errorf("while listing workers in db: %w", err)) } diff --git a/cmd/ateapi/internal/controlapi/workflow_delete.go b/cmd/ateapi/internal/controlapi/workflow_delete.go index 90ce7d507b..e97af85ba5 100644 --- a/cmd/ateapi/internal/controlapi/workflow_delete.go +++ b/cmd/ateapi/internal/controlapi/workflow_delete.go @@ -179,12 +179,10 @@ func (w *ActorWorkflow) ensureAteletTerminated(ctx context.Context, actorRef res if vol.GetStorageVolumeId() != "" { workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{ Name: vol.GetName(), - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: vol.GetStorageVolumeId(), - VolumeType: vol.GetVolumeType(), - VolumeContext: vol.GetVolumeContext(), - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: vol.GetStorageVolumeId(), + VolumeType: vol.GetVolumeType(), + VolumeContext: vol.GetVolumeContext(), }, }) } diff --git a/cmd/ateapi/internal/controlapi/workflow_delete_test.go b/cmd/ateapi/internal/controlapi/workflow_delete_test.go index 9192cdfe55..6c1d7f5a0a 100644 --- a/cmd/ateapi/internal/controlapi/workflow_delete_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_delete_test.go @@ -65,13 +65,6 @@ func TestDeleteActorWorkflow_ExecutionPaths(t *testing.T) { wantErr: true, wantCode: codes.FailedPrecondition, }, - { - name: "delete paused actor rejected when not any_state", - seedState: ateapipb.ActorState_ACTOR_STATE_PAUSED, - anyState: false, - wantErr: true, - wantCode: codes.FailedPrecondition, - }, { name: "any_state delete suspended actor succeeds", seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, @@ -84,12 +77,6 @@ func TestDeleteActorWorkflow_ExecutionPaths(t *testing.T) { anyState: true, wantErr: false, }, - { - name: "any_state delete paused actor succeeds", - seedState: ateapipb.ActorState_ACTOR_STATE_PAUSED, - anyState: true, - wantErr: false, - }, { name: "any_state delete crashed actor succeeds", seedState: ateapipb.ActorState_ACTOR_STATE_CRASHED, @@ -169,8 +156,6 @@ func TestEnsureMarkedDeleting_StateMatrix(t *testing.T) { ateapipb.ActorState_ACTOR_STATE_RUNNING: true, ateapipb.ActorState_ACTOR_STATE_RESUMING: true, ateapipb.ActorState_ACTOR_STATE_SUSPENDING: true, - ateapipb.ActorState_ACTOR_STATE_PAUSING: true, - ateapipb.ActorState_ACTOR_STATE_PAUSED: true, ateapipb.ActorState_ACTOR_STATE_SUSPENDED: true, ateapipb.ActorState_ACTOR_STATE_CRASHED: true, ateapipb.ActorState_ACTOR_STATE_REVERTING: true, diff --git a/cmd/ateapi/internal/controlapi/workflow_pause.go b/cmd/ateapi/internal/controlapi/workflow_pause.go index f64dd2c920..e8370a402b 100644 --- a/cmd/ateapi/internal/controlapi/workflow_pause.go +++ b/cmd/ateapi/internal/controlapi/workflow_pause.go @@ -60,9 +60,10 @@ func (w *ActorWorkflow) PauseActor(ctx context.Context, actorRef resources.Actor if err != nil { return nil, err } - if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_PAUSED { - // Fully paused already: FinalizePaused commits PAUSED and the cleared - // worker assignment in a single update, so there is nothing left to do. + if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDED && latestSnapshotIsLocalOnly(actor.GetStatus()) { + // Fully paused already: FinalizePaused commits SUSPENDED, the local + // snapshot's locality, and the cleared worker assignment in a single + // update, so there is nothing left to do. // This success reports no pool, and cannot: the previous attempt // released the worker, so the record names none (#957). return actor, nil @@ -108,19 +109,22 @@ func (w *ActorWorkflow) loadActorForPause(ctx context.Context, actorRef resource return actor, actorTemplate, nil } -// ensureMarkedPausing transitions a RUNNING actor to PAUSING, minting the -// local snapshot name. Skips when a previous attempt already marked the -// actor; the persisted name then stays authoritative for the rest of the +// ensureMarkedPausing transitions a RUNNING actor to SUSPENDING, minting the +// local snapshot name. A pause in progress is a SUSPENDING actor whose latest +// generation is a pending node-local snapshot; a suspend in progress has a +// pending durable one instead. Skips when a previous attempt already marked +// the actor; the persisted name then stays authoritative for the rest of the // workflow. func (w *ActorWorkflow) ensureMarkedPausing(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate) (_ *ateapipb.Actor, err error) { ctx, done := stepSpan(ctx, "MarkPausing") defer func() { err = done(err) }() - if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_PAUSING { - markSkipped(ctx, "actor already PAUSING") + if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDING && latestSnapshotIsPendingLocal(actor.GetStatus()) { + markSkipped(ctx, "actor already SUSPENDING for a pause") return actor, nil } - // The pause edge only exists from RUNNING. + // The pause edge only exists from RUNNING. A SUSPENDING actor that is not + // pausing is suspending, and that suspend owns it. if actor.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING { return nil, apierror.FailedPrecondition("MarkPausing prerequisite not met for Actor: %s (got: %v, want %s)", actorRef, actor.GetStatus().GetState(), ateapipb.ActorState_ACTOR_STATE_RUNNING) } @@ -132,7 +136,7 @@ func (w *ActorWorkflow) ensureMarkedPausing(ctx context.Context, actorRef resour snapshotUUID := resources.NewSnapshotName() storedActor, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error { - toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_PAUSING + toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_SUSPENDING // Increment last_assigned_generation for the new Pause request. gen := toUpdate.Status.LastAssignedGeneration + 1 toUpdate.Status.LastAssignedGeneration = gen @@ -168,7 +172,7 @@ func (w *ActorWorkflow) ensureAteletPaused(ctx context.Context, actorRef resourc assignment := actor.GetStatus().GetWorkerAssignment() if assignment == nil { - // Missing active worker pod reference in PAUSING state indicates corrupted store state. + // Missing active worker pod reference mid-pause indicates corrupted store state. return "", crashActorAndFail(ctx, w.store, actorRef, ateattr.OperationPause, crashMessageWorkerAssignmentMissing) } @@ -194,16 +198,16 @@ func (w *ActorWorkflow) ensureAteletPaused(ctx context.Context, actorRef resourc ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(), ActorTemplateName: actor.GetActorTemplate().GetName(), Spec: workloadSpec, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - Config: &ateletpb.CheckpointRequest_LocalConfig{ - LocalConfig: &ateletpb.LocalCheckpointConfiguration{ - SnapshotName: inProgressSnap.GetUuid(), - }, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: inProgressSnap.GetUuid(), + Fidelity: fidelityToAtelet(actorTemplate.GetSnapshotConfig().GetPreferredFidelity()), + }, + UploadConfig: &ateletpb.UploadConfig{ + Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED, }, - Fidelity: fidelityToAtelet(actorTemplate.GetSnapshotConfig().GetPreferredFidelity()), ActorUid: actor.GetMetadata().Uid, } - wireFidelity = ateattr.SnapshotFidelityValue(req.Fidelity) + wireFidelity = ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()) if _, err = client.Checkpoint(ctx, req); err != nil { return wireFidelity, handleAteletError(ctx, w.store, actorRef, ateattr.OperationPause, "Checkpoint", false, err) @@ -213,7 +217,9 @@ func (w *ActorWorkflow) ensureAteletPaused(ctx context.Context, actorRef resourc // ensurePausedFinalized releases the actor's worker (only when it is still // owned by this actor), records where the local snapshot lives, and commits -// PAUSED with the assignment cleared in a single update. When the worker's +// SUSPENDED with the assignment cleared in a single update. The actor keeps +// its assigned node: a paused actor is a SUSPENDED actor whose latest +// snapshot is node-local, and it resumes on that node. When the worker's // node name was lost or the latest local snapshot entry is missing, the actor // can never be resumed from local state, so it commits CRASHED instead and // fails the step with FailedPrecondition. It re-reads the actor first so an @@ -234,8 +240,8 @@ func (w *ActorWorkflow) ensurePausedFinalized(ctx context.Context, actorRef reso if latestActor.GetStatus().GetAssignedNode() == "" { // Without a node name we cannot record where the local snapshot lives, // so the actor can never be resumed (the scheduler would search for a - // worker on an unknown node forever). Crash it instead of leaving it - // stuck in PAUSED. + // worker on an unknown node forever). Crash it instead of committing + // SUSPENDED with a snapshot nothing can find. return nil, crashActorAndFail(ctx, w.store, actorRef, ateattr.OperationPause, crashMessageLocalSnapshotNodeUnknown) } if snapshotAtLatestGeneration(latestActor.GetStatus()) == nil { @@ -267,7 +273,7 @@ func (w *ActorWorkflow) ensurePausedFinalized(ctx context.Context, actorRef reso // precondition pins the version checked above, so an out-of-band write // since then fails the update instead of being overwritten. storedActor, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(latestActor), func(toUpdate *ateapipb.Actor) error { - toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_PAUSED + toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_SUSPENDED // Record the node locality of the local checkpoint and clear older local checkpoints. snap := snapshotAtLatestGeneration(toUpdate.Status) snap.Locality = toUpdate.GetStatus().GetAssignedNode() diff --git a/cmd/ateapi/internal/controlapi/workflow_pause_test.go b/cmd/ateapi/internal/controlapi/workflow_pause_test.go index 1c182dd46b..b11cbf7ced 100644 --- a/cmd/ateapi/internal/controlapi/workflow_pause_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_pause_test.go @@ -32,7 +32,7 @@ import ( // is unknown. // // Current behavior: AssignedNode is left empty, and the actor is crashed -// instead of left PAUSED, since a local snapshot with an unknown node can +// instead of left paused, since a local snapshot with an unknown node can // never be safely resumed. The step fails with FailedPrecondition rather than // reporting the pause as done. func TestEnsurePausedFinalized_WorkerGone(t *testing.T) { @@ -46,7 +46,7 @@ func TestEnsurePausedFinalized_WorkerGone(t *testing.T) { actor := &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_PAUSING, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, WorkerAssignment: &ateapipb.WorkerAssignment{ WorkerNamespace: "default", WorkerPool: "pool1", @@ -156,7 +156,7 @@ func TestEnsurePausedFinalized_AlreadyCrashed(t *testing.T) { // TestEnsurePausedFinalized_KeepsCrashFromWorkerDelete verifies that a crash // the DeleteWorker workflow commits while ensurePausedFinalized is freeing the // worker is kept: the step's update is pinned to the actor version it checked, -// so it fails Aborted instead of committing PAUSED over the crash. +// so it fails Aborted instead of committing SUSPENDED over the crash. func TestEnsurePausedFinalized_KeepsCrashFromWorkerDelete(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() @@ -269,8 +269,11 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { t.Fatalf("ensurePausedFinalized: %v", err) } - if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSED { - t.Fatalf("state = %v, want PAUSED", got.GetStatus().GetState()) + if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Fatalf("state = %v, want SUSPENDED", got.GetStatus().GetState()) + } + if !latestSnapshotIsLocalOnly(got.GetStatus()) { + t.Errorf("snapshots = %v, want the latest one local only", got.GetStatus().GetSnapshots()) } localSnap := findLatestLocalSnapshot(got.GetStatus()) if localSnap.GetFidelity() != tc.want { @@ -292,7 +295,7 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { // TestEnsurePausedFinalized_MissingLocalSnapshotClearsAssignment verifies that // if the latest generation snapshot entry is missing when ensurePausedFinalized // runs, the actor transitions to CRASHED and clears WorkerAssignment after -// releasing the worker rather than getting stuck in PAUSING, and the step +// releasing the worker rather than getting stuck SUSPENDING, and the step // fails. func TestEnsurePausedFinalized_MissingLocalSnapshotClearsAssignment(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) @@ -305,7 +308,7 @@ func TestEnsurePausedFinalized_MissingLocalSnapshotClearsAssignment(t *testing.T created := storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_PAUSING, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: "node1", WorkerAssignment: &ateapipb.WorkerAssignment{ Worker: &ateapipb.ObjectRef{Name: workerName}, @@ -357,11 +360,13 @@ func TestEnsurePausedFinalized_MissingLocalSnapshotClearsAssignment(t *testing.T // TestPauseActorWorkflow_RejectedAndIdempotentPaths covers the two // short-circuit paths of the pause workflow: rejection of the pause edge for -// a non-RUNNING actor and the idempotent fast-forward for a PAUSED one. +// a non-RUNNING actor and the idempotent fast-forward for a paused one. Both +// are SUSPENDED: a node-local latest snapshot is what tells them apart. func TestPauseActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { tests := []struct { name string seedState ateapipb.ActorState + seedOpts []func(*ateapipb.Actor) // wantErr true means PauseActor must fail with FailedPrecondition. wantErr bool // wantState is the stored state after the call. @@ -376,11 +381,12 @@ func TestPauseActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { wantState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, }, { - // Pausing a PAUSED actor succeeds idempotently via IsComplete + // Pausing a paused actor succeeds idempotently via IsComplete // fast-forward without calling atelet. name: "already paused succeeds", - seedState: ateapipb.ActorState_ACTOR_STATE_PAUSED, - wantState: ateapipb.ActorState_ACTOR_STATE_PAUSED, + seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, + seedOpts: []func(*ateapipb.Actor){withPausedShape("node-1")}, + wantState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, }, } for _, tc := range tests { @@ -390,7 +396,7 @@ func TestPauseActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { defer cleanup() w := newTestActorWorkflow(t, st, "ns", "tmpl1") - seedWorkflowActor(t, ctx, st, resources.ActorRef{Atespace: "team-a", Name: "id1"}, "ns", "tmpl1", tc.seedState) + seedWorkflowActor(t, ctx, st, resources.ActorRef{Atespace: "team-a", Name: "id1"}, "ns", "tmpl1", tc.seedState, tc.seedOpts...) actor, err := w.PauseActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}) if tc.wantErr { @@ -418,14 +424,15 @@ func TestPauseActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { } // TestEnsureMarkedPausing_StateMatrix verifies the pause edge's state gating -// against every actor state: RUNNING takes the edge, PAUSING skips (a -// previous attempt already marked the actor), everything else is rejected -// with FailedPrecondition. PAUSED is rejected here because the orchestrator -// early-returns before this step for a fully paused actor. +// against every actor state: RUNNING takes the edge, everything else is +// rejected with FailedPrecondition. SUSPENDED is rejected whatever its +// snapshots: the orchestrator early-returns before this step for a paused +// actor. SUSPENDING without a pending node-local snapshot is a suspend in +// progress, which owns the actor; a pause in progress is covered by +// TestEnsureMarkedPausing_ReentrySkipsPauseInProgress. func TestEnsureMarkedPausing_StateMatrix(t *testing.T) { allowed := map[ateapipb.ActorState]bool{ ateapipb.ActorState_ACTOR_STATE_RUNNING: true, - ateapipb.ActorState_ACTOR_STATE_PAUSING: true, // skipped, not re-marked } for _, seedState := range allActorStates { @@ -442,12 +449,57 @@ func TestEnsureMarkedPausing_StateMatrix(t *testing.T) { marked, err := w.ensureMarkedPausing(ctx, actorRef, actor, &ateapipb.ActorTemplate{}) assertPrerequisiteResult(t, seedState, err, allowed[seedState]) - if err == nil && marked.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSING { - t.Errorf("state %v: ensureMarkedPausing returned actor in %v, want PAUSING", seedState, marked.GetStatus().GetState()) + if err == nil && marked.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { + t.Errorf("state %v: ensureMarkedPausing returned actor in %v, want SUSPENDING", seedState, marked.GetStatus().GetState()) } } } +// TestEnsureMarkedPausing_ReentrySkipsPauseInProgress verifies a re-entered +// pause recognizes its own mark, a SUSPENDING actor whose latest generation +// is a pending node-local snapshot, and skips without minting another +// snapshot, while a running-origin suspend in progress (pending durable +// snapshot) is rejected. +func TestEnsureMarkedPausing_ReentrySkipsPauseInProgress(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + w := &ActorWorkflow{store: persistence} + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + + pausing := &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING}, + } + withPauseInProgress()(pausing) + created := storetest.MustCreateActor(t, ctx, persistence, pausing) + marked, err := w.ensureMarkedPausing(ctx, actorRef, created, &ateapipb.ActorTemplate{}) + if err != nil { + t.Fatalf("ensureMarkedPausing on a pause in progress: %v", err) + } + if got := marked.GetStatus().GetLastAssignedGeneration(); got != 1 { + t.Errorf("last_assigned_generation = %d, want 1 (no new snapshot minted on re-entry)", got) + } + if got := len(marked.GetStatus().GetSnapshots()); got != 1 { + t.Errorf("len(snapshots) = %d, want 1", got) + } + + suspending := &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: "id2"}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + LastAssignedGeneration: 1, + Snapshots: []*ateapipb.Snapshot{ + newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap", "gs://snapshots/s", ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + }, + } + created = storetest.MustCreateActor(t, ctx, persistence, suspending) + _, err = w.ensureMarkedPausing(ctx, resources.ActorRef{Atespace: actorRef.Atespace, Name: "id2"}, created, &ateapipb.ActorTemplate{}) + if got := apierror.Code(err); got != codes.FailedPrecondition { + t.Errorf("ensureMarkedPausing on a suspend in progress: code = %v, want FailedPrecondition (err: %v)", got, err) + } +} + func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { tests := []struct { name string @@ -477,7 +529,7 @@ func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { actor := &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_PAUSING, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, WorkerAssignment: &ateapipb.WorkerAssignment{ WorkerNamespace: "worker-ns", WorkerPool: "pool", @@ -499,8 +551,8 @@ func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { if err != nil { t.Fatalf("GetActor: %v", err) } - if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSING { - t.Errorf("state = %v, want unchanged PAUSING", stored.GetStatus().GetState()) + if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { + t.Errorf("state = %v, want unchanged SUSPENDING", stored.GetStatus().GetState()) } if gotSnap := snapshotAtLatestGeneration(stored.GetStatus()); gotSnap.GetUuid() != "actor-1-never-written" { t.Errorf("in-progress local snapshot uuid = %q, want preserved for debugging", gotSnap.GetUuid()) @@ -514,7 +566,7 @@ func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { } // TestPauseActor_CrashesWhenPausingActorMissingWorkerPod verifies that a -// PAUSING actor with no worker pod recorded is moved to CRASHED by +// pausing actor with no worker pod recorded is moved to CRASHED by // ensureAteletPaused's corrupted-assignment check and the pause fails with // FailedPrecondition. func TestPauseActor_CrashesWhenPausingActorMissingWorkerPod(t *testing.T) { @@ -523,7 +575,7 @@ func TestPauseActor_CrashesWhenPausingActorMissingWorkerPod(t *testing.T) { defer cleanup() w := newTestActorWorkflow(t, st, "ns", "tmpl1") - seedWorkflowActor(t, ctx, st, resources.ActorRef{Atespace: "team-a", Name: "id1"}, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_PAUSING) + seedWorkflowActor(t, ctx, st, resources.ActorRef{Atespace: "team-a", Name: "id1"}, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDING, withPauseInProgress()) _, err := w.PauseActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}) if got := apierror.Code(err); got != codes.FailedPrecondition { @@ -580,7 +632,7 @@ func TestEnsureAteletPaused_ForwardsPreferredFidelity(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, ActorTemplate: &ateapipb.ObjectRef{Atespace: "ns", Name: "tmpl1"}, Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_PAUSING, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: "node-1", WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, @@ -607,11 +659,14 @@ func TestEnsureAteletPaused_ForwardsPreferredFidelity(t *testing.T) { if req == nil { t.Fatal("expected Checkpoint RPC to be called") } - if got := req.GetFidelity(); got != tc.want { - t.Errorf("CheckpointRequest.Fidelity = %v, want %v", got, tc.want) + if got := req.GetSnapshot().GetFidelity(); got != tc.want { + t.Errorf("CheckpointRequest.Snapshot.Fidelity = %v, want %v", got, tc.want) + } + if got := req.GetSnapshot().GetSnapshotUuid(); got != "local-snap-1" { + t.Errorf("CheckpointRequest.Snapshot.SnapshotUuid = %q, want %q", got, "local-snap-1") } - if got := req.GetLocalConfig().GetSnapshotName(); got != "local-snap-1" { - t.Errorf("CheckpointRequest.LocalConfig.SnapshotName = %q, want %q", got, "local-snap-1") + if got := req.GetUploadConfig().GetMode(); got != ateletpb.UploadMode_UPLOAD_MODE_DEFERRED { + t.Errorf("CheckpointRequest.UploadConfig.Mode = %v, want UPLOAD_MODE_DEFERRED", got) } }) } diff --git a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go index c25d81d086..f6705c7e40 100644 --- a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go @@ -301,7 +301,6 @@ func TestReconcileAssignments_ReleasesByActorState(t *testing.T) { // touching the Actor. {name: "crashed unassigned is released", state: ateapipb.ActorState_ACTOR_STATE_CRASHED, pointsAt: unassigned, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantRelease: true}, {name: "suspended unassigned is released", state: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, pointsAt: unassigned, wantState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, wantRelease: true}, - {name: "paused unassigned is released", state: ateapipb.ActorState_ACTOR_STATE_PAUSED, pointsAt: unassigned, wantState: ateapipb.ActorState_ACTOR_STATE_PAUSED, wantRelease: true}, {name: "running elsewhere is released", state: ateapipb.ActorState_ACTOR_STATE_RUNNING, pointsAt: otherWorker, wantState: ateapipb.ActorState_ACTOR_STATE_RUNNING, wantRelease: true}, {name: "suspended elsewhere is released", state: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, pointsAt: otherWorker, wantState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, wantRelease: true}, {name: "resuming crashes", state: ateapipb.ActorState_ACTOR_STATE_RESUMING, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantRelease: true}, diff --git a/cmd/ateapi/internal/controlapi/workflow_resume.go b/cmd/ateapi/internal/controlapi/workflow_resume.go index 3ee70ac6c3..bb42bc03a0 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume.go @@ -38,6 +38,7 @@ import ( // resumeSnapshotSource is the boot source resolved once by loadActorForResume // and passed by value to the restore step — never mutated after resolution. type resumeSnapshotSource struct { + SnapshotUUID string // SnapshotURI is the storage location of the durable snapshot to restore // from: the actor's latest snapshot, including a tag borrowed at creation. // Zero means cold boot from the spec (unless the actor holds a local @@ -177,6 +178,7 @@ func (w *ActorWorkflow) loadActorForResume(ctx context.Context, actorRef resourc if src.SnapshotURI, err = resources.ParseSnapshotURI(st.GetObject().GetSnapshotUri()); err != nil { return nil, nil, src, apierror.DataLoss("Actor %s external snapshot: %v", actorRef, err) } + src.SnapshotUUID = snap.GetUuid() src.Fidelity = snap.GetFidelity() capturedUnder := snap.GetActorTemplateUid() src.TemplateReplaced = capturedUnder != "" && capturedUnder != actorTemplate.GetMetadata().GetUid() @@ -235,8 +237,9 @@ func (w *ActorWorkflow) ensureVolumesCreated(ctx context.Context, actorRef resou // A RESUMING actor was assigned by a previous attempt: its persisted // assignment is revalidated (worker still exists, not draining, still owned // by this actor UID, still eligible for the actor's constraints) and reused; -// a stale assignment crashes the actor. A SUSPENDED or PAUSED actor goes -// through scheduling, claiming the worker and persisting RESUMING with the +// a stale assignment crashes the actor. A SUSPENDED actor goes through +// scheduling (pinned to its assigned node when it holds a node-local +// snapshot), claiming the worker and persisting RESUMING with the // assignment. A version conflict there is retried under a bounded backoff // only after re-reading the actor and revalidating it can still be resumed — // the conflicting writer may have crashed, drained, or deleted it. @@ -252,9 +255,9 @@ func (w *ActorWorkflow) ensureWorkerAssigned(ctx context.Context, actorRef resou } markSkipped(ctx, "actor already RESUMING with a valid worker assignment") return actor, worker, nil - case ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ateapipb.ActorState_ACTOR_STATE_PAUSED: + case ateapipb.ActorState_ACTOR_STATE_SUSPENDED: default: - return nil, nil, apierror.FailedPrecondition("AssignWorker prerequisite not met for Actor: %s (got: %v, want %s or %s)", actorRef, actor.GetStatus().GetState(), ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ateapipb.ActorState_ACTOR_STATE_PAUSED) + return nil, nil, apierror.FailedPrecondition("AssignWorker prerequisite not met for Actor: %s (got: %v, want %s)", actorRef, actor.GetStatus().GetState(), ateapipb.ActorState_ACTOR_STATE_SUSPENDED) } var assignedActor *ateapipb.Actor @@ -550,7 +553,7 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour return nil, nil, err } switch fresh.GetStatus().GetState() { - case ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ateapipb.ActorState_ACTOR_STATE_PAUSED: + case ateapipb.ActorState_ACTOR_STATE_SUSPENDED: slog.InfoContext(ctx, "Retrying assignment due to actor version conflict", slog.Any("actor", actorRef)) return fresh, nil, err default: @@ -699,6 +702,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou slog.InfoContext(ctx, "Actor has snapshot; Restoring from snapshot") tele.SnapshotKind = ateattr.SnapshotKindLocal + fidelity := fidelityToAtelet(localSnap.GetFidelity()) req := &ateletpb.RestoreRequest{ WorkerPodUid: assignment.GetWorkerPodUid(), Atespace: actor.GetMetadata().GetAtespace(), @@ -706,18 +710,17 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(), ActorTemplateName: actor.GetActorTemplate().GetName(), Spec: workloadSpec, - SandboxAssets: sandboxAssets, - ActorUid: actor.GetMetadata().Uid, - EgressGateway: egressGateway, - CpuMilli: cpuMilli, - MemoryBytes: memBytes, - } - req.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - req.Config = &ateletpb.RestoreRequest_LocalConfig{ - LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: localSnap.GetUuid()}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: localSnap.GetUuid(), + Fidelity: fidelity, + }, + SandboxAssets: sandboxAssets, + ActorUid: actor.GetMetadata().Uid, + EgressGateway: egressGateway, + CpuMilli: cpuMilli, + MemoryBytes: memBytes, } - req.Fidelity = fidelityToAtelet(localSnap.GetFidelity()) - tele.WireFidelity = ateattr.SnapshotFidelityValue(req.Fidelity) + tele.WireFidelity = ateattr.SnapshotFidelityValue(fidelity) if _, err = client.Restore(ctx, req); err != nil { return tele, handleAteletError(ctx, w.store, actorRef, ateattr.OperationResume, "Restore", false, err) @@ -738,13 +741,13 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(), ActorTemplateName: actor.GetActorTemplate().GetName(), Spec: workloadSpec, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - Config: &ateletpb.RestoreRequest_ExternalConfig{ - ExternalConfig: &ateletpb.ExternalRestoreConfiguration{ + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: src.SnapshotUUID, + Fidelity: scope, + Object: &ateletpb.ObjectStorage{ SnapshotUri: src.SnapshotURI.String(), }, }, - Fidelity: scope, SandboxAssets: sandboxAssets, ActorUid: actor.GetMetadata().Uid, EgressGateway: egressGateway, diff --git a/cmd/ateapi/internal/controlapi/workflow_resume_test.go b/cmd/ateapi/internal/controlapi/workflow_resume_test.go index 0c4f00bfdc..48248f3aa7 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume_test.go @@ -547,8 +547,8 @@ func TestResumeActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { wantState ateapipb.ActorState }{ { - // The resume edge only exists from SUSPENDED, PAUSED, and - // RESUMING; a CRASHED actor is rejected by ensureWorkerAssigned + // The resume edge only exists from SUSPENDED and RESUMING; a + // CRASHED actor is rejected by ensureWorkerAssigned // and its state is left untouched. name: "crashed rejected", seedState: ateapipb.ActorState_ACTOR_STATE_CRASHED, @@ -616,9 +616,9 @@ func TestResumeActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { } // TestEnsureWorkerAssigned_RejectsNonResumableStates verifies the resume -// edge's state gating: every state outside SUSPENDED, PAUSED, and RESUMING -// is rejected with FailedPrecondition before any dependency is touched. -// (SUSPENDED/PAUSED assignment and RESUMING recovery are exercised by the +// edge's state gating: every state outside SUSPENDED and RESUMING is +// rejected with FailedPrecondition before any dependency is touched. +// (SUSPENDED assignment and RESUMING recovery are exercised by the // assignment-attempt and worker-validation tests; RUNNING never reaches this // step because the orchestrator early-returns.) func TestEnsureWorkerAssigned_RejectsNonResumableStates(t *testing.T) { @@ -626,7 +626,7 @@ func TestEnsureWorkerAssigned_RejectsNonResumableStates(t *testing.T) { w := &ActorWorkflow{} for _, st := range allActorStates { switch st { - case ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ateapipb.ActorState_ACTOR_STATE_PAUSED, ateapipb.ActorState_ACTOR_STATE_RESUMING: + case ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ateapipb.ActorState_ACTOR_STATE_RESUMING: continue } actor := &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: st}, Metadata: &ateapipb.ResourceMetadata{Name: "id1", Uid: "actor-uid-1"}} @@ -1161,9 +1161,11 @@ func newWireCaptureWorkflow(t *testing.T, persistence store.Interface) (*ActorWo // golden) so a missing permutation is visible by scanning. func TestResumeActor_AteletWireRequest(t *testing.T) { const localSnapshotName = "pause-snap-1" + // durableSnapshotName is the uuid of every durable snapshot a row seeds. + const durableSnapshotName = "snap-1" const malformedURI = "not-a-valid-snapshot-uri" - actorURI := someActorSnapshotURI(t, testStorageLocation, "team-a", "snap-1") + actorURI := someActorSnapshotURI(t, testStorageLocation, "team-a", durableSnapshotName) goldenURI := someActorSnapshotURI(t, "gs://bucket/golden-root", "ate-golden", "golden-1") fullScope := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY @@ -1172,7 +1174,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { unspecScope := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED extSnap := func(uri string, fidelity ateapipb.SnapshotFidelity) *ateapipb.Snapshot { - return newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, fidelity, "", "snap-1", uri, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + return newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, fidelity, "", durableSnapshotName, uri, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) } localSnap := func(name string, fidelity ateapipb.SnapshotFidelity) *ateapipb.Snapshot { return newLocalSnapshot(1, fidelity, "", name, "node-1") @@ -1181,7 +1183,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { // actorSeed is the actor status a row persists before resuming. type actorSeed struct { // localSnapshot seeds a completed local snapshot (the pause checkpoint); - // a non-nil value also parks the actor PAUSED instead of SUSPENDED. + // a non-nil value also pins the actor to the snapshot's node. localSnapshot *ateapipb.Snapshot // externalSnapshot seeds a completed durable snapshot. externalSnapshot *ateapipb.Snapshot @@ -1204,12 +1206,11 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { // must fire instead of Restore; otherwise exactly one Restore carrying // these wire values. type restoreWant struct { - code codes.Code - run bool - checkpointType ateletpb.CheckpointType - // snapshotName is LocalConfig.SnapshotName; snapshotURI is - // ExternalConfig.SnapshotUri. Both are asserted on every Restore, so - // a row also pins that the other config is absent. + code codes.Code + run bool + // snapshotName is Snapshot.SnapshotUuid and snapshotURI is + // Snapshot.Object.SnapshotUri. Both are asserted on every Restore, so + // a local row also pins that no durable copy is sent. snapshotName string snapshotURI string scope ateletpb.SnapshotFidelity @@ -1230,9 +1231,9 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { actor: actorSeed{externalSnapshot: extSnap(goldenURI, fullScope), tmplUID: "current"}, tmpl: templateSeed{golden: extSnap(goldenURI, fullScope)}, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: goldenURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + snapshotName: durableSnapshotName, + snapshotURI: goldenURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }, { @@ -1256,9 +1257,9 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { name: "06 Full durable snapshot restores itself in Full", actor: actorSeed{externalSnapshot: extSnap(actorURI, fullScope)}, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }, { @@ -1268,9 +1269,9 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { tmplUID: "current", }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }, { @@ -1283,18 +1284,18 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { tmplUID: "mismatch", }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, }, { name: "08a Rootfs durable snapshot restores as Rootfs", actor: actorSeed{externalSnapshot: extSnap(actorURI, rootfsScope)}, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, }, }, { @@ -1306,18 +1307,18 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { tmplUID: "mismatch", }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, }, { name: "09 Data durable snapshot restores as Data", actor: actorSeed{externalSnapshot: extSnap(actorURI, dataScope)}, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, }, { @@ -1327,9 +1328,9 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { actor: actorSeed{externalSnapshot: extSnap(actorURI, dataScope)}, tmpl: templateSeed{golden: extSnap(goldenURI, fullScope)}, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, }, { @@ -1339,9 +1340,9 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { name: "11 unspecified durable fidelity goes out unspecified", actor: actorSeed{externalSnapshot: extSnap(actorURI, unspecScope)}, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - snapshotURI: actorURI, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, }, }, { @@ -1355,9 +1356,8 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { localSnapshot: localSnap(localSnapshotName, fullScope), }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - snapshotName: localSnapshotName, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + snapshotName: localSnapshotName, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }, { @@ -1370,9 +1370,8 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { tmplUID: "current", }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - snapshotName: localSnapshotName, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + snapshotName: localSnapshotName, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }, { @@ -1381,9 +1380,8 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { localSnapshot: localSnap(localSnapshotName, dataScope), }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - snapshotName: localSnapshotName, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + snapshotName: localSnapshotName, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, }, { @@ -1395,9 +1393,8 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { golden: extSnap(goldenURI, fullScope), }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - snapshotName: localSnapshotName, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + snapshotName: localSnapshotName, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, }, { @@ -1409,9 +1406,8 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { externalSnapshot: extSnap(actorURI, fullScope), }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - snapshotName: localSnapshotName, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + snapshotName: localSnapshotName, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, }, { @@ -1426,9 +1422,8 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { tmplUID: "mismatch", }, want: restoreWant{ - checkpointType: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - snapshotName: localSnapshotName, - scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + snapshotName: localSnapshotName, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }, { @@ -1498,14 +1493,10 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { t.Fatal("created template has no UID; the matching tmplUID case would be vacuous") } - // A pause checkpoint is what parks an actor PAUSED; without one a - // non-running actor resumes from SUSPENDED. - actorState := ateapipb.ActorState_ACTOR_STATE_SUSPENDED - if tt.actor.localSnapshot != nil { - actorState = ateapipb.ActorState_ACTOR_STATE_PAUSED - } + // Paused and suspended actors are both SUSPENDED; a pause + // checkpoint is what makes the resume local. actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} - seedWorkflowActor(t, ctx, persistence, actorRef, "ns", "tmpl1", actorState, func(a *ateapipb.Actor) { + seedWorkflowActor(t, ctx, persistence, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, func(a *ateapipb.Actor) { a.Status.WorkerAssignment = wireTestAssignment() uid := tt.actor.tmplUID if uid == "current" { @@ -1565,16 +1556,13 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { if !proto.Equal(restore.GetSandboxAssets(), wantSandboxAssets) { t.Errorf("restore SandboxAssets = %v, want %v", restore.GetSandboxAssets(), wantSandboxAssets) } - if got := restore.GetType(); got != tt.want.checkpointType { - t.Errorf("restore type = %v, want %v", got, tt.want.checkpointType) - } - if got := restore.GetLocalConfig().GetSnapshotName(); got != tt.want.snapshotName { - t.Errorf("LocalConfig.SnapshotName = %q, want %q", got, tt.want.snapshotName) + if got := restore.GetSnapshot().GetSnapshotUuid(); got != tt.want.snapshotName { + t.Errorf("Snapshot.SnapshotUuid = %q, want %q", got, tt.want.snapshotName) } - if got := restore.GetExternalConfig().GetSnapshotUri(); got != tt.want.snapshotURI { - t.Errorf("ExternalConfig.SnapshotUri = %q, want %q", got, tt.want.snapshotURI) + if got := restore.GetSnapshot().GetObject().GetSnapshotUri(); got != tt.want.snapshotURI { + t.Errorf("Snapshot.Object.SnapshotUri = %q, want %q", got, tt.want.snapshotURI) } - if got := restore.GetFidelity(); got != tt.want.scope { + if got := restore.GetSnapshot().GetFidelity(); got != tt.want.scope { t.Errorf("restore scope = %v, want %v", got, tt.want.scope) } }) diff --git a/cmd/ateapi/internal/controlapi/workflow_revert.go b/cmd/ateapi/internal/controlapi/workflow_revert.go index 3675131ccd..44d6d30263 100644 --- a/cmd/ateapi/internal/controlapi/workflow_revert.go +++ b/cmd/ateapi/internal/controlapi/workflow_revert.go @@ -30,13 +30,23 @@ import ( "go.opentelemetry.io/otel/attribute" ) -// revertableStates are the states a revert is accepted from. +// revertableStates are the states a revert is accepted from. A SUSPENDED +// actor is also accepted when it is paused, that is, when its latest snapshot +// is node-local: the revert discards that snapshot and returns the actor to +// its durable one. See isRevertable. var revertableStates = []ateapipb.ActorState{ ateapipb.ActorState_ACTOR_STATE_RUNNING, - ateapipb.ActorState_ACTOR_STATE_PAUSED, ateapipb.ActorState_ACTOR_STATE_CRASHED, } +// isRevertable reports whether a revert is accepted from the actor's current +// state and snapshot shape. +func isRevertable(actor *ateapipb.Actor) bool { + state := actor.GetStatus().GetState() + return slices.Contains(revertableStates, state) || + (state == ateapipb.ActorState_ACTOR_STATE_SUSPENDED && latestSnapshotIsLocalOnly(actor.GetStatus())) +} + // RevertActor executes the workflow to discard an actor's current execution and // return it to SUSPENDED, leaving its external snapshot untouched for a later // resume to restore from. It is the recovery verb: it accepts a CRASHED actor, @@ -138,8 +148,8 @@ func (w *ActorWorkflow) ensureMarkedReverting(ctx context.Context, actorRef reso markSkipped(ctx, "actor already REVERTING") return actor, nil } - if !slices.Contains(revertableStates, st) { - return nil, apierror.FailedPrecondition("Actor %s is not in a revertable state (got: %v, want one of %v)", actorRef, st, revertableStates) + if !isRevertable(actor) { + return nil, apierror.FailedPrecondition("Actor %s is not in a revertable state (got: %v, want one of %v, or %s with a node-local snapshot)", actorRef, st, revertableStates, ateapipb.ActorState_ACTOR_STATE_SUSPENDED) } storedActor, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error { diff --git a/cmd/ateapi/internal/controlapi/workflow_revert_test.go b/cmd/ateapi/internal/controlapi/workflow_revert_test.go index f6b78d9503..c7fe7ca6bb 100644 --- a/cmd/ateapi/internal/controlapi/workflow_revert_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_revert_test.go @@ -31,11 +31,12 @@ import ( // // SUSPENDED is the load-bearing rejection: it is revert's own destination, and // also what a suspend that won the race leaves behind, so accepting it would -// report success for the opposite of what was asked. +// report success for the opposite of what was asked. A SUSPENDED actor that +// holds a node-local snapshot is paused, not reverted, and is accepted; see +// TestEnsureMarkedReverting_AcceptsPaused. func TestEnsureMarkedReverting_StateMatrix(t *testing.T) { allowed := map[ateapipb.ActorState]bool{ ateapipb.ActorState_ACTOR_STATE_RUNNING: true, - ateapipb.ActorState_ACTOR_STATE_PAUSED: true, ateapipb.ActorState_ACTOR_STATE_CRASHED: true, // Re-entry: a previous attempt already marked it, so the step skips // rather than re-committing. @@ -65,6 +66,31 @@ func TestEnsureMarkedReverting_StateMatrix(t *testing.T) { } } +// TestEnsureMarkedReverting_AcceptsPaused verifies a paused actor (SUSPENDED +// whose latest snapshot is node-local) takes the revert edge: the revert +// discards that snapshot and returns the actor to its durable one. +func TestEnsureMarkedReverting_AcceptsPaused(t *testing.T) { + ctx := context.Background() + st, cleanup := storetest.SetupTestStore(t) + defer cleanup() + w := newTestActorWorkflow(t, st, "ns", "tmpl1") + + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + seedWorkflowActor(t, ctx, st, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, withPausedShape("node-1")) + actor, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("get seeded actor: %v", err) + } + + updated, err := w.ensureMarkedReverting(ctx, actorRef, actor) + if err != nil { + t.Fatalf("ensureMarkedReverting: %v", err) + } + if got := updated.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_REVERTING { + t.Errorf("state = %v, want REVERTING", got) + } +} + // TestRevertActor_ReturnsActorToItsSnapshot covers the whole workflow from each // accepted origin: the actor lands SUSPENDED holding the same external snapshot // it started with, and every pointer to the discarded execution is gone. @@ -73,14 +99,22 @@ func TestEnsureMarkedReverting_StateMatrix(t *testing.T) { // exercised here; it needs the atelet rig, and the unit workflow is built with // a nil dialer on purpose so an unexpected dial fails loudly. func TestRevertActor_ReturnsActorToItsSnapshot(t *testing.T) { - for _, seedState := range []ateapipb.ActorState{ - ateapipb.ActorState_ACTOR_STATE_RUNNING, - ateapipb.ActorState_ACTOR_STATE_PAUSED, - ateapipb.ActorState_ACTOR_STATE_CRASHED, + for _, tc := range []struct { + name string + seedState ateapipb.ActorState + // paused seeds the paused shape: the latest snapshot is node-local + // and the actor is pinned to its node. Only meaningful with + // SUSPENDED, which the revert otherwise rejects. + paused bool + }{ + {name: "RUNNING", seedState: ateapipb.ActorState_ACTOR_STATE_RUNNING}, + {name: "paused", seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, paused: true}, + {name: "CRASHED", seedState: ateapipb.ActorState_ACTOR_STATE_CRASHED}, // A revert that died partway is retried, not rejected. - ateapipb.ActorState_ACTOR_STATE_REVERTING, + {name: "REVERTING", seedState: ateapipb.ActorState_ACTOR_STATE_REVERTING}, } { - t.Run(seedState.String(), func(t *testing.T) { + seedState := tc.seedState + t.Run(tc.name, func(t *testing.T) { ctx := context.Background() st, cleanup := storetest.SetupTestStore(t) defer cleanup() @@ -94,7 +128,9 @@ func TestRevertActor_ReturnsActorToItsSnapshot(t *testing.T) { } // The snapshot revert must preserve, plus the node-local state a - // pause left behind, which it must not. + // pause left behind, which it must not. A paused actor's latest + // snapshot is the completed local one; the other origins also carry + // the half-written local snapshot an interrupted pause leaves. const keptURI = "gs://snapshots/team-a/actors/keep/snapshot" mustUpdateActorStatus(t, ctx, st, actor, func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 3 @@ -103,6 +139,9 @@ func TestRevertActor_ReturnsActorToItsSnapshot(t *testing.T) { newLocalSnapshot(2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-1", "node-1"), newLocalSnapshot(3, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-in-progress", ""), } + if tc.paused { + s.Snapshots[2].Locality = "node-1" + } }) reverted, err := w.RevertActor(ctx, actorRef) diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend.go b/cmd/ateapi/internal/controlapi/workflow_suspend.go index 99c89fea5c..ff2cce1231 100644 --- a/cmd/ateapi/internal/controlapi/workflow_suspend.go +++ b/cmd/ateapi/internal/controlapi/workflow_suspend.go @@ -62,11 +62,13 @@ func (w *ActorWorkflow) SuspendActor(ctx context.Context, actorRef resources.Act if err != nil { return nil, err } - if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDED && !latestSnapshotIsLocalOnly(actor.GetStatus()) { // Fully suspended already: FinalizeSuspended commits SUSPENDED and the // cleared worker assignment in a single update, so there is nothing - // left to do. This success reports no pool, and cannot: the previous - // attempt released the worker, so the record names none (#957). + // left to do. A SUSPENDED actor whose latest snapshot is node-local is + // paused, not suspended, and goes on to upload it. This success + // reports no pool, and cannot: the previous attempt released the + // worker, so the record names none (#957). return actor, nil } // Decided before marking: once SUSPENDING is committed, the loaded status @@ -115,7 +117,7 @@ func (w *ActorWorkflow) loadActorForSuspend(ctx context.Context, actorRef resour return actor, actorTemplate, nil } -// ensureMarkedSuspending transitions a RUNNING or PAUSED actor to +// ensureMarkedSuspending transitions a RUNNING or paused actor to // SUSPENDING, minting the in-progress snapshot location and recording the // actor version the snapshot will capture. Skips when a previous attempt // already marked the actor; the persisted location and source version then @@ -125,21 +127,25 @@ func (w *ActorWorkflow) ensureMarkedSuspending(ctx context.Context, actorRef res defer func() { err = done(err) }() if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDING { + if latestSnapshotIsPendingLocal(actor.GetStatus()) { + // The pause workflow owns the actor until it finishes; a suspend + // can run once the actor is paused. + return nil, apierror.FailedPrecondition("MarkSuspending prerequisite not met for Actor: %s (a pause is in progress)", actorRef) + } markSkipped(ctx, "actor already SUSPENDING") return actor, nil } - if got := actor.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_RUNNING && got != ateapipb.ActorState_ACTOR_STATE_PAUSED { - return nil, apierror.FailedPrecondition("MarkSuspending prerequisite not met for Actor: %s (got: %v, want %s or %s)", actorRef, got, ateapipb.ActorState_ACTOR_STATE_RUNNING, ateapipb.ActorState_ACTOR_STATE_PAUSED) + state := actor.GetStatus().GetState() + paused := isPausedOriginSuspend(actor) + if state != ateapipb.ActorState_ACTOR_STATE_RUNNING && !paused { + return nil, apierror.FailedPrecondition("MarkSuspending prerequisite not met for Actor: %s (got: %v, want %s or %s with a node-local snapshot)", actorRef, state, ateapipb.ActorState_ACTOR_STATE_RUNNING, ateapipb.ActorState_ACTOR_STATE_SUSPENDED) } // A paused actor's durable snapshot is an upload of its local checkpoint, // so it keeps that checkpoint's name; a running actor's is a new one. var snapshotName string - if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_PAUSED { - localSnap := findLatestLocalSnapshot(actor.GetStatus()) - if localSnap == nil { - return nil, crashActorAndFail(ctx, w.store, actorRef, ateattr.OperationSuspend, crashMessageLocalSnapshotMissing) - } - snapshotName = localSnap.GetUuid() + if paused { + // paused implies a snapshot with a locality, so this is never nil. + snapshotName = findLatestLocalSnapshot(actor.GetStatus()).GetUuid() } else { snapshotName = resources.NewSnapshotName() } @@ -151,13 +157,12 @@ func (w *ActorWorkflow) ensureMarkedSuspending(ctx context.Context, actorRef res return nil, err } storedActor, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error { - wasPaused := toUpdate.Status.State == ateapipb.ActorState_ACTOR_STATE_PAUSED toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_SUSPENDING - if wasPaused { + if paused { // Promote the current generation's local snapshot to a pending durable snapshot. snap := findLatestLocalSnapshot(toUpdate.Status) if snap == nil { - return fmt.Errorf("actor %s is PAUSED but has no completed local snapshot", actorRef) + return fmt.Errorf("actor %s is paused but has no completed local snapshot", actorRef) } snap.ActorTemplateUid = actorTemplate.GetMetadata().GetUid() snap.Fidelity = preferredFidelity(actorRef.Atespace, actorTemplate) @@ -204,7 +209,7 @@ func preferredFidelity(atespace string, tmpl *ateapipb.ActorTemplate) ateapipb.S return tmpl.GetSnapshotConfig().GetPreferredFidelity() } -// isPausedOriginSuspend reports whether the suspend must upload a PAUSED +// isPausedOriginSuspend reports whether the suspend must upload a paused // actor's node-local snapshot instead of checkpointing a running workload. // A local snapshot alone does not mean paused-origin: resume restores from it // without deleting its files, so a RUNNING actor resumed from pause still @@ -212,7 +217,7 @@ func preferredFidelity(atespace string, tmpl *ateapipb.ActorTemplate) ateapipb.S // disambiguates — running-origin suspends keep their assignment until // finalize, paused actors never have one. func isPausedOriginSuspend(actor *ateapipb.Actor) bool { - if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_PAUSED { + if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDED && latestSnapshotIsLocalOnly(actor.GetStatus()) { return true } if actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_SUSPENDING && @@ -253,6 +258,7 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso // actor is currently running (recorded on-node at Run/Restore) and pins it // into the snapshot manifest. inProgressSnap := snapshotAtLatestGeneration(actor.GetStatus()) + fidelity := fidelityToAtelet(preferredFidelity(actor.GetMetadata().GetAtespace(), actorTemplate)) req := &ateletpb.CheckpointRequest{ WorkerPodUid: assignment.GetWorkerPodUid(), Atespace: actor.GetMetadata().GetAtespace(), @@ -260,16 +266,23 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(), ActorTemplateName: actor.GetActorTemplate().GetName(), Spec: workloadSpec, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - Config: &ateletpb.CheckpointRequest_ExternalConfig{ - ExternalConfig: &ateletpb.ExternalCheckpointConfiguration{ + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: inProgressSnap.GetUuid(), + Fidelity: fidelity, + Object: &ateletpb.ObjectStorage{ SnapshotUri: inProgressSnap.GetDurableSnapshot().GetObject().GetSnapshotUri(), }, }, - Fidelity: fidelityToAtelet(preferredFidelity(actor.GetMetadata().GetAtespace(), actorTemplate)), + // FinalizeSuspended marks this snapshot COMPLETED, so it must be + // durable before Checkpoint returns. IMMEDIATE temporarily keeps no + // local copy either (see UPLOAD_MODE_IMMEDIATE), which matches + // FinalizeSuspended dropping the actor's local snapshots. + UploadConfig: &ateletpb.UploadConfig{ + Mode: ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE, + }, ActorUid: actor.GetMetadata().Uid, } - wireFidelity = ateattr.SnapshotFidelityValue(req.Fidelity) + wireFidelity = ateattr.SnapshotFidelityValue(fidelity) if _, err = client.Checkpoint(ctx, req); err != nil { return wireFidelity, handleAteletError(ctx, w.store, actorRef, ateattr.OperationSuspend, "Checkpoint", false, err) @@ -277,14 +290,14 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso return wireFidelity, nil } -// ensurePausedSnapshotUploaded suspends a PAUSED actor by telling the atelet +// ensurePausedSnapshotUploaded suspends a paused actor by telling the atelet // on the node holding the local pause snapshot to upload it to the actor's // persisted in-progress snapshot location; no workload runs, so there is no // ateom to checkpoint. Retries re-send the same semantic request: the // destination is minted once and the upload overwrites deterministic object // names, with the remote manifest as the commit marker. func (w *ActorWorkflow) ensurePausedSnapshotUploaded(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate) (wireFidelity string, err error) { - ctx, done := stepSpan(ctx, "UploadPausedCheckpoint") + ctx, done := stepSpan(ctx, "PromoteSnapshot") defer func() { err = done(err) }() localSnap := findLatestLocalSnapshot(actor.GetStatus()) @@ -307,23 +320,26 @@ func (w *ActorWorkflow) ensurePausedSnapshotUploaded(ctx context.Context, actorR } client := ateletpb.NewAteomHerderClient(ateletConn) - inProgressSnap := snapshotAtLatestGeneration(actor.GetStatus()) - req := &ateletpb.UploadPausedCheckpointRequest{ - Atespace: actor.GetMetadata().GetAtespace(), - ActorName: actor.GetMetadata().GetName(), - ActorUid: actor.GetMetadata().GetUid(), - ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(), - ActorTemplateName: actor.GetActorTemplate().GetName(), - LocalSnapshotName: localSnap.GetUuid(), - DestinationSnapshotUri: inProgressSnap.GetDurableSnapshot().GetObject().GetSnapshotUri(), - // The commit scope, like a running-origin suspend; atelet converts - // from the captured scope in the snapshot's manifest where possible. - DesiredFidelity: fidelityToAtelet(preferredFidelity(actor.GetMetadata().GetAtespace(), actorTemplate)), - } - wireFidelity = ateattr.SnapshotFidelityValue(req.DesiredFidelity) - - if _, err = client.UploadPausedCheckpoint(ctx, req); err != nil { - return wireFidelity, handleAteletError(ctx, w.store, actorRef, ateattr.OperationSuspend, "UploadPausedCheckpoint", false, err) + // The template's preferred fidelity, as a running-origin suspend commits. + fidelity := fidelityToAtelet(preferredFidelity(actor.GetMetadata().GetAtespace(), actorTemplate)) + req := &ateletpb.PromoteSnapshotRequest{ + Atespace: actor.GetMetadata().GetAtespace(), + ActorName: actor.GetMetadata().GetName(), + ActorUid: actor.GetMetadata().GetUid(), + ActorTemplateAtespace: actor.GetActorTemplate().GetAtespace(), + ActorTemplateName: actor.GetActorTemplate().GetName(), + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: localSnap.GetUuid(), + Fidelity: fidelity, + Object: &ateletpb.ObjectStorage{ + SnapshotUri: snapshotAtLatestGeneration(actor.GetStatus()).GetDurableSnapshot().GetObject().GetSnapshotUri(), + }, + }, + } + wireFidelity = ateattr.SnapshotFidelityValue(fidelity) + + if _, err = client.PromoteSnapshot(ctx, req); err != nil { + return wireFidelity, handleAteletError(ctx, w.store, actorRef, ateattr.OperationSuspend, "PromoteSnapshot", false, err) } return wireFidelity, nil } diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go index 43f24b38e9..5eaf7d2dfa 100644 --- a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go @@ -26,9 +26,11 @@ import ( "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/objectstore/objectstoretest" "github.com/agent-substrate/substrate/internal/objectstoreplugin/objectstoreplugintest" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" + "google.golang.org/protobuf/proto" "k8s.io/client-go/tools/cache" ) @@ -146,15 +148,15 @@ func TestSuspendActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { // TestEnsureMarkedSuspending_StateMatrix verifies the suspend edge's state // gating against every actor state: RUNNING takes the edge (checkpoint the -// workload), PAUSED takes it too (upload the node-local pause snapshot), -// SUSPENDING skips (a previous attempt already marked the actor), everything -// else is rejected with FailedPrecondition. SUSPENDED is rejected here -// because the orchestrator early-returns before this step for a fully -// suspended actor. +// workload), SUSPENDING skips (a previous attempt already marked the actor), +// everything else is rejected with FailedPrecondition. SUSPENDED without a +// node-local snapshot is rejected here because the orchestrator +// early-returns before this step for a fully suspended actor; SUSPENDED with +// one is the paused shape, covered by +// TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot. func TestEnsureMarkedSuspending_StateMatrix(t *testing.T) { allowed := map[ateapipb.ActorState]bool{ ateapipb.ActorState_ACTOR_STATE_RUNNING: true, - ateapipb.ActorState_ACTOR_STATE_PAUSED: true, ateapipb.ActorState_ACTOR_STATE_SUSPENDING: true, // skipped, not re-marked } @@ -164,16 +166,9 @@ func TestEnsureMarkedSuspending_StateMatrix(t *testing.T) { w := &ActorWorkflow{store: persistence} actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} - actorStatus := &ateapipb.ActorStatus{State: seedState} - if seedState == ateapipb.ActorState_ACTOR_STATE_PAUSED { - actorStatus.LastAssignedGeneration = 1 - actorStatus.Snapshots = []*ateapipb.Snapshot{ - newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap-1", "node-1"), - } - } actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - Status: actorStatus, + Status: &ateapipb.ActorStatus{State: seedState}, }) tmpl := &ateapipb.ActorTemplate{ @@ -184,25 +179,92 @@ func TestEnsureMarkedSuspending_StateMatrix(t *testing.T) { if err == nil && marked.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { t.Errorf("state %v: ensureMarkedSuspending returned actor in %v, want SUSPENDING", seedState, marked.GetStatus().GetState()) } - if seedState == ateapipb.ActorState_ACTOR_STATE_PAUSED && err == nil { - wantURI, err := resources.NewActorSnapshotURI("gs://snapshots", actorRef.Atespace, actor.GetMetadata().GetUid(), "snap-1") - if err != nil { - t.Fatalf("NewActorSnapshotURI: %v", err) - } - pending := findLatestDurableSnapshot(marked.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) - gotURI := pending.GetDurableSnapshot().GetObject().GetSnapshotUri() - if gotURI != wantURI.String() { - t.Errorf("paused-origin in-progress SnapshotUri = %q, want %q (derived from local snapshot uuid)", gotURI, wantURI.String()) - } - // The upload promotes the pause snapshot in place rather than - // minting a new generation. - if pending.GetGeneration() != 1 || pending.GetUuid() != "snap-1" { - t.Errorf("paused-origin pending snapshot = generation %d, uuid %q; want the pause snapshot (generation 1, uuid %q)", pending.GetGeneration(), pending.GetUuid(), "snap-1") - } - if got := marked.GetStatus().GetLastAssignedGeneration(); got != 1 { - t.Errorf("paused-origin LastAssignedGeneration = %d, want 1", got) - } - } + } +} + +// TestEnsureMarkedSuspending_RejectsPauseInProgress verifies a suspend does +// not adopt a SUSPENDING actor whose latest generation is a pending +// node-local snapshot: that actor is mid-pause, and the pause owns it until +// it is paused. +func TestEnsureMarkedSuspending_RejectsPauseInProgress(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + w := &ActorWorkflow{store: persistence} + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + + pausing := &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING}, + } + withPauseInProgress()(pausing) + created := storetest.MustCreateActor(t, ctx, persistence, pausing) + + tmpl := &ateapipb.ActorTemplate{SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}} + _, err := w.ensureMarkedSuspending(ctx, actorRef, created, tmpl) + if got := apierror.Code(err); got != codes.FailedPrecondition { + t.Errorf("ensureMarkedSuspending on a pause in progress: code = %v, want FailedPrecondition (err: %v)", got, err) + } + stored, err := persistence.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + if got := stored.GetStatus().GetLastAssignedGeneration(); got != 1 { + t.Errorf("last_assigned_generation = %d, want 1 (the pause's snapshot untouched)", got) + } +} + +// TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot verifies the suspend +// edge from a paused actor (SUSPENDED whose latest snapshot is node-local): +// it takes the edge and promotes that snapshot in place to a pending durable +// one, keeping its generation and uuid, with the upload destination derived +// from the uuid. +func TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + w := &ActorWorkflow{store: persistence} + + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, + AssignedNode: "node-1", + LastAssignedGeneration: 1, + Snapshots: []*ateapipb.Snapshot{ + newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap-1", "node-1"), + }, + }, + }) + + tmpl := &ateapipb.ActorTemplate{ + SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}, + } + marked, err := w.ensureMarkedSuspending(ctx, actorRef, actor, tmpl) + if err != nil { + t.Fatalf("ensureMarkedSuspending: %v", err) + } + if got := marked.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { + t.Errorf("state = %v, want SUSPENDING", got) + } + wantURI, err := resources.NewActorSnapshotURI("gs://snapshots", actorRef.Atespace, actor.GetMetadata().GetUid(), "snap-1") + if err != nil { + t.Fatalf("NewActorSnapshotURI: %v", err) + } + pending := findLatestDurableSnapshot(marked.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) + gotURI := pending.GetDurableSnapshot().GetObject().GetSnapshotUri() + if gotURI != wantURI.String() { + t.Errorf("paused-origin in-progress SnapshotUri = %q, want %q (derived from local snapshot uuid)", gotURI, wantURI.String()) + } + // The upload promotes the pause snapshot in place rather than minting a + // new generation. + if pending.GetGeneration() != 1 || pending.GetUuid() != "snap-1" { + t.Errorf("paused-origin pending snapshot = generation %d, uuid %q; want the pause snapshot (generation 1, uuid %q)", pending.GetGeneration(), pending.GetUuid(), "snap-1") + } + if pending.GetLocality() != "node-1" { + t.Errorf("pending snapshot locality = %q, want node-1 kept until the upload finishes", pending.GetLocality()) + } + if got := marked.GetStatus().GetLastAssignedGeneration(); got != 1 { + t.Errorf("paused-origin LastAssignedGeneration = %d, want 1", got) } } @@ -320,11 +382,59 @@ func TestEnsureAteletSuspended_DialFailureLeavesActorRetryable(t *testing.T) { } } +// TestEnsureAteletSuspended_WireRequest verifies a running-origin suspend +// checkpoints to the actor's in-progress durable snapshot with an immediate +// upload, since FinalizeSuspended then marks that snapshot COMPLETED. +func TestEnsureAteletSuspended_WireRequest(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + w, atelet := newWireCaptureWorkflow(t, persistence) + + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} + snapshotName := "2026-01-01t00-00-00z-abc" + snapshotURI := someActorSnapshotURI(t, testStorageLocation, "team-a", snapshotName) + created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: "ns", Name: "tmpl1"}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), + LastAssignedGeneration: 1, + Snapshots: []*ateapipb.Snapshot{ + newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "tmpl-uid-1", snapshotName, snapshotURI, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + }, + }) + tmpl := &ateapipb.ActorTemplate{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "ns", Name: "tmpl1"}, + SnapshotConfig: &ateapipb.SnapshotConfig{PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, + } + + if _, err := w.ensureAteletSuspended(ctx, actorRef, created, tmpl); err != nil { + t.Fatalf("ensureAteletSuspended: %v", err) + } + req := atelet.checkpointRequest() + if req == nil { + t.Fatal("expected Checkpoint RPC to be called") + } + if got := req.GetSnapshot().GetSnapshotUuid(); got != snapshotName { + t.Errorf("CheckpointRequest.Snapshot.SnapshotUuid = %q, want %q", got, snapshotName) + } + if got := req.GetSnapshot().GetObject().GetSnapshotUri(); got != snapshotURI { + t.Errorf("CheckpointRequest.Snapshot.Object.SnapshotUri = %q, want %q", got, snapshotURI) + } + wantConfig := &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE} + if got := req.GetUploadConfig(); !proto.Equal(got, wantConfig) { + t.Errorf("CheckpointRequest.UploadConfig = %v, want %v", got, wantConfig) + } +} + // TestEnsureSuspendedFinalized_NoAssignment verifies finalization runs even when // the actor has no worker assignment: the external snapshot must be recorded on // the actor and the actor moved to SUSPENDED rather than silently left // SUSPENDING. This is the shape a paused-origin suspend (#791) produces — a -// PAUSED actor has no worker — and the regression test for finalization +// paused actor has no worker — and the regression test for finalization // previously living inside the worker-freeing branch. func TestEnsureSuspendedFinalized_NoAssignment(t *testing.T) { ctx := context.Background() @@ -730,8 +840,9 @@ func TestPreferredFidelity(t *testing.T) { // TestIsPausedOriginSuspend pins the paused-origin discriminator: a local // snapshot alone must not select the paused path, because a RUNNING actor -// resumed from pause still carries one; only PAUSED state, or SUSPENDING with -// no worker assignment, means the suspend uploads a local snapshot. +// resumed from pause still carries one; only the paused shape (SUSPENDED with +// a node-local latest snapshot), or SUSPENDING with no worker assignment, +// means the suspend uploads a local snapshot. func TestIsPausedOriginSuspend(t *testing.T) { assignment := &ateapipb.WorkerAssignment{WorkerNamespace: "ns", WorkerPool: "pool", WorkerPod: "pod-1"} localSnaps := []*ateapipb.Snapshot{ @@ -742,7 +853,8 @@ func TestIsPausedOriginSuspend(t *testing.T) { actor *ateapipb.Actor want bool }{ - {"paused actor", &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_PAUSED, AssignedNode: "node1", Snapshots: localSnaps}}, true}, + {"paused actor", &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, AssignedNode: "node1", LastAssignedGeneration: 1, Snapshots: localSnaps}}, true}, + {"suspended actor without a local snapshot", &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}}, false}, {"suspending retry of a paused-origin suspend", &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: "node1", Snapshots: localSnaps}}, true}, {"running actor resumed from pause", &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING, Snapshots: localSnaps, WorkerAssignment: assignment}}, false}, {"suspending retry of a running-origin suspend after a resume from pause", &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, Snapshots: localSnaps, WorkerAssignment: assignment}}, false}, @@ -824,39 +936,6 @@ func TestEnsurePausedSnapshotUploaded_Preconditions(t *testing.T) { }) } -// TestSuspendActor_PausedWithoutLocalSnapshotCrashes verifies a PAUSED actor -// whose LocalSnapshot is missing (corrupted store state: nothing records -// where the pause snapshot lives) is crashed by the suspend workflow rather -// than left flapping between PAUSED and SUSPENDING. -func TestSuspendActor_PausedWithoutLocalSnapshotCrashes(t *testing.T) { - ctx := context.Background() - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() - - // The template needs a snapshot location: MarkSuspending validates the - // destination URI before the workflow reaches the crash under test. - // newTestActorWorkflow's stored template carries one. - w := newTestActorWorkflow(t, st, "ns", "tmpl1") - - seedWorkflowActor(t, ctx, st, resources.ActorRef{Atespace: "team-a", Name: "id1"}, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_PAUSED) - - _, err := w.SuspendActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}) - if got := apierror.Code(err); got != codes.FailedPrecondition { - t.Fatalf("apierror.Code(err) = %v, want %v for PAUSED actor with no local snapshot record (err: %v)", got, codes.FailedPrecondition, err) - } - - got, err := st.GetActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}) - if err != nil { - t.Fatalf("GetActor failed: %v", err) - } - if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_CRASHED { - t.Errorf("stored state = %v, want %v", got.GetStatus().GetState(), ateapipb.ActorState_ACTOR_STATE_CRASHED) - } - if msg, want := got.GetStatus().GetCrash().GetMessage(), "suspend failed: "+crashMessageLocalSnapshotMissing; msg != want { - t.Errorf("crash message = %q, want %q", msg, want) - } -} - // TestEnsureSuspendedFinalized_MissingDurableSnapshotClearsAssignment verifies // that if the latest generation snapshot entry is missing when // ensureSuspendedFinalized runs, the actor transitions to CRASHED and clears diff --git a/cmd/ateapi/internal/controlapi/workflow_tag.go b/cmd/ateapi/internal/controlapi/workflow_tag.go index 0311d70449..af5bd847dc 100644 --- a/cmd/ateapi/internal/controlapi/workflow_tag.go +++ b/cmd/ateapi/internal/controlapi/workflow_tag.go @@ -218,6 +218,10 @@ func (w *ActorWorkflow) loadActorForTag(ctx context.Context, actorRef resources. if snap == nil { return nil, nil, apierror.FailedPrecondition("Actor %s holds no external snapshot to tag", actorRef) } + // The actor's latest snapshot is local only. It must be uploaded before it can be tagged. + if latestSnapshotIsLocalOnly(actor.GetStatus()) { + return nil, nil, apierror.FailedPrecondition("Actor %s holds no external snapshot of its latest state to tag: suspend it first", actorRef) + } // Every way an Actor comes to hold an external snapshot records the // template its guest state was built under: a suspend through // ensureSuspendedFinalized, a create from a tag through the tag's own UID. diff --git a/cmd/ateapi/internal/controlapi/workflow_tag_test.go b/cmd/ateapi/internal/controlapi/workflow_tag_test.go index bbfaf5fe97..1f72b4ed75 100644 --- a/cmd/ateapi/internal/controlapi/workflow_tag_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_tag_test.go @@ -161,7 +161,10 @@ func TestTagActorSnapshot_Preconditions(t *testing.T) { // under, as every real path to holding one does. Only the case that // exercises the missing record leaves it false. builtOnTemplate bool - wantCode codes.Code + // paused adds a node-local snapshot above the external one, the + // shape a pause leaves behind. + paused bool + wantCode codes.Code }{ { name: "the actor is not suspended", @@ -181,6 +184,14 @@ func TestTagActorSnapshot_Preconditions(t *testing.T) { withSnapshot: true, wantCode: codes.Internal, }, + { + name: "the actor is paused above its external snapshot", + state: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, + withSnapshot: true, + builtOnTemplate: true, + paused: true, + wantCode: codes.FailedPrecondition, + }, } for _, tt := range tests { @@ -208,6 +219,10 @@ func TestTagActorSnapshot_Preconditions(t *testing.T) { s.Snapshots = []*ateapipb.Snapshot{ newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmplUID, uri.Name(), uri.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), } + if tt.paused { + s.LastAssignedGeneration = 2 + s.Snapshots = append(s.Snapshots, newLocalSnapshot(2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmplUID, "paused", "node-1")) + } }) } @@ -223,6 +238,62 @@ func TestTagActorSnapshot_Preconditions(t *testing.T) { } } +// TestTagActorSnapshot_TagsNewestExternalSnapshot verifies that a tag is +// accepted when no snapshot is newer than the actor's external one, even when +// that external snapshot is not at last_assigned_generation. +func TestTagActorSnapshot_TagsNewestExternalSnapshot(t *testing.T) { + tests := []struct { + name string + // lastAssigned is the actor's last_assigned_generation. The external + // snapshot is always at generation extGen. + lastAssigned int32 + extGen int32 + // locality is set on the external snapshot, as when a paused actor's + // local snapshot was uploaded in place. + locality string + }{ + { + name: "a revert dropped the newest generation", + lastAssigned: 2, + extGen: 1, + }, + { + name: "a paused actor's local snapshot was uploaded at the same generation", + lastAssigned: 2, + extGen: 2, + locality: "node-1", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + template := seedSubstrateTemplate(t, ctx, persistence, "sub-tmpl") + w, objects := newFinalizeWorkflow(persistence) + + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} + actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: "team-a", Name: "sub-tmpl"}, + Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, + }) + uri := mustActorSnapshotURI(t, template, actor, "actor-1-snapshot") + objects.PutSnapshot(t, uri, "manifest.json") + mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) { + snap := newDurableSnapshot(tt.extGen, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, template.GetMetadata().GetUid(), uri.Name(), uri.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + snap.Locality = tt.locality + s.LastAssignedGeneration = tt.lastAssigned + s.Snapshots = []*ateapipb.Snapshot{snap} + }) + + if _, err := w.TagActorSnapshot(ctx, tagToCreate(actorRef, "v1")); err != nil { + t.Fatalf("TagActorSnapshot: %v", err) + } + }) + } +} + // TestTagActorSnapshot_RecreateAfterCopyFailure verifies what a create that // dies mid-copy leaves behind, and how a client gets past it. The row survives // as a pending tag naming the prefix the copy was writing into, so the objects diff --git a/cmd/ateapi/internal/controlapi/workflow_test.go b/cmd/ateapi/internal/controlapi/workflow_test.go index 0b5f79d982..c0e1876941 100644 --- a/cmd/ateapi/internal/controlapi/workflow_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_test.go @@ -57,7 +57,7 @@ func TestActorStateChangeRecords(t *testing.T) { wantStored: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, }, { - name: "pause marks the actor pausing", + name: "pause marks the actor suspending", seedState: ateapipb.ActorState_ACTOR_STATE_RUNNING, transition: func(t *testing.T, w *ActorWorkflow, ref resources.ActorRef, actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) { if _, err := w.ensureMarkedPausing(context.Background(), ref, actor, tmpl); err != nil { @@ -65,8 +65,8 @@ func TestActorStateChangeRecords(t *testing.T) { } }, wantOp: ateattr.OperationPause, - wantState: ateattr.ActorStatePausing, - wantStored: ateapipb.ActorState_ACTOR_STATE_PAUSING, + wantState: ateattr.ActorStateSuspending, + wantStored: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, }, { name: "delete marks the actor deleting", diff --git a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go index 2eb719c404..10470ca1b3 100644 --- a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go @@ -148,6 +148,31 @@ func seedWorkflowActor(t *testing.T, ctx context.Context, st store.Interface, ac storetest.MustCreateActor(t, ctx, st, actor) } +// withPausedShape seeds the shape a pause leaves behind: a node-local +// snapshot at the latest generation, pinned to node. Pair it with +// ACTOR_STATE_SUSPENDED to seed a paused actor. +func withPausedShape(node string) func(*ateapipb.Actor) { + return func(a *ateapipb.Actor) { + a.Status.AssignedNode = node + a.Status.LastAssignedGeneration = 1 + a.Status.Snapshots = []*ateapipb.Snapshot{ + newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "pause-snap", node), + } + } +} + +// withPauseInProgress seeds the shape a pause has between MarkPausing and +// FinalizePaused: a node-local snapshot at the latest generation that has no +// locality yet. Pair it with ACTOR_STATE_SUSPENDING. +func withPauseInProgress() func(*ateapipb.Actor) { + return func(a *ateapipb.Actor) { + a.Status.LastAssignedGeneration = 1 + a.Status.Snapshots = []*ateapipb.Snapshot{ + newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "pause-snap", ""), + } + } +} + // allActorStates enumerates every ActorState value, for exhaustive // CheckPrerequisite table tests. It is derived from the generated enum map so // states added to the proto are covered automatically. diff --git a/cmd/ateapi/internal/controlapi/workflow_worker_delete.go b/cmd/ateapi/internal/controlapi/workflow_worker_delete.go index 742f7179ce..d3919f8987 100644 --- a/cmd/ateapi/internal/controlapi/workflow_worker_delete.go +++ b/cmd/ateapi/internal/controlapi/workflow_worker_delete.go @@ -207,10 +207,13 @@ func (w *WorkerWorkflow) crashBoundActor(ctx context.Context, worker *ateapipb.W opName = ateattr.OperationResume case ateapipb.ActorState_ACTOR_STATE_SUSPENDING: opName = ateattr.OperationSuspend - case ateapipb.ActorState_ACTOR_STATE_PAUSING: - opName = ateattr.OperationPause + if latestSnapshotIsPendingLocal(actor.GetStatus()) { + opName = ateattr.OperationPause + } case ateapipb.ActorState_ACTOR_STATE_REVERTING: opName = ateattr.OperationRevert + case ateapipb.ActorState_ACTOR_STATE_DELETING: + opName = ateattr.OperationDelete } wasAlreadyCrashed := actor.GetStatus().GetState() == ateapipb.ActorState_ACTOR_STATE_CRASHED diff --git a/cmd/ateapi/internal/controlapi/workflow_worker_delete_test.go b/cmd/ateapi/internal/controlapi/workflow_worker_delete_test.go index 7f77931523..5c1493d4db 100644 --- a/cmd/ateapi/internal/controlapi/workflow_worker_delete_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_worker_delete_test.go @@ -159,8 +159,11 @@ func TestDeleteWorkerWorkflow_ReleasesBoundActor(t *testing.T) { // crashed is released again without being counted twice. func TestDeleteWorkerWorkflow_ReleasedActorStateTransitions(t *testing.T) { tests := []struct { - name string - start ateapipb.ActorState + name string + start ateapipb.ActorState + // pausing seeds the pause-in-progress shape (a pending node-local + // snapshot), which labels a SUSPENDING crash as a pause. + pausing bool wantState ateapipb.ActorState wantOp string wantMetric bool @@ -168,8 +171,9 @@ func TestDeleteWorkerWorkflow_ReleasedActorStateTransitions(t *testing.T) { {name: "running becomes crashed", start: ateapipb.ActorState_ACTOR_STATE_RUNNING, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantOp: ateattr.OperationUnknown, wantMetric: true}, {name: "resuming becomes crashed", start: ateapipb.ActorState_ACTOR_STATE_RESUMING, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantOp: ateattr.OperationResume, wantMetric: true}, {name: "suspending becomes crashed", start: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantOp: ateattr.OperationSuspend, wantMetric: true}, - {name: "pausing becomes crashed", start: ateapipb.ActorState_ACTOR_STATE_PAUSING, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantOp: ateattr.OperationPause, wantMetric: true}, + {name: "pausing becomes crashed", start: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, pausing: true, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantOp: ateattr.OperationPause, wantMetric: true}, {name: "reverting becomes crashed", start: ateapipb.ActorState_ACTOR_STATE_REVERTING, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantOp: ateattr.OperationRevert, wantMetric: true}, + {name: "deleting becomes crashed", start: ateapipb.ActorState_ACTOR_STATE_DELETING, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantOp: ateattr.OperationDelete, wantMetric: true}, {name: "suspended stays suspended", start: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, wantState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, wantMetric: false}, {name: "crashed is not counted twice", start: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantState: ateapipb.ActorState_ACTOR_STATE_CRASHED, wantMetric: false}, } @@ -183,7 +187,11 @@ func TestDeleteWorkerWorkflow_ReleasedActorStateTransitions(t *testing.T) { ctx := context.Background() wf, persistence := newWorkerDeleteWorkflow(t) seedAPIWorker(t, ctx, persistence, validWorker(apiWorkerName)) - actor := seedAPIActor(t, ctx, persistence, tc.start) + var opts []func(*ateapipb.Actor) + if tc.pausing { + opts = append(opts, withPauseInProgress()) + } + actor := seedAPIActor(t, ctx, persistence, tc.start, opts...) assignAPIWorker(t, ctx, persistence, apiWorkerName, actor.GetMetadata().GetUid()) if _, err := wf.DeleteWorker(ctx, apiWorkerName, store.DeletePreconditions{}); err != nil { diff --git a/cmd/ateapi/internal/controlapi/workload_spec.go b/cmd/ateapi/internal/controlapi/workload_spec.go index 2c666994bf..749cb0b497 100644 --- a/cmd/ateapi/internal/controlapi/workload_spec.go +++ b/cmd/ateapi/internal/controlapi/workload_spec.go @@ -57,10 +57,8 @@ func workloadSpecFromActorTemplate(actorTemplate *ateapipb.ActorTemplate, actor switch { case vol.GetDurableDir() != nil: workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{ - Name: vol.GetName(), - Source: &ateletpb.Volume_DurableDir{ - DurableDir: &ateletpb.DurableDirVolume{}, - }, + Name: vol.GetName(), + DurableDir: &ateletpb.DurableDirVolume{}, }) case vol.GetSystemInfo() != nil: @@ -76,19 +74,15 @@ func workloadSpecFromActorTemplate(actorTemplate *ateapipb.ActorTemplate, actor }) } ateletSystemInfo.DataSources = append(ateletSystemInfo.DataSources, &ateletpb.SystemInfoDataSource{ - DataSource: &ateletpb.SystemInfoDataSource_ActorMetadata{ - ActorMetadata: actorMetadata, - }, + ActorMetadata: actorMetadata, }) case dataSource.GetTrustBundle() != nil: // atelet resolves named trustBundles against its allowlist // and ClusterTrustBundle informer at write time ateletSystemInfo.DataSources = append(ateletSystemInfo.DataSources, &ateletpb.SystemInfoDataSource{ - DataSource: &ateletpb.SystemInfoDataSource_TrustBundle{ - TrustBundle: &ateletpb.TrustBundleDataSource{ - Names: dataSource.GetTrustBundle().GetNames(), - Path: dataSource.GetTrustBundle().GetPath(), - }, + TrustBundle: &ateletpb.TrustBundleDataSource{ + Names: dataSource.GetTrustBundle().GetNames(), + Path: dataSource.GetTrustBundle().GetPath(), }, }) default: @@ -96,19 +90,15 @@ func workloadSpecFromActorTemplate(actorTemplate *ateapipb.ActorTemplate, actor } } workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{ - Name: vol.GetName(), - Source: &ateletpb.Volume_SystemInfo{ - SystemInfo: ateletSystemInfo, - }, + Name: vol.GetName(), + SystemInfo: ateletSystemInfo, }) case vol.GetImage() != nil: workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{ Name: vol.GetName(), - Source: &ateletpb.Volume_Image{ - Image: &ateletpb.ImageVolumeSource{ - Reference: vol.GetImage().GetReference(), - }, + Image: &ateletpb.ImageVolumeSource{ + Reference: vol.GetImage().GetReference(), }, }) @@ -188,13 +178,11 @@ func appendExternalVolumes(workloadSpec *ateletpb.WorkloadSpec, template *ateapi } workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{ Name: vol.GetName(), - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: storageVolID, - VolumeType: volType, - VolumeContext: volCtx, - PublishContext: volumePublishContexts[vol.GetName()], - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: storageVolID, + VolumeType: volType, + VolumeContext: volCtx, + PublishContext: volumePublishContexts[vol.GetName()], }, }) } @@ -232,13 +220,13 @@ func toAteletActorMetadataField(in ateapipb.ActorMetadataField) ateletpb.ActorMe // toAteletWakeupProbe projects the template wakeup probe field onto the ateletpb wire // type. Returns nil when the source is nil so containers without a probe // stay unchanged on the wire. -func toAteletWakeupProbe(in *ateapipb.ContainerWakeupProbe) *ateletpb.WakeupProbe { +func toAteletWakeupProbe(in *ateapipb.WakeupProbe) *ateletpb.WakeupProbe { if in == nil { return nil } out := &ateletpb.WakeupProbe{TimeoutSeconds: in.GetTimeoutSeconds()} if in.GetHttpGet() != nil { - out.HttpGet = &ateletpb.HTTPGetAction{ + out.HttpGet = &ateletpb.HttpGetAction{ Path: in.GetHttpGet().GetPath(), Port: in.GetHttpGet().GetPort(), } diff --git a/cmd/ateapi/internal/controlapi/workload_spec_test.go b/cmd/ateapi/internal/controlapi/workload_spec_test.go index b39a90bbfd..f84f6f5e36 100644 --- a/cmd/ateapi/internal/controlapi/workload_spec_test.go +++ b/cmd/ateapi/internal/controlapi/workload_spec_test.go @@ -50,8 +50,8 @@ func TestWorkloadSpecFromActorTemplate(t *testing.T) { want: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ { - Name: "home", - Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}, + Name: "home", + DurableDir: &ateletpb.DurableDirVolume{}, }, }, Containers: []*ateletpb.Container{ @@ -100,19 +100,15 @@ func TestWorkloadSpecFromActorTemplate(t *testing.T) { Volumes: []*ateletpb.Volume{ { Name: "system-info", - Source: &ateletpb.Volume_SystemInfo{ - SystemInfo: &ateletpb.SystemInfoVolume{ - DataSources: []*ateletpb.SystemInfoDataSource{ - {DataSource: &ateletpb.SystemInfoDataSource_ActorMetadata{ - ActorMetadata: &ateletpb.ActorMetadataDataSource{ - Items: []*ateletpb.ActorMetadataItem{ - {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME, Path: "actor-name"}, - {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE, Path: "atespace"}, - {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID, Path: "identity/actor-uid"}, - }, - }, - }}, - }, + SystemInfo: &ateletpb.SystemInfoVolume{ + DataSources: []*ateletpb.SystemInfoDataSource{ + {ActorMetadata: &ateletpb.ActorMetadataDataSource{ + Items: []*ateletpb.ActorMetadataItem{ + {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME, Path: "actor-name"}, + {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE, Path: "atespace"}, + {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID, Path: "identity/actor-uid"}, + }, + }}, }, }, }, @@ -150,12 +146,12 @@ func TestWorkloadSpecFromActorTemplate(t *testing.T) { want: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ { - Name: "home", - Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}, + Name: "home", + DurableDir: &ateletpb.DurableDirVolume{}, }, { - Name: "agent", - Source: &ateletpb.Volume_Image{Image: &ateletpb.ImageVolumeSource{Reference: "example.com/agent@sha256:abc"}}, + Name: "agent", + Image: &ateletpb.ImageVolumeSource{Reference: "example.com/agent@sha256:abc"}, }, }, Containers: []*ateletpb.Container{ @@ -200,13 +196,9 @@ func TestWorkloadSpecFromActorTemplate(t *testing.T) { Volumes: []*ateletpb.Volume{ { Name: "system-info", - Source: &ateletpb.Volume_SystemInfo{ - SystemInfo: &ateletpb.SystemInfoVolume{ - DataSources: []*ateletpb.SystemInfoDataSource{ - {DataSource: &ateletpb.SystemInfoDataSource_TrustBundle{ - TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{"egress-trust"}, Path: "trust/ca.pem"}, - }}, - }, + SystemInfo: &ateletpb.SystemInfoVolume{ + DataSources: []*ateletpb.SystemInfoDataSource{ + {TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{"egress-trust"}, Path: "trust/ca.pem"}}, }, }, }, @@ -243,8 +235,8 @@ func TestWorkloadSpecFromActorTemplate(t *testing.T) { want: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ { - Name: "home", - Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}, + Name: "home", + DurableDir: &ateletpb.DurableDirVolume{}, }, }, Containers: []*ateletpb.Container{ @@ -272,8 +264,8 @@ func TestWorkloadSpecFromActorTemplate(t *testing.T) { want: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ { - Name: "home", - Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}, + Name: "home", + DurableDir: &ateletpb.DurableDirVolume{}, }, }, Containers: []*ateletpb.Container{{Name: "main", Image: "main"}}, @@ -349,8 +341,8 @@ func TestWorkloadSpecFromActorTemplatePropagatesWakeupProbe(t *testing.T) { { Name: "with-probe", Image: "main", - WakeupProbe: &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/health", Port: 8080}, + WakeupProbe: &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/health", Port: 8080}, TimeoutSeconds: 45, }, }, @@ -370,7 +362,7 @@ func TestWorkloadSpecFromActorTemplatePropagatesWakeupProbe(t *testing.T) { Name: "with-probe", Image: "main", WakeupProbe: &ateletpb.WakeupProbe{ - HttpGet: &ateletpb.HTTPGetAction{Path: "/health", Port: 8080}, + HttpGet: &ateletpb.HttpGetAction{Path: "/health", Port: 8080}, TimeoutSeconds: 45, }, }, @@ -456,13 +448,11 @@ func TestAppendExternalVolumes(t *testing.T) { Volumes: []*ateletpb.Volume{ { Name: "vol-1", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "vol-gce-pd-123", - VolumeType: "pd-standard", - VolumeContext: map[string]string{"foo": "bar"}, - PublishContext: tt.wantPublishCtx, - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "vol-gce-pd-123", + VolumeType: "pd-standard", + VolumeContext: map[string]string{"foo": "bar"}, + PublishContext: tt.wantPublishCtx, }, }, }, diff --git a/cmd/ateapi/internal/defaults/defaults_test.go b/cmd/ateapi/internal/defaults/defaults_test.go index 5a4c90042c..5ccb5ba159 100644 --- a/cmd/ateapi/internal/defaults/defaults_test.go +++ b/cmd/ateapi/internal/defaults/defaults_test.go @@ -64,47 +64,47 @@ func TestApply(t *testing.T) { }, { name: "wakeup probe gets timeout and path", in: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "main", WakeupProbe: &ateapipb.ContainerWakeupProbe{HttpGet: &ateapipb.HTTPGetAction{Port: 8080}}}, + {Name: "main", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 8080}}}, }}, want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "main", WakeupProbe: &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Port: 8080, Path: "/"}, + {Name: "main", WakeupProbe: &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Port: 8080, Path: "/"}, TimeoutSeconds: 30, }}, }}, }, { name: "wakeup probe without http_get gets only the timeout", in: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "main", WakeupProbe: &ateapipb.ContainerWakeupProbe{}}, + {Name: "main", WakeupProbe: &ateapipb.WakeupProbe{}}, }}, want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "main", WakeupProbe: &ateapipb.ContainerWakeupProbe{TimeoutSeconds: 30}}, + {Name: "main", WakeupProbe: &ateapipb.WakeupProbe{TimeoutSeconds: 30}}, }}, }, { name: "set wakeup probe fields are kept", in: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "main", WakeupProbe: &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Port: 8080, Path: "/healthz"}, + {Name: "main", WakeupProbe: &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Port: 8080, Path: "/healthz"}, TimeoutSeconds: 5, }}, }}, want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "main", WakeupProbe: &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Port: 8080, Path: "/healthz"}, + {Name: "main", WakeupProbe: &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Port: 8080, Path: "/healthz"}, TimeoutSeconds: 5, }}, }}, }, { name: "every container is defaulted", in: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "a", WakeupProbe: &ateapipb.ContainerWakeupProbe{HttpGet: &ateapipb.HTTPGetAction{Port: 1}}}, + {Name: "a", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 1}}}, {Name: "b"}, - {Name: "c", WakeupProbe: &ateapipb.ContainerWakeupProbe{HttpGet: &ateapipb.HTTPGetAction{Port: 3}}}, + {Name: "c", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 3}}}, }}, want: &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{ - {Name: "a", WakeupProbe: &ateapipb.ContainerWakeupProbe{HttpGet: &ateapipb.HTTPGetAction{Port: 1, Path: "/"}, TimeoutSeconds: 30}}, + {Name: "a", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 1, Path: "/"}, TimeoutSeconds: 30}}, {Name: "b"}, - {Name: "c", WakeupProbe: &ateapipb.ContainerWakeupProbe{HttpGet: &ateapipb.HTTPGetAction{Port: 3, Path: "/"}, TimeoutSeconds: 30}}, + {Name: "c", WakeupProbe: &ateapipb.WakeupProbe{HttpGet: &ateapipb.HttpGetAction{Port: 3, Path: "/"}, TimeoutSeconds: 30}}, }}, }, { name: "actor has no defaults", @@ -117,17 +117,17 @@ func TestApply(t *testing.T) { }, { name: "egress policy defaults http and https ports only", in: &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{ - {Http: &ateapipb.HTTPRule{Hostnames: []string{"a.example.com"}}}, - {Https: &ateapipb.HTTPSRule{Hostnames: []string{"b.example.com"}}}, - {Http: &ateapipb.HTTPRule{Hostnames: []string{"c.example.com"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, - {TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: []string{"d.example.com"}}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"a.example.com"}}}, + {Https: &ateapipb.HttpsRule{Hostnames: []string{"b.example.com"}}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"c.example.com"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, + {TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: []string{"d.example.com"}}}, {}, }}, want: &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{ - {Http: &ateapipb.HTTPRule{Hostnames: []string{"a.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80}}}}, - {Https: &ateapipb.HTTPSRule{Hostnames: []string{"b.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}, - {Http: &ateapipb.HTTPRule{Hostnames: []string{"c.example.com"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, - {TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: []string{"d.example.com"}}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"a.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80}}}}, + {Https: &ateapipb.HttpsRule{Hostnames: []string{"b.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"c.example.com"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, + {TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: []string{"d.example.com"}}}, {}, }}, }, { diff --git a/cmd/ateapi/internal/store/atepg/common_test.go b/cmd/ateapi/internal/store/atepg/common_test.go index 6edd774f3e..f85e8741a6 100644 --- a/cmd/ateapi/internal/store/atepg/common_test.go +++ b/cmd/ateapi/internal/store/atepg/common_test.go @@ -19,6 +19,7 @@ import ( "testing" "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -54,7 +55,11 @@ func setTestPolicyManager(t *testing.T, p *Persistence) { t.Fatalf("NewOpenFGAServer failed: %v", err) } t.Cleanup(fgaServer.Close) - _, policyManager, err := authz.New(t.Context(), p.pool, fgaServer, nil) + _, policyManager, err := authz.New(t.Context(), p.pool, fgaServer, nil, authz.SystemPrincipals{ + ControllerSPIFFEID: installdefaults.ControllerSPIFFEID(installdefaults.SystemNamespace), + EgressGatewaySPIFFEID: installdefaults.EgressSPIFFEID(installdefaults.SystemNamespace), + IngressRouterSPIFFEID: installdefaults.RouterSPIFFEID(installdefaults.SystemNamespace), + }) if err != nil { t.Fatalf("authz.New failed: %v", err) } diff --git a/cmd/ateapi/internal/store/storecontract/contract.go b/cmd/ateapi/internal/store/storecontract/contract.go index f356967998..2093146326 100644 --- a/cmd/ateapi/internal/store/storecontract/contract.go +++ b/cmd/ateapi/internal/store/storecontract/contract.go @@ -232,7 +232,7 @@ func runEgressPolicyContractTests(t *testing.T, setup func(t *testing.T) store.I policy := &ateapipb.EgressPolicy{ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "default"}, Rules: []*ateapipb.EgressRule{{ - Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80}}}, + Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80}}}, }}, } @@ -262,7 +262,7 @@ func runEgressPolicyContractTests(t *testing.T, setup func(t *testing.T) store.I updated, err := s.UpdateEgressPolicy(ctx, actorRef, store.PreconditionFrom(created), func(policy *ateapipb.EgressPolicy) error { policy.Metadata.Atespace = "other" policy.Metadata.Name = "other" - policy.Rules = []*ateapipb.EgressRule{{TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}} + policy.Rules = []*ateapipb.EgressRule{{TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}} return nil }) if err != nil || updated.GetMetadata().GetAtespace() != testAtespace || updated.GetMetadata().GetName() != "default" || updated.GetMetadata().GetVersion() != 2 || updated.GetMetadata().GetUid() != created.GetMetadata().GetUid() { @@ -749,7 +749,6 @@ func runActorContractTests(t *testing.T, setup func(t *testing.T) store.Interfac ateapipb.ActorState_ACTOR_STATE_CRASHED, ateapipb.ActorState_ACTOR_STATE_DELETING, ateapipb.ActorState_ACTOR_STATE_RUNNING, - ateapipb.ActorState_ACTOR_STATE_PAUSED, } { t.Run(state.String(), func(t *testing.T) { s := setup(t) @@ -3225,7 +3224,7 @@ func runUnknownFieldContractTests(t *testing.T, setup func(t *testing.T) store.I created, err := s.CreateEgressPolicy(ctx, ref, withUnknownField(&ateapipb.EgressPolicy{ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "default"}, Rules: []*ateapipb.EgressRule{withUnknownField(&ateapipb.EgressRule{ - Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80}}}, + Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80}}}, })}, })) if err != nil { diff --git a/cmd/ateapi/internal/store/storetest/storetest.go b/cmd/ateapi/internal/store/storetest/storetest.go index ed8e457d7b..96d7e17428 100644 --- a/cmd/ateapi/internal/store/storetest/storetest.go +++ b/cmd/ateapi/internal/store/storetest/storetest.go @@ -33,6 +33,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/atepg" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/dockerenv" + "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5/pgxpool" "github.com/testcontainers/testcontainers-go/modules/postgres" @@ -87,7 +88,7 @@ func SetupPostgresPersistence(t *testing.T) *atepg.Persistence { t.Fatalf("creating OpenFGA server: %v", err) } t.Cleanup(fgaServer.Close) - _, policyManager, err := authz.New(ctx, pool, fgaServer, nil) + _, policyManager, err := authz.New(ctx, pool, fgaServer, nil, DefaultSystemPrincipals()) if err != nil { t.Fatalf("initializing OpenFGA authz: %v", err) } @@ -95,6 +96,16 @@ func SetupPostgresPersistence(t *testing.T) *atepg.Persistence { return persistence } +// DefaultSystemPrincipals returns valid SystemPrincipals for tests that +// initialize authz.New without testing system component roles. +func DefaultSystemPrincipals() authz.SystemPrincipals { + return authz.SystemPrincipals{ + ControllerSPIFFEID: installdefaults.ControllerSPIFFEID(installdefaults.SystemNamespace), + EgressGatewaySPIFFEID: installdefaults.EgressSPIFFEID(installdefaults.SystemNamespace), + IngressRouterSPIFFEID: installdefaults.RouterSPIFFEID(installdefaults.SystemNamespace), + } +} + // MustCreateAtespace creates name unless it already exists. PostgreSQL enforces // the parent relationship for actor and snapshot records, so test fixtures use // this before seeding those resources. diff --git a/cmd/ateapi/internal/workerservice/certificate.go b/cmd/ateapi/internal/workerservice/certificate.go index 8d909c384d..457d8b2a13 100644 --- a/cmd/ateapi/internal/workerservice/certificate.go +++ b/cmd/ateapi/internal/workerservice/certificate.go @@ -28,6 +28,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/internal/substratex509" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -86,6 +87,18 @@ func (s *Server) MintAteomActorCertificate(ctx context.Context, req *ateapipb.Mi IsCA: false, } + err = substratex509.AddActorIdentityToCertificate( + &substratex509.ActorIdentity{ + Atespace: dbActor.GetMetadata().GetAtespace(), + ActorName: dbActor.GetMetadata().GetName(), + ActorUid: dbActor.GetMetadata().GetUid(), + }, + template, + ) + if err != nil { + return nil, fmt.Errorf("while adding Substrate extension: %w", err) + } + // Sign and return the actor cert. chain, err := s.actorIDCAPool.CreateCertificate(template, csr.PublicKey) if err != nil { diff --git a/cmd/ateapi/internal/workerservice/certificate_test.go b/cmd/ateapi/internal/workerservice/certificate_test.go index d75bab18e3..1bbeeeb084 100644 --- a/cmd/ateapi/internal/workerservice/certificate_test.go +++ b/cmd/ateapi/internal/workerservice/certificate_test.go @@ -30,6 +30,7 @@ import ( "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/substratex509" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/google/go-cmp/cmp" "google.golang.org/grpc/codes" ) @@ -135,8 +136,13 @@ func TestMintAteomActorCertificate(t *testing.T) { if err != nil { t.Fatalf("ActorIdentityFromCertificate: %v", err) } - if identity != nil { - t.Errorf("ActorIdentity = %+v, want nil (ateom certificates should not carry ActorIdentity)", identity) + wantIdentity := &substratex509.ActorIdentity{ + Atespace: "team-a", + ActorName: "my-actor", + ActorUid: actor.GetMetadata().GetUid(), + } + if diff := cmp.Diff(wantIdentity, identity); diff != "" { + t.Errorf("ActorIdentity mismatch (-want +got):\n%s", diff) } } diff --git a/cmd/ateapi/internal/workerservice/suspend.go b/cmd/ateapi/internal/workerservice/suspend.go index b3dea05528..702e6f6b51 100644 --- a/cmd/ateapi/internal/workerservice/suspend.go +++ b/cmd/ateapi/internal/workerservice/suspend.go @@ -124,7 +124,7 @@ func checkActorHostedBy(ctx context.Context, actor *ateapipb.Actor, workerName, return apierror.NotFound("Actor %s not found", actorRef) } // An Actor with no assignment has no worker to speak for it: it is - // SUSPENDED, PAUSED, or CRASHED, and nothing hosts it. + // SUSPENDED (paused or not) or CRASHED, and nothing hosts it. if assigned := actor.GetStatus().GetWorkerAssignment().GetWorker().GetName(); assigned != workerName { slog.WarnContext(ctx, "Refusing a suspend request for an actor hosted elsewhere", slog.String("worker", workerName), diff --git a/cmd/ateapi/internal/workerservice/suspend_test.go b/cmd/ateapi/internal/workerservice/suspend_test.go index 14f32a23ff..1423755d04 100644 --- a/cmd/ateapi/internal/workerservice/suspend_test.go +++ b/cmd/ateapi/internal/workerservice/suspend_test.go @@ -37,7 +37,7 @@ const ( // seedHostedActor stores an actor assigned to workerName, as the placement // path leaves one that is RUNNING. An empty workerName seeds an actor with no -// assignment at all, which is how a SUSPENDED, PAUSED, or CRASHED one looks. +// assignment at all, which is how a SUSPENDED or CRASHED one looks. func seedHostedActor(t *testing.T, st store.Interface, workerName string) *ateapipb.Actor { t.Helper() actorStatus := &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED} diff --git a/cmd/ateapi/main.go b/cmd/ateapi/main.go index ef592a0c0c..f2bcd717fa 100644 --- a/cmd/ateapi/main.go +++ b/cmd/ateapi/main.go @@ -41,6 +41,7 @@ import ( "github.com/agent-substrate/substrate/internal/oidcdiscovery" "github.com/agent-substrate/substrate/internal/oidcjwt" "github.com/agent-substrate/substrate/internal/pluginsocket" + "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/version" "github.com/agent-substrate/substrate/internal/volume" @@ -56,6 +57,7 @@ import ( "google.golang.org/grpc/credentials" "google.golang.org/grpc/keepalive" "google.golang.org/grpc/reflection" + "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" @@ -69,6 +71,7 @@ const minResyncInterval = 250 * time.Millisecond var ( listenAddr = pflag.String("grpc-listen-addr", ":443", "Address and port the gRPC server should listen on.") metricsListenAddr = pflag.String("metrics-listen-addr", ":9090", "Address and port the prometheus metrics server should listen on.") + healthListenAddr = pflag.String("health-listen-addr", ":9091", "Address and port the /readyz and /healthz probe server should listen on.") grpcServerCredBundle = pflag.String("grpc-server-cred-bundle", "", "File with the server TLS credential bundle.") authenticationConfigFile = pflag.String("authentication-config", "", "YAML file configuring trusted JWT providers.") @@ -83,6 +86,11 @@ var ( // (agent-substrate/substrate#2021) once it lands, so bootstrap owner // changes take effect without a restart. authzBootstrapOwners = pflag.StringSlice("authz-bootstrap-owners", nil, "Principal IDs that are always global owners while listed, independent of the stored global AccessPolicy. Removing an ID revokes its access on restart. At least one is required when --experimental-enable-authz is set.") + // These SPIFFE IDs hold the matching system component role when they call + // over mTLS. + controllerSPIFFEID = pflag.String("controller-spiffe-id", installdefaults.ControllerSPIFFEID(installdefaults.NamespaceFromPodEnv()), "mTLS SPIFFE ID of ate-controller. Holds the controller authz role, which manages workers.") + egressGatewaySPIFFEID = pflag.String("egress-gateway-spiffe-id", installdefaults.EgressSPIFFEID(installdefaults.NamespaceFromPodEnv()), "mTLS SPIFFE ID of atenet-egress. Holds the egress_gateway authz role, which reads and lists actors and reads their egress policies.") + ingressRouterSPIFFEID = pflag.String("ingress-router-spiffe-id", installdefaults.RouterSPIFFEID(installdefaults.NamespaceFromPodEnv()), "mTLS SPIFFE ID of atenet-router. Holds the ingress_router authz role, which resumes and lists actors.") actorIDJWTPoolFile = pflag.String("actor-id-jwt-pool", "", "The file that contains the serialized JWT authority pool for signing actor JWTs") actorJWTIssuer = pflag.String("actor-jwt-issuer", "", "Issuer URL placed in the iss claim of actor JWTs. Relying parties fetch /.well-known/openid-configuration to verify them. Must be https with no query or fragment. Required.") @@ -130,6 +138,11 @@ func main() { if err := validateActorJWTIssuer(*actorJWTIssuer); err != nil { serverboot.Fatal(ctx, "Invalid --actor-jwt-issuer", err) } + // atelet applies the same rule to EgressGateway.address on every Run and + // Restore; a value it would refuse must fail the rollout, not the actors. + if errs := resources.ValidateHostPort(field.NewPath("default-egress-gateway-address"), *defaultEgressGatewayAddress); len(errs) > 0 { + serverboot.Fatal(ctx, "Invalid --default-egress-gateway-address", errs.ToAggregate()) + } // Kept separate from ctx so that in-progress work (clients, informers) is // not cancelled the moment SIGTERM arrives. The drainOnShutdown @@ -151,6 +164,9 @@ func main() { serverboot.Fatal(ctx, "Failed to initialize metrics", err) } defer serverboot.ShutdownProvider("MeterProvider", mp.Shutdown) + // The probes have their own server, so /metrics starts here and serves + // during the store connect and the schema migrations. + go serverboot.StartMetricsServer(ctx, *metricsListenAddr) lp, err := serverboot.InitLogging(ctx, serverboot.LoggingOptions{ ServiceName: "ateapi", @@ -197,7 +213,11 @@ func main() { serverboot.Fatal(ctx, "Failed to create OpenFGA server", err) } defer fgaServer.Close() - authorizer, policyManager, err := authz.New(shutdownCtx, pool, fgaServer, *authzBootstrapOwners) + authorizer, policyManager, err := authz.New(shutdownCtx, pool, fgaServer, *authzBootstrapOwners, authz.SystemPrincipals{ + ControllerSPIFFEID: *controllerSPIFFEID, + EgressGatewaySPIFFEID: *egressGatewaySPIFFEID, + IngressRouterSPIFFEID: *ingressRouterSPIFFEID, + }) if err != nil { serverboot.Fatal(ctx, "Failed to initialize OpenFGA authz", err) } @@ -355,11 +375,7 @@ func main() { ateapipb.RegisterWorkerServiceServer(mux, workerservice.New(persistence, controlSrv, ateletSPIFFEID, actorIDCAPool)) readiness := &serverboot.Readiness{} - go serverboot.StartMetricsServer(ctx, serverboot.MetricsServerOptions{ - Addr: *metricsListenAddr, - Readiness: readiness, - EnableHealthz: true, - }) + go serverboot.StartHealthServer(ctx, *healthListenAddr, readiness) drainDone := drainOnShutdown(shutdownCtx, mux, readiness) diff --git a/cmd/atecontroller/internal/workersync/fakecontrol_test.go b/cmd/atecontroller/internal/workersync/fakecontrol_test.go index 10d59a5e83..06fbb31ef2 100644 --- a/cmd/atecontroller/internal/workersync/fakecontrol_test.go +++ b/cmd/atecontroller/internal/workersync/fakecontrol_test.go @@ -261,13 +261,13 @@ func (f *fakeControl) ListWorkers(_ context.Context, in *ateapipb.ListWorkersReq // retrying a failed page with the same token resumes from the same place. names := slices.Sorted(maps.Keys(f.workers)) start := 0 - if in.GetPageToken() != "" { + if in.GetOptions().GetPageToken() != "" { var err error - if start, err = strconv.Atoi(in.GetPageToken()); err != nil { - return nil, status.Errorf(codes.InvalidArgument, "malformed page token %q", in.GetPageToken()) + if start, err = strconv.Atoi(in.GetOptions().GetPageToken()); err != nil { + return nil, status.Errorf(codes.InvalidArgument, "malformed page token %q", in.GetOptions().GetPageToken()) } } - size := int(in.GetPageSize()) + size := int(in.GetOptions().GetPageSize()) if f.listPageSize > 0 { size = f.listPageSize } diff --git a/cmd/atecontroller/internal/workersync/syncer.go b/cmd/atecontroller/internal/workersync/syncer.go index f60291ab1d..a3551ed17b 100644 --- a/cmd/atecontroller/internal/workersync/syncer.go +++ b/cmd/atecontroller/internal/workersync/syncer.go @@ -574,7 +574,7 @@ func (s *WorkerPoolSyncer) listWorkersPageWithRetry(ctx context.Context, pageTok Cap: s.listCap, } for { - page, err := s.client.ListWorkers(ctx, &ateapipb.ListWorkersRequest{PageSize: 1000, PageToken: pageToken}) + page, err := s.client.ListWorkers(ctx, &ateapipb.ListWorkersRequest{Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: pageToken}}) if err == nil { return page, nil } diff --git a/cmd/atecontroller/internal/workersync/syncer_test.go b/cmd/atecontroller/internal/workersync/syncer_test.go index d8a1f77788..29b301d295 100644 --- a/cmd/atecontroller/internal/workersync/syncer_test.go +++ b/cmd/atecontroller/internal/workersync/syncer_test.go @@ -519,7 +519,7 @@ func TestSyncer_EnqueueRegisteredWorkers_StreamsPagesAndRetriesLatePage(t *testi // The second page fails once before serving. failed := false api.setListHook(func(req *ateapipb.ListWorkersRequest) error { - if req.GetPageToken() == "1" && !failed { + if req.GetOptions().GetPageToken() == "1" && !failed { failed = true return status.Error(codes.Unavailable, "transient failure on page") } diff --git a/cmd/atecontroller/main.go b/cmd/atecontroller/main.go index f9a4bbd298..87ce83a43a 100644 --- a/cmd/atecontroller/main.go +++ b/cmd/atecontroller/main.go @@ -36,6 +36,7 @@ import ( corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/client-go/kubernetes" clientgoscheme "k8s.io/client-go/kubernetes/scheme" @@ -44,6 +45,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/healthz" ctrlmetrics "sigs.k8s.io/controller-runtime/pkg/metrics" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) _ "k8s.io/client-go/plugin/pkg/client/auth" @@ -91,7 +93,14 @@ func init() { utilruntime.Must(clientv1alpha1.AddToScheme(scheme)) // Register our CRD } -const serviceName = "atecontroller" +const ( + serviceName = "atecontroller" + + // metricsPullAddr is the manager's metrics listener, and metricsOffAddr + // turns it off. + metricsPullAddr = ":8080" + metricsOffAddr = "0" +) // logr verbosity V(n) maps to slog level -n, so V(1) stays below Info until // --log-level=debug. logr carries no context, so these records have no trace IDs. @@ -99,6 +108,34 @@ func newControllerRuntimeLogger(h slog.Handler) logr.Logger { return logr.FromSlogHandler(h) } +// metricsBindAddr is the manager's metrics address. servePull is whether +// OTEL_METRICS_EXPORTER leaves the listener on. +func metricsBindAddr(servePull bool) string { + if servePull { + return metricsPullAddr + } + return metricsOffAddr +} + +// managerOptions configures the controller-runtime manager. +func managerOptions(egressMITMCAPool types.NamespacedName, servePull bool) ctrl.Options { + return ctrl.Options{ + Scheme: scheme, + Metrics: metricsserver.Options{BindAddress: metricsBindAddr(servePull)}, + Cache: cache.Options{ + ByObject: map[client.Object]cache.ByObject{ + &corev1.Secret{}: { + Namespaces: map[string]cache.Config{ + egressMITMCAPool.Namespace: { + FieldSelector: fields.OneTermEqualSelector("metadata.name", egressMITMCAPool.Name), + }, + }, + }, + }, + }, + } +} + func main() { pflag.Parse() if *showVersion { @@ -131,13 +168,15 @@ func main() { defer serverboot.ShutdownProvider("TracerProvider", tp.Shutdown) // controller-runtime records reconcile, workqueue, and runtime metrics into its - // own Prometheus registry, which the manager serves. On the OTLP path the - // bridged queue histograms are padded so the Telemetry API accepts idle ones. - mp, err := serverboot.InitMetricsBridged(ctx, serviceName, ctrlmetrics.Registry, padEmptyExponentialHistograms) + // own Prometheus registry. OTEL_METRICS_EXPORTER picks where they go: the + // OTLP push, which pads the bridged queue histograms so the Telemetry API + // accepts idle ones, and the manager's scrape listener. + mp, servePull, err := serverboot.InitMetricsBridged(ctx, serviceName, ctrlmetrics.Registry, padEmptyExponentialHistograms) if err != nil { serverboot.Fatal(ctx, "Failed to initialize metrics", err) } defer serverboot.ShutdownProvider("MeterProvider", mp.Shutdown) + slog.InfoContext(ctx, "Metrics listener", slog.Bool("enabled", servePull), slog.String("addr", metricsBindAddr(servePull))) k8sConfig := ctrl.GetConfigOrDie() k8sClient, err := kubernetes.NewForConfig(k8sConfig) @@ -184,20 +223,7 @@ func main() { // EgressMITMTrustReconciler watches the Secret `egress-mitm-ca-pool`. systemNamespace := installdefaults.NamespaceFromPodEnv() egressMITMCAPool := controllers.EgressMITMCAPoolRef(systemNamespace) - mgr, err := ctrl.NewManager(k8sConfig, ctrl.Options{ - Scheme: scheme, - Cache: cache.Options{ - ByObject: map[client.Object]cache.ByObject{ - &corev1.Secret{}: { - Namespaces: map[string]cache.Config{ - egressMITMCAPool.Namespace: { - FieldSelector: fields.OneTermEqualSelector("metadata.name", egressMITMCAPool.Name), - }, - }, - }, - }, - }, - }) + mgr, err := ctrl.NewManager(k8sConfig, managerOptions(egressMITMCAPool, servePull)) if err != nil { setupLog.Error(err, "unable to start manager") os.Exit(1) diff --git a/cmd/atecontroller/main_test.go b/cmd/atecontroller/main_test.go index 7d95bff394..0068d361b5 100644 --- a/cmd/atecontroller/main_test.go +++ b/cmd/atecontroller/main_test.go @@ -21,7 +21,11 @@ import ( "testing" prombridge "go.opentelemetry.io/contrib/bridges/prometheus" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/cache" ctrlmetrics "sigs.k8s.io/controller-runtime/pkg/metrics" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" ) func TestNewControllerRuntimeLoggerVerbosity(t *testing.T) { @@ -78,3 +82,61 @@ func TestBridgedRegistryProducesBeforeManagerStart(t *testing.T) { t.Errorf("no go_* family in %v", names) } } + +// The manager opens its metrics listener only when OTEL_METRICS_EXPORTER leaves +// the scrape endpoint on. +func TestManagerOptionsMetricsListener(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + servePull bool + wantAddr string + wantServed bool + }{ + {"serving", true, metricsPullAddr, true}, + {"disabled", false, metricsOffAddr, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + opts := managerOptions(types.NamespacedName{Namespace: "ate-system", Name: "pool"}, tt.servePull) + + if opts.Metrics.BindAddress != tt.wantAddr { + t.Errorf("metrics address = %q, want %q", opts.Metrics.BindAddress, tt.wantAddr) + } + srv, err := metricsserver.NewServer(opts.Metrics, nil, nil) + if err != nil { + t.Fatalf("NewServer: %v", err) + } + if served := srv != nil; served != tt.wantServed { + t.Errorf("serves a metrics listener = %t, want %t", served, tt.wantServed) + } + }) + } +} + +func TestManagerOptionsScopeSecretCache(t *testing.T) { + t.Parallel() + + ref := types.NamespacedName{Namespace: "ate-system", Name: "pool"} + opts := managerOptions(ref, true) + + var byObject cache.ByObject + found := false + for obj, by := range opts.Cache.ByObject { + if _, isSecret := obj.(*corev1.Secret); isSecret { + byObject, found = by, true + } + } + if !found { + t.Fatal("no cache scope for Secrets") + } + cfg, ok := byObject.Namespaces[ref.Namespace] + if !ok || len(byObject.Namespaces) != 1 { + t.Fatalf("Secret cache namespaces = %v, want only %q", byObject.Namespaces, ref.Namespace) + } + if got := cfg.FieldSelector.String(); got != "metadata.name="+ref.Name { + t.Errorf("Secret field selector = %q, want %q", got, "metadata.name="+ref.Name) + } +} diff --git a/cmd/atelet/imagevolume_test.go b/cmd/atelet/imagevolume_test.go index a4df056f00..e22c60cb86 100644 --- a/cmd/atelet/imagevolume_test.go +++ b/cmd/atelet/imagevolume_test.go @@ -72,7 +72,9 @@ func singleFileLayer(t *testing.T, path, body string) v1.Layer { return l } -func pushTestImage(t *testing.T, ref string, layers ...v1.Layer) { +// pushTestImage pushes the image and returns its digest-pinned reference, +// which is what validated workload specs carry. +func pushTestImage(t *testing.T, ref string, layers ...v1.Layer) string { t.Helper() img, err := mutate.AppendLayers(empty.Image, layers...) if err != nil { @@ -85,6 +87,11 @@ func pushTestImage(t *testing.T, ref string, layers ...v1.Layer) { if err := remote.Write(tag, img); err != nil { t.Fatalf("remote.Write(%q): %v", ref, err) } + digest, err := img.Digest() + if err != nil { + t.Fatalf("computing digest of %q: %v", ref, err) + } + return tag.Context().Name() + "@" + digest.String() } func newImageVolumeStore(t *testing.T) *imagecache.Store { @@ -104,8 +111,8 @@ func TestResolveImageVolumes_RecordsLayersAndDigest(t *testing.T) { pushTestImage(t, ref, singleFileLayer(t, "payload-binary", "binary")) volumes := []*ateletpb.Volume{{ - Name: "agent", - Source: &ateletpb.Volume_Image{Image: &ateletpb.ImageVolumeSource{Reference: ref}}, + Name: "agent", + Image: &ateletpb.ImageVolumeSource{Reference: ref}, }} mounts := []*ateletpb.VolumeMount{{Name: "agent", MountPath: "/ate"}} @@ -138,8 +145,8 @@ func TestResolveImageVolumes_MultiLayer(t *testing.T) { ) volumes := []*ateletpb.Volume{{ - Name: "agent", - Source: &ateletpb.Volume_Image{Image: &ateletpb.ImageVolumeSource{Reference: ref}}, + Name: "agent", + Image: &ateletpb.ImageVolumeSource{Reference: ref}, }} mounts := []*ateletpb.VolumeMount{{Name: "agent", MountPath: "/ate"}} @@ -161,8 +168,8 @@ func TestResolveImageVolumes_MultiLayer(t *testing.T) { // unused volume cannot fail the actor. func TestResolveImageVolumes_UnmountedVolumeNotPulled(t *testing.T) { volumes := []*ateletpb.Volume{{ - Name: "agent", - Source: &ateletpb.Volume_Image{Image: &ateletpb.ImageVolumeSource{Reference: "127.0.0.1:1/nope@sha256:abc"}}, + Name: "agent", + Image: &ateletpb.ImageVolumeSource{Reference: "127.0.0.1:1/nope@sha256:abc"}, }} got, err := resolveImageVolumes(t.Context(), newImageVolumeStore(t), volumes, nil) diff --git a/cmd/atelet/internal/apivalidation/validate.go b/cmd/atelet/internal/apivalidation/validate.go index cb69930019..f3a0b3de32 100644 --- a/cmd/atelet/internal/apivalidation/validate.go +++ b/cmd/atelet/internal/apivalidation/validate.go @@ -16,9 +16,11 @@ package apivalidation import ( "context" + "fmt" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" + atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -33,6 +35,31 @@ func ValidateMintActorCertificateRequest(ctx context.Context, req *ateletpb.Mint return Validate_MintActorCertificateRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) } +// ValidateRunRequest runs the generated validation for req. +func ValidateRunRequest(ctx context.Context, req *ateletpb.RunRequest) field.ErrorList { + return Validate_RunRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) +} + +// ValidateTerminateRequest runs the generated validation for req. +func ValidateTerminateRequest(ctx context.Context, req *ateletpb.TerminateRequest) field.ErrorList { + return Validate_TerminateRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) +} + +// ValidateCheckpointRequest runs the generated validation for req. +func ValidateCheckpointRequest(ctx context.Context, req *ateletpb.CheckpointRequest) field.ErrorList { + return Validate_CheckpointRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) +} + +// ValidateRestoreRequest runs the generated validation for req. +func ValidateRestoreRequest(ctx context.Context, req *ateletpb.RestoreRequest) field.ErrorList { + return Validate_RestoreRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) +} + +// ValidatePromoteSnapshotRequest runs the generated validation for req. +func ValidatePromoteSnapshotRequest(ctx context.Context, req *ateletpb.PromoteSnapshotRequest) field.ErrorList { + return Validate_PromoteSnapshotRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) +} + // ValidateRegisterWorkerRequest runs the generated validation for req. func ValidateRegisterWorkerRequest(ctx context.Context, req *ateletpb.RegisterWorkerRequest) field.ErrorList { return Validate_RegisterWorkerRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) @@ -45,6 +72,65 @@ func ValidateCustom_Limits(_ context.Context, _ operation.Operation, fldPath *fi return resources.ValidateLimit(fldPath, value.GetName(), value.GetQuantity()) } +func ValidateCustom_EgressGateway_Address(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateHostPort(fldPath, *value) +} + +// ValidateCustom_SandboxAssets_SandboxClass allows the sandbox classes of the +// SandboxConfig CRD. +func ValidateCustom_SandboxAssets_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + switch atev1alpha1.SandboxClass(*value) { + case atev1alpha1.SandboxClassGvisor, atev1alpha1.SandboxClassMicroVM: + return nil + } + return field.ErrorList{field.NotSupported(fldPath, *value, []string{ + string(atev1alpha1.SandboxClassGvisor), string(atev1alpha1.SandboxClassMicroVM), + })} +} + +func ValidateCustom_SandboxAssets_PauseImage(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidatePinnedImage(fldPath, *value) +} + +func ValidateCustom_AssetFile_Sha256(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + if err := resources.ValidateAssetHash(*value); err != nil { + return field.ErrorList{field.Invalid(fldPath, *value, err.Error())} + } + return nil +} + +// ValidateCustom_PromoteSnapshotRequest_Atespace rejects the golden atespace: +// golden actors are never paused, so a paused checkpoint is never promoted to +// a golden snapshot. +func ValidateCustom_PromoteSnapshotRequest_Atespace(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + if *value == resources.GoldenActorAtespace { + return field.ErrorList{field.Forbidden(fldPath, fmt.Sprintf("atespace %q holds golden actors, which are never paused", *value))} + } + return nil +} + +// ValidateCustom_CheckpointRequest requires snapshot.object when +// upload_config.mode is UPLOAD_MODE_IMMEDIATE. +func ValidateCustom_CheckpointRequest(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateletpb.CheckpointRequest) field.ErrorList { + if value.GetUploadConfig().GetMode() == ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE && value.GetSnapshot() != nil && value.GetSnapshot().GetObject() == nil { + return field.ErrorList{field.Required(fldPath.Child("snapshot", "object"), "required when upload_config.mode is UPLOAD_MODE_IMMEDIATE")} + } + return nil +} + +func ValidateCustom_ObjectStorage_SnapshotUri(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return validateSnapshotURI(fldPath, *value) +} + +// validateSnapshotURI requires a snapshot URI that resources.ParseSnapshotURI +// accepts. +func validateSnapshotURI(fldPath *field.Path, uri string) field.ErrorList { + if _, err := resources.ParseSnapshotURI(uri); err != nil { + return field.ErrorList{field.Invalid(fldPath, uri, err.Error())} + } + return nil +} + // ateDeepEqual is the deep-equal function declarative validation's generated // code calls by name; it delegates to resources.DeepEqual. func ateDeepEqual[T any](a, b T) bool { diff --git a/cmd/atelet/internal/apivalidation/validation_test.go b/cmd/atelet/internal/apivalidation/validation_test.go index 938a5b3e28..ef855df94a 100644 --- a/cmd/atelet/internal/apivalidation/validation_test.go +++ b/cmd/atelet/internal/apivalidation/validation_test.go @@ -21,9 +21,33 @@ import ( "testing" "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "github.com/agent-substrate/substrate/internal/resources" + "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/util/validation/field" ) +const testDigestImage = "example.com/app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + +const testSHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + +func validSandboxAssets(mutate ...func(*ateletpb.SandboxAssets)) *ateletpb.SandboxAssets { + a := &ateletpb.SandboxAssets{ + SandboxClass: "gvisor", + PauseImage: testDigestImage, + Assets: map[string]*ateletpb.ArchAssets{ + "amd64": {Files: map[string]*ateletpb.AssetFile{"gvisor": {Url: "gs://bucket/gvisor.tar.zstd", Sha256: testSHA256}}}, + "arm64": {Files: map[string]*ateletpb.AssetFile{"gvisor": {Url: "gs://bucket/gvisor.tar.zstd", Sha256: testSHA256}}}, + }, + } + for _, m := range mutate { + m(a) + } + return a +} + +// createOp is for tests of nested messages, which have no request wrapper. +var createOp = operation.Operation{Type: operation.Create} + func assertValidateErr(t *testing.T, got field.ErrorList, want field.ErrorList) { t.Helper() field.ErrorMatcher{}.ByType().ByField().ByOrigin().Test(t, want, got) @@ -171,6 +195,809 @@ func TestValidateMintActorCertificateRequest(t *testing.T) { } } +func TestValidateRunRequest(t *testing.T) { + valid := func(mutate ...func(*ateletpb.RunRequest)) *ateletpb.RunRequest { + r := &ateletpb.RunRequest{ + WorkerPodUid: "0f9a3b1c-2d4e-5f60-7182-93a4b5c6d7e8", + Atespace: "team-a", + ActorName: "actor-1", + ActorUid: "01234567-89ab-cdef-0123-456789abcdef", + ActorTemplateAtespace: "team-a", + ActorTemplateName: "tmpl-1", + Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker", Image: testDigestImage}}}, + SandboxAssets: validSandboxAssets(), + CpuMilli: 500, + MemoryBytes: 1 << 30, + } + for _, m := range mutate { + m(r) + } + return r + } + containerName := field.NewPath("spec", "containers").Index(0).Child("name") + + tests := []struct { + name string + obj *ateletpb.RunRequest + want field.ErrorList + }{{ + name: "valid", + obj: valid(), + }, { + name: "missing worker_pod_uid", + obj: valid(func(r *ateletpb.RunRequest) { r.WorkerPodUid = "" }), + want: field.ErrorList{field.Required(field.NewPath("worker_pod_uid"), "")}, + }, { + name: "invalid worker_pod_uid: path escape", + obj: valid(func(r *ateletpb.RunRequest) { r.WorkerPodUid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid worker_pod_uid: not a pod UID", + obj: valid(func(r *ateletpb.RunRequest) { r.WorkerPodUid = "ateom-1" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "missing atespace", + obj: valid(func(r *ateletpb.RunRequest) { r.Atespace = "" }), + want: field.ErrorList{field.Required(field.NewPath("atespace"), "")}, + }, { + name: "invalid atespace: path escape", + obj: valid(func(r *ateletpb.RunRequest) { r.Atespace = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing actor_name", + obj: valid(func(r *ateletpb.RunRequest) { r.ActorName = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_name"), "")}, + }, { + name: "invalid actor_name: path escape", + obj: valid(func(r *ateletpb.RunRequest) { r.ActorName = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing actor_uid", + obj: valid(func(r *ateletpb.RunRequest) { r.ActorUid = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_uid"), "")}, + }, { + name: "invalid actor_uid: path escape", + obj: valid(func(r *ateletpb.RunRequest) { r.ActorUid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid actor_template_atespace: uppercase", + obj: valid(func(r *ateletpb.RunRequest) { r.ActorTemplateAtespace = "Team-A" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_template_atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "invalid actor_template_name: trailing dash", + obj: valid(func(r *ateletpb.RunRequest) { r.ActorTemplateName = "tmpl-" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_template_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "unset template identity is allowed", + obj: valid(func(r *ateletpb.RunRequest) { + r.ActorTemplateAtespace = "" + r.ActorTemplateName = "" + }), + }, { + name: "missing spec", + obj: valid(func(r *ateletpb.RunRequest) { r.Spec = nil }), + want: field.ErrorList{field.Required(field.NewPath("spec"), "")}, + }, { + name: "missing sandbox_assets", + obj: valid(func(r *ateletpb.RunRequest) { r.SandboxAssets = nil }), + want: field.ErrorList{field.Required(field.NewPath("sandbox_assets"), "")}, + }, { + name: "sandbox asset errors surface at the sandbox_assets path", + obj: valid(func(r *ateletpb.RunRequest) { r.SandboxAssets.PauseImage = "registry.k8s.io/pause:3.10.2" }), + want: field.ErrorList{field.Invalid(field.NewPath("sandbox_assets", "pause_image"), nil, "")}, + }, { + name: "invalid container name: path escape", + obj: valid(func(r *ateletpb.RunRequest) { + r.Spec.Containers[0].Name = "../escape" + }), + want: field.ErrorList{field.Invalid(containerName, nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "duplicate container name", + obj: valid(func(r *ateletpb.RunRequest) { + r.Spec.Containers = append(r.Spec.Containers, &ateletpb.Container{Name: "worker", Image: testDigestImage}) + }), + want: field.ErrorList{field.Duplicate(field.NewPath("spec", "containers").Index(1), nil)}, + }, { + name: "egress gateway with a DNS name", + obj: valid(func(r *ateletpb.RunRequest) { + r.EgressGateway = &ateletpb.EgressGateway{Address: "atenet-egress.ate-system.svc:443"} + }), + }, { + name: "egress gateway with an IPv6 address", + obj: valid(func(r *ateletpb.RunRequest) { + r.EgressGateway = &ateletpb.EgressGateway{Address: "[fd00::1]:443"} + }), + }, { + name: "egress gateway missing address", + obj: valid(func(r *ateletpb.RunRequest) { + r.EgressGateway = &ateletpb.EgressGateway{} + }), + want: field.ErrorList{field.Required(field.NewPath("egress_gateway", "address"), "")}, + }, { + name: "egress gateway address without a port", + obj: valid(func(r *ateletpb.RunRequest) { + r.EgressGateway = &ateletpb.EgressGateway{Address: "atenet-egress.ate-system.svc"} + }), + want: field.ErrorList{field.Invalid(field.NewPath("egress_gateway", "address"), nil, "")}, + }, { + name: "egress gateway address too long", + obj: valid(func(r *ateletpb.RunRequest) { + r.EgressGateway = &ateletpb.EgressGateway{Address: strings.Repeat("a", 258) + ":443"} + }), + want: field.ErrorList{ + field.TooLong(field.NewPath("egress_gateway", "address"), nil, 261).WithOrigin("maxLength"), + field.Invalid(field.NewPath("egress_gateway", "address"), nil, ""), + }, + }, { + name: "negative cpu_milli", + obj: valid(func(r *ateletpb.RunRequest) { r.CpuMilli = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("minimum")}, + }, { + name: "cpu_milli at the bound", + obj: valid(func(r *ateletpb.RunRequest) { r.CpuMilli = 999999 }), + }, { + name: "cpu_milli over the bound", + obj: valid(func(r *ateletpb.RunRequest) { r.CpuMilli = 1000000 }), + want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("maximum")}, + }, { + name: "negative memory_bytes", + obj: valid(func(r *ateletpb.RunRequest) { r.MemoryBytes = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("memory_bytes"), nil, "").WithOrigin("minimum")}, + }, { + name: "unset sizes are allowed", + obj: valid(func(r *ateletpb.RunRequest) { + r.CpuMilli = 0 + r.MemoryBytes = 0 + }), + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, ValidateRunRequest(context.Background(), tt.obj), tt.want) + }) + } +} + +// TestValidateSandboxAssets covers SandboxAssets and the asset files it holds. +func TestValidateSandboxAssets(t *testing.T) { + file := func(url, sha string) *ateletpb.ArchAssets { + return &ateletpb.ArchAssets{Files: map[string]*ateletpb.AssetFile{"gvisor": {Url: url, Sha256: sha}}} + } + amd64 := field.NewPath("assets").Key("amd64") + amd64File := amd64.Child("files").Key("gvisor") + + tests := []struct { + name string + obj *ateletpb.SandboxAssets + want field.ErrorList + }{{ + name: "valid gvisor", + obj: validSandboxAssets(), + }, { + name: "valid microvm", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "microvm" }), + }, { + name: "missing sandbox_class", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "" }), + want: field.ErrorList{field.Required(field.NewPath("sandbox_class"), "")}, + }, { + name: "unknown sandbox_class", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "kvm" }), + want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_class"), nil, nil)}, + }, { + name: "sandbox_class is case-sensitive", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "GVISOR" }), + want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_class"), nil, nil)}, + }, { + name: "sandbox_class too long", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = strings.Repeat("g", 64) }), + want: field.ErrorList{ + field.TooLong(field.NewPath("sandbox_class"), nil, 63).WithOrigin("maxLength"), + field.NotSupported[string](field.NewPath("sandbox_class"), nil, nil), + }, + }, { + name: "valid: no pause_image (microvm has no pause container)", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.PauseImage = "" }), + }, { + name: "pause_image not pinned by digest", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.PauseImage = "registry.k8s.io/pause:3.10.2" }), + want: field.ErrorList{field.Invalid(field.NewPath("pause_image"), nil, "")}, + }, { + name: "missing assets", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets = nil }), + want: field.ErrorList{field.Required(field.NewPath("assets"), "")}, + }, { + name: "too many architectures", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { + for _, arch := range []string{"386", "arm", "loong64", "mips64", "ppc64le", "riscv64", "s390x"} { + a.Assets[arch] = file("gs://bucket/gvisor.tar.zstd", testSHA256) + } + }), + want: field.ErrorList{field.TooMany(field.NewPath("assets"), 9, 8).WithOrigin("maxProperties")}, + }, { + name: "architecture with no files", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = &ateletpb.ArchAssets{} }), + want: field.ErrorList{field.Required(amd64.Child("files"), "")}, + }, { + name: "missing url", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = file("", testSHA256) }), + want: field.ErrorList{field.Required(amd64File.Child("url"), "")}, + }, { + name: "missing sha256", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", "") }), + want: field.ErrorList{field.Required(amd64File.Child("sha256"), "")}, + }, { + name: "sha256 too short", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", "deadbeef") }), + want: field.ErrorList{field.Invalid(amd64File.Child("sha256"), nil, "")}, + }, { + name: "sha256 too long", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { + a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", testSHA256+"0") + }), + want: field.ErrorList{ + field.TooLong(amd64File.Child("sha256"), nil, 64).WithOrigin("maxLength"), + field.Invalid(amd64File.Child("sha256"), nil, ""), + }, + }, { + name: "sha256 with a path escape", + obj: validSandboxAssets(func(a *ateletpb.SandboxAssets) { + a.Assets["amd64"] = file("gs://bucket/gvisor.tar.zstd", "../"+testSHA256[3:]) + }), + want: field.ErrorList{field.Invalid(amd64File.Child("sha256"), nil, "")}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, Validate_SandboxAssets(context.Background(), createOp, nil, tt.obj, nil), tt.want) + }) + } +} + +func TestValidateCheckpointRequest(t *testing.T) { + const snapshotURI = "gs://bucket/root/atespaces/team-a/actors/01234567-89ab-cdef-0123-456789abcdef/snapshots/snap-1" + valid := func(mutate ...func(*ateletpb.CheckpointRequest)) *ateletpb.CheckpointRequest { + r := &ateletpb.CheckpointRequest{ + WorkerPodUid: "0f9a3b1c-2d4e-5f60-7182-93a4b5c6d7e8", + Atespace: "team-a", + ActorName: "actor-1", + ActorUid: "01234567-89ab-cdef-0123-456789abcdef", + ActorTemplateAtespace: "team-a", + ActorTemplateName: "tmpl-1", + Spec: &ateletpb.WorkloadSpec{}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: "snap-1", + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: snapshotURI}, + }, + UploadConfig: &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE}, + } + for _, m := range mutate { + m(r) + } + return r + } + deferred := func(name string) func(*ateletpb.CheckpointRequest) { + return func(r *ateletpb.CheckpointRequest) { + r.Snapshot.SnapshotUuid = name + r.Snapshot.Object = nil + r.UploadConfig = &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED} + } + } + + tests := []struct { + name string + obj *ateletpb.CheckpointRequest + want field.ErrorList + }{{ + name: "valid immediate", + obj: valid(), + }, { + name: "valid deferred local only", + obj: valid(deferred("pause-snap-1")), + }, { + name: "valid deferred with object storage", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.UploadConfig = &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED} + }), + }, { + name: "valid volumes fidelity", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + }), + }, { + name: "missing worker_pod_uid", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.WorkerPodUid = "" }), + want: field.ErrorList{field.Required(field.NewPath("worker_pod_uid"), "")}, + }, { + name: "invalid worker_pod_uid: path escape", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.WorkerPodUid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid worker_pod_uid: not a pod UID", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.WorkerPodUid = "ateom-1" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid atespace: path escape", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Atespace = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "invalid actor_name: uppercase", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.ActorName = "UPPER" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "invalid actor_uid: not a UUID", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.ActorUid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid actor_template_name: trailing dash", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.ActorTemplateName = "tmpl-" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_template_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "unset template identity is allowed", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.ActorTemplateAtespace = "" + r.ActorTemplateName = "" + }), + }, { + name: "missing spec", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Spec = nil }), + want: field.ErrorList{field.Required(field.NewPath("spec"), "")}, + }, { + name: "spec errors surface at the spec's path", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.Spec = &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "_pause", Image: testDigestImage}}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("spec", "containers").Index(0).Child("name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing snapshot", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot = nil }), + want: field.ErrorList{field.Required(field.NewPath("snapshot"), "")}, + }, { + name: "missing upload_config", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.UploadConfig = nil }), + want: field.ErrorList{field.Required(field.NewPath("upload_config"), "")}, + }, { + name: "unspecified upload mode", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.UploadConfig = &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_UNSPECIFIED} + }), + want: field.ErrorList{field.Required(field.NewPath("upload_config", "mode"), "")}, + }, { + name: "unknown upload mode", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.UploadConfig.Mode = ateletpb.UploadMode(3) }), + want: field.ErrorList{field.Invalid(field.NewPath("upload_config", "mode"), nil, "").WithOrigin("maximum")}, + }, { + name: "immediate upload without object storage", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot.Object = nil }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "object"), "")}, + }, { + name: "missing snapshot_uuid", + obj: valid(deferred("")), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "snapshot_uuid"), "")}, + }, { + name: "invalid snapshot_uuid: path escape", + obj: valid(deferred("../escape")), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "snapshot_uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing snapshot_uri", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot.Object.SnapshotUri = "" }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "object", "snapshot_uri"), "")}, + }, { + name: "invalid snapshot_uri: no bucket", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot.Object.SnapshotUri = "relative/path" }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "object", "snapshot_uri"), nil, "")}, + }, { + name: "snapshot_uri too long", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.Snapshot.Object.SnapshotUri = "gs://bucket/" + strings.Repeat("p", 2048) + }), + want: field.ErrorList{ + field.TooLong(field.NewPath("snapshot", "object", "snapshot_uri"), nil, 2048).WithOrigin("maxLength"), + field.Invalid(field.NewPath("snapshot", "object", "snapshot_uri"), nil, ""), + }, + }, { + name: "unspecified fidelity", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED + }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "fidelity"), "")}, + }, { + name: "unknown fidelity", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot.Fidelity = 4 }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "fidelity"), nil, "").WithOrigin("maximum")}, + }, { + name: "rootfs fidelity is forwarded for the runtime to judge", + obj: valid(func(r *ateletpb.CheckpointRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + }), + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, ValidateCheckpointRequest(context.Background(), tt.obj), tt.want) + }) + } +} + +func TestValidateRestoreRequest(t *testing.T) { + const snapshotURI = "gs://bucket/root/atespaces/team-a/actors/01234567-89ab-cdef-0123-456789abcdef/snapshots/snap-1" + valid := func(mutate ...func(*ateletpb.RestoreRequest)) *ateletpb.RestoreRequest { + r := &ateletpb.RestoreRequest{ + WorkerPodUid: "0f9a3b1c-2d4e-5f60-7182-93a4b5c6d7e8", + Atespace: "team-a", + ActorName: "actor-1", + ActorUid: "01234567-89ab-cdef-0123-456789abcdef", + ActorTemplateAtespace: "team-a", + ActorTemplateName: "tmpl-1", + Spec: &ateletpb.WorkloadSpec{}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: "snap-1", + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: snapshotURI}, + }, + SandboxAssets: validSandboxAssets(), + } + for _, m := range mutate { + m(r) + } + return r + } + local := func(name string) func(*ateletpb.RestoreRequest) { + return func(r *ateletpb.RestoreRequest) { + r.Snapshot.SnapshotUuid = name + r.Snapshot.Object = nil + } + } + + tests := []struct { + name string + obj *ateletpb.RestoreRequest + want field.ErrorList + }{{ + name: "valid with object storage", + obj: valid(), + }, { + name: "valid local only", + obj: valid(local("pause-snap-1")), + }, { + name: "valid volumes fidelity", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + }), + }, { + name: "valid size and egress", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.CpuMilli = 999999 + r.MemoryBytes = 1 << 30 + r.EgressGateway = &ateletpb.EgressGateway{Address: "10.0.0.1:15001"} + }), + }, { + name: "missing worker_pod_uid", + obj: valid(func(r *ateletpb.RestoreRequest) { r.WorkerPodUid = "" }), + want: field.ErrorList{field.Required(field.NewPath("worker_pod_uid"), "")}, + }, { + name: "invalid worker_pod_uid: path escape", + obj: valid(func(r *ateletpb.RestoreRequest) { r.WorkerPodUid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid worker_pod_uid: not a pod UID", + obj: valid(func(r *ateletpb.RestoreRequest) { r.WorkerPodUid = "ateom-1" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid atespace: path escape", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Atespace = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "invalid actor_name: uppercase", + obj: valid(func(r *ateletpb.RestoreRequest) { r.ActorName = "UPPER" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "invalid actor_uid: not a UUID", + obj: valid(func(r *ateletpb.RestoreRequest) { r.ActorUid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid actor_template_atespace: slash", + obj: valid(func(r *ateletpb.RestoreRequest) { r.ActorTemplateAtespace = "no/slashes" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_template_atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "unset template identity is allowed", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.ActorTemplateAtespace = "" + r.ActorTemplateName = "" + }), + }, { + name: "missing spec", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Spec = nil }), + want: field.ErrorList{field.Required(field.NewPath("spec"), "")}, + }, { + name: "spec errors surface at the spec's path", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.Spec = &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "_pause", Image: testDigestImage}}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("spec", "containers").Index(0).Child("name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing snapshot", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot = nil }), + want: field.ErrorList{field.Required(field.NewPath("snapshot"), "")}, + }, { + name: "missing snapshot_uuid", + obj: valid(local("")), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "snapshot_uuid"), "")}, + }, { + name: "invalid snapshot_uuid: path escape", + obj: valid(local("../escape")), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "snapshot_uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing snapshot_uri", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.Object.SnapshotUri = "" }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "object", "snapshot_uri"), "")}, + }, { + name: "invalid snapshot_uri: no bucket", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.Object.SnapshotUri = "relative/path" }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "object", "snapshot_uri"), nil, "")}, + }, { + name: "unspecified fidelity", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED + }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "fidelity"), "")}, + }, { + name: "unknown fidelity", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.Fidelity = ateletpb.SnapshotFidelity(4) }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "fidelity"), nil, "").WithOrigin("maximum")}, + }, { + name: "rootfs fidelity is forwarded for the runtime to judge", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + }), + }, { + name: "negative cpu_milli", + obj: valid(func(r *ateletpb.RestoreRequest) { r.CpuMilli = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("minimum")}, + }, { + name: "cpu_milli too large", + obj: valid(func(r *ateletpb.RestoreRequest) { r.CpuMilli = 1000000 }), + want: field.ErrorList{field.Invalid(field.NewPath("cpu_milli"), nil, "").WithOrigin("maximum")}, + }, { + name: "negative memory_bytes", + obj: valid(func(r *ateletpb.RestoreRequest) { r.MemoryBytes = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("memory_bytes"), nil, "").WithOrigin("minimum")}, + }, { + name: "empty egress gateway address", + obj: valid(func(r *ateletpb.RestoreRequest) { r.EgressGateway = &ateletpb.EgressGateway{} }), + want: field.ErrorList{field.Required(field.NewPath("egress_gateway", "address"), "")}, + }, { + name: "missing sandbox_assets", + obj: valid(func(r *ateletpb.RestoreRequest) { r.SandboxAssets = nil }), + want: field.ErrorList{field.Required(field.NewPath("sandbox_assets"), "")}, + }, { + name: "sandbox_assets errors surface at their path", + obj: valid(func(r *ateletpb.RestoreRequest) { + r.SandboxAssets = validSandboxAssets(func(a *ateletpb.SandboxAssets) { a.SandboxClass = "kata" }) + }), + want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_assets", "sandbox_class"), nil, nil)}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, ValidateRestoreRequest(context.Background(), tt.obj), tt.want) + }) + } +} + +func TestValidatePromoteSnapshotRequest(t *testing.T) { + valid := func(mutate ...func(*ateletpb.PromoteSnapshotRequest)) *ateletpb.PromoteSnapshotRequest { + r := &ateletpb.PromoteSnapshotRequest{ + Atespace: "team-a", + ActorName: "actor-1", + ActorUid: "01234567-89ab-cdef-0123-456789abcdef", + ActorTemplateAtespace: "team-a", + ActorTemplateName: "tmpl-1", + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: "pause-snap-1", + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{ + SnapshotUri: "gs://bucket/root/atespaces/team-a/actors/01234567-89ab-cdef-0123-456789abcdef/snapshots/snap-1", + }, + }, + } + for _, m := range mutate { + m(r) + } + return r + } + + tests := []struct { + name string + obj *ateletpb.PromoteSnapshotRequest + want field.ErrorList + }{{ + name: "valid", + obj: valid(), + }, { + name: "valid volumes fidelity", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + }), + }, { + name: "missing atespace", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Atespace = "" }), + want: field.ErrorList{field.Required(field.NewPath("atespace"), "")}, + }, { + name: "invalid atespace: path escape", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Atespace = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "golden atespace", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Atespace = resources.GoldenActorAtespace }), + want: field.ErrorList{field.Forbidden(field.NewPath("atespace"), "")}, + }, { + name: "invalid actor_name: uppercase", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.ActorName = "UPPER" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing actor_uid", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.ActorUid = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_uid"), "")}, + }, { + name: "invalid actor_uid: not a UUID", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.ActorUid = "actor-uid-1" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid actor_template_atespace: slash", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.ActorTemplateAtespace = "no/slashes" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_template_atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "unset template identity is allowed", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { + r.ActorTemplateAtespace = "" + r.ActorTemplateName = "" + }), + }, { + name: "missing snapshot", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot = nil }), + want: field.ErrorList{field.Required(field.NewPath("snapshot"), "")}, + }, { + name: "missing snapshot_uuid", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.SnapshotUuid = "" }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "snapshot_uuid"), "")}, + }, { + name: "invalid snapshot_uuid: path escape", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.SnapshotUuid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "snapshot_uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing snapshot.object", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.Object = nil }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "object"), "")}, + }, { + name: "missing snapshot_uri", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.Object.SnapshotUri = "" }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "object", "snapshot_uri"), "")}, + }, { + name: "invalid snapshot_uri", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.Object.SnapshotUri = "not-a-uri" }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "object", "snapshot_uri"), nil, "")}, + }, { + name: "snapshot_uri too long", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { + r.Snapshot.Object.SnapshotUri = "gs://bucket/" + strings.Repeat("p", 2048) + }), + want: field.ErrorList{ + field.TooLong(field.NewPath("snapshot", "object", "snapshot_uri"), nil, 2048).WithOrigin("maxLength"), + field.Invalid(field.NewPath("snapshot", "object", "snapshot_uri"), nil, ""), + }, + }, { + name: "unspecified fidelity", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED + }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "fidelity"), "")}, + }, { + name: "unknown fidelity", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { + r.Snapshot.Fidelity = 4 + }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "fidelity"), nil, "").WithOrigin("maximum")}, + }, { + name: "rootfs fidelity is forwarded for the runtime to judge", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { + r.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + }), + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, ValidatePromoteSnapshotRequest(context.Background(), tt.obj), tt.want) + }) + } +} + +func TestValidateTerminateRequest(t *testing.T) { + valid := func(mutate ...func(*ateletpb.TerminateRequest)) *ateletpb.TerminateRequest { + r := &ateletpb.TerminateRequest{ + WorkerPodUid: "0f9a3b1c-2d4e-5f60-7182-93a4b5c6d7e8", + Atespace: "team-a", + ActorName: "actor-1", + ActorUid: "01234567-89ab-cdef-0123-456789abcdef", + ActorTemplateAtespace: "team-a", + ActorTemplateName: "tmpl-1", + Spec: &ateletpb.WorkloadSpec{}, + } + for _, m := range mutate { + m(r) + } + return r + } + + tests := []struct { + name string + obj *ateletpb.TerminateRequest + want field.ErrorList + }{{ + name: "valid", + obj: valid(), + }, { + name: "unset worker_pod_uid is allowed: node-only cleanup", + obj: valid(func(r *ateletpb.TerminateRequest) { r.WorkerPodUid = "" }), + }, { + name: "invalid worker_pod_uid: path escape", + obj: valid(func(r *ateletpb.TerminateRequest) { r.WorkerPodUid = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid worker_pod_uid: not a pod UID", + obj: valid(func(r *ateletpb.TerminateRequest) { r.WorkerPodUid = "ateom-1" }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "missing atespace", + obj: valid(func(r *ateletpb.TerminateRequest) { r.Atespace = "" }), + want: field.ErrorList{field.Required(field.NewPath("atespace"), "")}, + }, { + name: "invalid atespace: uppercase", + obj: valid(func(r *ateletpb.TerminateRequest) { r.Atespace = "Team-A" }), + want: field.ErrorList{field.Invalid(field.NewPath("atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing actor_name", + obj: valid(func(r *ateletpb.TerminateRequest) { r.ActorName = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_name"), "")}, + }, { + name: "invalid actor_name: path escape", + obj: valid(func(r *ateletpb.TerminateRequest) { r.ActorName = "../escape" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing actor_uid", + obj: valid(func(r *ateletpb.TerminateRequest) { r.ActorUid = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_uid"), "")}, + }, { + name: "invalid actor_uid: not a uuid", + obj: valid(func(r *ateletpb.TerminateRequest) { r.ActorUid = "not-a-uuid" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + }, { + name: "invalid actor_template_atespace: uppercase", + obj: valid(func(r *ateletpb.TerminateRequest) { r.ActorTemplateAtespace = "Team-A" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_template_atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "invalid actor_template_name: trailing dash", + obj: valid(func(r *ateletpb.TerminateRequest) { r.ActorTemplateName = "tmpl-" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_template_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "spec errors surface at the spec's path", + obj: valid(func(r *ateletpb.TerminateRequest) { + r.Spec = &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "_pause", Image: testDigestImage}}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("spec", "containers").Index(0).Child("name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "unset template identity is allowed", + obj: valid(func(r *ateletpb.TerminateRequest) { + r.ActorTemplateAtespace = "" + r.ActorTemplateName = "" + }), + }, { + name: "unset spec is allowed", + obj: valid(func(r *ateletpb.TerminateRequest) { r.Spec = nil }), + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, ValidateTerminateRequest(context.Background(), tt.obj), tt.want) + }) + } +} + func TestValidateRegisterWorkerRequest(t *testing.T) { testRuntime := func() *ateletpb.SandboxRuntime { return &ateletpb.SandboxRuntime{ @@ -420,7 +1247,7 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { }, }, { name: "quantity too long", - obj: withLimits(&ateletpb.Limits{Name: "memory", Quantity: strings.Repeat("1", 33)}), + obj: withLimits(&ateletpb.Limits{Name: "memory", Quantity: "1." + strings.Repeat("0", 31)}), want: field.ErrorList{field.TooLong(limitsPath.Index(0).Child("quantity"), nil, 32).WithOrigin("maxLength")}, }, { name: "duplicate resource name", @@ -434,21 +1261,6 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { name: "malformed quantity", obj: withLimits(&ateletpb.Limits{Name: "cpu", Quantity: "not-a-quantity"}), want: field.ErrorList{field.Invalid(limitsPath.Index(0).Child("quantity"), nil, "")}, - }, { - name: "negative quantity", - obj: withLimits(&ateletpb.Limits{Name: "memory", Quantity: "-1Gi"}), - want: field.ErrorList{field.Invalid(limitsPath.Index(0).Child("quantity"), nil, "")}, - }, { - name: "zero quantity", - obj: withLimits(&ateletpb.Limits{Name: "memory", Quantity: "0"}), - want: field.ErrorList{field.Invalid(limitsPath.Index(0).Child("quantity"), nil, "")}, - }, { - name: "cpu below the bound", - obj: withLimits(&ateletpb.Limits{Name: "cpu", Quantity: "999"}), - }, { - name: "cpu at the bound", - obj: withLimits(&ateletpb.Limits{Name: "cpu", Quantity: "1000"}), - want: field.ErrorList{field.Invalid(limitsPath.Index(0).Child("quantity"), nil, "")}, }, { name: "too many limits", obj: withLimits( @@ -468,3 +1280,577 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { }) } } + +// TestValidateWorkloadSpec covers the rules WorkloadSpec owns: the volumes and +// containers lists. One nested case per list proves the element validators +// run; their own rules are covered by TestValidateVolume and +// TestValidateContainer. +func TestValidateWorkloadSpec(t *testing.T) { + ctr := func(name string) *ateletpb.Container { + return &ateletpb.Container{Name: name, Image: testDigestImage} + } + valid := func(mutate ...func(*ateletpb.WorkloadSpec)) *ateletpb.WorkloadSpec { + s := &ateletpb.WorkloadSpec{ + Volumes: []*ateletpb.Volume{{Name: "data", DurableDir: &ateletpb.DurableDirVolume{}}}, + Containers: []*ateletpb.Container{ctr("main"), ctr("sidecar")}, + } + for _, m := range mutate { + m(s) + } + return s + } + + tests := []struct { + name string + obj *ateletpb.WorkloadSpec + want field.ErrorList + }{{ + name: "valid", + obj: valid(), + }, { + name: "empty spec", + obj: &ateletpb.WorkloadSpec{}, + }, { + name: "no volumes: the delete flow may send containers only", + obj: valid(func(s *ateletpb.WorkloadSpec) { s.Volumes = nil }), + }, { + name: "duplicate volume names", + obj: valid(func(s *ateletpb.WorkloadSpec) { + s.Volumes = append(s.Volumes, &ateletpb.Volume{Name: "data", Image: &ateletpb.ImageVolumeSource{Reference: testDigestImage}}) + }), + want: field.ErrorList{field.Duplicate(field.NewPath("volumes").Index(1), nil)}, + }, { + name: "too many volumes", + obj: valid(func(s *ateletpb.WorkloadSpec) { + s.Volumes = nil + for i := range 33 { + s.Volumes = append(s.Volumes, &ateletpb.Volume{Name: fmt.Sprintf("v%d", i), DurableDir: &ateletpb.DurableDirVolume{}}) + } + }), + want: field.ErrorList{field.TooMany(field.NewPath("volumes"), 33, 32).WithOrigin("maxItems")}, + }, { + name: "volume errors surface at the volume's path", + obj: valid(func(s *ateletpb.WorkloadSpec) { s.Volumes[0].DurableDir = nil }), + want: field.ErrorList{field.Invalid(field.NewPath("volumes").Index(0), nil, "").WithOrigin("union")}, + }, { + name: "duplicate container names", + obj: valid(func(s *ateletpb.WorkloadSpec) { + s.Containers = []*ateletpb.Container{ctr("main"), ctr("main")} + }), + want: field.ErrorList{field.Duplicate(field.NewPath("containers").Index(1), nil)}, + }, { + name: "too many containers", + obj: valid(func(s *ateletpb.WorkloadSpec) { + s.Containers = nil + for _, n := range []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"} { + s.Containers = append(s.Containers, ctr(n)) + } + }), + want: field.ErrorList{field.TooMany(field.NewPath("containers"), 11, 10).WithOrigin("maxItems")}, + }, { + name: "container errors surface at the container's path", + obj: valid(func(s *ateletpb.WorkloadSpec) { + s.Containers[0].Resources = &ateletpb.ResourceLimits{CpuMillis: -1} + }), + want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(0).Child("resources", "cpu_millis"), nil, "").WithOrigin("minimum")}, + }, { + name: "mount references a declared volume", + obj: valid(func(s *ateletpb.WorkloadSpec) { + s.Containers[0].VolumeMounts = []*ateletpb.VolumeMount{{Name: "data", MountPath: "/data"}} + }), + }, { + name: "mount references an undeclared volume", + obj: valid(func(s *ateletpb.WorkloadSpec) { + s.Containers[1].VolumeMounts = []*ateletpb.VolumeMount{{Name: "data", MountPath: "/data"}, {Name: "ghost", MountPath: "/ghost"}} + }), + want: field.ErrorList{field.Invalid(field.NewPath("containers").Index(1).Child("volume_mounts").Index(1).Child("name"), nil, "")}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, Validate_WorkloadSpec(context.Background(), createOp, nil, tt.obj, nil), tt.want) + }) + } +} + +// TestValidateVolume covers Volume and every source it can hold, with errors +// asserted at their paths under the volume. +func TestValidateVolume(t *testing.T) { + durable := func(mutate ...func(*ateletpb.Volume)) *ateletpb.Volume { + v := &ateletpb.Volume{Name: "data", DurableDir: &ateletpb.DurableDirVolume{}} + for _, m := range mutate { + m(v) + } + return v + } + external := func(mutate ...func(*ateletpb.ExternalVolumeSource)) *ateletpb.Volume { + e := &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "projects/p/zones/z/disks/vol-1", + VolumeType: "substrate.io/mock", + VolumeContext: map[string]string{"fsType": "ext4"}, + } + for _, m := range mutate { + m(e) + } + return &ateletpb.Volume{Name: "data", External: e} + } + image := func(ref string) *ateletpb.Volume { + return &ateletpb.Volume{Name: "data", Image: &ateletpb.ImageVolumeSource{Reference: ref}} + } + systemInfo := func(ds ...*ateletpb.SystemInfoDataSource) *ateletpb.Volume { + return &ateletpb.Volume{Name: "data", SystemInfo: &ateletpb.SystemInfoVolume{DataSources: ds}} + } + bundle := func(name, path string) *ateletpb.SystemInfoDataSource { + return &ateletpb.SystemInfoDataSource{TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{name}, Path: path}} + } + item := func(f ateletpb.ActorMetadataField, path string) *ateletpb.ActorMetadataItem { + return &ateletpb.ActorMetadataItem{Field: f, Path: path} + } + metadata := func(items ...*ateletpb.ActorMetadataItem) *ateletpb.SystemInfoDataSource { + return &ateletpb.SystemInfoDataSource{ActorMetadata: &ateletpb.ActorMetadataDataSource{Items: items}} + } + fieldName := ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME + + extPath := field.NewPath("external") + dsPath := field.NewPath("system_info", "data_sources") + bundlePath := dsPath.Index(0).Child("trust_bundle") + itemsPath := dsPath.Index(0).Child("actor_metadata", "items") + + tests := []struct { + name string + obj *ateletpb.Volume + want field.ErrorList + }{ + // Volume. + { + name: "valid durable dir", + obj: durable(), + }, { + name: "missing name", + obj: durable(func(v *ateletpb.Volume) { v.Name = "" }), + want: field.ErrorList{field.Required(field.NewPath("name"), "")}, + }, { + name: "invalid name: uppercase", + obj: durable(func(v *ateletpb.Volume) { v.Name = "Data" }), + want: field.ErrorList{field.Invalid(field.NewPath("name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "no source set", + obj: durable(func(v *ateletpb.Volume) { v.DurableDir = nil }), + want: field.ErrorList{field.Invalid(nil, nil, "").WithOrigin("union")}, + }, { + name: "two sources set", + obj: durable(func(v *ateletpb.Volume) { + v.External = &ateletpb.ExternalVolumeSource{StorageVolumeId: "vol-1"} + }), + want: field.ErrorList{field.Invalid(nil, nil, "").WithOrigin("union")}, + }, + + // ExternalVolumeSource. + { + name: "valid external", + obj: external(), + }, { + name: "external: missing storage_volume_id", + obj: external(func(e *ateletpb.ExternalVolumeSource) { e.StorageVolumeId = "" }), + want: field.ErrorList{field.Required(extPath.Child("storage_volume_id"), "")}, + }, { + name: "external: storage_volume_id with a control character", + obj: external(func(e *ateletpb.ExternalVolumeSource) { e.StorageVolumeId = "vol\x01" }), + want: field.ErrorList{field.Invalid(extPath.Child("storage_volume_id"), nil, "")}, + }, { + name: "external: storage_volume_id too long", + obj: external(func(e *ateletpb.ExternalVolumeSource) { e.StorageVolumeId = strings.Repeat("x", 257) }), + want: field.ErrorList{field.TooLong(extPath.Child("storage_volume_id"), nil, 256).WithOrigin("maxLength")}, + }, { + name: "external: unset volume_type is allowed", + obj: external(func(e *ateletpb.ExternalVolumeSource) { e.VolumeType = "" }), + }, { + name: "external: invalid volume_type: uppercase", + obj: external(func(e *ateletpb.ExternalVolumeSource) { e.VolumeType = "substrate.io/Mock" }), + want: field.ErrorList{field.Invalid(extPath.Child("volume_type"), nil, "")}, + }, { + name: "external: volume_context with one long value, as a CSI driver may return", + obj: external(func(e *ateletpb.ExternalVolumeSource) { + e.VolumeContext = map[string]string{"k": strings.Repeat("v", 4000)} + }), + }, { + name: "external: volume_context over the size bound", + obj: external(func(e *ateletpb.ExternalVolumeSource) { + e.VolumeContext = map[string]string{"k": strings.Repeat("v", 40*1024)} + }), + want: field.ErrorList{field.TooLong(extPath.Child("volume_context"), nil, 40*1024)}, + }, { + name: "external: publish_context with many entries, as a CSI driver may return", + obj: external(func(e *ateletpb.ExternalVolumeSource) { + e.PublishContext = make(map[string]string) + for i := range 100 { + e.PublishContext[fmt.Sprintf("key-%03d", i)] = "v" + } + }), + }, { + name: "external: publish_context over the size bound", + obj: external(func(e *ateletpb.ExternalVolumeSource) { + e.PublishContext = map[string]string{"a": strings.Repeat("v", 20*1024), "b": strings.Repeat("v", 20*1024)} + }), + want: field.ErrorList{field.TooLong(extPath.Child("publish_context"), nil, 40*1024)}, + }, + + // ImageVolumeSource. + { + name: "valid image", + obj: image(testDigestImage), + }, { + name: "image: missing reference", + obj: image(""), + want: field.ErrorList{field.Required(field.NewPath("image", "reference"), "")}, + }, { + name: "image: reference not pinned by digest", + obj: image("example.com/app:v1"), + want: field.ErrorList{field.Invalid(field.NewPath("image", "reference"), nil, "")}, + }, + + // SystemInfoVolume and SystemInfoDataSource. + { + name: "valid system info", + obj: systemInfo(bundle("podcert", "a.pem"), bundle("podcert", "b.pem"), metadata(item(fieldName, "name"))), + }, { + name: "system info: data source with neither set", + obj: systemInfo(&ateletpb.SystemInfoDataSource{}), + want: field.ErrorList{field.Invalid(dsPath.Index(0), nil, "").WithOrigin("union")}, + }, { + name: "system info: data source with both set", + obj: systemInfo(&ateletpb.SystemInfoDataSource{ + ActorMetadata: &ateletpb.ActorMetadataDataSource{Items: []*ateletpb.ActorMetadataItem{item(fieldName, "name")}}, + TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{"podcert"}, Path: "p"}, + }), + want: field.ErrorList{field.Invalid(dsPath.Index(0), nil, "").WithOrigin("union")}, + }, { + name: "system info: duplicate path across trust bundles", + obj: systemInfo(bundle("podcert", "a.pem"), bundle("podcert", "a.pem")), + want: field.ErrorList{field.Duplicate(dsPath.Index(1).Child("trust_bundle", "path"), nil)}, + }, { + name: "system info: duplicate path between bundle and metadata item", + obj: systemInfo(bundle("podcert", "name"), metadata(item(fieldName, "name"))), + want: field.ErrorList{field.Duplicate(dsPath.Index(1).Child("actor_metadata", "items").Index(0).Child("path"), nil)}, + }, { + name: "system info: second actor_metadata entry", + obj: systemInfo(metadata(item(fieldName, "name")), metadata(item(fieldName, "name-2"))), + want: field.ErrorList{field.Forbidden(dsPath.Index(1).Child("actor_metadata"), "")}, + }, { + name: "system info: too many data sources", + obj: systemInfo( + bundle("podcert", "a"), bundle("podcert", "b"), bundle("podcert", "c"), + bundle("podcert", "d"), bundle("podcert", "e"), bundle("podcert", "f"), + bundle("podcert", "g"), bundle("podcert", "h"), bundle("podcert", "i"), + ), + want: field.ErrorList{field.TooMany(dsPath, 9, 8).WithOrigin("maxItems")}, + }, + + // TrustBundleDataSource. + { + name: "trust bundle: missing path", + obj: systemInfo(bundle("podcert", "")), + want: field.ErrorList{field.Required(bundlePath.Child("path"), "")}, + }, { + name: "trust bundle: absolute path", + obj: systemInfo(bundle("podcert", "/trust/bundle.pem")), + want: field.ErrorList{field.Invalid(bundlePath.Child("path"), nil, "")}, + }, { + name: "trust bundle: path too long", + obj: systemInfo(bundle("podcert", strings.Repeat("p", 256))), + want: field.ErrorList{field.TooLong(bundlePath.Child("path"), nil, 255).WithOrigin("maxLength")}, + }, { + name: "trust bundle: no names", + obj: systemInfo(&ateletpb.SystemInfoDataSource{TrustBundle: &ateletpb.TrustBundleDataSource{Path: "trust/bundle.pem"}}), + want: field.ErrorList{field.Required(bundlePath.Child("names"), "")}, + }, { + name: "trust bundle: several names", + obj: systemInfo(&ateletpb.SystemInfoDataSource{TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{"egress-mitm.ate.dev", "system-roots.ate.dev"}, Path: "trust/bundle.pem"}}), + }, { + name: "trust bundle: too many names", + obj: systemInfo(&ateletpb.SystemInfoDataSource{TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{"a", "b", "c", "d", "e", "f", "g", "h", "i"}, Path: "trust/bundle.pem"}}), + want: field.ErrorList{field.TooMany(bundlePath.Child("names"), 9, 8).WithOrigin("maxItems")}, + }, { + name: "trust bundle: duplicate names", + obj: systemInfo(&ateletpb.SystemInfoDataSource{TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{"a", "a"}, Path: "trust/bundle.pem"}}), + want: field.ErrorList{field.Duplicate(bundlePath.Child("names").Index(1), nil)}, + }, { + name: "trust bundle: empty name", + obj: systemInfo(bundle("", "trust/bundle.pem")), + want: field.ErrorList{field.TooShort(bundlePath.Child("names").Index(0), "", 1).WithOrigin("minLength")}, + }, { + name: "trust bundle: name too long", + obj: systemInfo(bundle(strings.Repeat("n", 254), "trust/bundle.pem")), + want: field.ErrorList{field.TooLong(bundlePath.Child("names").Index(0), nil, 253).WithOrigin("maxLength")}, + }, + + // ActorMetadataDataSource. + { + name: "actor metadata: several fields", + obj: systemInfo(metadata( + item(fieldName, "name"), + item(ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID, "ids/uid"), + )), + }, { + name: "actor metadata: empty items", + obj: systemInfo(metadata()), + want: field.ErrorList{field.Required(itemsPath, "")}, + }, { + name: "actor metadata: same field projected twice", + obj: systemInfo(metadata(item(fieldName, "a"), item(fieldName, "b"))), + want: field.ErrorList{field.Duplicate(itemsPath.Index(1), nil)}, + }, { + name: "actor metadata: unspecified field", + obj: systemInfo(metadata(item(ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UNSPECIFIED, "a"))), + want: field.ErrorList{field.Required(itemsPath.Index(0).Child("field"), "")}, + }, { + name: "actor metadata: field outside the enum", + obj: systemInfo(metadata(item(ateletpb.ActorMetadataField(4), "a"))), + want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("field"), nil, "").WithOrigin("maximum")}, + }, { + name: "actor metadata: escaping item path", + obj: systemInfo(metadata(item(fieldName, "../name"))), + want: field.ErrorList{field.Invalid(itemsPath.Index(0).Child("path"), nil, "")}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, Validate_Volume(context.Background(), createOp, nil, tt.obj, nil), tt.want) + }) + } +} + +// TestValidateContainer covers Container and every message it holds, with +// errors asserted at their paths under the container. +func TestValidateContainer(t *testing.T) { + valid := func(mutate ...func(*ateletpb.Container)) *ateletpb.Container { + c := &ateletpb.Container{ + Name: "main", + Image: testDigestImage, + Command: []string{"/bin/app"}, + Args: []string{"--serve"}, + Env: []*ateletpb.EnvEntry{{Name: "PORT", Value: "8080"}}, + VolumeMounts: []*ateletpb.VolumeMount{ + {Name: "data", MountPath: "/data"}, + {Name: "data", MountPath: "/mnt/data"}, + }, + } + for _, m := range mutate { + m(c) + } + return c + } + env := func(mutate func(*ateletpb.EnvEntry)) *ateletpb.Container { + return valid(func(c *ateletpb.Container) { mutate(c.Env[0]) }) + } + // mount replaces the mounts with a single one, so the nesting check + // between mounts stays out of the way. + mount := func(name, path string) *ateletpb.Container { + return valid(func(c *ateletpb.Container) { + c.VolumeMounts = []*ateletpb.VolumeMount{{Name: name, MountPath: path}} + }) + } + probe := func(mutate ...func(*ateletpb.WakeupProbe)) *ateletpb.Container { + p := &ateletpb.WakeupProbe{ + HttpGet: &ateletpb.HttpGetAction{Path: "/healthz", Port: 8080}, + TimeoutSeconds: 30, + } + for _, m := range mutate { + m(p) + } + return valid(func(c *ateletpb.Container) { c.WakeupProbe = p }) + } + caps := func(add, drop []string) *ateletpb.Container { + return valid(func(c *ateletpb.Container) { + c.SecurityContext = &ateletpb.SecurityContext{Capabilities: &ateletpb.Capabilities{Add: add, Drop: drop}} + }) + } + limits := func(r *ateletpb.ResourceLimits) *ateletpb.Container { + return valid(func(c *ateletpb.Container) { c.Resources = r }) + } + + envPath := field.NewPath("env").Index(0) + mountPath := field.NewPath("volume_mounts").Index(0) + httpGetPath := field.NewPath("wakeup_probe", "http_get") + capsPath := field.NewPath("security_context", "capabilities") + + tests := []struct { + name string + obj *ateletpb.Container + want field.ErrorList + }{ + // Container name, image, command, and args. + { + name: "valid: the same volume mounted at two paths", + obj: valid(), + }, { + name: "missing name", + obj: valid(func(c *ateletpb.Container) { c.Name = "" }), + want: field.ErrorList{field.Required(field.NewPath("name"), "")}, + }, { + name: "invalid name: uppercase", + obj: valid(func(c *ateletpb.Container) { c.Name = "Main" }), + want: field.ErrorList{field.Invalid(field.NewPath("name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing image", + obj: valid(func(c *ateletpb.Container) { c.Image = "" }), + want: field.ErrorList{field.Required(field.NewPath("image"), "")}, + }, { + name: "image not pinned by digest", + obj: valid(func(c *ateletpb.Container) { c.Image = "example.com/app:v1" }), + want: field.ErrorList{field.Invalid(field.NewPath("image"), nil, "")}, + }, { + name: "empty command and args are allowed", + obj: valid(func(c *ateletpb.Container) { + c.Command = nil + c.Args = nil + }), + }, { + name: "too many command items", + obj: valid(func(c *ateletpb.Container) { c.Command = make([]string, 65) }), + want: field.ErrorList{field.TooMany(field.NewPath("command"), 65, 64).WithOrigin("maxItems")}, + }, { + name: "arg over the length guardrail", + obj: valid(func(c *ateletpb.Container) { c.Args = []string{strings.Repeat("a", 4097)} }), + want: field.ErrorList{field.TooLong(field.NewPath("args").Index(0), nil, 4096).WithOrigin("maxLength")}, + }, + + // EnvEntry. + { + name: "duplicate env names", + obj: valid(func(c *ateletpb.Container) { + c.Env = append(c.Env, &ateletpb.EnvEntry{Name: "PORT", Value: "9"}) + }), + want: field.ErrorList{field.Duplicate(field.NewPath("env").Index(1), nil)}, + }, { + name: "env: missing name", + obj: env(func(e *ateletpb.EnvEntry) { e.Name = "" }), + want: field.ErrorList{field.Required(envPath.Child("name"), "")}, + }, { + name: "env: name with equals sign", + obj: env(func(e *ateletpb.EnvEntry) { e.Name = "A=B" }), + want: field.ErrorList{field.Invalid(envPath.Child("name"), nil, "")}, + }, { + name: "env: name too long", + obj: env(func(e *ateletpb.EnvEntry) { e.Name = strings.Repeat("N", 257) }), + want: field.ErrorList{field.TooLong(envPath.Child("name"), nil, 256).WithOrigin("maxLength")}, + }, { + name: "env: empty value is allowed", + obj: env(func(e *ateletpb.EnvEntry) { e.Value = "" }), + }, { + name: "env: value too long", + obj: env(func(e *ateletpb.EnvEntry) { e.Value = strings.Repeat("v", 32769) }), + want: field.ErrorList{field.TooLong(envPath.Child("value"), nil, 32768).WithOrigin("maxLength")}, + }, + + // VolumeMount. + { + name: "duplicate mount paths", + obj: valid(func(c *ateletpb.Container) { c.VolumeMounts[1].MountPath = "/data" }), + want: field.ErrorList{field.Duplicate(field.NewPath("volume_mounts").Index(1), nil)}, + }, { + name: "nested mount paths", + obj: valid(func(c *ateletpb.Container) { c.VolumeMounts[1].MountPath = "/data/nested" }), + want: field.ErrorList{field.Invalid(field.NewPath("volume_mounts").Index(1).Child("mount_path"), nil, "")}, + }, { + name: "mount: missing name", + obj: mount("", "/var/data"), + want: field.ErrorList{field.Required(mountPath.Child("name"), "")}, + }, { + name: "mount: invalid name: uppercase", + obj: mount("Data", "/var/data"), + want: field.ErrorList{field.Invalid(mountPath.Child("name"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "mount: missing mount_path", + obj: mount("data", ""), + want: field.ErrorList{field.Required(mountPath.Child("mount_path"), "")}, + }, { + name: "mount: relative mount_path", + obj: mount("data", "var/data"), + want: field.ErrorList{field.Invalid(mountPath.Child("mount_path"), nil, "")}, + }, + + // WakeupProbe and HttpGetAction. + { + name: "valid probe", + obj: probe(), + }, { + name: "probe: missing http_get", + obj: probe(func(p *ateletpb.WakeupProbe) { p.HttpGet = nil }), + want: field.ErrorList{field.Required(httpGetPath, "")}, + }, { + name: "probe: missing timeout", + obj: probe(func(p *ateletpb.WakeupProbe) { p.TimeoutSeconds = 0 }), + want: field.ErrorList{field.Required(field.NewPath("wakeup_probe", "timeout_seconds"), "")}, + }, { + name: "probe: timeout above the bound", + obj: probe(func(p *ateletpb.WakeupProbe) { p.TimeoutSeconds = 3601 }), + want: field.ErrorList{field.Invalid(field.NewPath("wakeup_probe", "timeout_seconds"), nil, "").WithOrigin("maximum")}, + }, { + name: "probe: missing path", + obj: probe(func(p *ateletpb.WakeupProbe) { p.HttpGet.Path = "" }), + want: field.ErrorList{field.Required(httpGetPath.Child("path"), "")}, + }, { + name: "probe: path without a leading slash", + obj: probe(func(p *ateletpb.WakeupProbe) { p.HttpGet.Path = "healthz" }), + want: field.ErrorList{field.Invalid(httpGetPath.Child("path"), nil, "")}, + }, { + name: "probe: missing port", + obj: probe(func(p *ateletpb.WakeupProbe) { p.HttpGet.Port = 0 }), + want: field.ErrorList{field.Required(httpGetPath.Child("port"), "")}, + }, { + name: "probe: port above the range", + obj: probe(func(p *ateletpb.WakeupProbe) { p.HttpGet.Port = 65536 }), + want: field.ErrorList{field.Invalid(httpGetPath.Child("port"), nil, "").WithOrigin("maximum")}, + }, + + // SecurityContext. + { + name: "valid capabilities", + obj: caps([]string{"NET_BIND_SERVICE"}, []string{"ALL"}), + }, { + name: "empty security context", + obj: valid(func(c *ateletpb.Container) { c.SecurityContext = &ateletpb.SecurityContext{} }), + }, { + name: "capabilities: add does not accept ALL", + obj: caps([]string{"ALL"}, nil), + want: field.ErrorList{field.Invalid(capsPath.Child("add").Index(0), nil, "")}, + }, { + name: "capabilities: lowercase rejected", + obj: caps(nil, []string{"net_bind_service"}), + want: field.ErrorList{field.Invalid(capsPath.Child("drop").Index(0), nil, "")}, + }, { + name: "capabilities: duplicate rejected by the set", + obj: caps([]string{"SYS_TIME", "SYS_TIME"}, nil), + want: field.ErrorList{field.Duplicate(capsPath.Child("add").Index(1), nil)}, + }, + + // ResourceLimits. + { + name: "valid limits", + obj: limits(&ateletpb.ResourceLimits{MemoryBytes: 1 << 30, CpuMillis: 500}), + }, { + name: "limits: zero means unset", + obj: limits(&ateletpb.ResourceLimits{}), + }, { + name: "limits: negative memory_bytes", + obj: limits(&ateletpb.ResourceLimits{MemoryBytes: -1}), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "memory_bytes"), nil, "").WithOrigin("minimum")}, + }, { + name: "limits: negative cpu_millis", + obj: limits(&ateletpb.ResourceLimits{CpuMillis: -1}), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "cpu_millis"), nil, "").WithOrigin("minimum")}, + }, { + name: "limits: cpu_millis at the cap", + obj: limits(&ateletpb.ResourceLimits{CpuMillis: 999999}), + }, { + name: "limits: cpu_millis above the cap", + obj: limits(&ateletpb.ResourceLimits{CpuMillis: 1000000}), + want: field.ErrorList{field.Invalid(field.NewPath("resources", "cpu_millis"), nil, "").WithOrigin("maximum")}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, Validate_Container(context.Background(), createOp, nil, tt.obj, nil), tt.want) + }) + } +} diff --git a/cmd/atelet/internal/apivalidation/workloadspec.go b/cmd/atelet/internal/apivalidation/workloadspec.go new file mode 100644 index 0000000000..c0e3f94693 --- /dev/null +++ b/cmd/atelet/internal/apivalidation/workloadspec.go @@ -0,0 +1,153 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Custom validations for the WorkloadSpec tree. The field rules are shared +// with the control plane through internal/resources: values arriving here +// already passed them at template creation, so a failure is an internal +// inconsistency, not a user error. + +package apivalidation + +import ( + "context" + + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "github.com/agent-substrate/substrate/internal/resources" + "k8s.io/apimachinery/pkg/api/operation" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/apimachinery/pkg/util/validation/field" +) + +// ValidateCustom_WorkloadSpec rejects container volume mounts that name a +// volume the spec does not declare, mirroring the template-side rule, so a +// dangling mount is refused at the RPC edge rather than after atelet has +// already reset the actor's directories and mounted its external volumes. +func ValidateCustom_WorkloadSpec(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateletpb.WorkloadSpec) field.ErrorList { + declared := sets.New[string]() + for _, vol := range value.GetVolumes() { + declared.Insert(vol.GetName()) + } + var errs field.ErrorList + for i, ctr := range value.GetContainers() { + for j, mount := range ctr.GetVolumeMounts() { + name := mount.GetName() + if name == "" { + continue // required is enforced by tags + } + if !declared.Has(name) { + errs = append(errs, field.Invalid( + fldPath.Child("containers").Index(i).Child("volume_mounts").Index(j).Child("name"), + name, "must reference a volume declared in the spec")) + } + } + } + return errs +} + +func ValidateCustom_Container_Image(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidatePinnedImage(fldPath, *value) +} + +func ValidateCustom_EnvEntry_Name(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateEnvVarName(fldPath, *value) +} + +func ValidateCustom_VolumeMount_MountPath(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateMountPath(fldPath, *value) +} + +// ValidateCustom_Container_VolumeMounts rejects nested mounts. Mount-path +// uniqueness is enforced by the list key. +func ValidateCustom_Container_VolumeMounts(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []*ateletpb.VolumeMount) field.ErrorList { + paths := make([]string, len(value)) + for i, m := range value { + paths[i] = m.GetMountPath() + } + return resources.ValidateNestedMountPaths(fldPath, paths) +} + +func ValidateCustom_Capabilities_Add(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList { + return resources.ValidateCapabilities(fldPath, value, false) +} + +func ValidateCustom_Capabilities_Drop(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList { + return resources.ValidateCapabilities(fldPath, value, true) +} + +func ValidateCustom_HttpGetAction_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateHTTPGetPath(fldPath, *value) +} + +func ValidateCustom_ExternalVolumeSource_StorageVolumeId(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateStorageVolumeID(fldPath, *value) +} + +func ValidateCustom_ExternalVolumeSource_VolumeType(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateVolumeType(fldPath, *value) +} + +func ValidateCustom_ExternalVolumeSource_VolumeContext(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ map[string]string) field.ErrorList { + return resources.ValidateCSIMap(fldPath, value) +} + +func ValidateCustom_ExternalVolumeSource_PublishContext(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ map[string]string) field.ErrorList { + return resources.ValidateCSIMap(fldPath, value) +} + +func ValidateCustom_ImageVolumeSource_Reference(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidatePinnedImage(fldPath, *value) +} + +func ValidateCustom_ActorMetadataItem_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateProjectedPath(fldPath, *value) +} + +func ValidateCustom_TrustBundleDataSource_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateProjectedPath(fldPath, *value) +} + +// ValidateCustom_SystemInfoVolume_DataSources allows at most one +// actor_metadata entry and requires every projected file path to be unique +// across all data sources: atelet writes them in order into one tree, so a +// repeated path silently clobbers the earlier file. +func ValidateCustom_SystemInfoVolume_DataSources(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []*ateletpb.SystemInfoDataSource) field.ErrorList { + var errs field.ErrorList + seen := sets.New[string]() + sawMetadata := false + for i, ds := range value { + switch { + case ds == nil: + case ds.TrustBundle != nil: + if seen.Has(ds.TrustBundle.Path) { + errs = append(errs, field.Duplicate(fldPath.Index(i).Child("trust_bundle", "path"), ds.TrustBundle.Path)) + } + seen.Insert(ds.TrustBundle.Path) + case ds.ActorMetadata != nil: + if sawMetadata { + errs = append(errs, field.Forbidden(fldPath.Index(i).Child("actor_metadata"), "at most one actor_metadata entry may appear")) + } + sawMetadata = true + for j, item := range ds.ActorMetadata.Items { + if item == nil { + continue + } + if seen.Has(item.Path) { + errs = append(errs, field.Duplicate(fldPath.Index(i).Child("actor_metadata", "items").Index(j).Child("path"), item.Path)) + } + seen.Insert(item.Path) + } + } + } + return errs +} diff --git a/cmd/atelet/internal/apivalidation/zz_generated.validation.go b/cmd/atelet/internal/apivalidation/zz_generated.validation.go index c77dd28826..4cf3036e60 100644 --- a/cmd/atelet/internal/apivalidation/zz_generated.validation.go +++ b/cmd/atelet/internal/apivalidation/zz_generated.validation.go @@ -29,48 +29,75 @@ import ( field "k8s.io/apimachinery/pkg/util/validation/field" ) -// Validate_AttributeEntry validates an instance of AttributeEntry according +// Validate_ActorMetadataDataSource validates an instance of ActorMetadataDataSource according // to declarative validation rules in the API schema. -func Validate_AttributeEntry( +func Validate_ActorMetadataDataSource( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.AttributeEntry) (errs field.ErrorList) { + obj, oldObj *ateletpb.ActorMetadataDataSource) (errs field.ErrorList) { - { // field ateletpb.AttributeEntry.Key + { // field ateletpb.ActorMetadataDataSource.Items fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj []*ateletpb.ActorMetadataItem, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.PtrSliceNoNils[ateletpb.ActorMetadataItem](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 8).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 128); len(e) != 0 { + if e := validate.MinItems(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + // lists with map semantics require unique keys + if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.ActorMetadataItem, b *ateletpb.ActorMetadataItem) bool { return a.Field == b.Field }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.ActorMetadataItem, b *ateletpb.ActorMetadataItem) bool { return a.Field == b.Field }, ateDeepEqual, Validate_ActorMetadataItem); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.AttributeEntry) *string { - return &oldObj.Key + func(oldObj *ateletpb.ActorMetadataDataSource) []*ateletpb.ActorMetadataItem { + return oldObj.Items }) - errs = append(errs, fn(fldPath.Child("key"), &obj.Key, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("items"), obj.Items, oldVal, oldObj != nil)...) } - { // field ateletpb.AttributeEntry.Value + return errs +} + +// Validate_ActorMetadataItem validates an instance of ActorMetadataItem according +// to declarative validation rules in the API schema. +func Validate_ActorMetadataItem( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.ActorMetadataItem) (errs field.ErrorList) { + + { // field ateletpb.ActorMetadataItem.Field fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj *ateletpb.ActorMetadataField, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -87,33 +114,22 @@ func Validate_AttributeEntry( if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.AttributeEntry) *string { - return &oldObj.Value + func(oldObj *ateletpb.ActorMetadataItem) *ateletpb.ActorMetadataField { + return &oldObj.Field }) - errs = append(errs, fn(fldPath.Child("value"), &obj.Value, oldVal, oldObj != nil)...) - } - - return errs -} - -// Validate_Limits validates an instance of Limits according -// to declarative validation rules in the API schema. -func Validate_Limits( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.Limits) (errs field.ErrorList) { - - // custom validation - if e := ValidateCustom_Limits(ctx, op, fldPath, obj, oldObj); len(e) != 0 { - errs = append(errs, e...) + errs = append(errs, fn(fldPath.Child("field"), &obj.Field, oldVal, oldObj != nil)...) } - { // field ateletpb.Limits.Name + { // field ateletpb.ActorMetadataItem.Path fn := func( fldPath *field.Path, obj, oldObj *string, @@ -133,60 +149,91 @@ func Validate_Limits( if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 16); len(e) != 0 { + // custom validation + if e := ValidateCustom_ActorMetadataItem_Path(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 255); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MinLength(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.Limits) *string { - return &oldObj.Name + func(oldObj *ateletpb.ActorMetadataItem) *string { + return &oldObj.Path }) - errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("path"), &obj.Path, oldVal, oldObj != nil)...) } - { // field ateletpb.Limits.Quantity + return errs +} + +// Validate_ArchAssets validates an instance of ArchAssets according +// to declarative validation rules in the API schema. +func Validate_ArchAssets( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.ArchAssets) (errs field.ErrorList) { + + { // field ateletpb.ArchAssets.Files fn := func( fldPath *field.Path, - obj, oldObj *string, + obj, oldObj map[string]*ateletpb.AssetFile, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + if ateDeepEqual(obj, oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.PtrMapNoNils[string, ateletpb.AssetFile](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxProperties(ctx, op, fldPath, obj, oldObj, 16).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.RequiredMap(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 32); len(e) != 0 { + if e := validate.EachMapKey(ctx, op, fldPath, obj, oldObj, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 64) + }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the map and call the value type's validation function + if e := validate.EachPtrMapVal(ctx, op, fldPath, obj, oldObj, ateDeepEqual, Validate_AssetFile); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.Limits) *string { - return &oldObj.Quantity + func(oldObj *ateletpb.ArchAssets) map[string]*ateletpb.AssetFile { + return oldObj.Files }) - errs = append(errs, fn(fldPath.Child("quantity"), &obj.Quantity, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("files"), obj.Files, oldVal, oldObj != nil)...) } return errs } -// Validate_MintActorCertificateRequest validates an instance of MintActorCertificateRequest according +// Validate_AssetFile validates an instance of AssetFile according // to declarative validation rules in the API schema. -func Validate_MintActorCertificateRequest( +func Validate_AssetFile( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.MintActorCertificateRequest) (errs field.ErrorList) { + obj, oldObj *ateletpb.AssetFile) (errs field.ErrorList) { - { // field ateletpb.MintActorCertificateRequest.ActorAtespace + { // field ateletpb.AssetFile.Url fn := func( fldPath *field.Path, obj, oldObj *string, @@ -206,19 +253,19 @@ func Validate_MintActorCertificateRequest( if earlyReturn { return // do not proceed } - if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 2048); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.MintActorCertificateRequest) *string { - return &oldObj.ActorAtespace + func(oldObj *ateletpb.AssetFile) *string { + return &oldObj.Url }) - errs = append(errs, fn(fldPath.Child("actor_atespace"), &obj.ActorAtespace, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("url"), &obj.Url, oldVal, oldObj != nil)...) } - { // field ateletpb.MintActorCertificateRequest.ActorName + { // field ateletpb.AssetFile.Sha256 fn := func( fldPath *field.Path, obj, oldObj *string, @@ -238,19 +285,32 @@ func Validate_MintActorCertificateRequest( if earlyReturn { return // do not proceed } - if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + // custom validation + if e := ValidateCustom_AssetFile_Sha256(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 64); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.MintActorCertificateRequest) *string { - return &oldObj.ActorName + func(oldObj *ateletpb.AssetFile) *string { + return &oldObj.Sha256 }) - errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("sha256"), &obj.Sha256, oldVal, oldObj != nil)...) } - { // field ateletpb.MintActorCertificateRequest.ActorUid + return errs +} + +// Validate_AttributeEntry validates an instance of AttributeEntry according +// to declarative validation rules in the API schema. +func Validate_AttributeEntry( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.AttributeEntry) (errs field.ErrorList) { + + { // field ateletpb.AttributeEntry.Key fn := func( fldPath *field.Path, obj, oldObj *string, @@ -270,63 +330,63 @@ func Validate_MintActorCertificateRequest( if earlyReturn { return // do not proceed } - if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 128); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.MintActorCertificateRequest) *string { - return &oldObj.ActorUid + func(oldObj *ateletpb.AttributeEntry) *string { + return &oldObj.Key }) - errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("key"), &obj.Key, oldVal, oldObj != nil)...) } - { // field ateletpb.MintActorCertificateRequest.CertificateSigningRequest + { // field ateletpb.AttributeEntry.Value fn := func( fldPath *field.Path, - obj, oldObj []byte, + obj, oldObj *string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } - if e := validate.MaxBytesSlice(ctx, op, fldPath, obj, oldObj, 16384); len(e) != 0 { + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.MintActorCertificateRequest) []byte { - return oldObj.CertificateSigningRequest + func(oldObj *ateletpb.AttributeEntry) *string { + return &oldObj.Value }) - errs = append(errs, fn(fldPath.Child("certificate_signing_request"), obj.CertificateSigningRequest, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("value"), &obj.Value, oldVal, oldObj != nil)...) } return errs } -// Validate_RegisterWorkerRequest validates an instance of RegisterWorkerRequest according +// Validate_Capabilities validates an instance of Capabilities according // to declarative validation rules in the API schema. -func Validate_RegisterWorkerRequest( +func Validate_Capabilities( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.RegisterWorkerRequest) (errs field.ErrorList) { + obj, oldObj *ateletpb.Capabilities) (errs field.ErrorList) { - { // field ateletpb.RegisterWorkerRequest.Capacity + { // field ateletpb.Capabilities.Add fn := func( fldPath *field.Path, - obj, oldObj *ateletpb.WorkerResources, + obj, oldObj []string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -336,28 +396,37 @@ func Validate_RegisterWorkerRequest( } // call field-attached validations earlyReturn := false - if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 64).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } if earlyReturn { return // do not proceed } - // call the type's validation function - errs = append(errs, Validate_WorkerResources(ctx, op, fldPath, obj, oldObj)...) + // custom validation + if e := ValidateCustom_Capabilities_Add(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + // lists with set semantics require unique values + if e := validate.ValSliceUnique(ctx, op, fldPath, obj, oldObj, validate.DirectEqual); len(e) != 0 { + errs = append(errs, e...) + } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RegisterWorkerRequest) *ateletpb.WorkerResources { - return oldObj.Capacity + func(oldObj *ateletpb.Capabilities) []string { + return oldObj.Add }) - errs = append(errs, fn(fldPath.Child("capacity"), obj.Capacity, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("add"), obj.Add, oldVal, oldObj != nil)...) } - { // field ateletpb.RegisterWorkerRequest.DefaultRuntime + { // field ateletpb.Capabilities.Drop fn := func( fldPath *field.Path, - obj, oldObj *ateletpb.SandboxRuntime, + obj, oldObj []string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -367,74 +436,80 @@ func Validate_RegisterWorkerRequest( } // call field-attached validations earlyReturn := false - if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 64).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } if earlyReturn { return // do not proceed } - // call the type's validation function - errs = append(errs, Validate_SandboxRuntime(ctx, op, fldPath, obj, oldObj)...) + // custom validation + if e := ValidateCustom_Capabilities_Drop(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + // lists with set semantics require unique values + if e := validate.ValSliceUnique(ctx, op, fldPath, obj, oldObj, validate.DirectEqual); len(e) != 0 { + errs = append(errs, e...) + } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RegisterWorkerRequest) *ateletpb.SandboxRuntime { - return oldObj.DefaultRuntime + func(oldObj *ateletpb.Capabilities) []string { + return oldObj.Drop }) - errs = append(errs, fn(fldPath.Child("default_runtime"), obj.DefaultRuntime, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("drop"), obj.Drop, oldVal, oldObj != nil)...) } - { // field ateletpb.RegisterWorkerRequest.RestorableRuntimes + return errs +} + +// Validate_CheckpointRequest validates an instance of CheckpointRequest according +// to declarative validation rules in the API schema. +func Validate_CheckpointRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.CheckpointRequest) (errs field.ErrorList) { + + // custom validation + if e := ValidateCustom_CheckpointRequest(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + + { // field ateletpb.CheckpointRequest.WorkerPodUid fn := func( fldPath *field.Path, - obj, oldObj []*ateletpb.SandboxRuntime, + obj, oldObj *string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.PtrSliceNoNils[ateletpb.SandboxRuntime](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } - if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } if earlyReturn { return // do not proceed } - // iterate the list and call the type's validation function - if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, nil, nil, Validate_SandboxRuntime); len(e) != 0 { + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RegisterWorkerRequest) []*ateletpb.SandboxRuntime { - return oldObj.RestorableRuntimes + func(oldObj *ateletpb.CheckpointRequest) *string { + return &oldObj.WorkerPodUid }) - errs = append(errs, fn(fldPath.Child("restorable_runtimes"), obj.RestorableRuntimes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("worker_pod_uid"), &obj.WorkerPodUid, oldVal, oldObj != nil)...) } - return errs -} - -// Validate_RequestActorSuspendRequest validates an instance of RequestActorSuspendRequest according -// to declarative validation rules in the API schema. -func Validate_RequestActorSuspendRequest( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.RequestActorSuspendRequest) (errs field.ErrorList) { - - { // field ateletpb.RequestActorSuspendRequest.ActorAtespace + { // field ateletpb.CheckpointRequest.Atespace fn := func( fldPath *field.Path, obj, oldObj *string, @@ -460,13 +535,13 @@ func Validate_RequestActorSuspendRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RequestActorSuspendRequest) *string { - return &oldObj.ActorAtespace + func(oldObj *ateletpb.CheckpointRequest) *string { + return &oldObj.Atespace }) - errs = append(errs, fn(fldPath.Child("actor_atespace"), &obj.ActorAtespace, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) } - { // field ateletpb.RequestActorSuspendRequest.ActorName + { // field ateletpb.CheckpointRequest.ActorName fn := func( fldPath *field.Path, obj, oldObj *string, @@ -492,13 +567,13 @@ func Validate_RequestActorSuspendRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RequestActorSuspendRequest) *string { + func(oldObj *ateletpb.CheckpointRequest) *string { return &oldObj.ActorName }) errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) } - { // field ateletpb.RequestActorSuspendRequest.ActorUid + { // field ateletpb.CheckpointRequest.ActorUid fn := func( fldPath *field.Path, obj, oldObj *string, @@ -524,77 +599,3473 @@ func Validate_RequestActorSuspendRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.RequestActorSuspendRequest) *string { + func(oldObj *ateletpb.CheckpointRequest) *string { return &oldObj.ActorUid }) errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) } - return errs -} - -// Validate_Resources validates an instance of Resources according -// to declarative validation rules in the API schema. -func Validate_Resources( - ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.Resources) (errs field.ErrorList) { - - { // field ateletpb.Resources.Limits + { // field ateletpb.CheckpointRequest.ActorTemplateAtespace fn := func( fldPath *field.Path, - obj, oldObj []*ateletpb.Limits, + obj, oldObj *string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.PtrSliceNoNils[ateletpb.Limits](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 2).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) - earlyReturn = true - } - if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } - // lists with map semantics require unique keys - if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, - func(a *ateletpb.Limits, b *ateletpb.Limits) bool { return a.Name == b.Name }); len(e) != 0 { - errs = append(errs, e...) - } - // iterate the list and call the type's validation function - if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, - func(a *ateletpb.Limits, b *ateletpb.Limits) bool { return a.Name == b.Name }, ateDeepEqual, Validate_Limits); len(e) != 0 { + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.Resources) []*ateletpb.Limits { - return oldObj.Limits + func(oldObj *ateletpb.CheckpointRequest) *string { + return &oldObj.ActorTemplateAtespace }) - errs = append(errs, fn(fldPath.Child("limits"), obj.Limits, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("actor_template_atespace"), &obj.ActorTemplateAtespace, oldVal, oldObj != nil)...) } - return errs + { // field ateletpb.CheckpointRequest.ActorTemplateName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.CheckpointRequest) *string { + return &oldObj.ActorTemplateName + }) + errs = append(errs, fn(fldPath.Child("actor_template_name"), &obj.ActorTemplateName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.CheckpointRequest.Spec + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.WorkloadSpec, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_WorkloadSpec(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.CheckpointRequest) *ateletpb.WorkloadSpec { + return oldObj.Spec + }) + errs = append(errs, fn(fldPath.Child("spec"), obj.Spec, oldVal, oldObj != nil)...) + } + + { // field ateletpb.CheckpointRequest.Snapshot + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.Snapshot, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_Snapshot(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.CheckpointRequest) *ateletpb.Snapshot { + return oldObj.Snapshot + }) + errs = append(errs, fn(fldPath.Child("snapshot"), obj.Snapshot, oldVal, oldObj != nil)...) + } + + { // field ateletpb.CheckpointRequest.UploadConfig + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.UploadConfig, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_UploadConfig(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.CheckpointRequest) *ateletpb.UploadConfig { + return oldObj.UploadConfig + }) + errs = append(errs, fn(fldPath.Child("upload_config"), obj.UploadConfig, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_Container validates an instance of Container according +// to declarative validation rules in the API schema. +func Validate_Container( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.Container) (errs field.ErrorList) { + + { // field ateletpb.Container.Name + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) *string { + return &oldObj.Name + }) + errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.Image + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_Container_Image(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 512); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) *string { + return &oldObj.Image + }) + errs = append(errs, fn(fldPath.Child("image"), &obj.Image, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.Command + fn := func( + fldPath *field.Path, + obj, oldObj []string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 64).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.EachValSliceVal(ctx, op, fldPath, obj, oldObj, nil, nil, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 4096) + }); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) []string { + return oldObj.Command + }) + errs = append(errs, fn(fldPath.Child("command"), obj.Command, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.Args + fn := func( + fldPath *field.Path, + obj, oldObj []string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 64).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.EachValSliceVal(ctx, op, fldPath, obj, oldObj, nil, nil, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 4096) + }); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) []string { + return oldObj.Args + }) + errs = append(errs, fn(fldPath.Child("args"), obj.Args, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.Env + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.EnvEntry, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.EnvEntry](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // lists with map semantics require unique keys + if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.EnvEntry, b *ateletpb.EnvEntry) bool { return a.Name == b.Name }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.EnvEntry, b *ateletpb.EnvEntry) bool { return a.Name == b.Name }, ateDeepEqual, Validate_EnvEntry); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) []*ateletpb.EnvEntry { + return oldObj.Env + }) + errs = append(errs, fn(fldPath.Child("env"), obj.Env, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.WakeupProbe + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.WakeupProbe, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_WakeupProbe(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) *ateletpb.WakeupProbe { + return oldObj.WakeupProbe + }) + errs = append(errs, fn(fldPath.Child("wakeup_probe"), obj.WakeupProbe, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.VolumeMounts + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.VolumeMount, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.VolumeMount](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_Container_VolumeMounts(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + // lists with map semantics require unique keys + if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.VolumeMount, b *ateletpb.VolumeMount) bool { return a.MountPath == b.MountPath }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.VolumeMount, b *ateletpb.VolumeMount) bool { return a.MountPath == b.MountPath }, ateDeepEqual, Validate_VolumeMount); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) []*ateletpb.VolumeMount { + return oldObj.VolumeMounts + }) + errs = append(errs, fn(fldPath.Child("volume_mounts"), obj.VolumeMounts, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.SecurityContext + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.SecurityContext, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SecurityContext(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) *ateletpb.SecurityContext { + return oldObj.SecurityContext + }) + errs = append(errs, fn(fldPath.Child("security_context"), obj.SecurityContext, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Container.Resources + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.ResourceLimits, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_ResourceLimits(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Container) *ateletpb.ResourceLimits { + return oldObj.Resources + }) + errs = append(errs, fn(fldPath.Child("resources"), obj.Resources, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_EgressGateway validates an instance of EgressGateway according +// to declarative validation rules in the API schema. +func Validate_EgressGateway( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.EgressGateway) (errs field.ErrorList) { + + { // field ateletpb.EgressGateway.Address + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_EgressGateway_Address(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 261); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.EgressGateway) *string { + return &oldObj.Address + }) + errs = append(errs, fn(fldPath.Child("address"), &obj.Address, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_EnvEntry validates an instance of EnvEntry according +// to declarative validation rules in the API schema. +func Validate_EnvEntry( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.EnvEntry) (errs field.ErrorList) { + + { // field ateletpb.EnvEntry.Name + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_EnvEntry_Name(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.EnvEntry) *string { + return &oldObj.Name + }) + errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) + } + + { // field ateletpb.EnvEntry.Value + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 32768); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.EnvEntry) *string { + return &oldObj.Value + }) + errs = append(errs, fn(fldPath.Child("value"), &obj.Value, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_ExternalVolumeSource validates an instance of ExternalVolumeSource according +// to declarative validation rules in the API schema. +func Validate_ExternalVolumeSource( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.ExternalVolumeSource) (errs field.ErrorList) { + + { // field ateletpb.ExternalVolumeSource.StorageVolumeId + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_ExternalVolumeSource_StorageVolumeId(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 256); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ExternalVolumeSource) *string { + return &oldObj.StorageVolumeId + }) + errs = append(errs, fn(fldPath.Child("storage_volume_id"), &obj.StorageVolumeId, oldVal, oldObj != nil)...) + } + + { // field ateletpb.ExternalVolumeSource.VolumeType + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_ExternalVolumeSource_VolumeType(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 253); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ExternalVolumeSource) *string { + return &oldObj.VolumeType + }) + errs = append(errs, fn(fldPath.Child("volume_type"), &obj.VolumeType, oldVal, oldObj != nil)...) + } + + { // field ateletpb.ExternalVolumeSource.VolumeContext + fn := func( + fldPath *field.Path, + obj, oldObj map[string]string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalMap(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_ExternalVolumeSource_VolumeContext(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ExternalVolumeSource) map[string]string { + return oldObj.VolumeContext + }) + errs = append(errs, fn(fldPath.Child("volume_context"), obj.VolumeContext, oldVal, oldObj != nil)...) + } + + { // field ateletpb.ExternalVolumeSource.PublishContext + fn := func( + fldPath *field.Path, + obj, oldObj map[string]string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalMap(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_ExternalVolumeSource_PublishContext(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ExternalVolumeSource) map[string]string { + return oldObj.PublishContext + }) + errs = append(errs, fn(fldPath.Child("publish_context"), obj.PublishContext, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_HttpGetAction validates an instance of HttpGetAction according +// to declarative validation rules in the API schema. +func Validate_HttpGetAction( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.HttpGetAction) (errs field.ErrorList) { + + { // field ateletpb.HttpGetAction.Path + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_HttpGetAction_Path(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 1024); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.HttpGetAction) *string { + return &oldObj.Path + }) + errs = append(errs, fn(fldPath.Child("path"), &obj.Path, oldVal, oldObj != nil)...) + } + + { // field ateletpb.HttpGetAction.Port + fn := func( + fldPath *field.Path, + obj, oldObj *int32, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 65535); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.HttpGetAction) *int32 { + return &oldObj.Port + }) + errs = append(errs, fn(fldPath.Child("port"), &obj.Port, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_ImageVolumeSource validates an instance of ImageVolumeSource according +// to declarative validation rules in the API schema. +func Validate_ImageVolumeSource( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.ImageVolumeSource) (errs field.ErrorList) { + + { // field ateletpb.ImageVolumeSource.Reference + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_ImageVolumeSource_Reference(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 512); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ImageVolumeSource) *string { + return &oldObj.Reference + }) + errs = append(errs, fn(fldPath.Child("reference"), &obj.Reference, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_Limits validates an instance of Limits according +// to declarative validation rules in the API schema. +func Validate_Limits( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.Limits) (errs field.ErrorList) { + + // custom validation + if e := ValidateCustom_Limits(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + + { // field ateletpb.Limits.Name + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 16); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Limits) *string { + return &oldObj.Name + }) + errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Limits.Quantity + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 32); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Limits) *string { + return &oldObj.Quantity + }) + errs = append(errs, fn(fldPath.Child("quantity"), &obj.Quantity, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_MintActorCertificateRequest validates an instance of MintActorCertificateRequest according +// to declarative validation rules in the API schema. +func Validate_MintActorCertificateRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.MintActorCertificateRequest) (errs field.ErrorList) { + + { // field ateletpb.MintActorCertificateRequest.ActorAtespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.MintActorCertificateRequest) *string { + return &oldObj.ActorAtespace + }) + errs = append(errs, fn(fldPath.Child("actor_atespace"), &obj.ActorAtespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.MintActorCertificateRequest.ActorName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.MintActorCertificateRequest) *string { + return &oldObj.ActorName + }) + errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.MintActorCertificateRequest.ActorUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.MintActorCertificateRequest) *string { + return &oldObj.ActorUid + }) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.MintActorCertificateRequest.CertificateSigningRequest + fn := func( + fldPath *field.Path, + obj, oldObj []byte, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.MaxBytesSlice(ctx, op, fldPath, obj, oldObj, 16384); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.MintActorCertificateRequest) []byte { + return oldObj.CertificateSigningRequest + }) + errs = append(errs, fn(fldPath.Child("certificate_signing_request"), obj.CertificateSigningRequest, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_ObjectStorage validates an instance of ObjectStorage according +// to declarative validation rules in the API schema. +func Validate_ObjectStorage( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.ObjectStorage) (errs field.ErrorList) { + + { // field ateletpb.ObjectStorage.SnapshotUri + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_ObjectStorage_SnapshotUri(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 2048); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ObjectStorage) *string { + return &oldObj.SnapshotUri + }) + errs = append(errs, fn(fldPath.Child("snapshot_uri"), &obj.SnapshotUri, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_PromoteSnapshotRequest validates an instance of PromoteSnapshotRequest according +// to declarative validation rules in the API schema. +func Validate_PromoteSnapshotRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.PromoteSnapshotRequest) (errs field.ErrorList) { + + { // field ateletpb.PromoteSnapshotRequest.Atespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_PromoteSnapshotRequest_Atespace(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.PromoteSnapshotRequest) *string { + return &oldObj.Atespace + }) + errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.PromoteSnapshotRequest.ActorName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.PromoteSnapshotRequest) *string { + return &oldObj.ActorName + }) + errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.PromoteSnapshotRequest.ActorUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.PromoteSnapshotRequest) *string { + return &oldObj.ActorUid + }) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.PromoteSnapshotRequest.ActorTemplateAtespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.PromoteSnapshotRequest) *string { + return &oldObj.ActorTemplateAtespace + }) + errs = append(errs, fn(fldPath.Child("actor_template_atespace"), &obj.ActorTemplateAtespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.PromoteSnapshotRequest.ActorTemplateName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.PromoteSnapshotRequest) *string { + return &oldObj.ActorTemplateName + }) + errs = append(errs, fn(fldPath.Child("actor_template_name"), &obj.ActorTemplateName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.PromoteSnapshotRequest.Snapshot + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.Snapshot, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + func() { // cohort = "object" + earlyReturn := false + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "object", + func(o *ateletpb.Snapshot) *ateletpb.ObjectStorage { return o.Object }, ateDeepEqual, validate.RequiredPointer).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "object", + func(o *ateletpb.Snapshot) *ateletpb.ObjectStorage { return o.Object }, ateDeepEqual, validate.OptionalPointer).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + }() + // call the type's validation function + errs = append(errs, Validate_Snapshot(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.PromoteSnapshotRequest) *ateletpb.Snapshot { + return oldObj.Snapshot + }) + errs = append(errs, fn(fldPath.Child("snapshot"), obj.Snapshot, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_RegisterWorkerRequest validates an instance of RegisterWorkerRequest according +// to declarative validation rules in the API schema. +func Validate_RegisterWorkerRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.RegisterWorkerRequest) (errs field.ErrorList) { + + { // field ateletpb.RegisterWorkerRequest.Capacity + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.WorkerResources, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_WorkerResources(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RegisterWorkerRequest) *ateletpb.WorkerResources { + return oldObj.Capacity + }) + errs = append(errs, fn(fldPath.Child("capacity"), obj.Capacity, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RegisterWorkerRequest.DefaultRuntime + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.SandboxRuntime, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SandboxRuntime(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RegisterWorkerRequest) *ateletpb.SandboxRuntime { + return oldObj.DefaultRuntime + }) + errs = append(errs, fn(fldPath.Child("default_runtime"), obj.DefaultRuntime, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RegisterWorkerRequest.RestorableRuntimes + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.SandboxRuntime, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.SandboxRuntime](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, nil, nil, Validate_SandboxRuntime); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RegisterWorkerRequest) []*ateletpb.SandboxRuntime { + return oldObj.RestorableRuntimes + }) + errs = append(errs, fn(fldPath.Child("restorable_runtimes"), obj.RestorableRuntimes, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_RequestActorSuspendRequest validates an instance of RequestActorSuspendRequest according +// to declarative validation rules in the API schema. +func Validate_RequestActorSuspendRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.RequestActorSuspendRequest) (errs field.ErrorList) { + + { // field ateletpb.RequestActorSuspendRequest.ActorAtespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RequestActorSuspendRequest) *string { + return &oldObj.ActorAtespace + }) + errs = append(errs, fn(fldPath.Child("actor_atespace"), &obj.ActorAtespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RequestActorSuspendRequest.ActorName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RequestActorSuspendRequest) *string { + return &oldObj.ActorName + }) + errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RequestActorSuspendRequest.ActorUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RequestActorSuspendRequest) *string { + return &oldObj.ActorUid + }) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_ResourceLimits validates an instance of ResourceLimits according +// to declarative validation rules in the API schema. +func Validate_ResourceLimits( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.ResourceLimits) (errs field.ErrorList) { + + { // field ateletpb.ResourceLimits.MemoryBytes + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ResourceLimits) *int64 { + return &oldObj.MemoryBytes + }) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + } + + { // field ateletpb.ResourceLimits.CpuMillis + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.ResourceLimits) *int64 { + return &oldObj.CpuMillis + }) + errs = append(errs, fn(fldPath.Child("cpu_millis"), &obj.CpuMillis, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_Resources validates an instance of Resources according +// to declarative validation rules in the API schema. +func Validate_Resources( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.Resources) (errs field.ErrorList) { + + { // field ateletpb.Resources.Limits + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.Limits, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.Limits](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 2).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // lists with map semantics require unique keys + if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.Limits, b *ateletpb.Limits) bool { return a.Name == b.Name }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.Limits, b *ateletpb.Limits) bool { return a.Name == b.Name }, ateDeepEqual, Validate_Limits); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Resources) []*ateletpb.Limits { + return oldObj.Limits + }) + errs = append(errs, fn(fldPath.Child("limits"), obj.Limits, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_RestoreRequest validates an instance of RestoreRequest according +// to declarative validation rules in the API schema. +func Validate_RestoreRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.RestoreRequest) (errs field.ErrorList) { + + { // field ateletpb.RestoreRequest.WorkerPodUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *string { + return &oldObj.WorkerPodUid + }) + errs = append(errs, fn(fldPath.Child("worker_pod_uid"), &obj.WorkerPodUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.Atespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *string { + return &oldObj.Atespace + }) + errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.ActorName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *string { + return &oldObj.ActorName + }) + errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.ActorUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *string { + return &oldObj.ActorUid + }) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.ActorTemplateAtespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *string { + return &oldObj.ActorTemplateAtespace + }) + errs = append(errs, fn(fldPath.Child("actor_template_atespace"), &obj.ActorTemplateAtespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.ActorTemplateName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *string { + return &oldObj.ActorTemplateName + }) + errs = append(errs, fn(fldPath.Child("actor_template_name"), &obj.ActorTemplateName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.Spec + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.WorkloadSpec, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_WorkloadSpec(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *ateletpb.WorkloadSpec { + return oldObj.Spec + }) + errs = append(errs, fn(fldPath.Child("spec"), obj.Spec, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.Snapshot + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.Snapshot, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_Snapshot(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *ateletpb.Snapshot { + return oldObj.Snapshot + }) + errs = append(errs, fn(fldPath.Child("snapshot"), obj.Snapshot, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.EgressGateway + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.EgressGateway, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_EgressGateway(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *ateletpb.EgressGateway { + return oldObj.EgressGateway + }) + errs = append(errs, fn(fldPath.Child("egress_gateway"), obj.EgressGateway, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.CpuMilli + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *int64 { + return &oldObj.CpuMilli + }) + errs = append(errs, fn(fldPath.Child("cpu_milli"), &obj.CpuMilli, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.MemoryBytes + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *int64 { + return &oldObj.MemoryBytes + }) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RestoreRequest.SandboxAssets + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.SandboxAssets, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SandboxAssets(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RestoreRequest) *ateletpb.SandboxAssets { + return oldObj.SandboxAssets + }) + errs = append(errs, fn(fldPath.Child("sandbox_assets"), obj.SandboxAssets, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_RunRequest validates an instance of RunRequest according +// to declarative validation rules in the API schema. +func Validate_RunRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.RunRequest) (errs field.ErrorList) { + + { // field ateletpb.RunRequest.WorkerPodUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *string { + return &oldObj.WorkerPodUid + }) + errs = append(errs, fn(fldPath.Child("worker_pod_uid"), &obj.WorkerPodUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.Atespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *string { + return &oldObj.Atespace + }) + errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.ActorName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *string { + return &oldObj.ActorName + }) + errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.ActorUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *string { + return &oldObj.ActorUid + }) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.ActorTemplateAtespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *string { + return &oldObj.ActorTemplateAtespace + }) + errs = append(errs, fn(fldPath.Child("actor_template_atespace"), &obj.ActorTemplateAtespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.ActorTemplateName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *string { + return &oldObj.ActorTemplateName + }) + errs = append(errs, fn(fldPath.Child("actor_template_name"), &obj.ActorTemplateName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.Spec + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.WorkloadSpec, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_WorkloadSpec(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *ateletpb.WorkloadSpec { + return oldObj.Spec + }) + errs = append(errs, fn(fldPath.Child("spec"), obj.Spec, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.SandboxAssets + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.SandboxAssets, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SandboxAssets(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *ateletpb.SandboxAssets { + return oldObj.SandboxAssets + }) + errs = append(errs, fn(fldPath.Child("sandbox_assets"), obj.SandboxAssets, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.EgressGateway + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.EgressGateway, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_EgressGateway(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *ateletpb.EgressGateway { + return oldObj.EgressGateway + }) + errs = append(errs, fn(fldPath.Child("egress_gateway"), obj.EgressGateway, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.CpuMilli + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 999999); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *int64 { + return &oldObj.CpuMilli + }) + errs = append(errs, fn(fldPath.Child("cpu_milli"), &obj.CpuMilli, oldVal, oldObj != nil)...) + } + + { // field ateletpb.RunRequest.MemoryBytes + fn := func( + fldPath *field.Path, + obj, oldObj *int64, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 0); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.RunRequest) *int64 { + return &oldObj.MemoryBytes + }) + errs = append(errs, fn(fldPath.Child("memory_bytes"), &obj.MemoryBytes, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_SandboxAssets validates an instance of SandboxAssets according +// to declarative validation rules in the API schema. +func Validate_SandboxAssets( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.SandboxAssets) (errs field.ErrorList) { + + { // field ateletpb.SandboxAssets.SandboxClass + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_SandboxAssets_SandboxClass(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SandboxAssets) *string { + return &oldObj.SandboxClass + }) + errs = append(errs, fn(fldPath.Child("sandbox_class"), &obj.SandboxClass, oldVal, oldObj != nil)...) + } + + { // field ateletpb.SandboxAssets.Assets + fn := func( + fldPath *field.Path, + obj, oldObj map[string]*ateletpb.ArchAssets, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrMapNoNils[string, ateletpb.ArchAssets](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxProperties(ctx, op, fldPath, obj, oldObj, 8).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.RequiredMap(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.EachMapKey(ctx, op, fldPath, obj, oldObj, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 16) + }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the map and call the value type's validation function + if e := validate.EachPtrMapVal(ctx, op, fldPath, obj, oldObj, ateDeepEqual, Validate_ArchAssets); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SandboxAssets) map[string]*ateletpb.ArchAssets { + return oldObj.Assets + }) + errs = append(errs, fn(fldPath.Child("assets"), obj.Assets, oldVal, oldObj != nil)...) + } + + { // field ateletpb.SandboxAssets.PauseImage + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_SandboxAssets_PauseImage(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 512); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SandboxAssets) *string { + return &oldObj.PauseImage + }) + errs = append(errs, fn(fldPath.Child("pause_image"), &obj.PauseImage, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_SandboxRuntime validates an instance of SandboxRuntime according +// to declarative validation rules in the API schema. +func Validate_SandboxRuntime( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.SandboxRuntime) (errs field.ErrorList) { + + { // field ateletpb.SandboxRuntime.SandboxClass + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SandboxRuntime) *string { + return &oldObj.SandboxClass + }) + errs = append(errs, fn(fldPath.Child("sandbox_class"), &obj.SandboxClass, oldVal, oldObj != nil)...) + } + + { // field ateletpb.SandboxRuntime.Version + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.VersionedSandboxCompat, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_VersionedSandboxCompat(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SandboxRuntime) *ateletpb.VersionedSandboxCompat { + return oldObj.Version + }) + errs = append(errs, fn(fldPath.Child("version"), obj.Version, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_SecurityContext validates an instance of SecurityContext according +// to declarative validation rules in the API schema. +func Validate_SecurityContext( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.SecurityContext) (errs field.ErrorList) { + + { // field ateletpb.SecurityContext.Capabilities + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.Capabilities, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_Capabilities(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SecurityContext) *ateletpb.Capabilities { + return oldObj.Capabilities + }) + errs = append(errs, fn(fldPath.Child("capabilities"), obj.Capabilities, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_Snapshot validates an instance of Snapshot according +// to declarative validation rules in the API schema. +func Validate_Snapshot( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.Snapshot) (errs field.ErrorList) { + + { // field ateletpb.Snapshot.SnapshotUuid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Snapshot) *string { + return &oldObj.SnapshotUuid + }) + errs = append(errs, fn(fldPath.Child("snapshot_uuid"), &obj.SnapshotUuid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Snapshot.Generation + fn := func( + fldPath *field.Path, + obj, oldObj *int32, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Snapshot) *int32 { + return &oldObj.Generation + }) + errs = append(errs, fn(fldPath.Child("generation"), &obj.Generation, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Snapshot.Fidelity + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.SnapshotFidelity, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Snapshot) *ateletpb.SnapshotFidelity { + return &oldObj.Fidelity + }) + errs = append(errs, fn(fldPath.Child("fidelity"), &obj.Fidelity, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Snapshot.Object + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.ObjectStorage, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_ObjectStorage(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Snapshot) *ateletpb.ObjectStorage { + return oldObj.Object + }) + errs = append(errs, fn(fldPath.Child("object"), obj.Object, oldVal, oldObj != nil)...) + } + + return errs +} + +var unionMembershipFor_github_com_agent_substrate_substrate_internal_proto_ateletpb_SystemInfoDataSource_ = validate.NewUnionMembership(validate.NewUnionMember("actor_metadata"), validate.NewUnionMember("trust_bundle")) + +// Validate_SystemInfoDataSource validates an instance of SystemInfoDataSource according +// to declarative validation rules in the API schema. +func Validate_SystemInfoDataSource( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.SystemInfoDataSource) (errs field.ErrorList) { + + if e := validate.Union(ctx, op, fldPath, obj, oldObj, unionMembershipFor_github_com_agent_substrate_substrate_internal_proto_ateletpb_SystemInfoDataSource_, + func(obj *ateletpb.SystemInfoDataSource) bool { + if obj == nil { + return false + } + return obj.ActorMetadata != nil + }, + func(obj *ateletpb.SystemInfoDataSource) bool { + if obj == nil { + return false + } + return obj.TrustBundle != nil + }); len(e) != 0 { + errs = append(errs, e...) + } + + { // field ateletpb.SystemInfoDataSource.ActorMetadata + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.ActorMetadataDataSource, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_ActorMetadataDataSource(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SystemInfoDataSource) *ateletpb.ActorMetadataDataSource { + return oldObj.ActorMetadata + }) + errs = append(errs, fn(fldPath.Child("actor_metadata"), obj.ActorMetadata, oldVal, oldObj != nil)...) + } + + { // field ateletpb.SystemInfoDataSource.TrustBundle + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.TrustBundleDataSource, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_TrustBundleDataSource(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SystemInfoDataSource) *ateletpb.TrustBundleDataSource { + return oldObj.TrustBundle + }) + errs = append(errs, fn(fldPath.Child("trust_bundle"), obj.TrustBundle, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_SystemInfoVolume validates an instance of SystemInfoVolume according +// to declarative validation rules in the API schema. +func Validate_SystemInfoVolume( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.SystemInfoVolume) (errs field.ErrorList) { + + { // field ateletpb.SystemInfoVolume.DataSources + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.SystemInfoDataSource, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.SystemInfoDataSource](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 8).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_SystemInfoVolume_DataSources(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, nil, nil, Validate_SystemInfoDataSource); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.SystemInfoVolume) []*ateletpb.SystemInfoDataSource { + return oldObj.DataSources + }) + errs = append(errs, fn(fldPath.Child("data_sources"), obj.DataSources, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_TerminateRequest validates an instance of TerminateRequest according +// to declarative validation rules in the API schema. +func Validate_TerminateRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.TerminateRequest) (errs field.ErrorList) { + + { // field ateletpb.TerminateRequest.WorkerPodUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TerminateRequest) *string { + return &oldObj.WorkerPodUid + }) + errs = append(errs, fn(fldPath.Child("worker_pod_uid"), &obj.WorkerPodUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.TerminateRequest.Atespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TerminateRequest) *string { + return &oldObj.Atespace + }) + errs = append(errs, fn(fldPath.Child("atespace"), &obj.Atespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.TerminateRequest.ActorName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TerminateRequest) *string { + return &oldObj.ActorName + }) + errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.TerminateRequest.ActorUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TerminateRequest) *string { + return &oldObj.ActorUid + }) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + } + + { // field ateletpb.TerminateRequest.ActorTemplateAtespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TerminateRequest) *string { + return &oldObj.ActorTemplateAtespace + }) + errs = append(errs, fn(fldPath.Child("actor_template_atespace"), &obj.ActorTemplateAtespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.TerminateRequest.ActorTemplateName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TerminateRequest) *string { + return &oldObj.ActorTemplateName + }) + errs = append(errs, fn(fldPath.Child("actor_template_name"), &obj.ActorTemplateName, oldVal, oldObj != nil)...) + } + + { // field ateletpb.TerminateRequest.Spec + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.WorkloadSpec, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_WorkloadSpec(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TerminateRequest) *ateletpb.WorkloadSpec { + return oldObj.Spec + }) + errs = append(errs, fn(fldPath.Child("spec"), obj.Spec, oldVal, oldObj != nil)...) + } + + return errs } -// Validate_SandboxRuntime validates an instance of SandboxRuntime according +// Validate_TrustBundleDataSource validates an instance of TrustBundleDataSource according // to declarative validation rules in the API schema. -func Validate_SandboxRuntime( +func Validate_TrustBundleDataSource( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.SandboxRuntime) (errs field.ErrorList) { + obj, oldObj *ateletpb.TrustBundleDataSource) (errs field.ErrorList) { - { // field ateletpb.SandboxRuntime.SandboxClass + { // field ateletpb.TrustBundleDataSource.Path + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_TrustBundleDataSource_Path(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 255); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MinLength(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TrustBundleDataSource) *string { + return &oldObj.Path + }) + errs = append(errs, fn(fldPath.Child("path"), &obj.Path, oldVal, oldObj != nil)...) + } + + { // field ateletpb.TrustBundleDataSource.Names + fn := func( + fldPath *field.Path, + obj, oldObj []string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 8).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.MinItems(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.EachValSliceVal(ctx, op, fldPath, obj, oldObj, validate.DirectEqual, nil, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MinLength(ctx, op, fldPath, obj, oldObj, 1) + }); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.EachValSliceVal(ctx, op, fldPath, obj, oldObj, validate.DirectEqual, nil, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 253) + }); len(e) != 0 { + errs = append(errs, e...) + } + // lists with set semantics require unique values + if e := validate.ValSliceUnique(ctx, op, fldPath, obj, oldObj, validate.DirectEqual); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.TrustBundleDataSource) []string { + return oldObj.Names + }) + errs = append(errs, fn(fldPath.Child("names"), obj.Names, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_UploadConfig validates an instance of UploadConfig according +// to declarative validation rules in the API schema. +func Validate_UploadConfig( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.UploadConfig) (errs field.ErrorList) { + + { // field ateletpb.UploadConfig.Mode + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.UploadMode, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 2); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.UploadConfig) *ateletpb.UploadMode { + return &oldObj.Mode + }) + errs = append(errs, fn(fldPath.Child("mode"), &obj.Mode, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_VersionedSandboxCompat validates an instance of VersionedSandboxCompat according +// to declarative validation rules in the API schema. +func Validate_VersionedSandboxCompat( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.VersionedSandboxCompat) (errs field.ErrorList) { + + { // field ateletpb.VersionedSandboxCompat.SchemaVersion + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 64); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.VersionedSandboxCompat) *string { + return &oldObj.SchemaVersion + }) + errs = append(errs, fn(fldPath.Child("schema_version"), &obj.SchemaVersion, oldVal, oldObj != nil)...) + } + + { // field ateletpb.VersionedSandboxCompat.Attributes + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.AttributeEntry, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.AttributeEntry](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // lists with map semantics require unique keys + if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.AttributeEntry, b *ateletpb.AttributeEntry) bool { return a.Key == b.Key }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.AttributeEntry, b *ateletpb.AttributeEntry) bool { return a.Key == b.Key }, ateDeepEqual, Validate_AttributeEntry); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.VersionedSandboxCompat) []*ateletpb.AttributeEntry { + return oldObj.Attributes + }) + errs = append(errs, fn(fldPath.Child("attributes"), obj.Attributes, oldVal, oldObj != nil)...) + } + + return errs +} + +var unionMembershipFor_github_com_agent_substrate_substrate_internal_proto_ateletpb_Volume_ = validate.NewUnionMembership(validate.NewUnionMember("durable_dir"), validate.NewUnionMember("external"), validate.NewUnionMember("system_info"), validate.NewUnionMember("image")) + +// Validate_Volume validates an instance of Volume according +// to declarative validation rules in the API schema. +func Validate_Volume( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.Volume) (errs field.ErrorList) { + + if e := validate.Union(ctx, op, fldPath, obj, oldObj, unionMembershipFor_github_com_agent_substrate_substrate_internal_proto_ateletpb_Volume_, + func(obj *ateletpb.Volume) bool { + if obj == nil { + return false + } + return obj.DurableDir != nil + }, + func(obj *ateletpb.Volume) bool { + if obj == nil { + return false + } + return obj.External != nil + }, + func(obj *ateletpb.Volume) bool { + if obj == nil { + return false + } + return obj.SystemInfo != nil + }, + func(obj *ateletpb.Volume) bool { + if obj == nil { + return false + } + return obj.Image != nil + }); len(e) != 0 { + errs = append(errs, e...) + } + + { // field ateletpb.Volume.Name fn := func( fldPath *field.Path, obj, oldObj *string, @@ -614,22 +4085,22 @@ func Validate_SandboxRuntime( if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 { + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.SandboxRuntime) *string { - return &oldObj.SandboxClass + func(oldObj *ateletpb.Volume) *string { + return &oldObj.Name }) - errs = append(errs, fn(fldPath.Child("sandbox_class"), &obj.SandboxClass, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) } - { // field ateletpb.SandboxRuntime.Version + { // field ateletpb.Volume.DurableDir fn := func( fldPath *field.Path, - obj, oldObj *ateletpb.VersionedSandboxCompat, + obj, oldObj *ateletpb.DurableDirVolume, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -639,34 +4110,121 @@ func Validate_SandboxRuntime( } // call field-attached validations earlyReturn := false - if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - errs = append(errs, e...) + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Volume) *ateletpb.DurableDirVolume { + return oldObj.DurableDir + }) + errs = append(errs, fn(fldPath.Child("durable_dir"), obj.DurableDir, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Volume.External + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.ExternalVolumeSource, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } if earlyReturn { return // do not proceed } // call the type's validation function - errs = append(errs, Validate_VersionedSandboxCompat(ctx, op, fldPath, obj, oldObj)...) + errs = append(errs, Validate_ExternalVolumeSource(ctx, op, fldPath, obj, oldObj)...) return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.SandboxRuntime) *ateletpb.VersionedSandboxCompat { - return oldObj.Version + func(oldObj *ateletpb.Volume) *ateletpb.ExternalVolumeSource { + return oldObj.External }) - errs = append(errs, fn(fldPath.Child("version"), obj.Version, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("external"), obj.External, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Volume.SystemInfo + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.SystemInfoVolume, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SystemInfoVolume(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Volume) *ateletpb.SystemInfoVolume { + return oldObj.SystemInfo + }) + errs = append(errs, fn(fldPath.Child("system_info"), obj.SystemInfo, oldVal, oldObj != nil)...) + } + + { // field ateletpb.Volume.Image + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.ImageVolumeSource, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_ImageVolumeSource(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.Volume) *ateletpb.ImageVolumeSource { + return oldObj.Image + }) + errs = append(errs, fn(fldPath.Child("image"), obj.Image, oldVal, oldObj != nil)...) } return errs } -// Validate_VersionedSandboxCompat validates an instance of VersionedSandboxCompat according +// Validate_VolumeMount validates an instance of VolumeMount according // to declarative validation rules in the API schema. -func Validate_VersionedSandboxCompat( +func Validate_VolumeMount( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateletpb.VersionedSandboxCompat) (errs field.ErrorList) { + obj, oldObj *ateletpb.VolumeMount) (errs field.ErrorList) { - { // field ateletpb.VersionedSandboxCompat.SchemaVersion + { // field ateletpb.VolumeMount.Name fn := func( fldPath *field.Path, obj, oldObj *string, @@ -686,63 +4244,127 @@ func Validate_VersionedSandboxCompat( if earlyReturn { return // do not proceed } - if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 64); len(e) != 0 { + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.VersionedSandboxCompat) *string { - return &oldObj.SchemaVersion + func(oldObj *ateletpb.VolumeMount) *string { + return &oldObj.Name }) - errs = append(errs, fn(fldPath.Child("schema_version"), &obj.SchemaVersion, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("name"), &obj.Name, oldVal, oldObj != nil)...) } - { // field ateletpb.VersionedSandboxCompat.Attributes + { // field ateletpb.VolumeMount.MountPath fn := func( fldPath *field.Path, - obj, oldObj []*ateletpb.AttributeEntry, + obj, oldObj *string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { - if ateDeepEqual(obj, oldObj) { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { return nil } } // call field-attached validations earlyReturn := false - if e := validate.PtrSliceNoNils[ateletpb.AttributeEntry](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } - if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + if earlyReturn { + return // do not proceed + } + // custom validation + if e := ValidateCustom_VolumeMount_MountPath(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 4096); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.VolumeMount) *string { + return &oldObj.MountPath + }) + errs = append(errs, fn(fldPath.Child("mount_path"), &obj.MountPath, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_WakeupProbe validates an instance of WakeupProbe according +// to declarative validation rules in the API schema. +func Validate_WakeupProbe( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.WakeupProbe) (errs field.ErrorList) { + + { // field ateletpb.WakeupProbe.HttpGet + fn := func( + fldPath *field.Path, + obj, oldObj *ateletpb.HttpGetAction, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } - if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_HttpGetAction(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.WakeupProbe) *ateletpb.HttpGetAction { + return oldObj.HttpGet + }) + errs = append(errs, fn(fldPath.Child("http_get"), obj.HttpGet, oldVal, oldObj != nil)...) + } + + { // field ateletpb.WakeupProbe.TimeoutSeconds + fn := func( + fldPath *field.Path, + obj, oldObj *int32, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } - // lists with map semantics require unique keys - if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, - func(a *ateletpb.AttributeEntry, b *ateletpb.AttributeEntry) bool { return a.Key == b.Key }); len(e) != 0 { + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3600); len(e) != 0 { errs = append(errs, e...) } - // iterate the list and call the type's validation function - if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, - func(a *ateletpb.AttributeEntry, b *ateletpb.AttributeEntry) bool { return a.Key == b.Key }, ateDeepEqual, Validate_AttributeEntry); len(e) != 0 { + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateletpb.VersionedSandboxCompat) []*ateletpb.AttributeEntry { - return oldObj.Attributes + func(oldObj *ateletpb.WakeupProbe) *int32 { + return &oldObj.TimeoutSeconds }) - errs = append(errs, fn(fldPath.Child("attributes"), obj.Attributes, oldVal, oldObj != nil)...) + errs = append(errs, fn(fldPath.Child("timeout_seconds"), &obj.TimeoutSeconds, oldVal, oldObj != nil)...) } return errs @@ -817,3 +4439,109 @@ func Validate_WorkerResources( return errs } + +// Validate_WorkloadSpec validates an instance of WorkloadSpec according +// to declarative validation rules in the API schema. +func Validate_WorkloadSpec( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.WorkloadSpec) (errs field.ErrorList) { + + // custom validation + if e := ValidateCustom_WorkloadSpec(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + + { // field ateletpb.WorkloadSpec.Containers + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.Container, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.Container](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 10).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // lists with map semantics require unique keys + if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.Container, b *ateletpb.Container) bool { return a.Name == b.Name }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.Container, b *ateletpb.Container) bool { return a.Name == b.Name }, ateDeepEqual, Validate_Container); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.WorkloadSpec) []*ateletpb.Container { + return oldObj.Containers + }) + errs = append(errs, fn(fldPath.Child("containers"), obj.Containers, oldVal, oldObj != nil)...) + } + + { // field ateletpb.WorkloadSpec.Volumes + fn := func( + fldPath *field.Path, + obj, oldObj []*ateletpb.Volume, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.PtrSliceNoNils[ateletpb.Volume](ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 32).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.OptionalSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // lists with map semantics require unique keys + if e := validate.PtrSliceUnique(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.Volume, b *ateletpb.Volume) bool { return a.Name == b.Name }); len(e) != 0 { + errs = append(errs, e...) + } + // iterate the list and call the type's validation function + if e := validate.EachPtrSliceVal(ctx, op, fldPath, obj, oldObj, + func(a *ateletpb.Volume, b *ateletpb.Volume) bool { return a.Name == b.Name }, ateDeepEqual, Validate_Volume); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.WorkloadSpec) []*ateletpb.Volume { + return oldObj.Volumes + }) + errs = append(errs, fn(fldPath.Child("volumes"), obj.Volumes, oldVal, oldObj != nil)...) + } + + return errs +} diff --git a/cmd/atelet/internal/ateletpath/ateletpath.go b/cmd/atelet/internal/ateletpath/ateletpath.go index 335f05c748..9a5ff54907 100644 --- a/cmd/atelet/internal/ateletpath/ateletpath.go +++ b/cmd/atelet/internal/ateletpath/ateletpath.go @@ -36,8 +36,9 @@ var ( SnapshotScratchDir = filepath.Join(nodepath.BasePath, "snapshot-scratch") ) -func RunSCBinaryPath(sha256 string) string { - return filepath.Join(nodepath.StaticFilesDir, "runsc-"+sha256) +// AssetPath returns the cache path for a single-file sandbox asset +func AssetPath(sha256 string) string { + return filepath.Join(nodepath.StaticFilesDir, "asset-"+sha256) } // GVisorReleaseDir is the directory a gVisor release tarball (gvisor.tar.bz2, diff --git a/cmd/atelet/internal/credentialprovider/credentialprovider_test.go b/cmd/atelet/internal/credentialprovider/credentialprovider_test.go index 53b11328a1..0bab7d1a64 100644 --- a/cmd/atelet/internal/credentialprovider/credentialprovider_test.go +++ b/cmd/atelet/internal/credentialprovider/credentialprovider_test.go @@ -33,33 +33,38 @@ import ( "github.com/google/go-containerregistry/pkg/name" ) -// fakePlugin writes an executable named name into dir that records each -// request it is handed (one JSON object per line in /.requests) and -// replies with response. It returns the requests file's path. +// linkPlugin symlinks / to a pre-existing executable script +// in testdata/. Pre-existing scripts are used instead of writing executable +// files during the test to avoid ETXTBSY races when parallel tests fork+exec +// plugins concurrently (https://go.dev/issue/22315). +func linkPlugin(t *testing.T, dir, pluginName, testdataScript string) { + t.Helper() + target, err := filepath.Abs(filepath.Join("testdata", testdataScript)) + if err != nil { + t.Fatalf("Failed to resolve testdata script %q: %v", testdataScript, err) + } + if err := os.Symlink(target, filepath.Join(dir, pluginName)); err != nil { + t.Fatalf("Failed to symlink plugin %q: %v", pluginName, err) + } +} + +// fakePlugin links an executable named pluginName into dir that records each +// request it is handed (one JSON object per line in /.requests) +// and replies with response. It returns the requests file's path. func fakePlugin(t *testing.T, dir, pluginName, response string) string { t.Helper() - requests := filepath.Join(dir, pluginName+".requests") - script := fmt.Sprintf(`#!/bin/sh -cat >> %q -echo >> %q -cat <<'RESPONSE' -%s -RESPONSE -`, requests, requests, response) - if err := os.WriteFile(filepath.Join(dir, pluginName), []byte(script), 0o700); err != nil { - t.Fatalf("Failed to write fake plugin: %v", err) - } - return requests + if err := os.WriteFile(filepath.Join(dir, pluginName+".response"), []byte(response), 0o600); err != nil { + t.Fatalf("Failed to write plugin response: %v", err) + } + linkPlugin(t, dir, pluginName, "fake-provider.sh") + return filepath.Join(dir, pluginName+".requests") } -// failingPlugin writes an executable that prints message to stderr and exits +// failingPlugin links an executable that prints an error to stderr and exits // non-zero. -func failingPlugin(t *testing.T, dir, pluginName, message string) { +func failingPlugin(t *testing.T, dir, pluginName string) { t.Helper() - script := fmt.Sprintf("#!/bin/sh\necho %q >&2\nexit 7\n", message) - if err := os.WriteFile(filepath.Join(dir, pluginName), []byte(script), 0o700); err != nil { - t.Fatalf("Failed to write failing plugin: %v", err) - } + linkPlugin(t, dir, pluginName, "failing-provider.sh") } // writeConfig writes a CredentialProviderConfig into dir and returns its path. @@ -159,7 +164,7 @@ func TestKeychainSkipsUnmatchedImage(t *testing.T) { dir := t.TempDir() // An image no provider claims must not exec anything, so point the config // at a plugin that would fail loudly if it ever ran. - failingPlugin(t, dir, "fake-provider", "plugin must not be invoked") + failingPlugin(t, dir, "fake-provider") kc, err := New(writeConfig(t, dir, gcpLikeConfig), dir) if err != nil { @@ -218,7 +223,7 @@ func TestKeychainSkipsFailingPlugin(t *testing.T) { { name: "plugin exits non-zero", plugin: func(t *testing.T, dir string) { - failingPlugin(t, dir, "fake-provider", "metadata server unreachable") + failingPlugin(t, dir, "fake-provider") }, wantLog: "metadata server unreachable", }, @@ -394,7 +399,7 @@ func TestKeychainCandidatesSkipInvalidAuthKey(t *testing.T) { func TestKeychainCandidatesSkipFailingProvider(t *testing.T) { t.Parallel() dir := t.TempDir() - failingPlugin(t, dir, "first-provider", "metadata server unreachable") + failingPlugin(t, dir, "first-provider") fakePlugin(t, dir, "second-provider", `{ "kind": "CredentialProviderResponse", "apiVersion": "credentialprovider.kubelet.k8s.io/v1", @@ -653,15 +658,7 @@ func TestKeychainPassesConfiguredEnv(t *testing.T) { dir := t.TempDir() // The plugin echoes an env var back as the username, so the assertion // covers both the configured env and the inherited process env. - script := `#!/bin/sh -cat > /dev/null -cat < /dev/null -cat < /dev/null +cat < /dev/null +cat <&2 +exit 7 diff --git a/cmd/atelet/internal/credentialprovider/testdata/fake-provider.sh b/cmd/atelet/internal/credentialprovider/testdata/fake-provider.sh new file mode 100755 index 0000000000..38ba6f4bec --- /dev/null +++ b/cmd/atelet/internal/credentialprovider/testdata/fake-provider.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -o errexit -o nounset -o pipefail + +cat >> "$0.requests" +echo >> "$0.requests" +cat "$0.response" diff --git a/cmd/atelet/internal/credentialprovider/testdata/sleep-provider.sh b/cmd/atelet/internal/credentialprovider/testdata/sleep-provider.sh new file mode 100755 index 0000000000..a0828483fe --- /dev/null +++ b/cmd/atelet/internal/credentialprovider/testdata/sleep-provider.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash + +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -o errexit -o nounset -o pipefail + +sleep 60 diff --git a/cmd/atelet/lifecycle_test.go b/cmd/atelet/lifecycle_test.go index c592bfb734..7d272cfd1e 100644 --- a/cmd/atelet/lifecycle_test.go +++ b/cmd/atelet/lifecycle_test.go @@ -172,8 +172,8 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { const ( atespace = "ate-demo" actorName = "counter" - actorUID = "actor-uid-1" - workerPodUID = "worker-pod-uid-1" + actorUID = "01234567-89ab-cdef-0123-456789abcdef" + workerPodUID = "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f" snapshotName = "pause-snap-1" ) @@ -182,7 +182,7 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { host := imageVolumeTestRegistry(t) image := host + "/actor:v1" - pushTestImage(t, image, singleFileLayer(t, "bin/app", "app")) + pinnedImage := pushTestImage(t, image, singleFileLayer(t, "bin/app", "app")) // A single "runsc" asset served from a fake bucket: enough to exercise the // content-addressed asset fetch without a gVisor release tarball. @@ -195,7 +195,7 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { } sandboxAssets := &ateletpb.SandboxAssets{ SandboxClass: "gvisor", - PauseImage: image, + PauseImage: pinnedImage, Assets: map[string]*ateletpb.ArchAssets{ runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{ runscAssetName: { @@ -206,7 +206,7 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { }, } spec := &ateletpb.WorkloadSpec{ - Containers: []*ateletpb.Container{{Name: "app", Image: image, Command: []string{"/bin/app"}}}, + Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}}, } if _, err := s.Run(ctx, &ateletpb.RunRequest{ @@ -231,11 +231,11 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { ActorTemplateName: "counter", WorkerPodUid: workerPodUID, Spec: spec, - Fidelity: fidelity, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - Config: &ateletpb.CheckpointRequest_LocalConfig{ - LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: snapshotName}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: fidelity, }, + UploadConfig: &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED}, }); err != nil { t.Fatalf("Checkpoint: %v", err) } @@ -254,10 +254,9 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { WorkerPodUid: workerPodUID, SandboxAssets: sandboxAssets, Spec: spec, - Fidelity: fidelity, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - Config: &ateletpb.RestoreRequest_LocalConfig{ - LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: snapshotName}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: fidelity, }, }); err != nil { t.Fatalf("Restore: %v", err) @@ -333,8 +332,8 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { const ( atespace = "ate-demo" actorName = "counter" - actorUID = "actor-uid-1" - workerPodUID = "worker-pod-uid-1" + actorUID = "01234567-89ab-cdef-0123-456789abcdef" + workerPodUID = "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f" snapshotName = "pause-snap-1" ) @@ -343,11 +342,9 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { host := imageVolumeTestRegistry(t) image := host + "/actor:v1" - pushTestImage(t, image, singleFileLayer(t, "bin/app", "app")) - checkpointPause := host + "/pause:v1" - pushTestImage(t, checkpointPause, singleFileLayer(t, "pause", "pause-v1")) - restorePause := host + "/pause:v2" - pushTestImage(t, restorePause, singleFileLayer(t, "pause", "pause-v2")) + pinnedImage := pushTestImage(t, image, singleFileLayer(t, "bin/app", "app")) + checkpointPause := pushTestImage(t, host+"/pause:v1", singleFileLayer(t, "pause", "pause-v1")) + restorePause := pushTestImage(t, host+"/pause:v2", singleFileLayer(t, "pause", "pause-v2")) runsc := []byte("runsc binary") s := &AteomHerder{ @@ -371,7 +368,7 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { } } spec := &ateletpb.WorkloadSpec{ - Containers: []*ateletpb.Container{{Name: "app", Image: image, Command: []string{"/bin/app"}}}, + Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}}, } if _, err := s.Run(ctx, &ateletpb.RunRequest{ @@ -395,11 +392,11 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { ActorTemplateName: "counter", WorkerPodUid: workerPodUID, Spec: spec, - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - Config: &ateletpb.CheckpointRequest_LocalConfig{ - LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: snapshotName}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, + UploadConfig: &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED}, }); err != nil { t.Fatalf("Checkpoint: %v", err) } @@ -425,10 +422,9 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { WorkerPodUid: workerPodUID, SandboxAssets: assetsWithPause(restorePause), Spec: spec, - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - Config: &ateletpb.RestoreRequest_LocalConfig{ - LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: snapshotName}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }); err != nil { t.Fatalf("Restore: %v", err) @@ -463,14 +459,14 @@ func TestActivationFailureBeforeRegistration(t *testing.T) { } { t.Run(tc.name, func(t *testing.T) { useTempNodeDirs(t) - const actorUID, snapshotName = "actor-uid-1", "pause-snap-1" + const actorUID, snapshotName = "0f6c2b0e-1b7e-4b3f-9a6a-1d2c3e4f5a61", "pause-snap-1" ctx := t.Context() store := newCTBStore(t) certA := string(testCertPEM(t)) store.set(t, certA) refresher := newSystemInfoVolumeRefresher(trustbundle.NewSource(store.lister.Get, nil), nil) spec := &ateletpb.WorkloadSpec{Volumes: []*ateletpb.Volume{{ - Name: "trust", Source: &ateletpb.Volume_SystemInfo{SystemInfo: trustVolumeSpec("ca.pem")}, + Name: "trust", SystemInfo: trustVolumeSpec("ca.pem"), }}} registered, err := refresher.Register(actorUID, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}, systemInfoVolumesFor(actorUID, spec)) if err != nil { @@ -519,14 +515,16 @@ func TestActivationFailureBeforeRegistration(t *testing.T) { } if tc.restore { _, err = s.Restore(ctx, &ateletpb.RestoreRequest{ - Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "worker-pod-uid-1", + Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f", SandboxAssets: assets, Spec: spec, - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - Config: &ateletpb.RestoreRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: snapshotName}}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + }, }) } else { _, err = s.Run(ctx, &ateletpb.RunRequest{ - Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "worker-pod-uid-1", + Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f", SandboxAssets: assets, Spec: spec, }) } @@ -577,6 +575,7 @@ func TestRunFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { refresher := newSystemInfoVolumeRefresher(trustbundle.NewSource(store.lister.Get, nil), nil) content := []byte("runsc binary") assetHash := fmt.Sprintf("%x", sha256.Sum256(content)) + const actorUID = "0f6c2b0e-1b7e-4b3f-9a6a-1d2c3e4f5a62" s := &AteomHerder{ anonGCSClient: fakeObjectStorage{data: content}, imageCache: newImageVolumeStore(t), @@ -584,18 +583,18 @@ func TestRunFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { } spec := &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{{ - Name: "trust", - Source: &ateletpb.Volume_SystemInfo{SystemInfo: trustVolumeSpec("ca.pem")}, + Name: "trust", + SystemInfo: trustVolumeSpec("ca.pem"), }}, } _, err := s.Run(t.Context(), &ateletpb.RunRequest{ Atespace: "team-a", ActorName: "actor-run", - ActorUid: "actor-uid-run", - WorkerPodUid: "worker-pod-uid-1", + ActorUid: actorUID, + WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f", SandboxAssets: &ateletpb.SandboxAssets{ SandboxClass: "gvisor", - PauseImage: "://invalid-image", + PauseImage: missingPauseImage(t), Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{ runscAssetName: {Url: "gs://test-bucket/runsc", Sha256: assetHash}, }}}, @@ -605,20 +604,29 @@ func TestRunFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { if err == nil || !strings.Contains(err.Error(), "while creating pause OCI bundle") { t.Fatalf("Run error = %v, want the post-registration OCI preparation failure", err) } - if _, err := os.ReadFile(filepath.Join(ateletpath.SystemInfoVolumeRoot("actor-uid-run", "trust"), "ca.pem")); err != nil { + if _, err := os.ReadFile(filepath.Join(ateletpath.SystemInfoVolumeRoot(actorUID, "trust"), "ca.pem")); err != nil { t.Fatalf("Register did not write its projection before OCI preparation failed: %v", err) } - if got := refresher.actors["actor-uid-run"]; got != nil { + if got := refresher.actors[actorUID]; got != nil { t.Fatalf("failed Run left its registration live: %p", got) } } +// missingPauseImage returns a digest-pinned reference that passes request +// validation but names a manifest the local test registry does not serve, so +// the pull fails only once OCI bundle preparation starts. +func missingPauseImage(t *testing.T) string { + t.Helper() + return imageVolumeTestRegistry(t) + "/pause@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" +} + func TestRestoreFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { useTempNodeDirs(t) const ( - actorUID = "actor-uid-restore-after" + actorUID = "0f6c2b0e-1b7e-4b3f-9a6a-1d2c3e4f5a63" snapshotName = "pause-snap-after" ) + pauseImage := missingPauseImage(t) store := newCTBStore(t) store.set(t, string(testCertPEM(t))) refresher := newSystemInfoVolumeRefresher(trustbundle.NewSource(store.lister.Get, nil), nil) @@ -626,14 +634,14 @@ func TestRestoreFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { assetHash := fmt.Sprintf("%x", sha256.Sum256(content)) writeLocalSnapshot(t, ateletpath.LocalSnapshotDir(actorUID, snapshotName), sandboxAssetsRecord{ SandboxClass: "gvisor", - PauseImage: "://invalid-image", + PauseImage: pauseImage, Assets: map[string]assetEntry{"runsc": {URL: "gs://test-bucket/runsc", SHA256: assetHash}}, SnapshotFiles: []string{"checkpoint.img"}, }, map[string]string{"checkpoint.img": "guest-memory"}) spec := &ateletpb.WorkloadSpec{Volumes: []*ateletpb.Volume{{ - Name: "trust", - Source: &ateletpb.Volume_SystemInfo{SystemInfo: trustVolumeSpec("ca.pem")}, + Name: "trust", + SystemInfo: trustVolumeSpec("ca.pem"), }}} s := &AteomHerder{ anonGCSClient: fakeObjectStorage{data: content}, @@ -644,19 +652,18 @@ func TestRestoreFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { Atespace: "team-a", ActorName: "actor-restore-after", ActorUid: actorUID, - WorkerPodUid: "worker-pod-uid-1", + WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f", SandboxAssets: &ateletpb.SandboxAssets{ SandboxClass: "gvisor", - PauseImage: "://invalid-image", + PauseImage: pauseImage, Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{ runscAssetName: {Url: "gs://test-bucket/runsc", Sha256: assetHash}, }}}, }, - Spec: spec, - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, - Config: &ateletpb.RestoreRequest_LocalConfig{ - LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: snapshotName}, + Spec: spec, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }) if err == nil || !strings.Contains(err.Error(), "while creating pause OCI bundle") { @@ -681,7 +688,7 @@ func TestTerminateWithoutWorkerPodUID(t *testing.T) { const ( atespace = "ate-demo" actorName = "counter" - actorUID = "actor-uid-1" + actorUID = "0f6c2b0e-1b7e-4b3f-9a6a-1d2c3e4f5a64" snapshotName = "pause-snap-1" ) @@ -705,7 +712,7 @@ func TestTerminateWithoutWorkerPodUID(t *testing.T) { systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil), } spec := &ateletpb.WorkloadSpec{ - Containers: []*ateletpb.Container{{Name: "app", Image: "example.com/app:v1"}}, + Containers: []*ateletpb.Container{{Name: "app", Image: "example.com/app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, } if _, err := s.Terminate(ctx, &ateletpb.TerminateRequest{ diff --git a/cmd/atelet/main.go b/cmd/atelet/main.go index 4db58a72f0..af81336599 100644 --- a/cmd/atelet/main.go +++ b/cmd/atelet/main.go @@ -30,6 +30,7 @@ import ( "syscall" "time" + "github.com/agent-substrate/substrate/cmd/atelet/internal/apivalidation" "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" "github.com/agent-substrate/substrate/cmd/atelet/internal/credentialprovider" "github.com/agent-substrate/substrate/cmd/atelet/internal/sparsefile" @@ -77,7 +78,7 @@ import ( "google.golang.org/grpc/status" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" - "k8s.io/apimachinery/pkg/util/validation/field" + kubeinformers "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" "k8s.io/client-go/tools/cache" @@ -87,13 +88,14 @@ import ( var ( port = pflag.Int("port", atelet.DefaultPort, "The port to listen on") metricsListenAddr = pflag.String("metrics-listen-addr", ":9090", "Address and port the prometheus metrics server should listen on.") + healthListenAddr = pflag.String("health-listen-addr", ":9091", "Address and port the /readyz and /healthz probe server should listen on.") grpcServerCredBundle = pflag.String("grpc-server-cred-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "Credential bundle atelet presents as its gRPC serving certificate.") clientCACerts = pflag.String("client-ca-certs", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify gRPC client certificates.") ateapiAddress = pflag.String("ateapi-address", "k8s:///api.ate-system.svc:443", "ateapi gRPC target used by the credential broker.") ateapiCAFile = pflag.String("ateapi-ca-file", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify ateapi.") - ateapiServerName = pflag.String("ateapi-server-name", "api.ate-system.svc", "DNS name expected on the ateapi certificate.") grpcInsecure = pflag.Bool("grpc-insecure", false, "Serve gRPC without transport security. Intended only for local clusters without Pod Certificates.") + ateapiServerName = pflag.String("ateapi-server-name", "api.ate-system.svc", "DNS name expected on the ateapi certificate.") // The kubelet already knows how to authenticate to its node's cloud // registry, via an exec plugin the node ships. Pointing atelet at the same @@ -169,11 +171,8 @@ func main() { // readiness flips to not-ready on SIGTERM so /readyz reports 503 while the // pod drains, while /healthz stays 200 for liveness. readiness := &serverboot.Readiness{} - go serverboot.StartMetricsServer(ctx, serverboot.MetricsServerOptions{ - Addr: *metricsListenAddr, - Readiness: readiness, - EnableHealthz: true, - }) + go serverboot.StartMetricsServer(ctx, *metricsListenAddr) + go serverboot.StartHealthServer(ctx, *healthListenAddr, readiness) // The OTLP relay lets the ateom pods on this node export telemetry over a // unix socket instead of their own network (see internal/otlprelay). Started @@ -314,37 +313,16 @@ func main() { ) go systemInfoVolumes.run(ctx) - // Pre-download sandbox assets as SandboxConfigs appear/change so the first - // Run/Restore on this node hits the cache. Best-effort: on failure the - // on-demand fetch in ensureSandboxAssets still covers correctness. - // - // We intentionally never WaitForCacheSync on this factory, so a failing - // list/watch (e.g. Forbidden while the ClusterRole rollout lags the - // binary) cannot hold up atelet startup: the reflector retries in the - // background and prewarm stays cold until it recovers. - ateFactory := externalversions.NewSharedInformerFactory(ateClient, 0) - sandboxConfigInformer := ateFactory.Api().V1alpha1().SandboxConfigs().Informer() - if _, err := startSandboxAssetPrewarm(ctx, sandboxConfigInformer, wmService, imageCache, microvmNodeCapable(hostDevRoot)); err != nil { - slog.ErrorContext(ctx, "Sandbox asset prewarm disabled", slog.Any("err", err)) - } - // Start after the informer is registered: the factory only runs informers - // that exist when Start is called. - ateFactory.Start(stopCh) lis, err := net.Listen("tcp", ":"+strconv.Itoa(*port)) if err != nil { serverboot.Fatal(ctx, "Failed to listen", err) } serverOpts := ateomHerderServerOptions(insecure.NewCredentials()) + nodeName := os.Getenv("NODE_NAME") if *grpcInsecure { slog.WarnContext(ctx, "Serving atelet gRPC without transport security") } else { - tlsCfg, err := ateletServerTLSConfig(*grpcServerCredBundle, *clientCACerts) - if err != nil { - serverboot.Fatal(ctx, "Failed to build server TLS config", err) - } - serverOpts = append(serverOpts, grpc.Creds(credentials.NewTLS(tlsCfg))) - dialOpts, err := ateapiauth.DialOptions(ateapiauth.ClientConfig{ K8sClient: k8sClient, CAFile: *ateapiCAFile, @@ -360,6 +338,10 @@ func main() { } defer ateapiConn.Close() + tlsCfg, err := ateletServerTLSConfig(*grpcServerCredBundle, *clientCACerts) + if err != nil { + serverboot.Fatal(ctx, "Failed to build server TLS config", err) + } ateletCert, err := credbundle.Parse(*grpcServerCredBundle) if err != nil { serverboot.Fatal(ctx, "Failed to load atelet Pod identity", err) @@ -371,6 +353,10 @@ func main() { if ateletIdentity == nil { serverboot.Fatal(ctx, "Failed to load atelet Pod identity", fmt.Errorf("credential bundle has no Pod identity")) } + + nodeName = ateletIdentity.NodeName + serverOpts = ateomHerderServerOptions(credentials.NewTLS(tlsCfg)) + ateomFacingTLS := tlsCfg.Clone() ateomFacingTLS.VerifyConnection = verifyClientOnSameNode(ateletIdentity) if err := os.Remove(nodepath.AteomSupportSocket); err != nil && !errors.Is(err, os.ErrNotExist) { @@ -384,7 +370,9 @@ func main() { if err := os.Chmod(nodepath.AteomSupportSocket, 0o600); err != nil { serverboot.Fatal(ctx, "Failed to restrict credential broker socket", err) } + ateomFacingSrv := grpc.NewServer(grpc.Creds(credentials.NewTLS(ateomFacingTLS))) + ateletpb.RegisterAteomSupportServer(ateomFacingSrv, &ateomSupportServer{ workers: ateapipb.NewWorkerServiceClient(ateapiConn), }) @@ -394,6 +382,53 @@ func main() { } }() } + if nodeName == "" { + serverboot.Fatal(ctx, "Failed to determine local node", fmt.Errorf("NODE_NAME must be set when --grpc-insecure is enabled")) + } + + // SandboxConfigs feed the asset prewarmer and workerSandboxSyncer. + // WorkerPools tell the syncer which SandboxConfig each local worker's pool + // uses. + // + // We intentionally never WaitForCacheSync on these factories, so a failing + // list/watch (e.g. Forbidden while the ClusterRole rollout lags the + // binary) cannot hold up atelet startup: the reflector retries in the + // background and the prewarmer and syncer wait until it recovers. + // + // 0 disables periodic resync. Both react to watch events and retry on their own. + ateFactory := externalversions.NewSharedInformerFactory(ateClient, 0) + // Watch only worker pods on this node, the same way the kubelet watches only its own pods. + // The apiserver indexes pods by spec.nodeName, so list and watch cost scales with + // the pods on this node, not the whole cluster. + workerPodFactory := kubeinformers.NewSharedInformerFactoryWithOptions(k8sClient, 0, + kubeinformers.WithTweakListOptions(func(o *metav1.ListOptions) { + o.FieldSelector = "spec.nodeName=" + nodeName + o.LabelSelector = workerPoolLabel + })) + // Pre-download sandbox assets as SandboxConfigs appear/change so the first + // Run/Restore on this node hits the cache. Best-effort: on failure the + // on-demand fetch in ensureSandboxAssets still covers correctness. + // + // TODO: remove once the worker sandbox syncer fetches assets through + // cmd/atelet/internal/filecache. + sandboxConfigInformer := ateFactory.Api().V1alpha1().SandboxConfigs().Informer() + if _, err := startSandboxAssetPrewarm(ctx, sandboxConfigInformer, wmService, imageCache, microvmNodeCapable(hostDevRoot)); err != nil { + slog.ErrorContext(ctx, "Sandbox asset prewarm disabled", slog.Any("err", err)) + } + // Keep a sandbox versions file in each local worker's directory: the + // runtime hashes of the enabled versions of its pool's SandboxConfig, + // rewritten when the config changes. + if _, err := startWorkerSandboxSyncer(ctx, + workerPodFactory.Core().V1().Pods(), + ateFactory.Api().V1alpha1().WorkerPools(), + ateFactory.Api().V1alpha1().SandboxConfigs(), + nodepath.AteomsDir()); err != nil { + slog.ErrorContext(ctx, "Worker sandbox versions syncer disabled", slog.Any("err", err)) + } + // Start after the informers are registered: a factory only runs + // informers that exist when Start is called. + ateFactory.Start(stopCh) + workerPodFactory.Start(stopCh) svr := grpc.NewServer(serverOpts...) ateletpb.RegisterAteomHerderServer(svr, wmService) @@ -530,8 +565,8 @@ func NewService( } func (s *AteomHerder) Run(ctx context.Context, req *ateletpb.RunRequest) (resp *ateletpb.RunResponse, err error) { - if err := validateRunRequest(req); err != nil { - return nil, apierror.InvalidArgument("%v", err) + if errs := apivalidation.ValidateRunRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) } actorUID := req.GetActorUid() @@ -651,8 +686,8 @@ func allocatedBytes(info os.FileInfo) int64 { } func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest) (_ *ateletpb.CheckpointResponse, err error) { - if err := validateCheckpointRequest(req); err != nil { - return nil, apierror.InvalidArgument("%v", err) + if errs := apivalidation.ValidateCheckpointRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) } actorUID := req.GetActorUid() @@ -666,7 +701,7 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe templateNamespace: req.GetActorTemplateAtespace(), templateName: req.GetActorTemplateName(), kind: checkpointSnapshotKind(req), - fidelity: ateattr.SnapshotFidelityValue(req.GetFidelity()), + fidelity: ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()), } attribution := resources.ActorAttribution{ Ref: actorRef, @@ -730,7 +765,7 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe RunscPath: runscPathFor(assetPaths), RuntimeAssetPaths: assetPaths, Spec: spec, - Fidelity: toAteomFidelity(req.GetFidelity()), + Fidelity: toAteomFidelity(req.GetSnapshot().GetFidelity()), ActorUid: actorUID, ActorDirs: ateletpath.ActorDirs(actorUID), }) @@ -752,7 +787,7 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe sandboxRec.ActorUID = req.GetActorUid() sandboxRec.ActorTemplateAtespace = req.GetActorTemplateAtespace() sandboxRec.ActorTemplateName = req.GetActorTemplateName() - sandboxRec.Fidelity = ateattr.SnapshotFidelityValue(req.GetFidelity()) + sandboxRec.Fidelity = ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()) // No earlier pause snapshot can ever be restored again, so remove them // all: the actor's current state was just captured by CheckpointWorkload, @@ -774,8 +809,8 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe // snapshots on both paths, so timing it as part of an external upload would // mix local disk deletion into the object-storage measurement. tPersist := time.Now() - switch req.GetType() { - case ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL: + switch mode := req.GetUploadConfig().GetMode(); mode { + case ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE: // CheckpointWorkload has already deleted the workload, so a retried // Checkpoint cannot redo this upload. It waits for the object-store plugin // as long as ctx allows, and a plugin error reaches the control plane as @@ -785,13 +820,13 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe dPersist = time.Since(tPersist) return nil, fmt.Errorf("while uploading external snapshot: %w", err) } - case ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL: + case ateletpb.UploadMode_UPLOAD_MODE_DEFERRED: if err := s.moveLocalCheckpoint(ctx, req, sandboxRec); err != nil { dPersist = time.Since(tPersist) return nil, fmt.Errorf("while moving to local snapshot: %w", err) } default: - return nil, fmt.Errorf("unexpected checkpoint type: %v", req.GetType()) + return nil, fmt.Errorf("unexpected upload mode: %v", mode) } dPersist = time.Since(tPersist) @@ -850,7 +885,8 @@ func (s *AteomHerder) moveLocalCheckpoint(ctx context.Context, req *ateletpb.Che if err != nil { return err } - localDir, err := filepath.Rel(actorDir, ateletpath.LocalSnapshotDir(req.GetActorUid(), req.GetLocalConfig().GetSnapshotName())) + snapshotName := req.GetSnapshot().GetSnapshotUuid() + localDir, err := filepath.Rel(actorDir, ateletpath.LocalSnapshotDir(req.GetActorUid(), snapshotName)) if err != nil { return err } @@ -893,12 +929,12 @@ func (s *AteomHerder) moveLocalCheckpoint(ctx context.Context, req *ateletpb.Che // shouldHaveSnapshots returns true if the checkpoint request is expected to produce snapshot files. func shouldHaveSnapshots(req *ateletpb.CheckpointRequest) bool { - if req.GetFidelity() != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + if req.GetSnapshot().GetFidelity() != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { return true } for _, vol := range req.GetSpec().GetVolumes() { - if _, ok := vol.GetSource().(*ateletpb.Volume_DurableDir); ok { + if vol.GetDurableDir() != nil { return true } } @@ -910,7 +946,7 @@ func shouldHaveSnapshots(req *ateletpb.CheckpointRequest) bool { // outage for as long as ctx allows rather than only pluginsocket's ready // wait, and their errors are not reported as retryable: see Checkpoint. func (s *AteomHerder) uploadExternalCheckpoint(ctx context.Context, req *ateletpb.CheckpointRequest, checkpointDir string, rec *sandboxAssetsRecord) error { - uri, err := resources.ParseSnapshotURI(req.GetExternalConfig().GetSnapshotUri()) + uri, err := resources.ParseSnapshotURI(req.GetSnapshot().GetObject().GetSnapshotUri()) if err != nil { return err } @@ -952,13 +988,14 @@ func (s *AteomHerder) uploadSnapshot(ctx context.Context, uri resources.Snapshot return nil } -// UploadPausedCheckpoint copies a paused actor's local checkpoint to object -// storage. It drives no ateom — the actor's sandbox is gone; the checkpoint -// files and their self-describing manifest already sit under the actor's -// local-checkpoints directory, written by an earlier local Checkpoint (pause). -func (s *AteomHerder) UploadPausedCheckpoint(ctx context.Context, req *ateletpb.UploadPausedCheckpointRequest) (_ *ateletpb.UploadPausedCheckpointResponse, err error) { - if err := validateUploadPausedCheckpointRequest(req); err != nil { - return nil, apierror.InvalidArgument("%v", err) +// PromoteSnapshot uploads a paused actor's local checkpoint to object +// storage so the actor can be suspended and is no longer tied to this node. It +// drives no ateom — the actor's sandbox is gone; the checkpoint files and their +// self-describing manifest already sit under the actor's local-checkpoints +// directory, written by an earlier local Checkpoint (pause). +func (s *AteomHerder) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnapshotRequest) (_ *ateletpb.PromoteSnapshotResponse, err error) { + if errs := apivalidation.ValidatePromoteSnapshotRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) } tStart := time.Now() @@ -969,7 +1006,7 @@ func (s *AteomHerder) UploadPausedCheckpoint(ctx context.Context, req *ateletpb. // Always the actor's durable latest: golden actors are never paused // (validation above rejects the golden atespace). kind: ateattr.SnapshotKindLatest, - fidelity: ateattr.SnapshotFidelityValue(req.GetDesiredFidelity()), + fidelity: ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()), } defer func() { s.instruments.recordCheckpoint(ctx, op, @@ -977,11 +1014,11 @@ func (s *AteomHerder) UploadPausedCheckpoint(ctx context.Context, req *ateletpb. phase{ateattr.SnapshotPhaseTotal, time.Since(tStart)}) }() - uri, err := resources.ParseSnapshotURI(req.GetDestinationSnapshotUri()) + uri, err := resources.ParseSnapshotURI(req.GetSnapshot().GetObject().GetSnapshotUri()) if err != nil { return nil, apierror.InvalidArgument("%v", err) } - localDir := ateletpath.LocalSnapshotDir(req.GetActorUid(), req.GetLocalSnapshotName()) + localDir := ateletpath.LocalSnapshotDir(req.GetActorUid(), req.GetSnapshot().GetSnapshotUuid()) tPersist := time.Now() sandboxClass, err := s.uploadLocalCheckpointDir(ctx, req, localDir, uri) @@ -991,20 +1028,21 @@ func (s *AteomHerder) UploadPausedCheckpoint(ctx context.Context, req *ateletpb. return nil, err } - // The uploaded snapshot supersedes every local pause snapshot of this - // actor; free the node's disk (best-effort, like Checkpoint). + // Best-effort: the snapshot is in object storage now and the actor will + // not resume from this node's local copy. Kept outside the persist window + // so local disk deletion does not mix into the upload measurement. if err := pruneLocalCheckpoints(ctx, req.GetActorUid()); err != nil { slog.WarnContext(ctx, "failed to prune uploaded local checkpoints", slog.String("actorUID", req.GetActorUid()), slog.Any("err", err)) } - return &ateletpb.UploadPausedCheckpointResponse{}, nil + return &ateletpb.PromoteSnapshotResponse{}, nil } -// uploadLocalCheckpointDir uploads the local checkpoint in localDir to uri, -// converting the captured scope to the requested one where possible. It +// uploadLocalCheckpointDir uploads the local checkpoint in localDir to uri. It // returns the sandbox class recorded in the snapshot manifest (empty when the // manifest was not read). Parameterized by localDir for tests. -func (s *AteomHerder) uploadLocalCheckpointDir(ctx context.Context, req *ateletpb.UploadPausedCheckpointRequest, localDir string, uri resources.SnapshotURI) (string, error) { +func (s *AteomHerder) uploadLocalCheckpointDir(ctx context.Context, req *ateletpb.PromoteSnapshotRequest, localDir string, uri resources.SnapshotURI) (string, error) { + localSnapshotName := req.GetSnapshot().GetSnapshotUuid() manifest, err := readSnapshotManifest(localDir) if errors.Is(err, os.ErrNotExist) { // The local snapshot is gone. A previous invocation may have uploaded @@ -1014,12 +1052,12 @@ func (s *AteomHerder) uploadLocalCheckpointDir(ctx context.Context, req *ateletp // unrecoverable. _, fetchErr := s.fetchManifest(ctx, uri.String()) if fetchErr == nil { - slog.InfoContext(ctx, "Local snapshot already uploaded and pruned; nothing to do", slog.String("snapshot_uri", req.GetDestinationSnapshotUri())) + slog.InfoContext(ctx, "Local snapshot already uploaded and pruned; nothing to do", slog.String("snapshot_uri", uri.String())) return "", nil } if status.Code(fetchErr) == codes.NotFound { return "", fmt.Errorf("local snapshot %q is gone and no uploaded copy exists: %w", - req.GetLocalSnapshotName(), fetchErr) + localSnapshotName, fetchErr) } return "", fmt.Errorf("while probing for an already-uploaded snapshot manifest: %w", fetchErr) } @@ -1034,13 +1072,13 @@ func (s *AteomHerder) uploadLocalCheckpointDir(ctx context.Context, req *ateletp capturedFidelity := rec.Fidelity if capturedFidelity == "" { - return rec.SandboxClass, apierror.FailedPrecondition("local snapshot %q has no fidelity recorded in its manifest; resume and pause the actor again before suspending it", req.GetLocalSnapshotName()) + return rec.SandboxClass, apierror.FailedPrecondition("local snapshot %q has no fidelity recorded in its manifest; resume and pause the actor again before suspending it", localSnapshotName) } // Pause and suspend capture the template's one preferred fidelity, and a // paused actor's template cannot change, so the upload is always of what // the pause captured. A mismatch means the store or the control plane // drifted; refuse rather than guess. - if desiredFidelity := ateattr.SnapshotFidelityValue(req.GetDesiredFidelity()); capturedFidelity != desiredFidelity { + if desiredFidelity := ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()); capturedFidelity != desiredFidelity { return rec.SandboxClass, apierror.FailedPrecondition("pause snapshot captured %s; upload asks for %s", capturedFidelity, desiredFidelity) } @@ -1059,8 +1097,8 @@ func readSnapshotManifest(dir string) ([]byte, error) { } func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (resp *ateletpb.RestoreResponse, err error) { - if err := validateRestoreRequest(req); err != nil { - return nil, apierror.InvalidArgument("%v", err) + if errs := apivalidation.ValidateRestoreRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) } actorUID := req.GetActorUid() @@ -1085,7 +1123,7 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) op := snapshotOp{ templateNamespace: req.GetActorTemplateAtespace(), templateName: req.GetActorTemplateName(), - fidelity: ateattr.SnapshotFidelityValue(req.GetFidelity()), + fidelity: ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()), sandboxClass: req.GetSandboxAssets().GetSandboxClass(), } attribution := resources.ActorAttribution{ @@ -1126,9 +1164,9 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) } checkpointDir := ateletpath.RestoreStateDir(actorUID) - directLocal := req.GetType() == ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL + directLocal := req.GetSnapshot().GetObject() == nil if directLocal { - checkpointDir = ateletpath.LocalSnapshotDir(actorUID, req.GetLocalConfig().GetSnapshotName()) + checkpointDir = ateletpath.LocalSnapshotDir(actorUID, req.GetSnapshot().GetSnapshotUuid()) } // Fetch the snapshot manifest stored beside the checkpoint images @@ -1142,25 +1180,22 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) } }() var sandboxRec *sandboxAssetsRecord - switch req.GetType() { - case ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL: - manifest, err := s.fetchManifest(ctx, req.GetExternalConfig().GetSnapshotUri()) + if directLocal { + manifest, err := readSnapshotManifest(checkpointDir) if err != nil { - return nil, fmt.Errorf("while fetching snapshot manifest: %w", err) + return nil, wrapFileSystemErr("while reading local snapshot manifest", err) } if sandboxRec, err = unmarshalSandboxRecord(manifest); err != nil { return nil, fmt.Errorf("while unmarshalling sandbox record: %w", err) } - case ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL: - manifest, err := readSnapshotManifest(ateletpath.LocalSnapshotDir(actorUID, req.GetLocalConfig().GetSnapshotName())) + } else { + manifest, err := s.fetchManifest(ctx, req.GetSnapshot().GetObject().GetSnapshotUri()) if err != nil { - return nil, wrapFileSystemErr("while reading local snapshot manifest", err) + return nil, fmt.Errorf("while fetching snapshot manifest: %w", err) } if sandboxRec, err = unmarshalSandboxRecord(manifest); err != nil { return nil, fmt.Errorf("while unmarshalling sandbox record: %w", err) } - default: - return nil, fmt.Errorf("unexpected checkpoint type: %v", req.GetType()) } dManifest = time.Since(tManifest) @@ -1198,18 +1233,11 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) } downloadErr = err }() - switch req.GetType() { - case ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL: - if err := s.downloadExternalCheckpoint(gctx, req.GetExternalConfig().GetSnapshotUri(), checkpointDir, sandboxRec.SnapshotFiles); err != nil { - return err - } - case ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL: + if directLocal { // Restore in place from LocalSnapshotDir; no staging. - if err := checkLocalSnapshotFiles(checkpointDir, sandboxRec.SnapshotFiles); err != nil { - return err - } + return checkLocalSnapshotFiles(checkpointDir, sandboxRec.SnapshotFiles) } - return nil + return s.downloadExternalCheckpoint(gctx, req.GetSnapshot().GetObject().GetSnapshotUri(), checkpointDir, sandboxRec.SnapshotFiles) }) g.Go(func() (err error) { defer func() { prepErr = err }() @@ -1269,7 +1297,7 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) RunscPath: runscPathFor(assetPaths), RuntimeAssetPaths: assetPaths, Spec: spec, - Fidelity: toAteomFidelity(req.GetFidelity()), + Fidelity: toAteomFidelity(req.GetSnapshot().GetFidelity()), ActorUid: req.GetActorUid(), ActorDirs: actorDirs, PreserveRestoreDir: directLocal, @@ -1297,8 +1325,8 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) // Terminate terminates any running workload on ateom, unmounts external volumes, // and resets actor directories on the node. func (s *AteomHerder) Terminate(ctx context.Context, req *ateletpb.TerminateRequest) (*ateletpb.TerminateResponse, error) { - if err := validateTerminateRequest(req); err != nil { - return nil, apierror.InvalidArgument("%v", err) + if errs := apivalidation.ValidateTerminateRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) } actorRef := resources.ActorRef{Atespace: req.GetAtespace(), Name: req.GetActorName()} @@ -1500,8 +1528,7 @@ func (s *AteomHerder) prepareOCIBundles( ) ([]*ateompb.ContainerSpec, error) { // Prepare host folders for volume types that need them. for _, vol := range spec.GetVolumes() { - switch vol.GetSource().(type) { - case *ateletpb.Volume_DurableDir: + if vol.GetDurableDir() != nil { volPath := ateletpath.DurableDirVolumeMountPoint(actorUID, vol.GetName()) if err := os.MkdirAll(volPath, 0o700); err != nil { return nil, fmt.Errorf("while creating %q: %w", volPath, err) @@ -1589,29 +1616,29 @@ func buildAteomWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*ateom return nil, fmt.Errorf("container %q mounts volume %q which is not defined in workload volumes", ctr.GetName(), volName) } - switch vol.GetSource().(type) { - case *ateletpb.Volume_DurableDir: + switch { + case vol.GetDurableDir() != nil: ddMounts = append(ddMounts, &ateompb.DurableDirVolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) - case *ateletpb.Volume_External: + case vol.GetExternal() != nil: csiMounts = append(csiMounts, &ateompb.VolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) - case *ateletpb.Volume_SystemInfo: + case vol.GetSystemInfo() != nil: siMounts = append(siMounts, &ateompb.SystemInfoVolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) - case *ateletpb.Volume_Image: + case vol.GetImage() != nil: imgMounts = append(imgMounts, &ateompb.ImageVolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) default: - return nil, fmt.Errorf("container %q mounts volume %q with unsupported source %T", ctr.GetName(), volName, vol.GetSource()) + return nil, fmt.Errorf("container %q mounts volume %q with no source set", ctr.GetName(), volName) } } container := &ateompb.Container{ @@ -1653,7 +1680,7 @@ func toAteomWakeupProbe(in *ateletpb.WakeupProbe) *ateompb.WakeupProbe { } out := &ateompb.WakeupProbe{} if hg := in.GetHttpGet(); hg != nil { - out.HttpGet = &ateompb.HTTPGetAction{ + out.HttpGet = &ateompb.HttpGetAction{ Path: hg.GetPath(), Port: hg.GetPort(), } @@ -1713,171 +1740,6 @@ func (d *AteomDialer) DialAteomPod(ctx context.Context, podUID string) (*grpc.Cl return conn, nil } -// validateRunRequest, validateCheckpointRequest, and validateRestoreRequest -// validate everything in their request that atelet turns into host filesystem -// paths, plus the request-specific fields. atelet listens on an insecure -// hostPort, so any reachable caller could otherwise smuggle a path separator -// or ".." through these fields and make atelet read/RemoveAll/write outside -// the intended directory tree, or collide bundles. Each RPC validates at its -// boundary, before any path is built. The field rules live in -// internal/resources so other components can apply them at their boundaries. -func validateRunRequest(req *ateletpb.RunRequest) error { - var errs field.ErrorList - errs = append(errs, resources.ValidateResourceName(req.GetAtespace(), field.NewPath("atespace"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorName(), field.NewPath("actor_name"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorUid(), field.NewPath("actor_uid"))...) - if len(errs) > 0 { - return errs.ToAggregate() - } - // TODO: Migrate all validations below to the validation framework. - if err := resources.ValidateWorkerPodUID(req.GetWorkerPodUid()); err != nil { - return err - } - names := make([]string, 0, len(req.GetSpec().GetContainers())) - for _, ctr := range req.GetSpec().GetContainers() { - names = append(names, ctr.GetName()) - } - return resources.ValidateContainerNames(names) -} - -func validateCheckpointRequest(req *ateletpb.CheckpointRequest) error { - var errs field.ErrorList - errs = append(errs, resources.ValidateResourceName(req.GetAtespace(), field.NewPath("atespace"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorName(), field.NewPath("actor_name"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorUid(), field.NewPath("actor_uid"))...) - if len(errs) > 0 { - return errs.ToAggregate() - } - // TODO: Migrate all validations below to the validation framework. - if err := resources.ValidateWorkerPodUID(req.GetWorkerPodUid()); err != nil { - return err - } - names := make([]string, 0, len(req.GetSpec().GetContainers())) - for _, ctr := range req.GetSpec().GetContainers() { - names = append(names, ctr.GetName()) - } - if err := resources.ValidateContainerNames(names); err != nil { - return err - } - - if err := validateFidelity(req.GetFidelity()); err != nil { - return err - } - - switch req.GetType() { - case ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL: - if _, err := resources.ParseSnapshotURI(req.GetExternalConfig().GetSnapshotUri()); err != nil { - return err - } - case ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL: - if !resources.IsValidResourceName(req.GetLocalConfig().GetSnapshotName()) { - return fmt.Errorf("invalid local snapshot name %q", req.GetLocalConfig().GetSnapshotName()) - } - default: - return fmt.Errorf("invalid checkpoint type: %v", req.GetType()) - } - return nil -} - -func validateRestoreRequest(req *ateletpb.RestoreRequest) error { - var errs field.ErrorList - errs = append(errs, resources.ValidateResourceName(req.GetAtespace(), field.NewPath("atespace"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorName(), field.NewPath("actor_name"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorUid(), field.NewPath("actor_uid"))...) - if len(errs) > 0 { - return errs.ToAggregate() - } - // TODO: Migrate all validations below to the validation framework. - if err := resources.ValidateWorkerPodUID(req.GetWorkerPodUid()); err != nil { - return err - } - names := make([]string, 0, len(req.GetSpec().GetContainers())) - for _, ctr := range req.GetSpec().GetContainers() { - names = append(names, ctr.GetName()) - } - if err := resources.ValidateContainerNames(names); err != nil { - return err - } - - if err := validateFidelity(req.GetFidelity()); err != nil { - return err - } - - if req.GetSandboxAssets() == nil { - return fmt.Errorf("missing sandbox_assets") - } - - switch req.GetType() { - case ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL: - if _, err := resources.ParseSnapshotURI(req.GetExternalConfig().GetSnapshotUri()); err != nil { - return err - } - case ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL: - if !resources.IsValidResourceName(req.GetLocalConfig().GetSnapshotName()) { - return fmt.Errorf("invalid local snapshot name %q", req.GetLocalConfig().GetSnapshotName()) - } - default: - return fmt.Errorf("invalid checkpoint type: %v", req.GetType()) - } - return nil -} - -func validateTerminateRequest(req *ateletpb.TerminateRequest) error { - var errs field.ErrorList - errs = append(errs, resources.ValidateResourceName(req.GetAtespace(), field.NewPath("atespace"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorName(), field.NewPath("actor_name"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorUid(), field.NewPath("actor_uid"))...) - if len(errs) > 0 { - return errs.ToAggregate() - } - if req.GetWorkerPodUid() != "" { - if err := resources.ValidateWorkerPodUID(req.GetWorkerPodUid()); err != nil { - return err - } - } - names := make([]string, 0, len(req.GetSpec().GetContainers())) - for _, ctr := range req.GetSpec().GetContainers() { - names = append(names, ctr.GetName()) - } - return resources.ValidateContainerNames(names) -} - -// validateFidelity accepts every level of the ladder. Which levels a sandbox -// runtime can actually serve is the runtime's call: atelet forwards the -// fidelity and ateom rejects what it cannot capture. -func validateFidelity(fidelity ateletpb.SnapshotFidelity) error { - switch fidelity { - case ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, - ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, - ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: - return nil - case ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED: - return fmt.Errorf("snapshot fidelity must be non-zero") - default: - return fmt.Errorf("invalid snapshot fidelity: %v", fidelity) - } -} - -func validateUploadPausedCheckpointRequest(req *ateletpb.UploadPausedCheckpointRequest) error { - var errs field.ErrorList - errs = append(errs, resources.ValidateResourceName(req.GetAtespace(), field.NewPath("atespace"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorName(), field.NewPath("actor_name"))...) - errs = append(errs, resources.ValidateResourceName(req.GetActorUid(), field.NewPath("actor_uid"))...) - errs = append(errs, resources.ValidateResourceName(req.GetLocalSnapshotName(), field.NewPath("local_snapshot_name"))...) - // Golden actors are never paused (the golden flow commits a running - // actor), so never promote a paused checkpoint to a golden snapshot. - if req.GetAtespace() == resources.GoldenActorAtespace { - errs = append(errs, field.Forbidden(field.NewPath("atespace"), fmt.Sprintf("atespace %q holds golden actors, which are never paused", req.GetAtespace()))) - } - if _, err := resources.ParseSnapshotURI(req.GetDestinationSnapshotUri()); err != nil { - errs = append(errs, field.Invalid(field.NewPath("destination_snapshot_uri"), req.GetDestinationSnapshotUri(), err.Error())) - } - if err := validateFidelity(req.GetDesiredFidelity()); err != nil { - errs = append(errs, field.Invalid(field.NewPath("desired_fidelity"), req.GetDesiredFidelity().String(), err.Error())) - } - return errs.ToAggregate() -} - // writeFileAtomic writes data to path by writing a temp file in the same // directory and renaming it over the target so readers never observe a // truncated or partially written value. diff --git a/cmd/atelet/main_test.go b/cmd/atelet/main_test.go index 1cbb89367a..95dac97ac3 100644 --- a/cmd/atelet/main_test.go +++ b/cmd/atelet/main_test.go @@ -538,216 +538,6 @@ func TestWriteFileAtomic(t *testing.T) { }) } -// validRunRequest, validCheckpointRequest, and validRestoreRequest build -// requests whose every field passes validation; the per-request tests below -// break one field per case. -func validRunRequest() *ateletpb.RunRequest { - return &ateletpb.RunRequest{ - Atespace: "ate-demo", - ActorName: "counter-1", - ActorTemplateAtespace: "ate-demo", - ActorTemplateName: "counter", - WorkerPodUid: "422938ba-8860-4983-a25d-d6bcb0a69d4e", - ActorUid: "123e4567-e89b-12d3-a456-426614174000", - Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker"}}}, - } -} - -func validCheckpointRequest() *ateletpb.CheckpointRequest { - return &ateletpb.CheckpointRequest{ - Atespace: "ate-demo", - ActorName: "counter-1", - ActorTemplateAtespace: "ate-demo", - ActorTemplateName: "counter", - WorkerPodUid: "422938ba-8860-4983-a25d-d6bcb0a69d4e", - ActorUid: "123e4567-e89b-12d3-a456-426614174000", - Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker"}}}, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - Config: &ateletpb.CheckpointRequest_ExternalConfig{ - ExternalConfig: &ateletpb.ExternalCheckpointConfiguration{ - SnapshotUri: testSnapshotURI, - }, - }, - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, - } -} - -func validRestoreRequest() *ateletpb.RestoreRequest { - return &ateletpb.RestoreRequest{ - Atespace: "ate-demo", - ActorName: "counter-1", - ActorTemplateAtespace: "ate-demo", - ActorTemplateName: "counter", - WorkerPodUid: "422938ba-8860-4983-a25d-d6bcb0a69d4e", - ActorUid: "123e4567-e89b-12d3-a456-426614174000", - Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker"}}}, - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - Config: &ateletpb.RestoreRequest_ExternalConfig{ - ExternalConfig: &ateletpb.ExternalRestoreConfiguration{ - SnapshotUri: testSnapshotURI, - }, - }, - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, - SandboxAssets: &ateletpb.SandboxAssets{ - SandboxClass: "gvisor", - PauseImage: testPauseImage, - }, - } -} - -func TestValidateRunRequest(t *testing.T) { - tests := []struct { - name string - mutate func(*ateletpb.RunRequest) - wantErr bool - }{ - {"valid", func(*ateletpb.RunRequest) {}, false}, - {"invalid worker pod uid", func(r *ateletpb.RunRequest) { r.WorkerPodUid = "../escape" }, true}, - {"invalid atespace", func(r *ateletpb.RunRequest) { r.Atespace = "../escape" }, true}, - {"invalid actor name", func(r *ateletpb.RunRequest) { r.ActorName = "../escape" }, true}, - {"invalid actor uid", func(r *ateletpb.RunRequest) { r.ActorUid = "../escape" }, true}, - {"any actor template identity accepted", func(r *ateletpb.RunRequest) { r.ActorTemplateAtespace, r.ActorTemplateName = "Not_Valid", "Not_Valid" }, false}, - {"empty actor template identity accepted", func(r *ateletpb.RunRequest) { r.ActorTemplateAtespace, r.ActorTemplateName = "", "" }, false}, - {"invalid container name", func(r *ateletpb.RunRequest) { - r.Spec.Containers = []*ateletpb.Container{{Name: "../escape"}} - }, true}, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - req := validRunRequest() - tc.mutate(req) - if err := validateRunRequest(req); (err != nil) != tc.wantErr { - t.Errorf("validateRunRequest err = %v, wantErr %v", err, tc.wantErr) - } - }) - } -} - -// Checkpoint and Restore must reject a bad snapshot URI even when -// every common field is valid. -func TestValidateCheckpointRequest(t *testing.T) { - makeReq := func(opts ...func(*ateletpb.CheckpointRequest)) *ateletpb.CheckpointRequest { - r := validCheckpointRequest() - for _, opt := range opts { - opt(r) - } - return r - } - - tests := []struct { - name string - req *ateletpb.CheckpointRequest - wantErr bool - }{ - {"valid", makeReq(), false}, - {"empty snapshot uri", makeReq(func(r *ateletpb.CheckpointRequest) { r.GetExternalConfig().SnapshotUri = "" }), true}, - {"bucketless snapshot uri", makeReq(func(r *ateletpb.CheckpointRequest) { r.GetExternalConfig().SnapshotUri = "relative/path" }), true}, - {"invalid worker pod uid", makeReq(func(r *ateletpb.CheckpointRequest) { r.WorkerPodUid = "../escape" }), true}, - {"invalid atespace", makeReq(func(r *ateletpb.CheckpointRequest) { r.Atespace = "../escape" }), true}, - {"invalid actor name", makeReq(func(r *ateletpb.CheckpointRequest) { r.ActorName = "../escape" }), true}, - {"invalid actor uid", makeReq(func(r *ateletpb.CheckpointRequest) { r.ActorUid = "../escape" }), true}, - {"any actor template identity accepted", makeReq(func(r *ateletpb.CheckpointRequest) { - r.ActorTemplateAtespace, r.ActorTemplateName = "Not_Valid", "Not_Valid" - }), false}, - {"empty actor template identity accepted", makeReq(func(r *ateletpb.CheckpointRequest) { r.ActorTemplateAtespace, r.ActorTemplateName = "", "" }), false}, - {"invalid container name", makeReq(func(r *ateletpb.CheckpointRequest) { - r.Spec.Containers = []*ateletpb.Container{{Name: "../escape"}} - }), true}, - {"invalid local snapshot prefix", makeReq(func(r *ateletpb.CheckpointRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.CheckpointRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: ""}} - }), true}, - {"local snapshot name escapes its directory", makeReq(func(r *ateletpb.CheckpointRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.CheckpointRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: "../escape"}} - }), true}, - {"nested local snapshot prefix", makeReq(func(r *ateletpb.CheckpointRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.CheckpointRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: "pause/2"}} - }), true}, - {"traversal local snapshot prefix", makeReq(func(r *ateletpb.CheckpointRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.CheckpointRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: ".."}} - }), true}, - {"unspecified snapshot type", makeReq(func(r *ateletpb.CheckpointRequest) { r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_UNSPECIFIED }), true}, - {"unspecified snapshot fidelity", makeReq(func(r *ateletpb.CheckpointRequest) { - r.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED - }), true}, - {"invalid snapshot fidelity", makeReq(func(r *ateletpb.CheckpointRequest) { r.Fidelity = ateletpb.SnapshotFidelity(23) }), true}, - {"rootfs fidelity", makeReq(func(r *ateletpb.CheckpointRequest) { - r.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS - }), false}, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - if err := validateCheckpointRequest(tc.req); (err != nil) != tc.wantErr { - t.Errorf("validateCheckpointRequest err = %v, wantErr %v", err, tc.wantErr) - } - }) - } -} - -func TestValidateRestoreRequest(t *testing.T) { - makeReq := func(opts ...func(*ateletpb.RestoreRequest)) *ateletpb.RestoreRequest { - r := validRestoreRequest() - for _, opt := range opts { - opt(r) - } - return r - } - - tests := []struct { - name string - req *ateletpb.RestoreRequest - wantErr bool - }{ - {"valid", makeReq(), false}, - {"missing sandbox assets", makeReq(func(r *ateletpb.RestoreRequest) { r.SandboxAssets = nil }), true}, - {"empty snapshot uri", makeReq(func(r *ateletpb.RestoreRequest) { r.GetExternalConfig().SnapshotUri = "" }), true}, - {"bucketless snapshot uri", makeReq(func(r *ateletpb.RestoreRequest) { r.GetExternalConfig().SnapshotUri = "relative/path" }), true}, - {"invalid worker pod uid", makeReq(func(r *ateletpb.RestoreRequest) { r.WorkerPodUid = "../escape" }), true}, - {"invalid atespace", makeReq(func(r *ateletpb.RestoreRequest) { r.Atespace = "../escape" }), true}, - {"invalid actor name", makeReq(func(r *ateletpb.RestoreRequest) { r.ActorName = "../escape" }), true}, - {"invalid actor uid", makeReq(func(r *ateletpb.RestoreRequest) { r.ActorUid = "../escape" }), true}, - {"any actor template identity accepted", makeReq(func(r *ateletpb.RestoreRequest) { - r.ActorTemplateAtespace, r.ActorTemplateName = "Not_Valid", "Not_Valid" - }), false}, - {"empty actor template identity accepted", makeReq(func(r *ateletpb.RestoreRequest) { r.ActorTemplateAtespace, r.ActorTemplateName = "", "" }), false}, - {"invalid container name", makeReq(func(r *ateletpb.RestoreRequest) { - r.Spec.Containers = []*ateletpb.Container{{Name: "../escape"}} - }), true}, - {"invalid local snapshot prefix", makeReq(func(r *ateletpb.RestoreRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.RestoreRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: ""}} - }), true}, - {"local snapshot name escapes its directory", makeReq(func(r *ateletpb.RestoreRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.RestoreRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: "../escape"}} - }), true}, - {"nested local snapshot prefix", makeReq(func(r *ateletpb.RestoreRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.RestoreRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: "pause/2"}} - }), true}, - {"traversal local snapshot prefix", makeReq(func(r *ateletpb.RestoreRequest) { - r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL - r.Config = &ateletpb.RestoreRequest_LocalConfig{LocalConfig: &ateletpb.LocalCheckpointConfiguration{SnapshotName: ".."}} - }), true}, - {"unspecified snapshot type", makeReq(func(r *ateletpb.RestoreRequest) { r.Type = ateletpb.CheckpointType_CHECKPOINT_TYPE_UNSPECIFIED }), true}, - {"unspecified snapshot fidelity", makeReq(func(r *ateletpb.RestoreRequest) { r.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED }), true}, - {"invalid snapshot fidelity", makeReq(func(r *ateletpb.RestoreRequest) { r.Fidelity = ateletpb.SnapshotFidelity(23) }), true}, - {"rootfs fidelity", makeReq(func(r *ateletpb.RestoreRequest) { - r.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS - }), false}, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - if err := validateRestoreRequest(tc.req); (err != nil) != tc.wantErr { - t.Errorf("validateRestoreRequest err = %v, wantErr %v", err, tc.wantErr) - } - }) - } -} - // Every valid atelet scope must map to its ateom counterpart. func TestToAteomFidelity(t *testing.T) { tests := []struct { @@ -782,7 +572,7 @@ func TestFetchAssetRejectsBadHash(t *testing.T) { // Invalid (8 chars, not 64) but separator-free, so it resolves to a normal // filename inside the temp StaticFilesDir. const badHash = "deadbeef" - if err := os.WriteFile(ateletpath.RunSCBinaryPath(badHash), []byte("planted"), 0o755); err != nil { + if err := os.WriteFile(ateletpath.AssetPath(badHash), []byte("planted"), 0o755); err != nil { t.Fatalf("planting cache file: %v", err) } @@ -847,7 +637,7 @@ func TestFetchAssetStreaming(t *testing.T) { if err == nil { t.Fatal("fetchAsset accepted an over-cap asset") } - if _, err := os.Stat(ateletpath.RunSCBinaryPath(goodHash)); !errors.Is(err, os.ErrNotExist) { + if _, err := os.Stat(ateletpath.AssetPath(goodHash)); !errors.Is(err, os.ErrNotExist) { t.Errorf("over-cap download left a file at the cache path (stat err = %v)", err) } }) @@ -861,7 +651,7 @@ func TestFetchAssetStreaming(t *testing.T) { if err == nil { t.Fatal("fetchAsset accepted a hash mismatch") } - if _, err := os.Stat(ateletpath.RunSCBinaryPath(wrongHash)); !errors.Is(err, os.ErrNotExist) { + if _, err := os.Stat(ateletpath.AssetPath(wrongHash)); !errors.Is(err, os.ErrNotExist) { t.Errorf("mismatched download left a file at the cache path (stat err = %v)", err) } }) @@ -913,7 +703,10 @@ func TestRPCBoundariesReject(t *testing.T) { ctx := context.Background() badUID := "../escape" // valid actor ref, invalid worker pod UID const okAtespace, okID, okActorUID = "ate-demo", "counter-1", "123e4567-e89b-12d3-a456-426614174000" - okSpec := &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker"}}} + okSpec := &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{ + Name: "worker", + Image: "example.com/app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + }}} wantInvalidArgument := func(t *testing.T, rpc string, err error) { t.Helper() @@ -964,7 +757,7 @@ func TestBuildAteomWorkloadSpecForwardsWakeupProbe(t *testing.T) { Name: "with-probe", Image: "main", WakeupProbe: &ateletpb.WakeupProbe{ - HttpGet: &ateletpb.HTTPGetAction{Path: "/health", Port: 8080}, + HttpGet: &ateletpb.HttpGetAction{Path: "/health", Port: 8080}, TimeoutSeconds: 45, }, }, @@ -978,7 +771,7 @@ func TestBuildAteomWorkloadSpecForwardsWakeupProbe(t *testing.T) { { Name: "with-probe", WakeupProbe: &ateompb.WakeupProbe{ - HttpGet: &ateompb.HTTPGetAction{Path: "/health", Port: 8080}, + HttpGet: &ateompb.HttpGetAction{Path: "/health", Port: 8080}, TimeoutSeconds: 45, }, }, @@ -1020,10 +813,10 @@ func TestBuildAteomWorkloadSpecForwardsContainerSpecs(t *testing.T) { func TestBuildAteomWorkloadSpecForwardsDurableDirMounts(t *testing.T) { in := &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ - {Name: "data", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, - {Name: "cache", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, - {Name: "scratch", Source: &ateletpb.Volume_External{External: &ateletpb.ExternalVolumeSource{}}}, - {Name: "system-info", Source: &ateletpb.Volume_SystemInfo{SystemInfo: &ateletpb.SystemInfoVolume{}}}, + {Name: "data", DurableDir: &ateletpb.DurableDirVolume{}}, + {Name: "cache", DurableDir: &ateletpb.DurableDirVolume{}}, + {Name: "scratch", External: &ateletpb.ExternalVolumeSource{}}, + {Name: "system-info", SystemInfo: &ateletpb.SystemInfoVolume{}}, }, Containers: []*ateletpb.Container{ { @@ -1089,7 +882,7 @@ func TestBuildAteomWorkloadSpecValidation(t *testing.T) { name: "missing volume definition", in: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ - {Name: "data", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, + {Name: "data", DurableDir: &ateletpb.DurableDirVolume{}}, }, Containers: []*ateletpb.Container{ { @@ -1117,14 +910,14 @@ func TestBuildAteomWorkloadSpecValidation(t *testing.T) { }, }, }, - wantErr: `container "ctr" mounts volume "data" with unsupported source `, + wantErr: `container "ctr" mounts volume "data" with no source set`, }, { name: "duplicate volume names", in: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ - {Name: "data", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, - {Name: "data", Source: &ateletpb.Volume_External{External: &ateletpb.ExternalVolumeSource{}}}, + {Name: "data", DurableDir: &ateletpb.DurableDirVolume{}}, + {Name: "data", External: &ateletpb.ExternalVolumeSource{}}, }, Containers: []*ateletpb.Container{ { @@ -1407,9 +1200,9 @@ func TestDrainOnShutdownForceStopsAfterTimeout(t *testing.T) { func TestBuildAteomWorkloadSpec_ImageVolumeMounts(t *testing.T) { spec := &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ - {Name: "agent", Source: &ateletpb.Volume_Image{Image: &ateletpb.ImageVolumeSource{}}}, - {Name: "data", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, - {Name: "ext", Source: &ateletpb.Volume_External{External: &ateletpb.ExternalVolumeSource{}}}, + {Name: "agent", Image: &ateletpb.ImageVolumeSource{}}, + {Name: "data", DurableDir: &ateletpb.DurableDirVolume{}}, + {Name: "ext", External: &ateletpb.ExternalVolumeSource{}}, }, Containers: []*ateletpb.Container{{ Name: "app", @@ -1508,16 +1301,42 @@ func writeLocalSnapshot(t *testing.T, dir string, rec sandboxAssetsRecord, conte } } -func validUploadPausedCheckpointRequest() *ateletpb.UploadPausedCheckpointRequest { - return &ateletpb.UploadPausedCheckpointRequest{ - Atespace: "ate-demo", - ActorName: "counter-1", - ActorUid: "123e4567-e89b-12d3-a456-426614174000", - ActorTemplateAtespace: "ate-demo", - ActorTemplateName: "counter", - LocalSnapshotName: "pause-snap-1", - DestinationSnapshotUri: pausedSnapshotURI, - DesiredFidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, +func immediateUploadConfig() *ateletpb.UploadConfig { + return &ateletpb.UploadConfig{ + Mode: ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE, + } +} + +func validCheckpointRequest() *ateletpb.CheckpointRequest { + return &ateletpb.CheckpointRequest{ + Atespace: "ate-demo", + ActorName: "counter-1", + ActorTemplateAtespace: "ate-demo", + ActorTemplateName: "counter", + WorkerPodUid: "422938ba-8860-4983-a25d-d6bcb0a69d4e", + ActorUid: "123e4567-e89b-12d3-a456-426614174000", + Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker"}}}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: "snap-1", + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: testSnapshotURI}, + }, + UploadConfig: immediateUploadConfig(), + } +} + +func validPromoteSnapshotRequest() *ateletpb.PromoteSnapshotRequest { + return &ateletpb.PromoteSnapshotRequest{ + Atespace: "ate-demo", + ActorName: "counter-1", + ActorUid: "123e4567-e89b-12d3-a456-426614174000", + ActorTemplateAtespace: "ate-demo", + ActorTemplateName: "counter", + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: "pause-snap-1", + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: pausedSnapshotURI}, + }, } } @@ -1558,7 +1377,7 @@ func TestUploadLocalCheckpointDir(t *testing.T) { "config.json": "cfg", "memory-ranges": "mem", "data.tar": "data", }) - if _, err := s.uploadLocalCheckpointDir(ctx, validUploadPausedCheckpointRequest(), dir, uri); err != nil { + if _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), dir, uri); err != nil { t.Fatalf("uploadLocalCheckpointDir: %v", err) } want := []string{ @@ -1587,8 +1406,8 @@ func TestUploadLocalCheckpointDir(t *testing.T) { Fidelity: ateattr.SnapshotFidelityRootfs, }, map[string]string{"rootfs-upper-app.tar": "upper", "data.tar": "data"}) - req := validUploadPausedCheckpointRequest() - req.DesiredFidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + req := validPromoteSnapshotRequest() + req.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS if _, err := s.uploadLocalCheckpointDir(ctx, req, dir, uri); err != nil { t.Fatalf("uploadLocalCheckpointDir: %v", err) } @@ -1627,8 +1446,8 @@ func TestUploadLocalCheckpointDir(t *testing.T) { "config.json": "cfg", "memory-ranges": "mem", "data.tar": "data", }) - req := validUploadPausedCheckpointRequest() - req.DesiredFidelity = tc.desired + req := validPromoteSnapshotRequest() + req.Snapshot.Fidelity = tc.desired _, err := s.uploadLocalCheckpointDir(ctx, req, dir, uri) if got := apierror.Code(err); got != codes.FailedPrecondition { t.Fatalf("status.Code = %v (err %v), want FailedPrecondition", got, err) @@ -1649,8 +1468,8 @@ func TestUploadLocalCheckpointDir(t *testing.T) { SnapshotFiles: []string{"data.tar"}, }, map[string]string{"data.tar": "data"}) - req := validUploadPausedCheckpointRequest() - req.DesiredFidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + req := validPromoteSnapshotRequest() + req.Snapshot.Fidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES _, err := s.uploadLocalCheckpointDir(ctx, req, dir, uri) if got := apierror.Code(err); got != codes.FailedPrecondition { t.Fatalf("status.Code = %v (err %v), want FailedPrecondition for a scope-less manifest", got, err) @@ -1666,7 +1485,7 @@ func TestUploadLocalCheckpointDir(t *testing.T) { }} s := newPluginHerder(t, store) - if _, err := s.uploadLocalCheckpointDir(ctx, validUploadPausedCheckpointRequest(), filepath.Join(t.TempDir(), "never-created"), uri); err != nil { + if _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), filepath.Join(t.TempDir(), "never-created"), uri); err != nil { t.Fatalf("uploadLocalCheckpointDir: %v", err) } }) @@ -1674,7 +1493,7 @@ func TestUploadLocalCheckpointDir(t *testing.T) { t.Run("gone locally and remotely crashes the actor", func(t *testing.T) { s := newPluginHerder(t, &recordingObjectStorage{}) - _, err := s.uploadLocalCheckpointDir(ctx, validUploadPausedCheckpointRequest(), filepath.Join(t.TempDir(), "never-created"), uri) + _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), filepath.Join(t.TempDir(), "never-created"), uri) if err == nil { t.Fatal("uploadLocalCheckpointDir succeeded, want error") } @@ -1690,54 +1509,13 @@ func TestUploadLocalCheckpointDir(t *testing.T) { "config.json": "cfg", "memory-ranges": "mem", "data.tar": "data", }) - _, err := s.uploadLocalCheckpointDir(ctx, validUploadPausedCheckpointRequest(), dir, uri) + _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), dir, uri) if err == nil { t.Fatal("uploadLocalCheckpointDir succeeded, want error") } }) } -func TestValidateUploadPausedCheckpointRequest(t *testing.T) { - tests := []struct { - name string - mutate func(*ateletpb.UploadPausedCheckpointRequest) - wantErr bool - }{ - {"valid", func(*ateletpb.UploadPausedCheckpointRequest) {}, false}, - {"valid volumes fidelity", func(r *ateletpb.UploadPausedCheckpointRequest) { - r.DesiredFidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES - }, false}, - {"valid rootfs fidelity", func(r *ateletpb.UploadPausedCheckpointRequest) { - r.DesiredFidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS - }, false}, - {"invalid atespace", func(r *ateletpb.UploadPausedCheckpointRequest) { r.Atespace = "../escape" }, true}, - {"golden atespace rejected", func(r *ateletpb.UploadPausedCheckpointRequest) { r.Atespace = resources.GoldenActorAtespace }, true}, - {"invalid actor name", func(r *ateletpb.UploadPausedCheckpointRequest) { r.ActorName = "UPPER" }, true}, - {"invalid actor uid", func(r *ateletpb.UploadPausedCheckpointRequest) { r.ActorUid = "" }, true}, - {"any actor template identity accepted", func(r *ateletpb.UploadPausedCheckpointRequest) { r.ActorTemplateAtespace = "no/slashes" }, false}, - {"empty actor template identity accepted", func(r *ateletpb.UploadPausedCheckpointRequest) { - r.ActorTemplateAtespace, r.ActorTemplateName = "", "" - }, false}, - {"invalid snapshot name", func(r *ateletpb.UploadPausedCheckpointRequest) { r.LocalSnapshotName = "../escape" }, true}, - {"invalid snapshot uri", func(r *ateletpb.UploadPausedCheckpointRequest) { r.DestinationSnapshotUri = "not-a-uri" }, true}, - {"unspecified fidelity", func(r *ateletpb.UploadPausedCheckpointRequest) { - r.DesiredFidelity = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED - }, true}, - {"invalid fidelity", func(r *ateletpb.UploadPausedCheckpointRequest) { - r.DesiredFidelity = ateletpb.SnapshotFidelity(23) - }, true}, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - req := validUploadPausedCheckpointRequest() - tc.mutate(req) - if err := validateUploadPausedCheckpointRequest(req); (err != nil) != tc.wantErr { - t.Errorf("validateUploadPausedCheckpointRequest err = %v, wantErr %v", err, tc.wantErr) - } - }) - } -} - func TestShouldHaveSnapshots(t *testing.T) { tests := []struct { name string @@ -1747,24 +1525,24 @@ func TestShouldHaveSnapshots(t *testing.T) { { name: "full scope always expects snapshots", req: &ateletpb.CheckpointRequest{ - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Snapshot: &ateletpb.Snapshot{Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, }, want: true, }, { name: "rootfs fidelity always expects snapshots", req: &ateletpb.CheckpointRequest{ - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, + Snapshot: &ateletpb.Snapshot{Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS}, }, want: true, }, { name: "data scope with durable volumes expects snapshots", req: &ateletpb.CheckpointRequest{ - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + Snapshot: &ateletpb.Snapshot{Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, Spec: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ - {Name: "durable", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, + {Name: "durable", DurableDir: &ateletpb.DurableDirVolume{}}, }, }, }, @@ -1773,10 +1551,10 @@ func TestShouldHaveSnapshots(t *testing.T) { { name: "data scope with only CSI volumes does not expect snapshots", req: &ateletpb.CheckpointRequest{ - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + Snapshot: &ateletpb.Snapshot{Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, Spec: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ - {Name: "csi", Source: &ateletpb.Volume_External{External: &ateletpb.ExternalVolumeSource{}}}, + {Name: "csi", External: &ateletpb.ExternalVolumeSource{}}, }, }, }, @@ -1785,11 +1563,11 @@ func TestShouldHaveSnapshots(t *testing.T) { { name: "data scope with both durable and CSI volumes expects snapshots", req: &ateletpb.CheckpointRequest{ - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + Snapshot: &ateletpb.Snapshot{Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, Spec: &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ - {Name: "durable", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, - {Name: "csi", Source: &ateletpb.Volume_External{External: &ateletpb.ExternalVolumeSource{}}}, + {Name: "durable", DurableDir: &ateletpb.DurableDirVolume{}}, + {Name: "csi", External: &ateletpb.ExternalVolumeSource{}}, }, }, }, @@ -1798,7 +1576,7 @@ func TestShouldHaveSnapshots(t *testing.T) { { name: "data scope with no volumes does not expect snapshots", req: &ateletpb.CheckpointRequest{ - Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + Snapshot: &ateletpb.Snapshot{Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, Spec: &ateletpb.WorkloadSpec{}, }, want: false, diff --git a/cmd/atelet/metrics.go b/cmd/atelet/metrics.go index b7ae516955..804020545f 100644 --- a/cmd/atelet/metrics.go +++ b/cmd/atelet/metrics.go @@ -159,7 +159,7 @@ func assetsAfterCollateral(prepFailedPhase string, assets time.Duration) time.Du // the actor that wrote the snapshot. An empty result means the manifest has not // been read yet, so the kind is not knowable. func restoreSnapshotKind(req *ateletpb.RestoreRequest, rec *sandboxAssetsRecord) string { - if req.GetType() == ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL { + if req.GetSnapshot().GetObject() == nil { return ateattr.SnapshotKindLocal } if rec == nil { @@ -177,7 +177,7 @@ func restoreSnapshotKind(req *ateletpb.RestoreRequest, rec *sandboxAssetsRecord) // writes the node-local one, a suspend the actor's durable latest, and a commit // by an actor in the golden atespace the template's golden image. func checkpointSnapshotKind(req *ateletpb.CheckpointRequest) string { - if req.GetType() == ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL { + if req.GetUploadConfig().GetMode() == ateletpb.UploadMode_UPLOAD_MODE_DEFERRED { return ateattr.SnapshotKindLocal } if req.GetAtespace() == resources.GoldenActorAtespace { diff --git a/cmd/atelet/metrics_test.go b/cmd/atelet/metrics_test.go index 5c58a7ed3b..8aaf437113 100644 --- a/cmd/atelet/metrics_test.go +++ b/cmd/atelet/metrics_test.go @@ -280,6 +280,13 @@ func TestAssetsAfterCollateral(t *testing.T) { } func TestRestoreSnapshotKind(t *testing.T) { + localSnapshot := &ateletpb.Snapshot{ + SnapshotUuid: "snap-1", + } + externalSnapshot := &ateletpb.Snapshot{ + SnapshotUuid: "snap-1", + Object: &ateletpb.ObjectStorage{SnapshotUri: "gs://bucket/snap"}, + } tests := []struct { name string req *ateletpb.RestoreRequest @@ -288,37 +295,37 @@ func TestRestoreSnapshotKind(t *testing.T) { }{ { name: "local pause snapshot", - req: &ateletpb.RestoreRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL}, + req: &ateletpb.RestoreRequest{Snapshot: localSnapshot}, rec: &sandboxAssetsRecord{Atespace: "team-a"}, want: ateattr.SnapshotKindLocal, }, { name: "local restore is classifiable before the manifest is read", - req: &ateletpb.RestoreRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL}, + req: &ateletpb.RestoreRequest{Snapshot: localSnapshot}, rec: nil, want: ateattr.SnapshotKindLocal, }, { name: "external snapshot written by a golden actor", - req: &ateletpb.RestoreRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL}, + req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: &sandboxAssetsRecord{Atespace: resources.GoldenActorAtespace}, want: ateattr.SnapshotKindGolden, }, { name: "external snapshot written by a tenant actor", - req: &ateletpb.RestoreRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL}, + req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: &sandboxAssetsRecord{Atespace: "team-a"}, want: ateattr.SnapshotKindLatest, }, { name: "manifest predating the identity fields", - req: &ateletpb.RestoreRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL}, + req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: &sandboxAssetsRecord{}, want: ateattr.SnapshotKindLatest, }, { name: "external kind is unknowable until the manifest is read", - req: &ateletpb.RestoreRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL}, + req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: nil, want: "", }, @@ -333,6 +340,8 @@ func TestRestoreSnapshotKind(t *testing.T) { } func TestCheckpointSnapshotKind(t *testing.T) { + deferred := &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED} + immediate := &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE} tests := []struct { name string req *ateletpb.CheckpointRequest @@ -340,22 +349,22 @@ func TestCheckpointSnapshotKind(t *testing.T) { }{ { name: "pause writes the node-local snapshot", - req: &ateletpb.CheckpointRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, Atespace: "team-a"}, + req: &ateletpb.CheckpointRequest{UploadConfig: deferred, Atespace: "team-a"}, want: ateattr.SnapshotKindLocal, }, { name: "suspend writes the actor's durable snapshot", - req: &ateletpb.CheckpointRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, Atespace: "team-a"}, + req: &ateletpb.CheckpointRequest{UploadConfig: immediate, Atespace: "team-a"}, want: ateattr.SnapshotKindLatest, }, { name: "a golden actor's commit writes the template's golden", - req: &ateletpb.CheckpointRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, Atespace: resources.GoldenActorAtespace}, + req: &ateletpb.CheckpointRequest{UploadConfig: immediate, Atespace: resources.GoldenActorAtespace}, want: ateattr.SnapshotKindGolden, }, { name: "a local checkpoint in the golden atespace is still local", - req: &ateletpb.CheckpointRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL, Atespace: resources.GoldenActorAtespace}, + req: &ateletpb.CheckpointRequest{UploadConfig: deferred, Atespace: resources.GoldenActorAtespace}, want: ateattr.SnapshotKindLocal, }, } diff --git a/cmd/atelet/sandbox_assets.go b/cmd/atelet/sandbox_assets.go index 305c2f667e..8b70d56ab1 100644 --- a/cmd/atelet/sandbox_assets.go +++ b/cmd/atelet/sandbox_assets.go @@ -196,11 +196,11 @@ func runscPathFor(paths map[string]string) string { // the shared static-files cache and returns its local path. On a cache hit it // returns immediately. func (s *AteomHerder) fetchAsset(ctx context.Context, entry assetEntry) (string, error) { - if err := resources.ValidateRunscHash(entry.SHA256); err != nil { + if err := resources.ValidateAssetHash(entry.SHA256); err != nil { return "", wrapFileSystemErr("while validating asset hash", err) } - localPath := ateletpath.RunSCBinaryPath(entry.SHA256) + localPath := ateletpath.AssetPath(entry.SHA256) _, err := os.Stat(localPath) if err == nil { slog.DebugContext(ctx, "Sandbox asset cache hit", slog.String("path", localPath)) @@ -233,7 +233,7 @@ func (s *AteomHerder) fetchAsset(ctx context.Context, entry assetEntry) (string, // the shared static-files cache, returning the local path of the extracted // `runsc` binary. func (s *AteomHerder) fetchGVisorRelease(ctx context.Context, entry assetEntry) (string, error) { - if err := resources.ValidateRunscHash(entry.SHA256); err != nil { + if err := resources.ValidateAssetHash(entry.SHA256); err != nil { return "", wrapFileSystemErr("while validating asset hash", err) } diff --git a/cmd/atelet/sandbox_assets_plugin_test.go b/cmd/atelet/sandbox_assets_plugin_test.go index c72b25980b..15ebfe7418 100644 --- a/cmd/atelet/sandbox_assets_plugin_test.go +++ b/cmd/atelet/sandbox_assets_plugin_test.go @@ -227,7 +227,7 @@ func TestFetchAssetThroughPlugin(t *testing.T) { t.Run("cache hit", func(t *testing.T) { f := newAssetHerderFixture(t, "s3") sum := f.put(t, "kata-assets/vmlinux", content) - cached := ateletpath.RunSCBinaryPath(sum) + cached := ateletpath.AssetPath(sum) if err := os.WriteFile(cached, content, 0o755); err != nil { t.Fatal(err) } diff --git a/cmd/atelet/sandbox_prewarm.go b/cmd/atelet/sandbox_prewarm.go index 1395643cea..64f6b290ff 100644 --- a/cmd/atelet/sandbox_prewarm.go +++ b/cmd/atelet/sandbox_prewarm.go @@ -57,6 +57,12 @@ var prewarmTimeout = 5 * time.Minute // permanently broken config forever. const prewarmMaxRetries = 8 +// prewarmFinished observes the result of each prewarm attempt, by config +// name, once the worker's call has returned. A no-op in production; a var so +// a test can wait for the worker's own prewarm call to return before it tears +// down what that call writes into. +var prewarmFinished = func(string, error) {} + // sandboxAssetFetcher is the one slice of AteomHerder the prewarmer needs. // Prewarming through the same method as the Run/Restore path keeps the two // fetches from ever diverging on cache layout or validation. @@ -226,6 +232,7 @@ func (p *sandboxPrewarmer) process(ctx context.Context, name string) { } if err == nil { err = p.prewarm(ctx, cfg) + prewarmFinished(name, err) } if err == nil { p.queue.Forget(name) diff --git a/cmd/atelet/sandbox_prewarm_test.go b/cmd/atelet/sandbox_prewarm_test.go index 7ff74a939e..3492369736 100644 --- a/cmd/atelet/sandbox_prewarm_test.go +++ b/cmd/atelet/sandbox_prewarm_test.go @@ -380,7 +380,7 @@ func TestPrewarmDefaultVersionOnly(t *testing.T) { if err := p.prewarm(ctx, cfg); err != nil { t.Fatalf("prewarm: %v", err) } - if _, err := os.Stat(ateletpath.RunSCBinaryPath(otherSHA)); !errors.Is(err, os.ErrNotExist) { + if _, err := os.Stat(ateletpath.AssetPath(otherSHA)); !errors.Is(err, os.ErrNotExist) { t.Errorf("non-default version asset was prewarmed (stat err = %v)", err) } @@ -402,10 +402,26 @@ func TestPrewarmDefaultVersionOnly(t *testing.T) { // fake clientset flows through the informer into the prewarm worker, which // lands the asset in the static-files cache without any Run/Restore request. func TestSandboxAssetPrewarmDownloads(t *testing.T) { - origDir, origJitter := nodepath.StaticFilesDir, prewarmMaxJitter + origDir, origJitter, origFinished := nodepath.StaticFilesDir, prewarmMaxJitter, prewarmFinished nodepath.StaticFilesDir = t.TempDir() prewarmMaxJitter = 0 - t.Cleanup(func() { nodepath.StaticFilesDir, prewarmMaxJitter = origDir, origJitter }) + // The worker's own prewarm call returns only when the asset has landed + // and the pause-image pull has finished: it waits for the pull on a live + // context, while the pull runs detached from that context (see + // imagecache.Store.EnsureImage). Waiting for the call keeps the pull's + // last writes from racing the TempDir cleanup. + prewarmed := make(chan error, 1) + prewarmFinished = func(_ string, err error) { + // Non-blocking: a failed attempt is retried, and the worker must not + // block on a result nobody reads. + select { + case prewarmed <- err: + default: + } + } + t.Cleanup(func() { + nodepath.StaticFilesDir, prewarmMaxJitter, prewarmFinished = origDir, origJitter, origFinished + }) content := []byte("runsc binary bytes") sha := fmt.Sprintf("%x", sha256.Sum256(content)) @@ -434,17 +450,16 @@ func TestSandboxAssetPrewarmDownloads(t *testing.T) { defer close(stopCh) factory.Start(stopCh) - wantPath := ateletpath.RunSCBinaryPath(sha) - deadline := time.Now().Add(10 * time.Second) - for { - if _, err := os.Stat(wantPath); err == nil { - return - } else if !errors.Is(err, os.ErrNotExist) { - t.Fatalf("stat %s: %v", wantPath, err) - } - if time.Now().After(deadline) { - t.Fatalf("asset never prewarmed to %s", wantPath) + select { + case err := <-prewarmed: + if err != nil { + t.Fatalf("prewarm: %v", err) } - time.Sleep(10 * time.Millisecond) + case <-time.After(30 * time.Second): + t.Fatal("prewarm never finished") + } + wantPath := ateletpath.AssetPath(sha) + if _, err := os.Stat(wantPath); err != nil { + t.Fatalf("asset not prewarmed to %s: %v", wantPath, err) } } diff --git a/cmd/atelet/snapshot_transfer_test.go b/cmd/atelet/snapshot_transfer_test.go index 5ef23cb368..b2f89d7fb0 100644 --- a/cmd/atelet/snapshot_transfer_test.go +++ b/cmd/atelet/snapshot_transfer_test.go @@ -190,7 +190,7 @@ func TestObjectStorePluginErrorCodeThroughAtelet(t *testing.T) { {"paused upload", true, func(ctx context.Context, s *AteomHerder) error { dir := t.TempDir() writeLocalSnapshot(t, dir, rec, map[string]string{"a": "a"}) - _, err := s.uploadLocalCheckpointDir(ctx, validUploadPausedCheckpointRequest(), dir, uri) + _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), dir, uri) return err }}, {"checkpoint upload", false, func(ctx context.Context, s *AteomHerder) error { @@ -256,7 +256,7 @@ func TestObjectStorePluginErrorKeepsCredentialsOut(t *testing.T) { {"paused upload", "", func(ctx context.Context, s *AteomHerder) error { dir := t.TempDir() writeLocalSnapshot(t, dir, rec, map[string]string{"a": "a"}) - _, err := s.uploadLocalCheckpointDir(ctx, validUploadPausedCheckpointRequest(), dir, uri) + _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), dir, uri) return err }}, } @@ -408,7 +408,7 @@ func TestCheckpointUploadWaitsOutPluginOutage(t *testing.T) { } } - if _, err := s.uploadLocalCheckpointDir(ctx, validUploadPausedCheckpointRequest(), pausedDir, uri); apierror.Code(err) != codes.Unavailable { + if _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), pausedDir, uri); apierror.Code(err) != codes.Unavailable { t.Fatalf("paused upload during the outage = %v, want %s", err, codes.Unavailable) } diff --git a/cmd/atelet/systeminfovolume.go b/cmd/atelet/systeminfovolume.go index ebded77046..05f1c444e3 100644 --- a/cmd/atelet/systeminfovolume.go +++ b/cmd/atelet/systeminfovolume.go @@ -168,17 +168,17 @@ func (r *systemInfoVolumeRefresher) DeregisterOwned(owner *registeredActor) { // collectData builds the volume's contents keyed by volume-relative path. func (r *systemInfoVolumeRefresher) collectData(ref resources.ActorRef, actorUID string, si *ateletpb.SystemInfoVolume) (payload map[string][]byte, err error) { payload = map[string][]byte{} - for _, dataSourceAny := range si.GetDataSources() { - switch dataSource := dataSourceAny.GetDataSource().(type) { - case *ateletpb.SystemInfoDataSource_TrustBundle: - tb := dataSource.TrustBundle + for _, dataSource := range si.GetDataSources() { + switch { + case dataSource.GetTrustBundle() != nil: + tb := dataSource.GetTrustBundle() pemBundle, err := r.bundles.Combined(tb.GetNames()) if err != nil { return nil, fmt.Errorf("system-info projection %q: %w", tb.GetPath(), err) } payload[tb.GetPath()] = pemBundle - case *ateletpb.SystemInfoDataSource_ActorMetadata: - for _, item := range dataSource.ActorMetadata.GetItems() { + case dataSource.GetActorMetadata() != nil: + for _, item := range dataSource.GetActorMetadata().GetItems() { var value string switch item.GetField() { case ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME: diff --git a/cmd/atelet/systeminfovolume_test.go b/cmd/atelet/systeminfovolume_test.go index d08928d77c..c5286c3b33 100644 --- a/cmd/atelet/systeminfovolume_test.go +++ b/cmd/atelet/systeminfovolume_test.go @@ -111,9 +111,7 @@ func (s *ctbStore) remove(t *testing.T) { func trustVolumeSpec(relPath string) *ateletpb.SystemInfoVolume { return &ateletpb.SystemInfoVolume{ DataSources: []*ateletpb.SystemInfoDataSource{ - {DataSource: &ateletpb.SystemInfoDataSource_TrustBundle{ - TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{trustbundle.EgressName}, Path: relPath}, - }}, + {TrustBundle: &ateletpb.TrustBundleDataSource{Names: []string{trustbundle.EgressName}, Path: relPath}}, }, } } @@ -121,13 +119,11 @@ func trustVolumeSpec(relPath string) *ateletpb.SystemInfoVolume { func metadataVolumeSpec() *ateletpb.SystemInfoVolume { return &ateletpb.SystemInfoVolume{ DataSources: []*ateletpb.SystemInfoDataSource{ - {DataSource: &ateletpb.SystemInfoDataSource_ActorMetadata{ - ActorMetadata: &ateletpb.ActorMetadataDataSource{ - Items: []*ateletpb.ActorMetadataItem{ - {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME, Path: "actor-name"}, - {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE, Path: "atespace"}, - {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID, Path: "identity/actor-uid"}, - }, + {ActorMetadata: &ateletpb.ActorMetadataDataSource{ + Items: []*ateletpb.ActorMetadataItem{ + {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME, Path: "actor-name"}, + {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE, Path: "atespace"}, + {Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID, Path: "identity/actor-uid"}, }, }}, }, @@ -687,9 +683,9 @@ func TestSystemInfoVolumeRefresher_FailedReplacementDropsRegistration(t *testing func TestSystemInfoVolumesFor(t *testing.T) { spec := &ateletpb.WorkloadSpec{Volumes: []*ateletpb.Volume{ - {Name: "data", Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}}}, - {Name: "trust", Source: &ateletpb.Volume_SystemInfo{SystemInfo: trustVolumeSpec("ca.pem")}}, - {Name: "meta", Source: &ateletpb.Volume_SystemInfo{SystemInfo: metadataVolumeSpec()}}, + {Name: "data", DurableDir: &ateletpb.DurableDirVolume{}}, + {Name: "trust", SystemInfo: trustVolumeSpec("ca.pem")}, + {Name: "meta", SystemInfo: metadataVolumeSpec()}, }} got := systemInfoVolumesFor("uid-1", spec) if len(got) != 2 || got[0].Name != "trust" || got[1].Name != "meta" { diff --git a/cmd/atelet/volumes_test.go b/cmd/atelet/volumes_test.go index e3b13ac9b3..cab35adcb2 100644 --- a/cmd/atelet/volumes_test.go +++ b/cmd/atelet/volumes_test.go @@ -91,27 +91,21 @@ func TestUnmountExternalVolumes(t *testing.T) { extVol1 := &ateletpb.Volume{ Name: "vol-1", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "mock-vol-1", - VolumeType: "mock-driver", - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "mock-vol-1", + VolumeType: "mock-driver", }, } extVol2 := &ateletpb.Volume{ Name: "vol-2", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "mock-vol-2", - VolumeType: "mock-driver", - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "mock-vol-2", + VolumeType: "mock-driver", }, } durableVol := &ateletpb.Volume{ - Name: "durable-1", - Source: &ateletpb.Volume_DurableDir{ - DurableDir: &ateletpb.DurableDirVolume{}, - }, + Name: "durable-1", + DurableDir: &ateletpb.DurableDirVolume{}, } t.Run("success", func(t *testing.T) { @@ -177,29 +171,23 @@ func TestMountExternalVolumes(t *testing.T) { extVol1 := &ateletpb.Volume{ Name: "vol-1", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "mock-vol-1", - VolumeType: "mock-driver", - VolumeContext: map[string]string{"key": "val1"}, - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "mock-vol-1", + VolumeType: "mock-driver", + VolumeContext: map[string]string{"key": "val1"}, }, } extVol2 := &ateletpb.Volume{ Name: "vol-2", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "mock-vol-2", - VolumeType: "mock-driver", - VolumeContext: map[string]string{"key": "val2"}, - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "mock-vol-2", + VolumeType: "mock-driver", + VolumeContext: map[string]string{"key": "val2"}, }, } durableVol := &ateletpb.Volume{ - Name: "durable-1", - Source: &ateletpb.Volume_DurableDir{ - DurableDir: &ateletpb.DurableDirVolume{}, - }, + Name: "durable-1", + DurableDir: &ateletpb.DurableDirVolume{}, } t.Run("success mounts external volumes and creates mount directory", func(t *testing.T) { @@ -275,11 +263,9 @@ func TestMountExternalVolumes(t *testing.T) { unknownVol := &ateletpb.Volume{ Name: "vol-unknown", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "mock-vol-unknown", - VolumeType: "unknown-driver", - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "mock-vol-unknown", + VolumeType: "unknown-driver", }, } @@ -394,20 +380,16 @@ func TestVolumeHostDirectoryCleanup(t *testing.T) { extVol1 := &ateletpb.Volume{ Name: "vol-1", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "mock-vol-1", - VolumeType: "mock-driver", - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "mock-vol-1", + VolumeType: "mock-driver", }, } extVol2 := &ateletpb.Volume{ Name: "vol-2", - Source: &ateletpb.Volume_External{ - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: "mock-vol-2", - VolumeType: "mock-driver", - }, + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: "mock-vol-2", + VolumeType: "mock-driver", }, } diff --git a/cmd/atelet/worker_sandbox_syncer.go b/cmd/atelet/worker_sandbox_syncer.go new file mode 100644 index 0000000000..b8094ae63a --- /dev/null +++ b/cmd/atelet/worker_sandbox_syncer.go @@ -0,0 +1,362 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +// This file decides when each worker pod on this node is synced: +// - start: wire the syncer to the Pod, WorkerPool and SandboxConfig +// informers +// - handlers: turn informer events into pod UIDs on the queue, with a +// random delay when a pool or config change reaches every node at once +// - process: for each UID, sync the pod, or remove its worker directory +// once the pod is gone +// - finish: forget the UID, or retry it with backoff up to +// syncMaxRetries times +// +// worker_sandbox_versions.go builds the sandbox config version file for each +// worker pod. + +import ( + "context" + "errors" + "fmt" + "log/slog" + "math/rand/v2" + "os" + "path/filepath" + "time" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/labels" + coreinformers "k8s.io/client-go/informers/core/v1" + corelisters "k8s.io/client-go/listers/core/v1" + "k8s.io/client-go/tools/cache" + "k8s.io/client-go/util/workqueue" + + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/api/v1alpha1" + informersv1alpha1 "github.com/agent-substrate/substrate/pkg/client/informers/externalversions/api/v1alpha1" +) + +// podUIDIndex indexes this node's worker pods by UID, the syncer's queue key. +// The key is the UID, not namespace/name, because worker directories are +// named by UID, and a directory left on disk is all the syncer has of a pod +// that is gone. +const podUIDIndex = "uid" + +// orphanMinAge is how old a worker directory found on disk at startup must be +// before the syncer looks at it. The grace covers an ateom that made its own +// directory before its pod reached atelet's cache. +const orphanMinAge = time.Minute + +// syncMaxJitter spreads the fleet's file rewrites after a WorkerPool or +// SandboxConfig change. Every atelet observes the change within about a +// second, and each rewrite makes the worker register again, so without +// jitter every worker in the fleet would hit ateapi at once. The jitter is +// applied as the enqueue delay, so duplicate events inside the window +// collapse into one queue entry. A pod bound to this node is +// enqueued without it, since only this atelet acts on it and its worker waits +// for the file. +const syncMaxJitter = 30 * time.Second + +// syncMaxRetries bounds how often one pod's failed sync is retried before it +// is dropped. The pod is synced again only when its pool switches to another +// SandboxConfig, the config's spec changes, or atelet restarts. The backoff +// doubles from one second, so this covers a few minutes of informer lag or an +// unresolved pool without retrying forever. +const syncMaxRetries = 8 + +// workerSandboxSyncer keeps a sandbox versions file in each local worker's +// directory under ateomsDir: the runtimes of the enabled versions of the +// SandboxConfig the worker's pool uses. It creates a worker's directory once +// it writes the pod's file and removes it when the pod is gone. +type workerSandboxSyncer struct { + resolver *workerSandboxResolver + // pods holds this node's worker pods, indexed by podUIDIndex. + pods cache.Indexer + // queue carries pod UIDs. One worker drains it, so a pod's file is never + // written and its directory removed at the same time. + queue workqueue.TypedRateLimitingInterface[string] + ateomsDir string // nodepath.AteomsDir(); a temp dir in tests + minAge time.Duration + maxJitter time.Duration +} + +func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, ateomsDir string) *workerSandboxSyncer { + return &workerSandboxSyncer{ + resolver: resolver, + pods: pods, + // Failures are informer lag and unresolved pools, both on the scale + // of seconds to minutes. + queue: workqueue.NewTypedRateLimitingQueue( + workqueue.NewTypedItemExponentialFailureRateLimiter[string](time.Second, 5*time.Minute)), + ateomsDir: ateomsDir, + minAge: orphanMinAge, + maxJitter: syncMaxJitter, + } +} + +// startWorkerSandboxSyncer wires a syncer to the given informers, which the +// caller starts, and runs it until ctx ends. pods must hold only this node's +// worker pods and must not have started yet. Worker directories live under +// ateomsDir. +// +// The syncer never holds up atelet startup: until the informers sync, or +// while a list/watch fails (e.g. Forbidden while an RBAC rollout lags the +// binary), files wait and existing ones stay as they are. +func startWorkerSandboxSyncer(ctx context.Context, pods coreinformers.PodInformer, workerPools informersv1alpha1.WorkerPoolInformer, configs informersv1alpha1.SandboxConfigInformer, ateomsDir string) (*workerSandboxSyncer, error) { + if err := pods.Informer().AddIndexers(cache.Indexers{podUIDIndex: func(obj any) ([]string, error) { + pod, ok := obj.(*corev1.Pod) + if !ok { + return nil, nil + } + return []string{string(pod.UID)}, nil + }}); err != nil { + return nil, fmt.Errorf("while indexing worker pods by UID: %w", err) + } + resolver := &workerSandboxResolver{ + workerPools: workerPools.Lister(), + configs: configs.Lister(), + } + s := newWorkerSandboxSyncer(resolver, pods.Informer().GetIndexer(), ateomsDir) + if err := watch(ctx, "worker pod", pods.Informer(), s.podHandler()); err != nil { + return nil, err + } + if err := watch(ctx, "WorkerPool", workerPools.Informer(), s.workerPoolHandler()); err != nil { + return nil, err + } + if err := watch(ctx, "SandboxConfig", configs.Informer(), s.sandboxConfigHandler()); err != nil { + return nil, err + } + go s.run(ctx, pods.Informer().HasSynced, workerPools.Informer().HasSynced, configs.Informer().HasSynced) + slog.InfoContext(ctx, "Worker sandbox versions syncer started") + return s, nil +} + +// watch registers handler on inf and logs inf's list/watch failures. +func watch(ctx context.Context, kind string, inf cache.SharedIndexInformer, handler cache.ResourceEventHandler) error { + if err := inf.SetWatchErrorHandler(func(_ *cache.Reflector, err error) { + slog.WarnContext(ctx, kind+" list/watch failed; workers on this node cannot update their sandbox versions until it recovers", slog.Any("err", err)) + }); err != nil { + // Fails only on an already-started informer, where the default + // reflector logging applies. + slog.InfoContext(ctx, "Could not set watch error handler", slog.String("kind", kind), slog.Any("err", err)) + } + if _, err := inf.AddEventHandler(handler); err != nil { + return fmt.Errorf("while registering the %s handler: %w", kind, err) + } + return nil +} + +// podHandler enqueues a pod when it is bound here and when it is gone. The +// initial list replays every existing pod as an Add, so a restarted atelet +// rewrites every file from current state. +func (s *workerSandboxSyncer) podHandler() cache.ResourceEventHandler { + enqueue := func(obj any) { + if d, ok := obj.(cache.DeletedFinalStateUnknown); ok { + obj = d.Obj + } + if pod, ok := obj.(*corev1.Pod); ok { + s.queue.Add(string(pod.UID)) + } + } + return cache.ResourceEventHandlerFuncs{AddFunc: enqueue, DeleteFunc: enqueue} +} + +// workerPoolHandler enqueues a pool's local pods when it appears and when the +// config its workers run changes. +func (s *workerSandboxSyncer) workerPoolHandler() cache.ResourceEventHandler { + return cache.ResourceEventHandlerFuncs{ + AddFunc: func(obj any) { + if pool, ok := obj.(*v1alpha1.WorkerPool); ok { + s.enqueuePool(pool, 0) + } + }, + UpdateFunc: func(oldObj, newObj any) { + oldPool, oldOK := oldObj.(*v1alpha1.WorkerPool) + newPool, newOK := newObj.(*v1alpha1.WorkerPool) + if oldOK && newOK && poolConfigName(oldPool) != poolConfigName(newPool) { + s.enqueuePool(newPool, s.maxJitter) + } + }, + } +} + +// sandboxConfigHandler enqueues the local pods of every pool that runs the +// config, when it appears and when its spec changes. +func (s *workerSandboxSyncer) sandboxConfigHandler() cache.ResourceEventHandler { + enqueue := func(cfg *v1alpha1.SandboxConfig) { + pools, err := s.resolver.workerPools.List(labels.Everything()) + if err != nil { + return + } + for _, pool := range pools { + if poolConfigName(pool) == cfg.Name { + s.enqueuePool(pool, s.maxJitter) + } + } + } + return cache.ResourceEventHandlerFuncs{ + AddFunc: func(obj any) { + if cfg, ok := obj.(*v1alpha1.SandboxConfig); ok { + enqueue(cfg) + } + }, + UpdateFunc: func(oldObj, newObj any) { + oldCfg, oldOK := oldObj.(*v1alpha1.SandboxConfig) + newCfg, newOK := newObj.(*v1alpha1.SandboxConfig) + // Generation moves on spec changes only. + if oldOK && newOK && oldCfg.Generation != newCfg.Generation { + enqueue(newCfg) + } + }, + } +} + +// enqueuePool enqueues every local worker pod of pool, each after a random +// delay below maxDelay when it is not zero. +func (s *workerSandboxSyncer) enqueuePool(pool *v1alpha1.WorkerPool, maxDelay time.Duration) { + pods, err := corelisters.NewPodLister(s.pods).Pods(pool.Namespace).List(labels.SelectorFromSet(labels.Set{workerPoolLabel: pool.Name})) + if err != nil { + return + } + for _, pod := range pods { + if maxDelay > 0 { + s.queue.AddAfter(string(pod.UID), rand.N(maxDelay)) + continue + } + s.queue.Add(string(pod.UID)) + } +} + +// enqueueExistingDirs enqueues the UID of every worker directory on disk, so +// one left by a pod that went away while atelet was down is removed. A +// directory younger than minAge is enqueued once it reaches that age. +func (s *workerSandboxSyncer) enqueueExistingDirs(ctx context.Context) { + entries, err := os.ReadDir(s.ateomsDir) + if err != nil { + if !errors.Is(err, os.ErrNotExist) { + slog.WarnContext(ctx, "Could not list worker directories", slog.String("dir", s.ateomsDir), slog.Any("err", err)) + } + return + } + for _, e := range entries { + if !e.IsDir() || resources.ValidateWorkerPodUID(e.Name()) != nil { + continue + } + info, err := e.Info() + if err != nil { + // Removed since the listing. + continue + } + if age := time.Since(info.ModTime()); age < s.minAge { + s.queue.AddAfter(e.Name(), s.minAge-age) + continue + } + s.queue.Add(e.Name()) + } +} + +// run drains the queue until ctx ends. It starts only once every cache in +// synced has synced: +// - before the pod cache syncs, every pod looks gone +// - before the WorkerPool and SandboxConfig caches sync, no pod resolves +// and each sync would spend a retry +func (s *workerSandboxSyncer) run(ctx context.Context, synced ...cache.InformerSynced) { + go func() { + <-ctx.Done() + s.queue.ShutDown() + }() + if !cache.WaitForCacheSync(ctx.Done(), synced...) { + return + } + s.enqueueExistingDirs(ctx) + for { + uid, shutdown := s.queue.Get() + if shutdown { + return + } + s.process(ctx, uid) + } +} + +// process brings the directory of the pod with the given UID up to date: +// written from its pool's current config while the pod exists, removed once +// it is gone. +func (s *workerSandboxSyncer) process(ctx context.Context, uid string) { + defer s.queue.Done(uid) + pod, err := s.podByUID(uid) + switch { + case err != nil: + case pod == nil: + err = s.removeDir(ctx, uid) + case pod.DeletionTimestamp != nil: + // The Delete event removes the directory. + default: + err = s.syncPod(ctx, pod) + } + s.finish(ctx, uid, err) +} + +// podByUID returns the cached pod with the given UID, or nil if there is none. +func (s *workerSandboxSyncer) podByUID(uid string) (*corev1.Pod, error) { + objs, err := s.pods.ByIndex(podUIDIndex, uid) + if err != nil { + // Only a missing index fails, and startWorkerSandboxSyncer adds it. + return nil, fmt.Errorf("%w: %w", errPermanent, err) + } + if len(objs) == 0 { + return nil, nil + } + pod, ok := objs[0].(*corev1.Pod) + if !ok { + return nil, fmt.Errorf("%w: pod cache holds a %T", errPermanent, objs[0]) + } + return pod, nil +} + +// finish forgets uid on success or errPermanent, and otherwise retries it with +// backoff up to syncMaxRetries times. A sync that fails before it writes +// leaves the existing file in place. +func (s *workerSandboxSyncer) finish(ctx context.Context, uid string, err error) { + switch { + case err == nil: + s.queue.Forget(uid) + case errors.Is(err, errPermanent): + slog.WarnContext(ctx, "Cannot sync worker sandbox versions; not retrying until the pool's SandboxConfig changes or atelet restarts", slog.String("pod_uid", uid), slog.Any("err", err)) + s.queue.Forget(uid) + case ctx.Err() != nil: + // Shutting down, not a sync failure. + case s.queue.NumRequeues(uid) < syncMaxRetries: + slog.WarnContext(ctx, "Could not sync worker sandbox versions; will retry", + slog.String("pod_uid", uid), slog.Int("retries", s.queue.NumRequeues(uid)), slog.Any("err", err)) + s.queue.AddRateLimited(uid) + default: + slog.WarnContext(ctx, "Could not sync worker sandbox versions; giving up until the pool's SandboxConfig changes or atelet restarts", + slog.String("pod_uid", uid), slog.Int("retries", syncMaxRetries), slog.Any("err", err)) + s.queue.Forget(uid) + } +} + +func (s *workerSandboxSyncer) removeDir(ctx context.Context, uid string) error { + if err := os.RemoveAll(s.podDir(uid)); err != nil { + return fmt.Errorf("while removing worker directory: %w", err) + } + slog.DebugContext(ctx, "Removed the directory of a worker pod that is gone", slog.String("pod_uid", uid)) + return nil +} + +func (s *workerSandboxSyncer) podDir(uid string) string { + return filepath.Join(s.ateomsDir, uid) +} diff --git a/cmd/atelet/worker_sandbox_syncer_test.go b/cmd/atelet/worker_sandbox_syncer_test.go new file mode 100644 index 0000000000..0832838689 --- /dev/null +++ b/cmd/atelet/worker_sandbox_syncer_test.go @@ -0,0 +1,680 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "errors" + "os" + "path/filepath" + "slices" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + k8sruntime "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + kubeinformers "k8s.io/client-go/informers" + kubefake "k8s.io/client-go/kubernetes/fake" + "k8s.io/client-go/tools/cache" + "k8s.io/client-go/util/workqueue" + + "github.com/agent-substrate/substrate/internal/sandboxversions" + "github.com/agent-substrate/substrate/pkg/api/v1alpha1" + atefake "github.com/agent-substrate/substrate/pkg/client/clientset/versioned/fake" + "github.com/agent-substrate/substrate/pkg/client/informers/externalversions" + listersv1alpha1 "github.com/agent-substrate/substrate/pkg/client/listers/api/v1alpha1" +) + +// workerPod is a worker pod on this node with the given UID, in pool when +// pool is not "". +func workerPod(uid, pool string) *corev1.Pod { + pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: "workers", Name: "worker-" + uid, UID: types.UID(uid)}} + if pool != "" { + pod.Labels = map[string]string{workerPoolLabel: pool} + } + return pod +} + +// resolvedPool is a gVisor WorkerPool whose class resolved to config, or has +// not resolved when config is "". +func resolvedPool(name, config string) *v1alpha1.WorkerPool { + pool := &v1alpha1.WorkerPool{ObjectMeta: metav1.ObjectMeta{Namespace: "workers", Name: name}} + pool.Spec.SandboxClasses = []v1alpha1.WorkerPoolSandboxClass{{Name: v1alpha1.SandboxClassGvisor}} + class := v1alpha1.WorkerPoolSandboxClassStatus{Name: v1alpha1.SandboxClassGvisor} + if config != "" { + class.ConfigRef = &v1alpha1.SandboxConfigReference{Name: config} + } + pool.Status.SandboxClasses = []v1alpha1.WorkerPoolSandboxClassStatus{class} + return pool +} + +// newTestSyncer serves objs from synced caches and keeps worker directories +// under a temp dir. +func newTestSyncer(t *testing.T, objs ...any) *workerSandboxSyncer { + t.Helper() + newIndexer := func() cache.Indexer { + return cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}) + } + pods := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{ + cache.NamespaceIndex: cache.MetaNamespaceIndexFunc, + podUIDIndex: func(obj any) ([]string, error) { return []string{string(obj.(*corev1.Pod).UID)}, nil }, + }) + pools, configs := newIndexer(), newIndexer() + for _, obj := range objs { + var err error + switch obj.(type) { + case *corev1.Pod: + err = pods.Add(obj) + case *v1alpha1.WorkerPool: + err = pools.Add(obj) + case *v1alpha1.SandboxConfig: + err = configs.Add(obj) + default: + t.Fatalf("unexpected object %T", obj) + } + if err != nil { + t.Fatal(err) + } + } + s := newWorkerSandboxSyncer(&workerSandboxResolver{ + workerPools: listersv1alpha1.NewWorkerPoolLister(pools), + configs: listersv1alpha1.NewSandboxConfigLister(configs), + }, pods, t.TempDir()) + // Tests that check the jitter set it. + s.maxJitter = 0 + return s +} + +// drain returns the UIDs waiting in the syncer's queue, sorted. +func drain(s *workerSandboxSyncer) []string { + var got []string + for s.queue.Len() > 0 { + uid, _ := s.queue.Get() + got = append(got, uid) + s.queue.Done(uid) + } + slices.Sort(got) + return got +} + +func TestWorkerSandboxSyncerEvents(t *testing.T) { + otherNSPod := workerPod("pod-other-ns", "pool") + otherNSPod.Namespace = "other" + // Every test runs over these pods: two of "pool", one of "pool-b", and + // one of a same-named pool in another namespace. + pods := []any{workerPod("pod-a", "pool"), workerPod("pod-b", "pool"), workerPod("pod-c", "pool-b"), otherNSPod} + pools := []any{resolvedPool("pool", "gvisor-release"), resolvedPool("pool-b", "gvisor-other")} + withConfig := func(pool *v1alpha1.WorkerPool, config string) *v1alpha1.WorkerPool { + pool = pool.DeepCopy() + pool.Status.SandboxClasses[0].ConfigRef = &v1alpha1.SandboxConfigReference{Name: config} + return pool + } + withGeneration := func(cfg *v1alpha1.SandboxConfig, gen int64) *v1alpha1.SandboxConfig { + cfg = cfg.DeepCopy() + cfg.Generation = gen + return cfg + } + + tests := []struct { + name string + event func(s *workerSandboxSyncer) + want []string + }{{ + name: "pod added", + event: func(s *workerSandboxSyncer) { s.podHandler().OnAdd(workerPod("pod-new", "pool"), false) }, + want: []string{"pod-new"}, + }, { + name: "pod deleted", + event: func(s *workerSandboxSyncer) { s.podHandler().OnDelete(workerPod("pod-gone", "pool")) }, + want: []string{"pod-gone"}, + }, { + name: "pod deleted with a tombstone", + event: func(s *workerSandboxSyncer) { + s.podHandler().OnDelete(cache.DeletedFinalStateUnknown{Key: "workers/worker-pod-gone", Obj: workerPod("pod-gone", "pool")}) + }, + want: []string{"pod-gone"}, + }, { + name: "pod updated", + event: func(s *workerSandboxSyncer) { + s.podHandler().OnUpdate(workerPod("pod-a", "pool"), workerPod("pod-a", "pool")) + }, + }, { + name: "pool added", + event: func(s *workerSandboxSyncer) { + s.workerPoolHandler().OnAdd(resolvedPool("pool", "gvisor-release"), false) + }, + want: []string{"pod-a", "pod-b"}, + }, { + name: "pool configRef changed", + event: func(s *workerSandboxSyncer) { + s.workerPoolHandler().OnUpdate(resolvedPool("pool", "gvisor-release"), withConfig(resolvedPool("pool", ""), "gvisor-next")) + }, + want: []string{"pod-a", "pod-b"}, + }, { + name: "pool configRef resolved", + event: func(s *workerSandboxSyncer) { + s.workerPoolHandler().OnUpdate(resolvedPool("pool", ""), resolvedPool("pool", "gvisor-release")) + }, + want: []string{"pod-a", "pod-b"}, + }, { + name: "pool changed elsewhere", + event: func(s *workerSandboxSyncer) { + changed := resolvedPool("pool", "gvisor-release") + changed.Status.Replicas = 3 + changed.Status.SandboxClasses[0].PendingConfigRef = &v1alpha1.SandboxConfigReference{Name: "gvisor-next"} + s.workerPoolHandler().OnUpdate(resolvedPool("pool", "gvisor-release"), changed) + }, + }, { + // Pods are namespaced; a same-named pool elsewhere is another pool. + name: "same-named pool in another namespace", + event: func(s *workerSandboxSyncer) { + other := resolvedPool("pool", "") + other.Namespace = "other" + s.workerPoolHandler().OnUpdate(other, withConfig(other, "gvisor-release")) + }, + want: []string{"pod-other-ns"}, + }, { + name: "config added", + event: func(s *workerSandboxSyncer) { s.sandboxConfigHandler().OnAdd(multiVersionConfig(), false) }, + want: []string{"pod-a", "pod-b"}, + }, { + name: "config spec changed", + event: func(s *workerSandboxSyncer) { + s.sandboxConfigHandler().OnUpdate(withGeneration(multiVersionConfig(), 1), withGeneration(multiVersionConfig(), 2)) + }, + want: []string{"pod-a", "pod-b"}, + }, { + name: "config update without a spec change", + event: func(s *workerSandboxSyncer) { + s.sandboxConfigHandler().OnUpdate(withGeneration(multiVersionConfig(), 1), withGeneration(multiVersionConfig(), 1)) + }, + }, { + name: "config no pool uses", + event: func(s *workerSandboxSyncer) { + cfg := multiVersionConfig() + cfg.Name = "gvisor-unused" + s.sandboxConfigHandler().OnAdd(cfg, false) + }, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := newTestSyncer(t, append(pods, pools...)...) + tt.event(s) + if diff := cmp.Diff(tt.want, drain(s)); diff != "" { + t.Errorf("enqueued pods mismatch (-want +got):\n%s", diff) + } + }) + } +} + +// TestWorkerSandboxSyncerJitter verifies that pool and config changes, which +// every node sees at once, enqueue pods after a random delay, and that events +// only this node acts on do not. +func TestWorkerSandboxSyncerJitter(t *testing.T) { + pods := []any{workerPod("pod-a", "pool"), workerPod("pod-b", "pool")} + pool := resolvedPool("pool", "gvisor-release") + withGeneration := func(gen int64) *v1alpha1.SandboxConfig { + cfg := multiVersionConfig() + cfg.Generation = gen + return cfg + } + + tests := []struct { + name string + event func(s *workerSandboxSyncer) + // want is enqueued right away, wantLater once the jitter has passed. + want []string + wantLater []string + }{{ + name: "pod added", + event: func(s *workerSandboxSyncer) { s.podHandler().OnAdd(workerPod("pod-new", "pool"), false) }, + want: []string{"pod-new"}, + }, { + name: "pod deleted", + event: func(s *workerSandboxSyncer) { s.podHandler().OnDelete(workerPod("pod-gone", "pool")) }, + want: []string{"pod-gone"}, + }, { + // A pool usually reaches the cache when atelet starts. + name: "pool added", + event: func(s *workerSandboxSyncer) { s.workerPoolHandler().OnAdd(pool, false) }, + want: []string{"pod-a", "pod-b"}, + }, { + name: "pool configRef changed", + event: func(s *workerSandboxSyncer) { + s.workerPoolHandler().OnUpdate(resolvedPool("pool", ""), pool) + }, + wantLater: []string{"pod-a", "pod-b"}, + }, { + name: "config added", + event: func(s *workerSandboxSyncer) { s.sandboxConfigHandler().OnAdd(multiVersionConfig(), false) }, + wantLater: []string{"pod-a", "pod-b"}, + }, { + name: "config spec changed", + event: func(s *workerSandboxSyncer) { + s.sandboxConfigHandler().OnUpdate(withGeneration(1), withGeneration(2)) + }, + wantLater: []string{"pod-a", "pod-b"}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := newTestSyncer(t, append(pods, pool)...) + s.maxJitter = 50 * time.Millisecond + tt.event(s) + if diff := cmp.Diff(tt.want, drain(s)); diff != "" { + t.Errorf("UIDs enqueued right away mismatch (-want +got):\n%s", diff) + } + time.Sleep(2 * s.maxJitter) + if diff := cmp.Diff(tt.wantLater, drain(s)); diff != "" { + t.Errorf("UIDs enqueued after the jitter mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestWorkerSandboxSyncerProcess(t *testing.T) { + ctx := context.Background() + deleting := workerPod("pod-a", "pool") + deleting.DeletionTimestamp = &metav1.Time{Time: time.Now()} + stale := wantVersions() + stale.DefaultRuntime = wantRuntime("v0") + + tests := []struct { + name string + objs []any + // existing is written to pod-a's file before processing, when set. + existing *sandboxversions.File + // priorRetries is how often pod-a has been retried before. + priorRetries int + // want is the file afterwards; nil means no file. + want *sandboxversions.File + wantDir bool + // wantUntouched is whether the existing file must not be rewritten. + wantUntouched bool + wantRetries int + }{{ + name: "writes the file", + objs: []any{workerPod("pod-a", "pool"), resolvedPool("pool", "gvisor-release"), multiVersionConfig()}, + want: wantVersions(), + wantDir: true, + }, { + name: "replaces a stale file", + objs: []any{workerPod("pod-a", "pool"), resolvedPool("pool", "gvisor-release"), multiVersionConfig()}, + existing: stale, + want: wantVersions(), + wantDir: true, + }, { + name: "leaves an equal file alone", + objs: []any{workerPod("pod-a", "pool"), resolvedPool("pool", "gvisor-release"), multiVersionConfig()}, + existing: wantVersions(), + want: wantVersions(), + wantDir: true, + wantUntouched: true, + }, { + // The Delete event removes the directory. + name: "pod being deleted", + objs: []any{deleting, resolvedPool("pool", "gvisor-release"), multiVersionConfig()}, + existing: stale, + want: stale, + wantDir: true, + }, { + name: "pod gone", + objs: []any{resolvedPool("pool", "gvisor-release"), multiVersionConfig()}, + existing: stale, + }, { + name: "pod gone, no directory", + objs: []any{resolvedPool("pool", "gvisor-release"), multiVersionConfig()}, + }, { + name: "unresolved pool keeps the existing file", + objs: []any{workerPod("pod-a", "pool"), multiVersionConfig()}, + existing: stale, + want: stale, + wantDir: true, + wantRetries: 1, + }, { + name: "unresolved pool retries up to syncMaxRetries", + objs: []any{workerPod("pod-a", "pool"), multiVersionConfig()}, + priorRetries: syncMaxRetries - 1, + wantRetries: syncMaxRetries, + }, { + name: "unresolved pool gives up after syncMaxRetries", + objs: []any{workerPod("pod-a", "pool"), multiVersionConfig()}, + priorRetries: syncMaxRetries, + }, { + name: "pod of no pool is not retried", + objs: []any{workerPod("pod-a", ""), resolvedPool("pool", "gvisor-release"), multiVersionConfig()}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := newTestSyncer(t, tt.objs...) + limiter := workqueue.NewTypedItemExponentialFailureRateLimiter[string](time.Second, 5*time.Minute) + for range tt.priorRetries { + limiter.When("pod-a") + } + s.queue = workqueue.NewTypedRateLimitingQueue(limiter) + dir := s.podDir("pod-a") + if tt.existing != nil { + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + if _, err := sandboxversions.Write(s.versionsFile("pod-a"), tt.existing); err != nil { + t.Fatal(err) + } + } + var fileBefore time.Time + if _, err := os.Stat(dir); err == nil { + file, err := os.Stat(s.versionsFile("pod-a")) + if err != nil { + t.Fatal(err) + } + fileBefore = file.ModTime() + // A rewrite must land on a later mtime to be told apart. + time.Sleep(10 * time.Millisecond) + } + s.queue.Add("pod-a") + uid, _ := s.queue.Get() + + s.process(ctx, uid) + + if _, err := os.Stat(dir); (err == nil) != tt.wantDir { + t.Errorf("directory exists = %v (%v), want %v", err == nil, err, tt.wantDir) + } + got, err := sandboxversions.Read(s.versionsFile("pod-a")) + if err != nil && !errors.Is(err, os.ErrNotExist) { + t.Fatal(err) + } + if diff := cmp.Diff(tt.want, got); diff != "" { + t.Errorf("file mismatch (-want +got):\n%s", diff) + } + if tt.wantUntouched { + if after, err := os.Stat(s.versionsFile("pod-a")); err != nil || !after.ModTime().Equal(fileBefore) { + t.Errorf("an equal file was rewritten (mtime %v -> %v, %v)", fileBefore, after.ModTime(), err) + } + } + if got := s.queue.NumRequeues("pod-a"); got != tt.wantRetries { + t.Errorf("NumRequeues = %d, want %d", got, tt.wantRetries) + } + requeued := make(chan struct{}) + go func() { + defer close(requeued) + if uid, shutdown := s.queue.Get(); !shutdown { + s.queue.Done(uid) + } + }() + select { + case <-requeued: + if tt.wantRetries == 0 { + t.Error("item was requeued") + } + case <-time.After(500 * time.Millisecond): + } + s.queue.ShutDown() + <-requeued + }) + } +} + +func TestWorkerSandboxSyncerExistingDirs(t *testing.T) { + const ( + oldDir = "old dir" + youngDir = "young dir" + file = "file" + ) + tests := []struct { + name string + entries map[string]string // name -> oldDir, youngDir or file + // want is enqueued right away, wantLater once minAge has passed. + want []string + wantLater []string + }{{ + name: "worker directories", + entries: map[string]string{"pod-a": oldDir, "pod-b": oldDir}, + want: []string{"pod-a", "pod-b"}, + }, { + // Its ateom may have made it before the pod reached the cache. + name: "a young directory waits for minAge", + entries: map[string]string{"pod-a": oldDir, "pod-b": youngDir}, + want: []string{"pod-a"}, + wantLater: []string{"pod-b"}, + }, { + name: "names that are no pod UID and files are skipped", + entries: map[string]string{"pod-a": oldDir, "Not_A_UID": oldDir, "stray": file}, + want: []string{"pod-a"}, + }, { + name: "no directory yet", + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := newTestSyncer(t) + // Short enough to observe a delayed enqueue. + s.minAge = 50 * time.Millisecond + if tt.entries == nil { + s.ateomsDir = filepath.Join(s.ateomsDir, "missing") + } + for name, kind := range tt.entries { + path := filepath.Join(s.ateomsDir, name) + var err error + switch kind { + case file: + err = os.WriteFile(path, nil, 0o600) + case youngDir: + err = os.Mkdir(path, 0o700) + case oldDir: + if err = os.Mkdir(path, 0o700); err == nil { + old := time.Now().Add(-2 * orphanMinAge) + err = os.Chtimes(path, old, old) + } + } + if err != nil { + t.Fatal(err) + } + } + s.enqueueExistingDirs(context.Background()) + if diff := cmp.Diff(tt.want, drain(s)); diff != "" { + t.Errorf("enqueued UIDs mismatch (-want +got):\n%s", diff) + } + time.Sleep(2 * s.minAge) + if diff := cmp.Diff(tt.wantLater, drain(s)); diff != "" { + t.Errorf("UIDs enqueued after minAge mismatch (-want +got):\n%s", diff) + } + }) + } +} + +// waitFor fails t unless cond holds within 10 seconds. +func waitFor(t *testing.T, what string, cond func() bool) { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for !cond() { + if time.Now().After(deadline) { + t.Fatalf("timed out waiting for %s", what) + } + time.Sleep(10 * time.Millisecond) + } +} + +// exists reports whether path exists. +func exists(path string) bool { + _, err := os.Stat(path) + return err == nil +} + +// mkOldDir creates a worker directory old enough for the syncer to judge. +func mkOldDir(t *testing.T, path string) { + t.Helper() + if err := os.Mkdir(path, 0o700); err != nil { + t.Fatal(err) + } + old := time.Now().Add(-2 * orphanMinAge) + if err := os.Chtimes(path, old, old); err != nil { + t.Fatal(err) + } +} + +// TestWorkerSandboxSyncerRun verifies that run drains the queue only once +// every cache has synced: before the pod cache syncs every pod looks gone, and +// before the WorkerPool cache syncs no pod resolves. It also verifies that run +// stops when ctx ends. +func TestWorkerSandboxSyncerRun(t *testing.T) { + tests := []struct { + name string + podsSynced bool + poolsSynced bool + // wantOrphan is whether the directory of a pod that is gone remains. + wantOrphan bool + // wantFile is whether the directory of a live pod gets its file. + wantFile bool + }{{ + name: "no cache synced", + wantOrphan: true, + }, { + name: "only the pod cache synced", + podsSynced: true, + wantOrphan: true, + }, { + name: "every cache synced", + podsSynced: true, + poolsSynced: true, + wantFile: true, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := newTestSyncer(t, workerPod("pod-live", "pool"), resolvedPool("pool", "gvisor-release"), multiVersionConfig()) + mkOldDir(t, s.podDir("pod-live")) + mkOldDir(t, s.podDir("pod-gone")) + // A pod Add event queued before the caches sync. + s.queue.Add("pod-live") + matches := func() bool { + return exists(s.podDir("pod-gone")) == tt.wantOrphan && exists(s.versionsFile("pod-live")) == tt.wantFile + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + stopped := make(chan struct{}) + go func() { + defer close(stopped) + s.run(ctx, func() bool { return tt.podsSynced }, func() bool { return tt.poolsSynced }) + }() + if tt.podsSynced && tt.poolsSynced { + waitFor(t, "the directories to be synced", matches) + } else { + // cache.WaitForCacheSync polls every 100ms. + time.Sleep(300 * time.Millisecond) + if !matches() { + t.Errorf("before every cache synced: pod-gone directory exists = %v, pod-live file exists = %v; want %v, %v", + exists(s.podDir("pod-gone")), exists(s.versionsFile("pod-live")), tt.wantOrphan, tt.wantFile) + } + } + + cancel() + select { + case <-stopped: + case <-time.After(10 * time.Second): + t.Fatal("run did not stop when ctx ended") + } + }) + } +} + +// TestStartWorkerSandboxSyncer runs the syncer against fake API servers +// through the informer wiring main uses: the pod UID index, the event +// handlers and the cache sync checks. +func TestStartWorkerSandboxSyncer(t *testing.T) { + tests := []struct { + name string + // pods exist before the informers start. + pods []*corev1.Pod + // orphans are directories left by pods that are gone. + orphans []string + // change runs once the files of pods are written. + change func(ctx context.Context, kube *kubefake.Clientset) error + wantFiles []string + wantNoDirs []string + }{{ + name: "writes the file of a pod already on the node", + pods: []*corev1.Pod{workerPod("pod-a", "pool")}, + wantFiles: []string{"pod-a"}, + }, { + name: "writes the file of a pod bound later", + change: func(ctx context.Context, kube *kubefake.Clientset) error { + _, err := kube.CoreV1().Pods("workers").Create(ctx, workerPod("pod-b", "pool"), metav1.CreateOptions{}) + return err + }, + wantFiles: []string{"pod-b"}, + }, { + name: "removes the directory of a deleted pod", + pods: []*corev1.Pod{workerPod("pod-a", "pool")}, + change: func(ctx context.Context, kube *kubefake.Clientset) error { + return kube.CoreV1().Pods("workers").Delete(ctx, "worker-pod-a", metav1.DeleteOptions{}) + }, + wantNoDirs: []string{"pod-a"}, + }, { + name: "removes a directory left by a pod that went away while atelet was down", + pods: []*corev1.Pod{workerPod("pod-a", "pool")}, + orphans: []string{"pod-gone"}, + wantFiles: []string{"pod-a"}, + wantNoDirs: []string{"pod-gone"}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + dir := t.TempDir() + for _, uid := range tt.orphans { + mkOldDir(t, filepath.Join(dir, uid)) + } + var podObjs []k8sruntime.Object + for _, pod := range tt.pods { + podObjs = append(podObjs, pod) + } + kube := kubefake.NewClientset(podObjs...) + ate := atefake.NewSimpleClientset(resolvedPool("pool", "gvisor-release"), multiVersionConfig()) + podFactory := kubeinformers.NewSharedInformerFactory(kube, 0) + ateFactory := externalversions.NewSharedInformerFactory(ate, 0) + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(func() { + cancel() + podFactory.Shutdown() + ateFactory.Shutdown() + }) + + s, err := startWorkerSandboxSyncer(ctx, + podFactory.Core().V1().Pods(), + ateFactory.Api().V1alpha1().WorkerPools(), + ateFactory.Api().V1alpha1().SandboxConfigs(), + dir) + if err != nil { + t.Fatalf("startWorkerSandboxSyncer() error = %v", err) + } + ateFactory.Start(ctx.Done()) + podFactory.Start(ctx.Done()) + + for _, pod := range tt.pods { + waitFor(t, "the file of "+string(pod.UID), func() bool { return exists(s.versionsFile(string(pod.UID))) }) + } + if tt.change != nil { + if err := tt.change(ctx, kube); err != nil { + t.Fatal(err) + } + } + for _, uid := range tt.wantFiles { + waitFor(t, "the file of "+uid, func() bool { return exists(s.versionsFile(uid)) }) + } + for _, uid := range tt.wantNoDirs { + waitFor(t, "the directory of "+uid+" to be removed", func() bool { return !exists(s.podDir(uid)) }) + } + }) + } +} diff --git a/cmd/atelet/worker_sandbox_versions.go b/cmd/atelet/worker_sandbox_versions.go new file mode 100644 index 0000000000..f6037a7966 --- /dev/null +++ b/cmd/atelet/worker_sandbox_versions.go @@ -0,0 +1,282 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +// This file decides how one worker pod's sandbox versions file is built, +// in the steps syncPod runs: +// - resolve: find the SandboxConfig of the pod's WorkerPool +// - plan: pick the config's versions this node can run +// - build: hash each version's assets and pause image digests into its +// sandbox_runtime_hash +// - write: replace the file in the worker directory if its content changed +// +// worker_sandbox_syncer.go decides when each pod is synced, and creates and +// removes the worker directories. + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "log/slog" + "os" + "path/filepath" + "runtime" + "slices" + "strings" + + "github.com/google/go-containerregistry/pkg/name" + corev1 "k8s.io/api/core/v1" + + "github.com/agent-substrate/substrate/internal/hardware" + "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/sandboxversions" + "github.com/agent-substrate/substrate/pkg/api/v1alpha1" + listersv1alpha1 "github.com/agent-substrate/substrate/pkg/client/listers/api/v1alpha1" +) + +// errPermanent marks a failure that retrying cannot fix, so the syncer drops +// the item instead of backing off until the next event for the pod: +// - the pod has no worker pool label +// - the config's default version is missing, has no assets for this +// node's architecture, or has a pause image not pinned by digest +var errPermanent = errors.New("permanent failure") + +// sandboxPlan is what one worker's sandbox versions file is built from. +type sandboxPlan struct { + config string + versions []versionPlan // the default first +} + +// versionPlan is one SandboxConfig version projected onto this node's +// architecture. +type versionPlan struct { + name string + isDefault bool + rec *sandboxAssetsRecord +} + +// workerSandboxResolver finds the SandboxConfig a worker on this node runs: +// the worker pod's WorkerPool, and the config the atecontroller resolved for +// the pool's sandbox class into the pool's status. +type workerSandboxResolver struct { + workerPools listersv1alpha1.WorkerPoolLister + configs listersv1alpha1.SandboxConfigLister +} + +// poolConfigName returns the SandboxConfig the atecontroller confirmed for the +// class the pool's workers run, or "" if it has not resolved one yet. It never +// returns pendingConfigRef: that config is not confirmed yet. +func poolConfigName(pool *v1alpha1.WorkerPool) string { + class := pool.Spec.DefaultSandboxClass() + for _, c := range pool.Status.SandboxClasses { + if c.Name == class && c.ConfigRef != nil { + return c.ConfigRef.Name + } + } + return "" +} + +// resolve returns the sandbox class and SandboxConfig of a worker pod on this +// node. Apart from a pod in no pool, every failure can succeed once an +// informer catches up or the atecontroller resolves the pool. +func (r *workerSandboxResolver) resolve(pod *corev1.Pod) (v1alpha1.SandboxClass, *v1alpha1.SandboxConfig, error) { + poolName := pod.Labels[workerPoolLabel] + if poolName == "" { + return "", nil, fmt.Errorf("%w: worker pod %s/%s has no %s label", errPermanent, pod.Namespace, pod.Name, workerPoolLabel) + } + pool, err := r.workerPools.WorkerPools(pod.Namespace).Get(poolName) + if err != nil { + return "", nil, fmt.Errorf("while getting WorkerPool %s/%s: %w", pod.Namespace, poolName, err) + } + configName := poolConfigName(pool) + if configName == "" { + return "", nil, fmt.Errorf("WorkerPool %s/%s has not resolved a SandboxConfig for sandbox class %q", pod.Namespace, poolName, pool.Spec.DefaultSandboxClass()) + } + cfg, err := r.configs.Get(configName) + if err != nil { + return "", nil, fmt.Errorf("while getting SandboxConfig %q: %w", configName, err) + } + return pool.Spec.DefaultSandboxClass(), cfg, nil +} + +// plan resolves pod's SandboxConfig and the versions of it this node can run. +func (r *workerSandboxResolver) plan(pod *corev1.Pod) (*sandboxPlan, error) { + _, cfg, err := r.resolve(pod) + if err != nil { + return nil, err + } + vers, err := planVersions(cfg) + if err != nil { + return nil, err + } + return &sandboxPlan{config: cfg.Name, versions: vers}, nil +} + +// planVersions returns the versions of cfg a worker on this node may use: the +// default version first, then the other Enabled versions in spec order. +// Versions with no assets for this node's architecture are left out, except +// the default, which every worker of the pool must be able to start actors on. +func planVersions(cfg *v1alpha1.SandboxConfig) ([]versionPlan, error) { + def, ok := cfg.Spec.DefaultVersionConfig() + if !ok { + // The CRD rejects this shape, so this should not be reachable. + return nil, fmt.Errorf("%w: SandboxConfig %q has no enabled version %q named by its defaultVersion", errPermanent, cfg.Name, cfg.Spec.DefaultVersion) + } + defRec, ok := recordForArch(cfg, def) + if !ok { + return nil, fmt.Errorf("%w: SandboxConfig %q, default version %q has no assets for architecture %q", errPermanent, cfg.Name, def.Name, runtime.GOARCH) + } + vers := []versionPlan{{name: def.Name, isDefault: true, rec: defRec}} + for i := range cfg.Spec.Versions { + v := &cfg.Spec.Versions[i] + if v.Name == def.Name || (v.State != nil && *v.State == v1alpha1.SandboxVersionStateDisabled) { + continue + } + if rec, ok := recordForArch(cfg, v); ok { + vers = append(vers, versionPlan{name: v.Name, rec: rec}) + } + } + return vers, nil +} + +// recordForArch projects the assets of one version of a SandboxConfig onto +// this node's architecture, mirroring recordFromRequest. It reports false when +// the version lists no assets for this architecture. +func recordForArch(cfg *v1alpha1.SandboxConfig, ver *v1alpha1.SandboxVersionConfig) (*sandboxAssetsRecord, bool) { + files := ver.Assets[runtime.GOARCH] + if len(files) == 0 { + return nil, false + } + rec := &sandboxAssetsRecord{ + SandboxClass: string(cfg.Spec.SandboxClass), + PauseImage: ver.PauseImage, + Assets: make(map[string]assetEntry, len(files)), + } + for name, f := range files { + rec.Assets[name] = assetEntry{URL: f.URL, SHA256: f.SHA256} + } + return rec, true +} + +// buildFile returns the file content naming the runtimes of the versions in +// plan. The hashes come from the SandboxConfig spec alone, so it does no IO. +// A version whose runtime hash equals an earlier one's is left out, so the +// file lists each runtime once. +// +// The file does not promise a runtime's assets are on the node. The +// prewarmer fetches each config's default version, and Run/Restore fetch any +// other version on demand. +// +// TODO: fetch the default version's assets and pause image through +// cmd/atelet/internal/filecache here, and remove sandbox_prewarm.go. Other +// versions stay fetched on demand at Restore. +func buildFile(ctx context.Context, plan *sandboxPlan) (*sandboxversions.File, error) { + out := &sandboxversions.File{} + seen := make(map[string]bool, len(plan.versions)) + for _, v := range plan.versions { + h, err := versionRuntimeHash(v.rec) + if err != nil { + err = fmt.Errorf("SandboxConfig %q, version %q: %w", plan.config, v.name, err) + if v.isDefault { + // No worker of the pool could start actors on this node. + return nil, fmt.Errorf("%w: %w", errPermanent, err) + } + slog.WarnContext(ctx, "Leaving a sandbox version out of the worker sandbox versions file", slog.Any("err", err)) + continue + } + if seen[h] { + continue + } + seen[h] = true + attrs := sandboxversions.Attributes{hardware.AttrSandboxRuntimeHash: h} + if v.isDefault { + out.DefaultRuntime = attrs + } else { + out.RestorableRuntimes = append(out.RestorableRuntimes, attrs) + } + } + return out, nil +} + +// versionRuntimeHash returns the runtime hash of one version. The CRD +// requires every pause image to be pinned by digest, which is the digest the +// image cache resolves the ref to, so the hash needs no pull. +func versionRuntimeHash(rec *sandboxAssetsRecord) (string, error) { + var pauseDigest string + if rec.PauseImage != "" { + ref, err := name.NewDigest(rec.PauseImage) + if err != nil { + return "", fmt.Errorf("pause image %q is not pinned by digest: %w", rec.PauseImage, err) + } + pauseDigest = ref.DigestStr() + } + return runtimeHash(rec, pauseDigest), nil +} + +// runtimeHash is the sandbox_runtime_hash of a version: a sha256 over one +// sorted "key=value" line per asset digest and pause image digest. Version names and URLs are left out, so the same binaries under +// another name or in another bucket are the same runtime. +func runtimeHash(rec *sandboxAssetsRecord, pauseDigest string) string { + lines := make([]string, 0, len(rec.Assets)+1) + for name, a := range rec.Assets { + lines = append(lines, "asset/"+name+"="+a.SHA256) + } + if pauseDigest != "" { + lines = append(lines, "pause_image="+pauseDigest) + } + slices.Sort(lines) + sum := sha256.Sum256([]byte(strings.Join(lines, "\n"))) + return "sha256:" + hex.EncodeToString(sum[:]) +} + +// syncPod writes pod's sandbox versions file: +// - plan: resolve the pool's config and its versions for this node +// - build: hash each version's runtime +// - write: replace the file if its content changed +func (s *workerSandboxSyncer) syncPod(ctx context.Context, pod *corev1.Pod) error { + plan, err := s.resolver.plan(pod) + if err != nil { + return err + } + want, err := buildFile(ctx, plan) + if err != nil { + return err + } + return s.write(ctx, string(pod.UID), plan.config, want) +} + +func (s *workerSandboxSyncer) write(ctx context.Context, uid, config string, want *sandboxversions.File) error { + if err := os.MkdirAll(s.podDir(uid), 0o700); err != nil { + return fmt.Errorf("while creating worker directory: %w", err) + } + wrote, err := sandboxversions.Write(s.versionsFile(uid), want) + if err != nil { + return err + } + if wrote { + slog.InfoContext(ctx, "Wrote worker sandbox versions", + slog.String("pod_uid", uid), + slog.String("sandbox_config", config), + slog.String("default_runtime_hash", want.DefaultRuntime[hardware.AttrSandboxRuntimeHash]), + slog.Int("restorable_runtimes", len(want.RestorableRuntimes))) + } + return nil +} + +func (s *workerSandboxSyncer) versionsFile(uid string) string { + return filepath.Join(s.podDir(uid), nodepath.SandboxVersionsFileName) +} diff --git a/cmd/atelet/worker_sandbox_versions_test.go b/cmd/atelet/worker_sandbox_versions_test.go new file mode 100644 index 0000000000..e64666dab6 --- /dev/null +++ b/cmd/atelet/worker_sandbox_versions_test.go @@ -0,0 +1,414 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "crypto/sha256" + "errors" + "fmt" + "runtime" + "testing" + + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + + "github.com/agent-substrate/substrate/internal/hardware" + "github.com/agent-substrate/substrate/internal/sandboxversions" + "github.com/agent-substrate/substrate/pkg/api/v1alpha1" +) + +// pauseDigest is the digest of the pause image of multiVersionConfig's +// version with the given name. +func pauseDigest(name string) string { + return fmt.Sprintf("sha256:%x", sha256.Sum256([]byte("pause-"+name))) +} + +// multiVersionConfig is a gVisor config whose default version "v2" is listed +// after an Enabled "v1", a Disabled "v0", and an Enabled "other-arch" with no +// assets for this node. +func multiVersionConfig() *v1alpha1.SandboxConfig { + version := func(name, sha string, state *v1alpha1.SandboxVersionState) v1alpha1.SandboxVersionConfig { + return v1alpha1.SandboxVersionConfig{ + Name: name, + State: state, + PauseImage: "registry.k8s.io/pause@" + pauseDigest(name), + Assets: map[string]map[string]v1alpha1.AssetFile{ + runtime.GOARCH: {gvisorAssetName: {URL: "gs://bucket/" + name, SHA256: sha}}, + }, + } + } + otherArch := version("other-arch", "sha-other", nil) + otherArch.Assets = map[string]map[string]v1alpha1.AssetFile{"other-arch": otherArch.Assets[runtime.GOARCH]} + return &v1alpha1.SandboxConfig{ + ObjectMeta: metav1.ObjectMeta{Name: "gvisor-release"}, + Spec: v1alpha1.SandboxConfigSpec{ + SandboxClass: v1alpha1.SandboxClassGvisor, + DefaultVersion: "v2", + Versions: []v1alpha1.SandboxVersionConfig{ + version("v0", "sha-v0", ptr.To(v1alpha1.SandboxVersionStateDisabled)), + version("v1", "sha-v1", ptr.To(v1alpha1.SandboxVersionStateEnabled)), + otherArch, + version("v2", "sha-v2", nil), + }, + }, + } +} + +func TestRecordForArch(t *testing.T) { + sha := fmt.Sprintf("%x", sha256.Sum256([]byte("runsc"))) + tests := []struct { + name string + assets map[string]map[string]v1alpha1.AssetFile + want *sandboxAssetsRecord + }{{ + name: "assets for this architecture", + assets: map[string]map[string]v1alpha1.AssetFile{runtime.GOARCH: {runscAssetName: {URL: "gs://bucket/runsc", SHA256: sha}}}, + want: &sandboxAssetsRecord{ + SandboxClass: string(v1alpha1.SandboxClassGvisor), + PauseImage: "registry.k8s.io/pause@sha256:abc", + Assets: map[string]assetEntry{runscAssetName: {URL: "gs://bucket/runsc", SHA256: sha}}, + }, + }, { + name: "assets for another architecture only", + assets: map[string]map[string]v1alpha1.AssetFile{"other-arch": {runscAssetName: {URL: "gs://bucket/runsc", SHA256: sha}}}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + cfg := gvisorConfig("gvisor-default", "", "") + cfg.Spec.Versions[0].Assets = tt.assets + got, ok := recordForArch(cfg, &cfg.Spec.Versions[0]) + if ok != (tt.want != nil) { + t.Fatalf("recordForArch() ok = %v, want %v", ok, tt.want != nil) + } + if diff := cmp.Diff(tt.want, got); diff != "" { + t.Errorf("recordForArch() mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestRuntimeHash(t *testing.T) { + gvisor := map[string]assetEntry{gvisorAssetName: {URL: "gs://bucket/gvisor", SHA256: "d547d81401461fd1c679c5c4fa0a6c2b8ef7dc3c22ce23c9e25dcc4c69cfd06f"}} + pause := "sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4" + + // Snapshots record the hash, so a change to any of these values strands + // every snapshot taken before it. + tests := []struct { + name string + rec *sandboxAssetsRecord + pauseDigest string + want string + }{{ + name: "gvisor asset and pause image", + rec: &sandboxAssetsRecord{SandboxClass: "gvisor", PauseImage: "registry.k8s.io/pause:3.10.2@" + pause, Assets: gvisor}, + pauseDigest: pause, + want: "sha256:7197896bb429f0b824accbc635a38c9ed5666c160070b833bbdc6e036ccd4f76", + }, { + // Only the digests count, not the version's other fields. + name: "another URL and pause image ref", + rec: &sandboxAssetsRecord{SandboxClass: "gvisor", PauseImage: "mirror.example.com/pause@" + pause, Assets: map[string]assetEntry{gvisorAssetName: {URL: "gs://other/gvisor", SHA256: gvisor[gvisorAssetName].SHA256}}}, + pauseDigest: pause, + want: "sha256:7197896bb429f0b824accbc635a38c9ed5666c160070b833bbdc6e036ccd4f76", + }, { + name: "no pause image", + rec: &sandboxAssetsRecord{SandboxClass: "microvm", Assets: gvisor}, + want: "sha256:f86e6f2bd8e25ac31a22e51e6e57e0eb6741eb0b0df77082989cc1e40f906fe1", + }, { + name: "the asset under another name", + rec: &sandboxAssetsRecord{SandboxClass: "gvisor", Assets: map[string]assetEntry{runscAssetName: gvisor[gvisorAssetName]}}, + pauseDigest: pause, + want: "sha256:91e241c70c94d67b16127f1ba9c239e76ed1e91cb99fd9587b071cb8ccf7d3d8", + }, { + name: "several assets", + rec: &sandboxAssetsRecord{SandboxClass: "microvm", Assets: map[string]assetEntry{ + "kernel": {SHA256: "aaaa"}, + "rootfs": {SHA256: "bbbb"}, + }}, + want: "sha256:fc253c8ca0d24c485ef86a377c0cd2cf97fbf922e9cabf290971c55f051158ba", + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := runtimeHash(tt.rec, tt.pauseDigest); got != tt.want { + t.Errorf("runtimeHash() = %q, want %q", got, tt.want) + } + }) + } +} + +func TestPoolConfigName(t *testing.T) { + tests := []struct { + name string + pool *v1alpha1.WorkerPool + want string + }{{ + name: "confirmed config of the pool's class", + pool: resolvedPool("pool", "gvisor-default"), + want: "gvisor-default", + }, { + name: "not resolved yet", + pool: resolvedPool("pool", ""), + }, { + name: "only a pending config", + pool: func() *v1alpha1.WorkerPool { + pool := resolvedPool("pool", "") + pool.Status.SandboxClasses[0].PendingConfigRef = &v1alpha1.SandboxConfigReference{Name: "gvisor-next"} + return pool + }(), + }, { + name: "status resolved another class than the pool's", + pool: func() *v1alpha1.WorkerPool { + pool := resolvedPool("pool", "gvisor-default") + pool.Spec.SandboxClasses[0].Name = v1alpha1.SandboxClassMicroVM + return pool + }(), + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := poolConfigName(tt.pool); got != tt.want { + t.Errorf("poolConfigName() = %q, want %q", got, tt.want) + } + }) + } +} + +func TestWorkerSandboxResolverResolve(t *testing.T) { + cfg := gvisorConfig("gvisor-default", "gs://bucket/runsc", "abc") + + tests := []struct { + name string + objs []any + pod *corev1.Pod + wantConfig string + wantErr bool + wantPermanent bool + }{{ + name: "the config the pool resolved for the pool's class", + objs: []any{resolvedPool("pool", "gvisor-default"), cfg}, + pod: workerPod("pod-a", "pool"), + wantConfig: "gvisor-default", + }, { + name: "pod belongs to no pool", + objs: []any{resolvedPool("pool", "gvisor-default"), cfg}, + pod: workerPod("pod-a", ""), + wantErr: true, + wantPermanent: true, + }, { + name: "pool not found", + objs: []any{cfg}, + pod: workerPod("pod-a", "pool"), + wantErr: true, + }, { + name: "pool has not resolved a config", + objs: []any{resolvedPool("pool", ""), cfg}, + pod: workerPod("pod-a", "pool"), + wantErr: true, + }, { + // A pending config is one the atecontroller has claimed but not yet + // confirmed exists and is not being deleted. + name: "pool is only switching to a config", + objs: []any{func() *v1alpha1.WorkerPool { + pool := resolvedPool("pool", "") + pool.Status.SandboxClasses[0].PendingConfigRef = &v1alpha1.SandboxConfigReference{Name: "gvisor-default"} + return pool + }(), cfg}, + pod: workerPod("pod-a", "pool"), + wantErr: true, + }, { + // The class comes from the pool's spec, not from the status entry. + name: "status resolved another class than the pool's", + objs: []any{func() *v1alpha1.WorkerPool { + pool := resolvedPool("pool", "gvisor-default") + pool.Spec.SandboxClasses[0].Name = v1alpha1.SandboxClassMicroVM + return pool + }(), cfg}, + pod: workerPod("pod-a", "pool"), + wantErr: true, + }, { + name: "config not found", + objs: []any{resolvedPool("pool", "gvisor-missing"), cfg}, + pod: workerPod("pod-a", "pool"), + wantErr: true, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := newTestSyncer(t, tt.objs...) + class, got, err := s.resolver.resolve(tt.pod) + if (err != nil) != tt.wantErr { + t.Fatalf("resolve() error = %v, want error %v", err, tt.wantErr) + } + if errors.Is(err, errPermanent) != tt.wantPermanent { + t.Errorf("resolve() error = %v, want permanent %v", err, tt.wantPermanent) + } + if err != nil { + return + } + if got.Name != tt.wantConfig || class != v1alpha1.SandboxClassGvisor { + t.Errorf("resolve() = %q, %q, want %q, %q", class, got.Name, v1alpha1.SandboxClassGvisor, tt.wantConfig) + } + }) + } +} + +func TestVersionRuntimeHash(t *testing.T) { + gvisor := map[string]assetEntry{gvisorAssetName: {URL: "gs://bucket/gvisor", SHA256: "d547d81401461fd1c679c5c4fa0a6c2b8ef7dc3c22ce23c9e25dcc4c69cfd06f"}} + pause := "sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4" + + tests := []struct { + name string + pauseImage string + want string + wantErr bool + }{{ + name: "pause image pinned by digest", + pauseImage: "registry.k8s.io/pause:3.10.2@" + pause, + want: runtimeHash(&sandboxAssetsRecord{Assets: gvisor}, pause), + }, { + name: "pause image pinned by digest only", + pauseImage: "gcr.io/gke-release/pause@" + pause, + want: runtimeHash(&sandboxAssetsRecord{Assets: gvisor}, pause), + }, { + name: "no pause image", + want: runtimeHash(&sandboxAssetsRecord{Assets: gvisor}, ""), + }, { + // The CRD rejects this shape. + name: "pause image by tag", + pauseImage: "registry.k8s.io/pause:3.10.2", + wantErr: true, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := versionRuntimeHash(&sandboxAssetsRecord{SandboxClass: "gvisor", PauseImage: tt.pauseImage, Assets: gvisor}) + if (err != nil) != tt.wantErr { + t.Fatalf("versionRuntimeHash() error = %v, want error %v", err, tt.wantErr) + } + if got != tt.want { + t.Errorf("versionRuntimeHash() = %q, want %q", got, tt.want) + } + }) + } +} + +// wantRuntime is the runtime buildFile builds from multiVersionConfig's +// version with the given name. +func wantRuntime(name string) sandboxversions.Attributes { + rec := &sandboxAssetsRecord{Assets: map[string]assetEntry{gvisorAssetName: {SHA256: "sha-" + name}}} + return sandboxversions.Attributes{hardware.AttrSandboxRuntimeHash: runtimeHash(rec, pauseDigest(name))} +} + +// wantVersions is the file content for multiVersionConfig. +func wantVersions() *sandboxversions.File { + return &sandboxversions.File{ + DefaultRuntime: wantRuntime("v2"), + RestorableRuntimes: []sandboxversions.Attributes{wantRuntime("v1")}, + } +} + +// planAndBuild runs the plan and build steps of syncPod for pod. +func planAndBuild(s *workerSandboxSyncer, pod *corev1.Pod) (*sandboxversions.File, error) { + plan, err := s.resolver.plan(pod) + if err != nil { + return nil, err + } + return buildFile(context.Background(), plan) +} + +func TestPlanAndBuild(t *testing.T) { + defaultNoArch := multiVersionConfig() + defaultNoArch.Spec.Versions[3].Assets = map[string]map[string]v1alpha1.AssetFile{"other-arch": defaultNoArch.Spec.Versions[3].Assets[runtime.GOARCH]} + + onlyDefault := &sandboxversions.File{DefaultRuntime: wantRuntime("v2")} + + // sameAsDefault has v1 run the default version's binaries. + sameAsDefault := multiVersionConfig() + sameAsDefault.Spec.Versions[1].PauseImage = sameAsDefault.Spec.Versions[3].PauseImage + sameAsDefault.Spec.Versions[1].Assets = sameAsDefault.Spec.Versions[3].Assets + + // v1Twice adds an Enabled "v3" running v1's binaries. + v1Twice := multiVersionConfig() + v3 := *v1Twice.Spec.Versions[1].DeepCopy() + v3.Name = "v3" + v1Twice.Spec.Versions = append(v1Twice.Spec.Versions, v3) + + // The CRD rejects a pause image not pinned by digest. + tagged := func(i int) *v1alpha1.SandboxConfig { + cfg := multiVersionConfig() + cfg.Spec.Versions[i].PauseImage = "registry.k8s.io/pause:3.10.2" + return cfg + } + + tests := []struct { + name string + cfg *v1alpha1.SandboxConfig + want *sandboxversions.File + wantErr bool + wantPermanent bool + }{{ + // Disabled versions and versions with nothing for this node's + // architecture are left out; the default leads. + name: "default first, then the other enabled versions", + cfg: multiVersionConfig(), + want: wantVersions(), + }, { + // The proto keeps default_runtime out of restorable_runtimes. + name: "a version with the default's binaries is left out", + cfg: sameAsDefault, + want: onlyDefault, + }, { + name: "versions with equal binaries are listed once", + cfg: v1Twice, + want: wantVersions(), + }, { + name: "default version has no assets for this architecture", + cfg: defaultNoArch, + wantErr: true, + wantPermanent: true, + }, { + // No worker of the pool could start actors on this node. + name: "default version pause image not pinned by digest", + cfg: tagged(3), + wantErr: true, + wantPermanent: true, + }, { + name: "non-default version pause image not pinned by digest is left out", + cfg: tagged(1), + want: onlyDefault, + }, { + name: "surfaces an unresolved config", + wantErr: true, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + objs := []any{resolvedPool("pool", "gvisor-release")} + if tt.cfg != nil { + objs = append(objs, tt.cfg) + } + s := newTestSyncer(t, objs...) + + got, err := planAndBuild(s, workerPod("pod-a", "pool")) + if (err != nil) != tt.wantErr { + t.Fatalf("planAndBuild() error = %v, want error %v", err, tt.wantErr) + } + if errors.Is(err, errPermanent) != tt.wantPermanent { + t.Errorf("planAndBuild() error = %v, want permanent %v", err, tt.wantPermanent) + } + if diff := cmp.Diff(tt.want, got); diff != "" { + t.Errorf("planAndBuild() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/cmd/atenet/internal/router/README.md b/cmd/atenet/internal/router/README.md index 9714653ea9..717499df69 100644 --- a/cmd/atenet/internal/router/README.md +++ b/cmd/atenet/internal/router/README.md @@ -214,7 +214,17 @@ list). ## status page -Serve a `/statusz` page on port 8080. +The unauthenticated `/statusz` dashboard binds to `127.0.0.1:4040` by default +(`--status-address` and `--status-port`). It is not published by the router +Service. Access it through a pod port-forward: + +```sh +kubectl -n ate-system port-forward deployment/atenet-router 4040:4040 +``` + +Open `http://127.0.0.1:4040/statusz` (or add `?format=json`). Binding +`--status-address` to a non-loopback address exposes the diagnostic data to +other pods. Contents: @@ -222,3 +232,10 @@ Contents: * Command line args * Last 100 queries served * Build tag + +Health checks use `/healthz` (liveness) and `/readyz` (readiness) on the health +listener, configured by `--health-listen-addr` (default `:9091`). For a GKE +Gateway, migrate any `HealthCheckPolicy` using port 4040 and `/statusz` to port +9091 and `/readyz`, so draining pods also stop receiving traffic. +[GKE probes Pod IPs directly](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-gateway-resources#configure_health_checks), +so port 9091 does not need to be published by the router Service. diff --git a/cmd/atenet/internal/router/cmd.go b/cmd/atenet/internal/router/cmd.go index b10ec9ad2d..966bd73597 100644 --- a/cmd/atenet/internal/router/cmd.go +++ b/cmd/atenet/internal/router/cmd.go @@ -54,6 +54,7 @@ func bindRouterFlags(fs *pflag.FlagSet, cfg *routerConfig) { fs.StringVar(&cfg.LogLevel, "log-level", "info", "Log level: debug, info, warn, error") fs.StringVar(&cfg.MetricsAddr, "metrics-listen-addr", ":9090", "Address and port the prometheus metrics server should listen on.") + fs.StringVar(&cfg.HealthAddr, "health-listen-addr", ":9091", "Address and port the /readyz and /healthz probe server should listen on.") fs.StringVar(&cfg.AtenetRouter, "atenet-dataplane", string(atenetRouterEnvoy), "Atenet ingress and egress dataplane: envoy or agentgateway") fs.StringVar(&cfg.Namespace, "namespace", "default", "Target operations namespace") fs.StringVar(&cfg.RouterServiceName, "router-service-name", installdefaults.RouterServiceName, "Service name of this atenet-router in the operations namespace. Override when the deployment renames the Service.") @@ -64,8 +65,9 @@ func bindRouterFlags(fs *pflag.FlagSet, cfg *routerConfig) { fs.IntVar(&cfg.ConnectTLSPort, "port-connect-tls", 8444, "TCP port for CONNECT-tunneled traffic entering through the router dataplane over TLS. --port-https also defaults to 8443, and both listeners are commonly enabled at once, so --port-connect-tls defaults to a different port (8444) rather than colliding with it") fs.IntVar(&cfg.XdsPort, "port-xds", 18000, "TCP port listening for the xDS dynamic Envoy connections") fs.IntVar(&cfg.ExtprocPort, "port-extproc", 50051, "Listen port for the External Processing (ext_proc) server the dataplane calls") - fs.StringVar(&cfg.ExtprocAddr, "extproc-address", "127.0.0.1", "Address of the External Processing (ext_proc) server: both the address it binds and the address the co-located dataplane is told to dial. Defaults to loopback, which keeps it unreachable from other pods; readiness is probed via /readyz on the metrics port, not this one. Empty binds every interface") + fs.StringVar(&cfg.ExtprocAddr, "extproc-address", "127.0.0.1", "Address of the External Processing (ext_proc) server: both the address it binds and the address the co-located dataplane is told to dial. Defaults to loopback, which keeps it unreachable from other pods; readiness is probed via /readyz on the health port, not this one. Empty binds every interface") fs.IntVar(&cfg.StatusPort, "status-port", 4040, "Port to serve /statusz on (set <= 0 to disable serving status)") + fs.StringVar(&cfg.StatusAddr, "status-address", "127.0.0.1", "Address to bind the unauthenticated /statusz dashboard to. Defaults to loopback; use /healthz and /readyz on the health port for health checks. Empty binds every interface") fs.DurationVar(&cfg.HealthInterval, "health-interval", 1*time.Second, "Interval for checking health of dependent services") fs.IntVar(&cfg.HttpsPort, "port-https", 8443, "TCP port for HTTPS workload traffic entering through the router dataplane") fs.StringVar(&cfg.EnvoyCertPath, "envoy-cert-path", "", "Path to the Envoy certificate file.") @@ -83,6 +85,7 @@ func bindRouterFlags(fs *pflag.FlagSet, cfg *routerConfig) { fs.StringVar(&cfg.UpstreamCredentialBundlePath, "upstream-credential-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "PEM credential bundle (cert+key) the router presents as the client cert when dialing the actor's atunnel ingress server over mTLS. Empty disables upstream mTLS (legacy plaintext pod-IP:80).") fs.StringVar(&cfg.UpstreamTrustBundlePath, "upstream-trust-bundle", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "PEM trust bundle used to validate the actor's atunnel ingress server certificate.") fs.StringVar(&cfg.UpstreamSpiffePrefix, "upstream-spiffe-prefix", "spiffe://cluster.local/", "SPIFFE URI SAN prefix (trust domain) the actor's atunnel server cert must match. Empty falls back to default SAN check against the dialed pod IP (which SPIFFE-only certs never match).") + fs.BoolVar(&cfg.UpstreamConnect, "upstream-connect", false, "Reach atunnel through mTLS CONNECT tunnels on port 8443 instead of HTTPS on 443 (Envoy dataplane only). Requires --upstream-credential-bundle") fs.StringVar(&cfg.ActorIdentityCAFile, "actor-identity-ca-file", "", "PEM trust bundle for the actor-identity CA, used to verify the actor client certificates presented on egress CONNECTs. Required by the egress gateway's ext_proc sidecar; empty (the default) leaves egress authentication unconfigured and every egress CONNECT is denied.") diff --git a/cmd/atenet/internal/router/config.go b/cmd/atenet/internal/router/config.go index dc3aa927d0..6d4e60c4b8 100644 --- a/cmd/atenet/internal/router/config.go +++ b/cmd/atenet/internal/router/config.go @@ -155,6 +155,7 @@ type routerConfig struct { ExtprocPort int ExtprocAddr string StatusPort int + StatusAddr string HealthInterval time.Duration HttpsPort int // ConnectPlainTextPort and ConnectTLSPort are the plaintext and TLS @@ -173,6 +174,8 @@ type routerConfig struct { // UpstreamSpiffePrefix validates the actor's atunnel server cert by its // SPIFFE URI SAN prefix (trust domain) instead of the dialed pod IP. UpstreamSpiffePrefix string + // UpstreamConnect tunnels actor traffic to atunnel's CONNECT listener. + UpstreamConnect bool // ActorIdentityCAFile is the PEM trust bundle for the actor-identity CA, // used by the egress gateway's ext_proc sidecar to verify the actor client @@ -195,6 +198,7 @@ type routerConfig struct { LogLevel string MetricsAddr string + HealthAddr string // OtlpCollectorAddress is the OTLP gRPC collector that Envoy reports // tracing spans to, as host:port or an http:// URL. It defaults to // OTEL_EXPORTER_OTLP_ENDPOINT — Envoy gets its whole configuration over @@ -319,6 +323,9 @@ func (c routerConfig) validate() error { return fmt.Errorf("--extproc-max-requests (%d) must be >= --parked-request-max (%d): a circuit breaker below the parking lot silently truncates it with Envoy-generated 503s", c.ExtProcMaxRequests, c.ParkedRequest.Max) } + if c.UpstreamConnect && c.UpstreamCredentialBundlePath == "" { + return fmt.Errorf("--upstream-connect requires --upstream-credential-bundle: atunnel's CONNECT listener accepts only mTLS") + } if c.EgressPolicyCacheTTL < 0 { return fmt.Errorf("--egress-policy-cache-ttl must not be negative, got %s (0 disables the cache)", c.EgressPolicyCacheTTL) } diff --git a/cmd/atenet/internal/router/config_test.go b/cmd/atenet/internal/router/config_test.go index 679e9be8f3..99a738c2a4 100644 --- a/cmd/atenet/internal/router/config_test.go +++ b/cmd/atenet/internal/router/config_test.go @@ -15,13 +15,11 @@ package router import ( - "os" "strings" "testing" "time" - appsv1 "k8s.io/api/apps/v1" - "sigs.k8s.io/yaml" + "github.com/spf13/pflag" "github.com/agent-substrate/substrate/cmd/atenet/internal/router/ingress" ) @@ -107,6 +105,15 @@ func TestRouterConfigValidate(t *testing.T) { cfg: routerConfig{Mode: ModeIngress, ParkedRequest: ingress.ParkedRequestConfig{Max: ingress.DefaultParkedRequestMax}, DrainTimeout: -time.Second}, wantErr: "--drain-timeout must not be negative", }, + { + name: "upstream-connect with the upstream credential bundle accepted", + cfg: routerConfig{Mode: ModeIngress, UpstreamConnect: true, UpstreamCredentialBundlePath: "/run/bundle.pem"}, + }, + { + name: "upstream-connect without the upstream credential bundle rejected", + cfg: routerConfig{Mode: ModeIngress, UpstreamConnect: true}, + wantErr: "--upstream-connect requires --upstream-credential-bundle", + }, { name: "negative drain-delay rejected", cfg: routerConfig{Mode: ModeIngress, ParkedRequest: ingress.ParkedRequestConfig{Max: ingress.DefaultParkedRequestMax}, DrainDelay: -time.Second}, @@ -419,58 +426,40 @@ const routerManifestPath = "../../../../manifests/ate-install/atenet-router.yaml // manifest rather than in Go: --drain-delay on the router container, which is // how long it keeps serving after SIGTERM, and the pod's // terminationGracePeriodSeconds, which is the whole budget. -// -// --drain-delay is read back through the router command's own flag set, so a -// renamed or retyped flag fails here rather than being parsed by a private -// copy of the syntax. func routerShutdownBudget(t *testing.T) (drainDelay, grace time.Duration) { t.Helper() - raw, err := os.ReadFile(routerManifestPath) + pod := findDeployment(t, routerManifestPath, "atenet-router").Spec.Template.Spec + if pod.TerminationGracePeriodSeconds == nil { + t.Fatalf("%s sets no terminationGracePeriodSeconds on the atenet-router pod", routerManifestPath) + } + d, err := routerManifestFlags(t).GetDuration("drain-delay") if err != nil { - t.Fatalf("reading %s: %v", routerManifestPath, err) + t.Fatalf("reading --drain-delay from the manifest args: %v", err) } - for _, doc := range strings.Split(string(raw), "\n---\n") { - var head struct { - Kind string `json:"kind"` - Metadata struct { - Name string `json:"name"` - } `json:"metadata"` - } - if err := yaml.Unmarshal([]byte(doc), &head); err != nil { - t.Fatalf("parsing a document of %s: %v", routerManifestPath, err) - } - if head.Kind != "Deployment" || head.Metadata.Name != "atenet-router" { + return d, time.Duration(*pod.TerminationGracePeriodSeconds) * time.Second +} + +// routerManifestFlags returns the router command's flags parsed from the +// atenet-router container's args in the manifest. Reading them back through +// the command's own flag set means a renamed or retyped flag fails here rather +// than being parsed by a private copy of the syntax. +func routerManifestFlags(t *testing.T) *pflag.FlagSet { + t.Helper() + for _, c := range findDeployment(t, routerManifestPath, "atenet-router").Spec.Template.Spec.Containers { + if c.Name != "atenet-router" { continue } - var deployment appsv1.Deployment - if err := yaml.Unmarshal([]byte(doc), &deployment); err != nil { - t.Fatalf("decoding the atenet-router Deployment from %s: %v", routerManifestPath, err) - } - pod := deployment.Spec.Template.Spec - if pod.TerminationGracePeriodSeconds == nil { - t.Fatalf("%s sets no terminationGracePeriodSeconds on the atenet-router pod", routerManifestPath) + cmd := NewRouterCmd() + if len(c.Args) == 0 || c.Args[0] != cmd.Name() { + t.Fatalf("the atenet-router container's args are %v; they should invoke the %q subcommand", c.Args, cmd.Name()) } - for _, c := range pod.Containers { - if c.Name != "atenet-router" { - continue - } - cmd := NewRouterCmd() - if len(c.Args) == 0 || c.Args[0] != cmd.Name() { - t.Fatalf("the atenet-router container's args are %v; they should invoke the %q subcommand", c.Args, cmd.Name()) - } - if err := cmd.ParseFlags(c.Args[1:]); err != nil { - t.Fatalf("the atenet-router container's flags are not ones the binary accepts: %v", err) - } - d, err := cmd.Flags().GetDuration("drain-delay") - if err != nil { - t.Fatalf("reading --drain-delay from the manifest args: %v", err) - } - return d, time.Duration(*pod.TerminationGracePeriodSeconds) * time.Second + if err := cmd.ParseFlags(c.Args[1:]); err != nil { + t.Fatalf("the atenet-router container's flags are not ones the binary accepts: %v", err) } - t.Fatalf("the atenet-router Deployment in %s has no container named atenet-router", routerManifestPath) + return cmd.Flags() } - t.Fatalf("%s has no Deployment named atenet-router", routerManifestPath) - return 0, 0 + t.Fatalf("the atenet-router Deployment in %s has no container named atenet-router", routerManifestPath) + return nil } func TestRouterConfigDrainTimeout(t *testing.T) { diff --git a/cmd/atenet/internal/router/dataplane.go b/cmd/atenet/internal/router/dataplane.go index f2bc3e8c71..c3c4f0d9ab 100644 --- a/cmd/atenet/internal/router/dataplane.go +++ b/cmd/atenet/internal/router/dataplane.go @@ -100,6 +100,7 @@ func (s *RouterServer) startEnvoyDataplane(ctx context.Context, g *errgroup.Grou slog.WarnContext(ctx, "Ingress client authentication is optional: any client that presents no certificate, or uses a plaintext listener, can reach any actor. Set --ingress-auth-mode=static-mtls to require mTLS") } xdsSrv.SetUpstreamTls(s.cfg.UpstreamCredentialBundlePath, s.cfg.UpstreamTrustBundlePath, s.cfg.UpstreamSpiffePrefix) + xdsSrv.SetUpstreamConnect(s.cfg.UpstreamConnect) // The snapshot is a pure function of the configuration applied above, so it // is built exactly once; an Envoy that connects later is served from the diff --git a/cmd/atenet/internal/router/egress/actorjwt/actorjwt.go b/cmd/atenet/internal/router/egress/actorjwt/actorjwt.go index 3c65c0783e..4f79ca14a1 100644 --- a/cmd/atenet/internal/router/egress/actorjwt/actorjwt.go +++ b/cmd/atenet/internal/router/egress/actorjwt/actorjwt.go @@ -46,9 +46,9 @@ func New(client ateapipb.ControlClient) *Minter { } // Token returns a JWT for the actor ref names, bound to src's audiences and -// lifetime. MintActorJWT errors are returned wrapped. -func (m *Minter) Token(ctx context.Context, ref resources.ActorRef, src *ateapipb.ActorJWTSource) (string, error) { - req := &ateapipb.MintActorJWTRequest{ +// lifetime. MintActorJwt errors are returned wrapped. +func (m *Minter) Token(ctx context.Context, ref resources.ActorRef, src *ateapipb.ActorJwtSource) (string, error) { + req := &ateapipb.MintActorJwtRequest{ Actor: ref.ToObjectRef(), Audiences: slices.Sorted(slices.Values(src.GetAudiences())), ExpirationSeconds: src.GetExpirationSeconds(), @@ -80,15 +80,15 @@ func (m *Minter) Token(ctx context.Context, ref resources.ActorRef, src *ateapip // mint sends req and caches the token under key, unless a flight that just // finished has already stored one. -func (m *Minter) mint(ctx context.Context, key string, ref resources.ActorRef, req *ateapipb.MintActorJWTRequest) (string, error) { +func (m *Minter) mint(ctx context.Context, key string, ref resources.ActorRef, req *ateapipb.MintActorJwtRequest) (string, error) { if jwt, ok := m.cached(key); ok { return jwt, nil } - resp, err := m.client.MintActorJWT(ctx, req) + resp, err := m.client.MintActorJwt(ctx, req) if err != nil { return "", fmt.Errorf("minting an actor JWT for %s: %w", ref, err) } - reuseFor := time.Until(resp.GetExpiresAt().AsTime()) - time.Duration(req.GetExpirationSeconds())*time.Second/3 + reuseFor := time.Until(resp.GetExpireTime().AsTime()) - time.Duration(req.GetExpirationSeconds())*time.Second/3 m.tokens.Set(key, resp.GetActorJwt(), reuseFor) return resp.GetActorJwt(), nil } @@ -101,7 +101,7 @@ func (m *Minter) cached(key string) (string, bool) { // tokenKey identifies the token req mints by req's own encoding. Deterministic // encoding is stable within one binary, which is all an in-process cache needs. -func tokenKey(req *ateapipb.MintActorJWTRequest) (string, error) { +func tokenKey(req *ateapipb.MintActorJwtRequest) (string, error) { b, err := proto.MarshalOptions{Deterministic: true}.Marshal(req) if err != nil { return "", fmt.Errorf("encoding the token key: %w", err) diff --git a/cmd/atenet/internal/router/egress/actorjwt/actorjwt_test.go b/cmd/atenet/internal/router/egress/actorjwt/actorjwt_test.go index 71a79dfc73..f7194887a4 100644 --- a/cmd/atenet/internal/router/egress/actorjwt/actorjwt_test.go +++ b/cmd/atenet/internal/router/egress/actorjwt/actorjwt_test.go @@ -39,18 +39,18 @@ import ( var testActor = resources.ActorRef{Atespace: "default", Name: "my-actor"} -// fakeControl's nth MintActorJWT call returns "jwt-", expiring +// fakeControl's nth MintActorJwt call returns "jwt-", expiring // expiration_seconds from now. err, when set, is returned instead, and gate, // when non-nil, blocks each call until it is closed. type fakeControl struct { ateapipb.UnimplementedControlServer calls atomic.Int32 - last atomic.Pointer[ateapipb.MintActorJWTRequest] + last atomic.Pointer[ateapipb.MintActorJwtRequest] err error gate chan struct{} } -func (f *fakeControl) MintActorJWT(ctx context.Context, req *ateapipb.MintActorJWTRequest) (*ateapipb.MintActorJWTResponse, error) { +func (f *fakeControl) MintActorJwt(ctx context.Context, req *ateapipb.MintActorJwtRequest) (*ateapipb.MintActorJwtResponse, error) { n := f.calls.Add(1) f.last.Store(req) if f.gate != nil { @@ -63,9 +63,9 @@ func (f *fakeControl) MintActorJWT(ctx context.Context, req *ateapipb.MintActorJ if f.err != nil { return nil, f.err } - return &ateapipb.MintActorJWTResponse{ - ActorJwt: fmt.Sprintf("jwt-%d", n), - ExpiresAt: timestamppb.New(time.Now().Add(time.Duration(req.GetExpirationSeconds()) * time.Second)), + return &ateapipb.MintActorJwtResponse{ + ActorJwt: fmt.Sprintf("jwt-%d", n), + ExpireTime: timestamppb.New(time.Now().Add(time.Duration(req.GetExpirationSeconds()) * time.Second)), }, nil } @@ -89,12 +89,12 @@ func newMinter(t *testing.T, ctl *fakeControl) *Minter { return New(ateapipb.NewControlClient(conn)) } -func jwtSource(lifetime int64, audiences ...string) *ateapipb.ActorJWTSource { - return &ateapipb.ActorJWTSource{Audiences: audiences, ExpirationSeconds: lifetime} +func jwtSource(lifetime int64, audiences ...string) *ateapipb.ActorJwtSource { + return &ateapipb.ActorJwtSource{Audiences: audiences, ExpirationSeconds: lifetime} } // wantToken calls m.Token and fails the test unless it returns want. -func wantToken(t *testing.T, m *Minter, ref resources.ActorRef, src *ateapipb.ActorJWTSource, want string) { +func wantToken(t *testing.T, m *Minter, ref resources.ActorRef, src *ateapipb.ActorJwtSource, want string) { t.Helper() got, err := m.Token(context.Background(), ref, src) if err != nil { @@ -112,13 +112,13 @@ func TestReuseTokenUntilAThirdOfItsLifetimeRemains(t *testing.T) { src := jwtSource(900, "https://b.example", "https://a.example") wantToken(t, m, testActor, src, "jwt-1") - want := &ateapipb.MintActorJWTRequest{ + want := &ateapipb.MintActorJwtRequest{ Actor: &ateapipb.ObjectRef{Atespace: "default", Name: "my-actor"}, Audiences: []string{"https://a.example", "https://b.example"}, ExpirationSeconds: 900, } if got := ctl.last.Load(); !proto.Equal(got, want) { - t.Errorf("MintActorJWT request = %v, want %v", got, want) + t.Errorf("MintActorJwt request = %v, want %v", got, want) } time.Sleep(600*time.Second - time.Nanosecond) @@ -146,7 +146,7 @@ func TestDoNotCacheErrors(t *testing.T) { src := jwtSource(900, "a") if _, err := m.Token(context.Background(), testActor, src); status.Code(err) != codes.Unavailable { - t.Fatalf("Token = %v, want the Unavailable MintActorJWT error", err) + t.Fatalf("Token = %v, want the Unavailable MintActorJwt error", err) } ctl.err = nil wantToken(t, m, testActor, src, "jwt-2") @@ -170,7 +170,7 @@ func TestCollapseConcurrentMints(t *testing.T) { close(ctl.gate) wg.Wait() if calls := ctl.calls.Load(); calls != 1 { - t.Errorf("MintActorJWT calls = %d, want 1 for %d concurrent callers", calls, callers) + t.Errorf("MintActorJwt calls = %d, want 1 for %d concurrent callers", calls, callers) } }) } @@ -198,7 +198,7 @@ func TestMintOutlivesCanceledCaller(t *testing.T) { synctest.Wait() wantToken(t, m, testActor, jwtSource(900, "a"), "jwt-1") if calls := ctl.calls.Load(); calls != 1 { - t.Errorf("MintActorJWT calls = %d, want 1", calls) + t.Errorf("MintActorJwt calls = %d, want 1", calls) } }) } diff --git a/cmd/atenet/internal/router/egress/credentials.go b/cmd/atenet/internal/router/egress/credentials.go index 733ba95474..996c361335 100644 --- a/cmd/atenet/internal/router/egress/credentials.go +++ b/cmd/atenet/internal/router/egress/credentials.go @@ -119,7 +119,7 @@ func (h *Handler) applyEffects(ctx context.Context, ref resources.ActorRef, dest // actorJWT mints the JWT src asks for. Every error it returns is already a // client-facing denial. -func (h *Handler) actorJWT(ctx context.Context, ref resources.ActorRef, dest egresspolicy.Destination, src *ateapipb.ActorJWTSource) ([]byte, error) { +func (h *Handler) actorJWT(ctx context.Context, ref resources.ActorRef, dest egresspolicy.Destination, src *ateapipb.ActorJwtSource) ([]byte, error) { jwt, err := h.actorJWTs.Token(ctx, ref, src) if err != nil { slog.ErrorContext(ctx, "egress denied: actor JWT mint failed", diff --git a/cmd/atenet/internal/router/egress/credentials_test.go b/cmd/atenet/internal/router/egress/credentials_test.go index 49630191c9..57fbcb09bd 100644 --- a/cmd/atenet/internal/router/egress/credentials_test.go +++ b/cmd/atenet/internal/router/egress/credentials_test.go @@ -114,11 +114,11 @@ func TestInjectionOnRequestLegs(t *testing.T) { // A request that does not carry the header a rule replaces goes out unchanged: // nothing is fetched, added, or denied, even where injecting would fail. func TestInjectionSkippedWithoutHeader(t *testing.T) { - jwtPolicy := &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Https: &ateapipb.HTTPSRule{ + jwtPolicy := &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Https: &ateapipb.HttpsRule{ Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{ReplaceHeaders: []*ateapipb.CredentialHeader{{ Header: "authorization", - ActorJwt: &ateapipb.ActorJWTSource{Audiences: []string{"https://api.example.com"}}, + ActorJwt: &ateapipb.ActorJwtSource{Audiences: []string{"https://api.example.com"}}, }}}, }}}} for _, tc := range []struct { @@ -154,7 +154,7 @@ func TestInjectionSkippedWithoutHeader(t *testing.T) { // Of a rule's replacements, only those whose header the request carries are // applied, matching the header name case-insensitively. func TestInjectionReplacesOnlyCarriedHeaders(t *testing.T) { - policy := &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Https: &ateapipb.HTTPSRule{ + policy := &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Https: &ateapipb.HttpsRule{ Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{ReplaceHeaders: []*ateapipb.CredentialHeader{ {Header: "authorization", Prefix: "Bearer ", CredentialUri: "ate-secret://k8s/default/token"}, @@ -259,7 +259,7 @@ func actorJWTHeader(header string) *ateapipb.CredentialHeader { return &ateapipb.CredentialHeader{ Header: header, Prefix: "Bearer ", - ActorJwt: &ateapipb.ActorJWTSource{Audiences: []string{"https://api.example.com"}, ExpirationSeconds: 900}, + ActorJwt: &ateapipb.ActorJwtSource{Audiences: []string{"https://api.example.com"}, ExpirationSeconds: 900}, } } @@ -270,9 +270,9 @@ func replaceHeadersPolicy(leg string, entries ...*ateapipb.CredentialHeader) *at hosts := []string{"api.example.com"} effects := &ateapipb.HttpRuleEffects{ReplaceHeaders: entries} if leg == extproc.EgressCleartextFilterChainName { - return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Http: &ateapipb.HTTPRule{Hostnames: hosts, Effects: effects}}}} + return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{Hostnames: hosts, Effects: effects}}}} } - return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Https: &ateapipb.HTTPSRule{Hostnames: hosts, Effects: effects}}}} + return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{Https: &ateapipb.HttpsRule{Hostnames: hosts, Effects: effects}}}} } var requestLegs = []string{extproc.EgressTLSMITMFilterChainName, extproc.EgressCleartextFilterChainName} @@ -308,13 +308,13 @@ func TestActorJWTInjection(t *testing.T) { if h0.GetAppendAction() != corev3.HeaderValueOption_OVERWRITE_IF_EXISTS_OR_ADD { t.Errorf("append action = %v, want OVERWRITE_IF_EXISTS_OR_ADD", h0.GetAppendAction()) } - want := &ateapipb.MintActorJWTRequest{ + want := &ateapipb.MintActorJwtRequest{ Actor: &ateapipb.ObjectRef{Atespace: testEgressAtespace, Name: testEgressActor}, Audiences: []string{"https://api.example.com"}, ExpirationSeconds: 900, } if got := client.lastMint.Load(); !proto.Equal(got, want) { - t.Errorf("MintActorJWT request = %v, want %v", got, want) + t.Errorf("MintActorJwt request = %v, want %v", got, want) } }) } @@ -332,7 +332,7 @@ func TestActorJWTInjectionSkippedWithoutHeader(t *testing.T) { t.Errorf("got %d injected headers, want 0", len(got)) } if calls := client.mintCalls.Load(); calls != 0 { - t.Errorf("MintActorJWT calls = %d, want 0", calls) + t.Errorf("MintActorJwt calls = %d, want 0", calls) } } diff --git a/cmd/atenet/internal/router/egress/egress_test.go b/cmd/atenet/internal/router/egress/egress_test.go index 1b1febf633..c14ea93bb7 100644 --- a/cmd/atenet/internal/router/egress/egress_test.go +++ b/cmd/atenet/internal/router/egress/egress_test.go @@ -185,10 +185,10 @@ type egressMockClient struct { // closed, so a test can hold several callers on one fetch. policyGate chan struct{} - // The nth MintActorJWT call returns "jwt-", expiring + // The nth MintActorJwt call returns "jwt-", expiring // expiration_seconds from now. mintErr, when set, is returned instead. mintCalls atomic.Int32 - lastMint atomic.Pointer[ateapipb.MintActorJWTRequest] + lastMint atomic.Pointer[ateapipb.MintActorJwtRequest] mintErr error } @@ -220,15 +220,15 @@ func (m *egressMockClient) GetActorEgressPolicy(ctx context.Context, req *ateapi return m.policy, nil } -func (m *egressMockClient) MintActorJWT(_ context.Context, req *ateapipb.MintActorJWTRequest, _ ...grpc.CallOption) (*ateapipb.MintActorJWTResponse, error) { +func (m *egressMockClient) MintActorJwt(_ context.Context, req *ateapipb.MintActorJwtRequest, _ ...grpc.CallOption) (*ateapipb.MintActorJwtResponse, error) { n := m.mintCalls.Add(1) m.lastMint.Store(req) if m.mintErr != nil { return nil, m.mintErr } - return &ateapipb.MintActorJWTResponse{ - ActorJwt: fmt.Sprintf("jwt-%d", n), - ExpiresAt: timestamppb.New(time.Now().Add(time.Duration(req.GetExpirationSeconds()) * time.Second)), + return &ateapipb.MintActorJwtResponse{ + ActorJwt: fmt.Sprintf("jwt-%d", n), + ExpireTime: timestamppb.New(time.Now().Add(time.Duration(req.GetExpirationSeconds()) * time.Second)), }, nil } @@ -240,31 +240,31 @@ func allowAllPolicy() *ateapipb.EgressPolicy { func httpPolicy(patterns ...string) *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ - Http: &ateapipb.HTTPRule{Hostnames: patterns}, + Http: &ateapipb.HttpRule{Hostnames: patterns}, }}} } func httpPolicyOnPorts(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ - Http: &ateapipb.HTTPRule{Hostnames: patterns, Ports: ports}, + Http: &ateapipb.HttpRule{Hostnames: patterns, Ports: ports}, }}} } func httpsPolicy(patterns ...string) *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ - Https: &ateapipb.HTTPSRule{Hostnames: patterns}, + Https: &ateapipb.HttpsRule{Hostnames: patterns}, }}} } func httpsPolicyOnPorts(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ - Https: &ateapipb.HTTPSRule{Hostnames: patterns, Ports: ports}, + Https: &ateapipb.HttpsRule{Hostnames: patterns, Ports: ports}, }}} } func passthroughPolicy(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ - TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: patterns, Ports: ports}, + TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: patterns, Ports: ports}, }}} } diff --git a/cmd/atenet/internal/router/egress/request_test.go b/cmd/atenet/internal/router/egress/request_test.go index 43eed50ebd..505472f4c9 100644 --- a/cmd/atenet/internal/router/egress/request_test.go +++ b/cmd/atenet/internal/router/egress/request_test.go @@ -52,14 +52,14 @@ func sampleEffects() *ateapipb.HttpRuleEffects { // the authorization header, which only the MITM leg can honor. func credentialInjectionPolicySample(pattern string) *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ - Https: &ateapipb.HTTPSRule{Hostnames: []string{pattern}, Effects: sampleEffects()}, + Https: &ateapipb.HttpsRule{Hostnames: []string{pattern}, Effects: sampleEffects()}, }}} } // cleartextInjectionPolicy is the same replacement on an http rule. func cleartextInjectionPolicy(pattern string) *ateapipb.EgressPolicy { return &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ - Http: &ateapipb.HTTPRule{Hostnames: []string{pattern}, Effects: sampleEffects()}, + Http: &ateapipb.HttpRule{Hostnames: []string{pattern}, Effects: sampleEffects()}, }}} } diff --git a/cmd/atenet/internal/router/egresspolicymanifest_test.go b/cmd/atenet/internal/router/egresspolicymanifest_test.go index ba73a0b573..ed2708ac28 100644 --- a/cmd/atenet/internal/router/egresspolicymanifest_test.go +++ b/cmd/atenet/internal/router/egresspolicymanifest_test.go @@ -441,9 +441,6 @@ func TestEgressManifestsGiveEachTunnelItsOwnInnerPool(t *testing.T) { continue } internal++ - if ok, _ := cluster["connection_pool_per_downstream_connection"].(bool); !ok { - t.Errorf("internal cluster %q shares its pool across tunnels; want connection_pool_per_downstream_connection: true", str(cluster, "name")) - } tcp := child(child(cluster, "typed_extension_protocol_options"), "envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions") if got := str(tcp, "idle_timeout"); got == "" || strings.Trim(got, "0.s") == "" { t.Errorf("internal cluster %q idle_timeout is %q; want a non-zero timeout so a pool its tunnel left behind is freed", str(cluster, "name"), got) @@ -452,9 +449,6 @@ func TestEgressManifestsGiveEachTunnelItsOwnInnerPool(t *testing.T) { if internal == 0 { t.Fatal("no cluster targets an internal listener") } - if got := str(hcm(outerChain(t, tree)), "codec_type"); got != "HTTP1" { - t.Errorf("egress chain codec_type is %q, want HTTP1: any other codec can carry several CONNECTs on one downstream connection, and they would share a pool", got) - } }) } } @@ -719,15 +713,46 @@ func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing. t.Fatalf("%s is set by %d filters, want exactly the CONNECT leg's copy of its answer", extproc.EgressPolicyMetadataNamespace, len(writers)) } entry := writers[0] + var gotKeys []string + actionsByKey := map[string]node{} at := -1 for i, f := range filters { if str(f, "name") != setFilterStateFilter { continue } for _, v := range list(child(f, "typed_config"), "on_request_headers") { - if str(v, "object_key") == extproc.EgressPolicyMetadataNamespace { + key := str(v, "object_key") + gotKeys = append(gotKeys, key) + actionsByKey[key] = v + if key == extproc.EgressPolicyMetadataNamespace { at = i } + if got := str(v, "shared_with_upstream"); got != "ONCE" { + t.Errorf("%s has shared_with_upstream %q, want ONCE", key, got) + } + } + } + // Verify that expected state is shared with the upstream. + wantKeys := []string{ + extproc.ActorIdentityFilterStateKey, + extproc.ConnectAuthorityFilterStateKey, + extproc.UpstreamDynamicPortFilterStateKey, + extproc.EgressPolicyMetadataNamespace, + } + if !slices.Equal(gotKeys, wantKeys) { + t.Errorf("outer chain set_filter_state actions write keys %v, want %v", gotKeys, wantKeys) + } + // These keys determine pool selection and their values are bound to the pool at the + // time of pool creation. All CONNECT tunnels that have the same values for these filter + // state keys will share to pool. + for _, key := range []string{ + extproc.ActorIdentityFilterStateKey, + extproc.ConnectAuthorityFilterStateKey, + extproc.EgressPolicyMetadataNamespace, + } { + // Make sure factory creates hashable filter state objects, so it impacts pool selection. + if got := str(actionsByKey[key], "factory_key"); got != "envoy.hashable_string" { + t.Errorf("%s uses factory %q, want envoy.hashable_string", key, got) } } if at < 0 { @@ -740,15 +765,9 @@ func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing. if got := str(child(format, "text_format_source"), "inline_string"); got != extproc.EgressPolicyMetadataFormat { t.Errorf("%s is set from %q, want %q", extproc.EgressPolicyMetadataNamespace, got, extproc.EgressPolicyMetadataFormat) } - if got := str(entry, "factory_key"); got != "envoy.string" { - t.Errorf("%s uses factory %q, want envoy.string, the string accessor the module reads", extproc.EgressPolicyMetadataNamespace, got) - } if skip, _ := entry["skip_if_empty"].(bool); !skip { t.Errorf("%s is not skip_if_empty; an absent answer would be written as unparseable JSON", extproc.EgressPolicyMetadataNamespace) } - if str(entry, "shared_with_upstream") == "" { - t.Errorf("%s is not shared with upstream; the inner listener would never see it", extproc.EgressPolicyMetadataNamespace) - } } // Denied connections match no chain, so the listener access log is their only diff --git a/cmd/atenet/internal/router/envoyadmin_manifest_test.go b/cmd/atenet/internal/router/envoyadmin_manifest_test.go index 7ccd302648..93e684eb33 100644 --- a/cmd/atenet/internal/router/envoyadmin_manifest_test.go +++ b/cmd/atenet/internal/router/envoyadmin_manifest_test.go @@ -356,20 +356,37 @@ func parseEnvoyAdminBootstrap(t *testing.T, raw string) envoyAdminBootstrap { } func findDeployment(t *testing.T, path, name string) appsv1.Deployment { + t.Helper() + var d appsv1.Deployment + manifestObject(t, path, "Deployment", name, &d) + return d +} + +// manifestObject decodes the object with the given kind and name in the +// manifest at path into into. +func manifestObject(t *testing.T, path, kind, name string, into any) { t.Helper() raw, err := os.ReadFile(path) if err != nil { t.Fatalf("reading %s: %v", path, err) } for _, doc := range strings.Split(string(raw), "\n---\n") { - var d appsv1.Deployment - if err := yaml.Unmarshal([]byte(doc), &d); err != nil { + var head struct { + Kind string `json:"kind"` + Metadata struct { + Name string `json:"name"` + } `json:"metadata"` + } + if err := yaml.Unmarshal([]byte(doc), &head); err != nil { t.Fatalf("parsing a document of %s: %v", path, err) } - if d.Kind == "Deployment" && d.Name == name { - return d + if head.Kind != kind || head.Metadata.Name != name { + continue + } + if err := yaml.Unmarshal([]byte(doc), into); err != nil { + t.Fatalf("decoding the %s %s from %s: %v", name, kind, path, err) } + return } - t.Fatalf("%s has no Deployment named %s", path, name) - return appsv1.Deployment{} + t.Fatalf("%s has no %s named %s", path, kind, name) } diff --git a/cmd/atenet/internal/router/extproc/attributes.go b/cmd/atenet/internal/router/extproc/attributes.go index f347e58d43..e5721d2a5a 100644 --- a/cmd/atenet/internal/router/extproc/attributes.go +++ b/cmd/atenet/internal/router/extproc/attributes.go @@ -143,8 +143,9 @@ const UpstreamDynamicHostFilterStateKey = "envoy.upstream.dynamic_host" const EgressPolicyMetadataFormat = "%DYNAMIC_METADATA(" + EgressPolicyMetadataNamespace + ")%" // OriginalDstFilterStateKey is Envoy's filter-state key for the address an -// ORIGINAL_DST cluster dials. The gateway never writes it and dials only by -// name; the request legs read it as an attribute. +// ORIGINAL_DST cluster dials. The ingress router sets it for actor tunnels. +// The egress gateway never writes it and dials only by name; its request legs +// read it as an attribute. const OriginalDstFilterStateKey = "envoy.network.transport_socket.original_dst_address" // OriginalDstIPAttribute and OriginalDstPortAttribute are the CEL expressions diff --git a/cmd/atenet/internal/router/health.go b/cmd/atenet/internal/router/health.go index 6c90e2a6a7..19f7802196 100644 --- a/cmd/atenet/internal/router/health.go +++ b/cmd/atenet/internal/router/health.go @@ -229,7 +229,7 @@ func (rh *routerHealth) checkAteAPI(ctx context.Context) (bool, string) { timeoutCtx, cancel := context.WithTimeout(ctx, dependencyHealthCheckTimeout) defer cancel() - _, err := rh.apiClient.ListActors(timeoutCtx, &ateapipb.ListActorsRequest{PageSize: 1}) + _, err := rh.apiClient.ListActors(timeoutCtx, &ateapipb.ListActorsRequest{Options: &ateapipb.ListOptions{PageSize: 1}}) if err != nil { return false, err.Error() } diff --git a/cmd/atenet/internal/router/router.go b/cmd/atenet/internal/router/router.go index 36ac759664..7afad07a3d 100644 --- a/cmd/atenet/internal/router/router.go +++ b/cmd/atenet/internal/router/router.go @@ -158,11 +158,8 @@ func (s *RouterServer) Run(ctx context.Context) error { // pod drains — dropping it from the Service endpoints — while /healthz // stays 200 for liveness. readiness := &serverboot.Readiness{} - go serverboot.StartMetricsServer(ctx, serverboot.MetricsServerOptions{ - Addr: s.cfg.MetricsAddr, - Readiness: readiness, - EnableHealthz: true, - }) + go serverboot.StartMetricsServer(ctx, s.cfg.MetricsAddr) + go serverboot.StartHealthServer(ctx, s.cfg.HealthAddr, readiness) dialOpts, err := ateapiauth.DialOptions(ateapiauth.ClientConfig{ K8sClient: s.clientset, @@ -260,6 +257,17 @@ func (s *RouterServer) Run(ctx context.Context) error { s.health = newRouterHealth(s.cfg.HealthInterval, s.clientset, s.apiClient, s.cfg) + // Bound before the background servers below start, so a bad + // --status-address fails startup without leaving them running. + var statusListener net.Listener + if s.cfg.StatusPort > 0 { + statusListener, err = s.cfg.listenStatus() + if err != nil { + return err + } + defer statusListener.Close() + } + // The ingress control plane — the xDS server — configures the *ingress* // dataplane. The egress gateway is statically configured, so an egress-only // instance does not run it. @@ -285,7 +293,7 @@ func (s *RouterServer) Run(ctx context.Context) error { // co-located in the same pod and always dials over loopback, so binding // loopback keeps the ext_proc server unreachable from other pods on the // flat pod network, where nothing authenticates the caller. Readiness is - // probed via /readyz on the metrics port, not this one, so a loopback bind + // probed via /readyz on the health port, not this one, so a loopback bind // does not break it. An empty address binds every interface. extprocListenAddr := net.JoinHostPort(s.cfg.ExtprocAddr, strconv.Itoa(s.cfg.ExtprocPort)) slog.InfoContext(ctx, "Starting ExtProc Server", slog.String("address", extprocListenAddr)) @@ -300,12 +308,7 @@ func (s *RouterServer) Run(ctx context.Context) error { }) // Start HTTP status endpoint - if s.cfg.StatusPort > 0 { - listener, err := net.Listen("tcp", fmt.Sprintf(":%d", s.cfg.StatusPort)) - if err != nil { - return fmt.Errorf("failed binding Router HTTP status server port: %w", err) - } - + if statusListener != nil { mux := http.NewServeMux() mux.HandleFunc("/statusz", s.handleStatusz) @@ -315,7 +318,7 @@ func (s *RouterServer) Run(ctx context.Context) error { g.Go(func() error { go func() { - if err := httpServer.Serve(listener); err != nil && !errors.Is(err, http.ErrServerClosed) { + if err := httpServer.Serve(statusListener); err != nil && !errors.Is(err, http.ErrServerClosed) { slog.ErrorContext(ctx, "status HTTP server exited unexpectedly", slog.Any("err", err)) } }() diff --git a/cmd/atenet/internal/router/status.go b/cmd/atenet/internal/router/status.go index 9fe196908f..2afae673a1 100644 --- a/cmd/atenet/internal/router/status.go +++ b/cmd/atenet/internal/router/status.go @@ -20,9 +20,11 @@ import ( "encoding/json" "fmt" "html/template" + "net" "net/http" "os" "runtime/debug" + "strconv" "strings" "time" @@ -63,6 +65,17 @@ type FormattedQuery struct { Duration string `json:"duration"` } +// listenStatus binds the /statusz listener on --status-address and +// --status-port. +func (c routerConfig) listenStatus() (net.Listener, error) { + addr := net.JoinHostPort(c.StatusAddr, strconv.Itoa(c.StatusPort)) + lis, err := net.Listen("tcp", addr) + if err != nil { + return nil, fmt.Errorf("failed binding Router HTTP status server address %q: %w", addr, err) + } + return lis, nil +} + func (s *RouterServer) getRouterIP(ctx context.Context) string { if s.clientset == nil { return "Offline Mode (No Cluster IP)" diff --git a/cmd/atenet/internal/router/status_test.go b/cmd/atenet/internal/router/status_test.go index 6fccee90d6..02f3055a5c 100644 --- a/cmd/atenet/internal/router/status_test.go +++ b/cmd/atenet/internal/router/status_test.go @@ -34,6 +34,8 @@ import ( "testing" "time" + corev1 "k8s.io/api/core/v1" + "github.com/agent-substrate/substrate/cmd/atenet/internal/router/extproc" "github.com/agent-substrate/substrate/cmd/atenet/internal/router/ingress" ) @@ -58,15 +60,21 @@ func TestStatuszEndpoint(t *testing.T) { // Run() dials ateapi in mtls mode, which requires real TLS material; generate it. caPath, clientCertPath := writeTestTLSMaterial(t) + statusAddr, err := NewRouterCmd().Flags().GetString("status-address") + if err != nil { + t.Fatal(err) + } cfg := routerConfig{ Mode: ModeIngress, Namespace: "default", StatusPort: httpPort, + StatusAddr: statusAddr, HttpPort: 8080, XdsPort: 18000, ExtprocPort: 50051, ExtProcMaxRequests: defaultExtProcMaxRequests, MetricsAddr: "127.0.0.1:0", + HealthAddr: "127.0.0.1:0", ParkedRequest: ingress.DefaultParkedRequestConfig(), Auth: authConfig{ AteapiCAFile: caPath, @@ -169,6 +177,37 @@ func TestStatuszEndpoint(t *testing.T) { } } +// The manifest's router args, defaults included, keep /statusz off the pod IP. +func TestRouterManifestBindsStatusToLoopback(t *testing.T) { + addr, err := routerManifestFlags(t).GetString("status-address") + if err != nil { + t.Fatalf("reading --status-address from the manifest args: %v", err) + } + // Port 0 binds any free port; only the address is under test. + lis, err := routerConfig{StatusAddr: addr}.listenStatus() + if err != nil { + t.Fatal(err) + } + defer lis.Close() + if ip := lis.Addr().(*net.TCPAddr).IP; !ip.IsLoopback() { + t.Errorf("--status-address %q in %s binds %s, which other pods can reach", addr, routerManifestPath, ip) + } +} + +func TestRouterServiceDoesNotExposeStatus(t *testing.T) { + statusPort, err := routerManifestFlags(t).GetInt("status-port") + if err != nil { + t.Fatalf("reading --status-port from the manifest args: %v", err) + } + var service corev1.Service + manifestObject(t, routerManifestPath, "Service", "atenet-router", &service) + for _, port := range service.Spec.Ports { + if port.Name == "status" || int(port.Port) == statusPort || port.TargetPort.IntValue() == statusPort || port.TargetPort.StrVal == "status" { + t.Errorf("router Service exposes the status server: %+v", port) + } + } +} + // writeTestKubeconfig writes a kubeconfig pointing at an unreachable local // server and returns its path. It satisfies client construction; any request // made through the resulting clientset fails with a connection error. diff --git a/cmd/atenet/internal/router/xds.go b/cmd/atenet/internal/router/xds.go index b50dbefdab..a3b3089426 100644 --- a/cmd/atenet/internal/router/xds.go +++ b/cmd/atenet/internal/router/xds.go @@ -51,6 +51,8 @@ import ( setfilterstatev3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/http/set_filter_state/v3" originaldstv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/listener/original_dst/v3" hcmv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/http_connection_manager/v3" + setfilterstatenetworkv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/set_filter_state/v3" + tcpproxyv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/tcp_proxy/v3" internalupstreamv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/internal_upstream/v3" rawbufferv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/raw_buffer/v3" tlsv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/tls/v3" @@ -109,6 +111,12 @@ const ( // request headers for atunnel to authorize. OriginalDstClusterName = "actor_original_dst" + // With upstream CONNECT, requests go through an actor_tunnel cluster into + // the actor_connect listener, which tunnels them to atunnel. + ActorTunnelHTTP1ClusterName = "actor_tunnel_http1" + ActorTunnelHTTP2ClusterName = "actor_tunnel_http2" + ActorConnectName = "actor_connect" + WildcardIP = "0.0.0.0" ConnectUpgradeType = "CONNECT" MainInternalName = "main_internal" @@ -183,6 +191,24 @@ const envoyDefaultStreamIdleTimeout = 5 * time.Minute // one is wide enough to read as deliberate in a config dump. const routeIdleTimeoutMargin = 30 * time.Second +// Filter state that carries the routing decision into actor_connect. +// tunnelKeyFilterStateKey is the only hashable value and Envoy pools tunnels by +// it, so a tunnel only carries one actor and port. A second hashable value +// would split the pool, since Envoy hashes them in map order. +const ( + tunnelKeyFilterStateKey = "dev.ate.tunnel.key" + tunnelWorkerFilterStateKey = "dev.ate.tunnel.worker" + tunnelActorFilterStateKey = "dev.ate.tunnel.actor" + tunnelPortFilterStateKey = "dev.ate.tunnel.port" +) + +// tunnelCircuitBreakerLimit lifts Envoy's default of 1024: there is a tunnel +// per active actor and port, and idle ones count too. +const tunnelCircuitBreakerLimit = 65536 + +// tunnelIdleTimeout closes unused tunnels, which hold actor connections. +const tunnelIdleTimeout = 60 * time.Second + // XdsServer implements an aggregated discovery service server for dynamic Envoy router nodes. type XdsServer struct { xdsPort int @@ -213,6 +239,8 @@ type XdsServer struct { // cert carries only a spiffe:// URI SAN, so without this Envoy's default // SAN check against the dialed IP fails ("verify SAN list"). upstreamSpiffePrefix string + // upstreamConnect: see SetUpstreamConnect. + upstreamConnect bool // Downstream (client-facing) mTLS. When downstreamClientCAPath is set, // every TLS ingress listener requests a client certificate and rejects @@ -380,6 +408,15 @@ func (x *XdsServer) SetUpstreamTls(credentialBundlePath, trustBundlePath, spiffe x.upstreamSpiffePrefix = spiffePrefix } +// SetUpstreamConnect makes actor traffic reach atunnel through CONNECT tunnels +// on atunnel.DefaultConnectPort, one per actor and port. It needs +// SetUpstreamTls, since that listener only accepts mTLS. +func (x *XdsServer) SetUpstreamConnect(enabled bool) { + x.mu.Lock() + defer x.mu.Unlock() + x.upstreamConnect = enabled +} + // SetOtlpCollector enables Envoy-side tracing pointed at the OTLP gRPC // collector. addr empty disables tracing. See normalizeOtlpCollector for the // accepted forms. @@ -486,7 +523,15 @@ func (x *XdsServer) UpdateSnapshot() error { // Clusters clusters := []types.Resource{ x.buildCluster(), - x.buildOriginalDstCluster(), + } + if x.upstreamConnect { + clusters = append(clusters, + buildActorTunnelCluster(ActorTunnelHTTP1ClusterName, false), + buildActorTunnelCluster(ActorTunnelHTTP2ClusterName, true), + x.buildActorConnectCluster(), + ) + } else { + clusters = append(clusters, x.buildOriginalDstCluster()) } if connectEnabled { clusters = append(clusters, x.buildMainInternalCluster()) @@ -505,6 +550,9 @@ func (x *XdsServer) UpdateSnapshot() error { if x.ingressPort > 0 { listeners = append(listeners, x.buildListener()) } + if x.upstreamConnect { + listeners = append(listeners, buildActorConnectListener()) + } if connectEnabled { listeners = append(listeners, x.buildMainInternalListener()) } @@ -764,28 +812,21 @@ func rawBuffer() *corev3.TransportSocket { } } -// MainInternalCluster is a simple cluster that sends traffic to the main_internal -// internal listener (used only for CONNECT requests). -func (x *XdsServer) buildMainInternalCluster() *clusterv3.Cluster { - return &clusterv3.Cluster{ - Name: MainInternalName, - ClusterDiscoveryType: &clusterv3.Cluster_Type{ - Type: clusterv3.Cluster_STATIC, - }, - LoadAssignment: &endpointv3.ClusterLoadAssignment{ - ClusterName: MainInternalName, - Endpoints: []*endpointv3.LocalityLbEndpoints{ - { - LbEndpoints: []*endpointv3.LbEndpoint{ - { - HostIdentifier: &endpointv3.LbEndpoint_Endpoint{ - Endpoint: &endpointv3.Endpoint{ - Address: &corev3.Address{ - Address: &corev3.Address_EnvoyInternalAddress{ - EnvoyInternalAddress: &corev3.EnvoyInternalAddress{ - AddressNameSpecifier: &corev3.EnvoyInternalAddress_ServerListenerName{ - ServerListenerName: MainInternalName, - }, +// internalListenerAssignment points a cluster at an internal listener. +func internalListenerAssignment(clusterName, listenerName string) *endpointv3.ClusterLoadAssignment { + return &endpointv3.ClusterLoadAssignment{ + ClusterName: clusterName, + Endpoints: []*endpointv3.LocalityLbEndpoints{ + { + LbEndpoints: []*endpointv3.LbEndpoint{ + { + HostIdentifier: &endpointv3.LbEndpoint_Endpoint{ + Endpoint: &endpointv3.Endpoint{ + Address: &corev3.Address{ + Address: &corev3.Address_EnvoyInternalAddress{ + EnvoyInternalAddress: &corev3.EnvoyInternalAddress{ + AddressNameSpecifier: &corev3.EnvoyInternalAddress_ServerListenerName{ + ServerListenerName: listenerName, }, }, }, @@ -796,6 +837,18 @@ func (x *XdsServer) buildMainInternalCluster() *clusterv3.Cluster { }, }, }, + } +} + +// MainInternalCluster is a simple cluster that sends traffic to the main_internal +// internal listener (used only for CONNECT requests). +func (x *XdsServer) buildMainInternalCluster() *clusterv3.Cluster { + return &clusterv3.Cluster{ + Name: MainInternalName, + ClusterDiscoveryType: &clusterv3.Cluster_Type{ + Type: clusterv3.Cluster_STATIC, + }, + LoadAssignment: internalListenerAssignment(MainInternalName, MainInternalName), TransportSocket: &corev3.TransportSocket{ Name: "internal_upstream", ConfigType: &corev3.TransportSocket_TypedConfig{ @@ -866,51 +919,191 @@ func (x *XdsServer) buildOriginalDstCluster() *clusterv3.Cluster { return cluster } +// buildActorTunnelCluster feeds the actor_connect listener, one tunnel per +// connection. The actor receives this protocol as is, so it is fixed per +// cluster: HTTP/2 for gRPC, HTTP/1.1 otherwise. +func buildActorTunnelCluster(name string, http2 bool) *clusterv3.Cluster { + explicit := &httpv3.HttpProtocolOptions_ExplicitHttpConfig{ + ProtocolConfig: &httpv3.HttpProtocolOptions_ExplicitHttpConfig_HttpProtocolOptions{ + HttpProtocolOptions: &corev3.Http1ProtocolOptions{}, + }, + } + if http2 { + explicit.ProtocolConfig = &httpv3.HttpProtocolOptions_ExplicitHttpConfig_Http2ProtocolOptions{ + Http2ProtocolOptions: &corev3.Http2ProtocolOptions{}, + } + } + return &clusterv3.Cluster{ + Name: name, + ClusterDiscoveryType: &clusterv3.Cluster_Type{ + Type: clusterv3.Cluster_STATIC, + }, + LoadAssignment: internalListenerAssignment(name, ActorConnectName), + TransportSocket: &corev3.TransportSocket{ + Name: "internal_upstream", + ConfigType: &corev3.TransportSocket_TypedConfig{ + TypedConfig: newAny(&internalupstreamv3.InternalUpstreamTransport{ + TransportSocket: rawBuffer(), + }), + }, + }, + CircuitBreakers: tunnelCircuitBreakers(), + TypedExtensionProtocolOptions: map[string]*anypb.Any{ + httpProtocolOptionsName: newAny(&httpv3.HttpProtocolOptions{ + CommonHttpProtocolOptions: &corev3.HttpProtocolOptions{ + IdleTimeout: durationpb.New(tunnelIdleTimeout), + }, + UpstreamProtocolOptions: &httpv3.HttpProtocolOptions_ExplicitHttpConfig_{ + ExplicitHttpConfig: explicit, + }, + }), + }, + } +} + +// buildActorConnectCluster dials atunnel's CONNECT port on the worker that +// actor_connect puts in filter state. HTTP/2 lets a worker's tunnels share a +// connection. +func (x *XdsServer) buildActorConnectCluster() *clusterv3.Cluster { + return &clusterv3.Cluster{ + Name: ActorConnectName, + ConnectTimeout: durationpb.New(5 * time.Second), + ClusterDiscoveryType: &clusterv3.Cluster_Type{ + Type: clusterv3.Cluster_ORIGINAL_DST, + }, + LbPolicy: clusterv3.Cluster_CLUSTER_PROVIDED, + LbConfig: &clusterv3.Cluster_OriginalDstLbConfig_{ + OriginalDstLbConfig: &clusterv3.Cluster_OriginalDstLbConfig{ + UpstreamPortOverride: wrapperspb.UInt32(atunnel.DefaultConnectPort), + }, + }, + TransportSocket: x.buildUpstreamTransportSocket(), + CircuitBreakers: tunnelCircuitBreakers(), + TypedExtensionProtocolOptions: map[string]*anypb.Any{ + httpProtocolOptionsName: newAny(&httpv3.HttpProtocolOptions{ + UpstreamProtocolOptions: &httpv3.HttpProtocolOptions_ExplicitHttpConfig_{ + ExplicitHttpConfig: &httpv3.HttpProtocolOptions_ExplicitHttpConfig{ + ProtocolConfig: &httpv3.HttpProtocolOptions_ExplicitHttpConfig_Http2ProtocolOptions{ + Http2ProtocolOptions: &corev3.Http2ProtocolOptions{}, + }, + }, + }, + }), + }, + } +} + +func tunnelCircuitBreakers() *clusterv3.CircuitBreakers { + return &clusterv3.CircuitBreakers{ + Thresholds: []*clusterv3.CircuitBreakers_Thresholds{{ + Priority: corev3.RoutingPriority_DEFAULT, + MaxConnections: wrapperspb.UInt32(tunnelCircuitBreakerLimit), + MaxPendingRequests: wrapperspb.UInt32(tunnelCircuitBreakerLimit), + MaxRequests: wrapperspb.UInt32(tunnelCircuitBreakerLimit), + }}, + } +} + func (x *XdsServer) buildRoutes() *routev3.RouteConfiguration { + var routes []*routev3.Route + if x.upstreamConnect { + routes = x.buildActorTunnelRoutes() + } else { + routes = []*routev3.Route{ + { + Match: &routev3.RouteMatch{ + PathSpecifier: &routev3.RouteMatch_Prefix{ + Prefix: "/", + }, + }, + Action: &routev3.Route_Route{ + Route: x.workloadRouteAction(OriginalDstClusterName), + }, + RequestHeadersToAdd: []*corev3.HeaderValueOption{ + { + Header: &corev3.HeaderValue{ + Key: atunnel.TargetPortHeader, + Value: fmt.Sprintf("%%DYNAMIC_METADATA(%s:%s)%%", + ingress.OriginalDstMetadataKey, ingress.OriginalDstPortKey), + }, + AppendAction: corev3.HeaderValueOption_OVERWRITE_IF_EXISTS_OR_ADD, + }, + }, + }, + } + } return &routev3.RouteConfiguration{ Name: RouteName, VirtualHosts: []*routev3.VirtualHost{ { Name: "local_service", Domains: []string{"*"}, - Routes: []*routev3.Route{ - { - Match: &routev3.RouteMatch{ - PathSpecifier: &routev3.RouteMatch_Prefix{ - Prefix: "/", - }, - }, - Action: &routev3.Route_Route{ - Route: &routev3.RouteAction{ - ClusterSpecifier: &routev3.RouteAction_Cluster{ - Cluster: OriginalDstClusterName, - }, - // Two separate limits: Timeout bounds the upstream response, - // IdleTimeout bounds a stream with no activity on it, and - // routeIdleTimeout keeps the second from firing before the - // first. This route also serves CONNECT-tunneled traffic - // re-injected via main_internal, where Envoy applies Timeout to - // the whole tunnel lifetime, so a long-lived tunnel needs - // --route-timeout raised like a long LLM turn does. - Timeout: durationpb.New(x.routeTimeout), - IdleTimeout: durationpb.New(x.routeIdleTimeout()), - }, - }, - RequestHeadersToAdd: []*corev3.HeaderValueOption{ - { - Header: &corev3.HeaderValue{ - Key: atunnel.TargetPortHeader, - Value: fmt.Sprintf("%%DYNAMIC_METADATA(%s:%s)%%", - ingress.OriginalDstMetadataKey, ingress.OriginalDstPortKey), - }, - AppendAction: corev3.HeaderValueOption_OVERWRITE_IF_EXISTS_OR_ADD, - }, - }, + Routes: routes, + }, + }, + } +} + +func (x *XdsServer) workloadRouteAction(cluster string) *routev3.RouteAction { + return &routev3.RouteAction{ + ClusterSpecifier: &routev3.RouteAction_Cluster{ + Cluster: cluster, + }, + // Two separate limits: Timeout bounds the upstream response, + // IdleTimeout bounds a stream with no activity on it, and + // routeIdleTimeout keeps the second from firing before the + // first. This route also serves CONNECT-tunneled traffic + // re-injected via main_internal, where Envoy applies Timeout to + // the whole tunnel lifetime, so a long-lived tunnel needs + // --route-timeout raised like a long LLM turn does. + Timeout: durationpb.New(x.routeTimeout), + IdleTimeout: durationpb.New(x.routeIdleTimeout()), + } +} + +// buildActorTunnelRoutes sends gRPC over HTTP/2 and everything else over +// HTTP/1.1, so HTTP/1.1-only actors keep working. A route can't see the +// client's protocol, so gRPC-shaped HTTP/1.1 requests go out as HTTP/2 too. +func (x *XdsServer) buildActorTunnelRoutes() []*routev3.Route { + route := func(cluster string, headers ...*routev3.HeaderMatcher) *routev3.Route { + return &routev3.Route{ + Match: &routev3.RouteMatch{ + PathSpecifier: &routev3.RouteMatch_Prefix{ + Prefix: "/", + }, + Headers: headers, + }, + Action: &routev3.Route_Route{ + Route: x.workloadRouteAction(cluster), + }, + // Nothing past this point strips it before the actor. + RequestHeadersToRemove: []string{atenet.TargetActorHeader}, + } + } + grpcHeaders := []*routev3.HeaderMatcher{ + { + Name: ":method", + HeaderMatchSpecifier: &routev3.HeaderMatcher_StringMatch{ + StringMatch: &matcherv3.StringMatcher{ + MatchPattern: &matcherv3.StringMatcher_Exact{Exact: "POST"}, + }, + }, + }, + { + Name: "content-type", + HeaderMatchSpecifier: &routev3.HeaderMatcher_StringMatch{ + StringMatch: &matcherv3.StringMatcher{ + MatchPattern: &matcherv3.StringMatcher_SafeRegex{ + SafeRegex: &matcherv3.RegexMatcher{Regex: `(?i)^application/grpc([+;].*)?$`}, }, }, }, }, } + return []*routev3.Route{ + route(ActorTunnelHTTP2ClusterName, grpcHeaders...), + route(ActorTunnelHTTP1ClusterName), + } } func (x *XdsServer) buildMainInternalListener() *listenerv3.Listener { @@ -947,38 +1140,30 @@ func (x *XdsServer) buildMainInternalListener() *listenerv3.Listener { } } -// actorRoutingFilterStateFilter captures the actor routing header so -// main_internal can read it across the CONNECT internal-listener hop. -func actorRoutingFilterStateFilter(captureAuthority bool) *hcmv3.HttpFilter { - values := make([]*setfilterstatecommonv3.FilterStateValue, 0, 2) - routingFields := []struct{ key, format string }{ - {extproc.TargetActorFilterStateKey, "%CEL(request.headers['" + atenet.TargetActorHeader + "'])%"}, - } - if captureAuthority { - routingFields = append(routingFields, struct{ key, format string }{ - extproc.ConnectAuthorityFilterStateKey, "%REQ(" + extproc.AuthorityHeader + ")%", - }) - } - for _, routingField := range routingFields { - values = append(values, &setfilterstatecommonv3.FilterStateValue{ - Key: &setfilterstatecommonv3.FilterStateValue_ObjectKey{ - ObjectKey: routingField.key, - }, - FactoryKey: "envoy.string", - Value: &setfilterstatecommonv3.FilterStateValue_FormatString{ - FormatString: &corev3.SubstitutionFormatString{ - Format: &corev3.SubstitutionFormatString_TextFormatSource{ - TextFormatSource: &corev3.DataSource{ - Specifier: &corev3.DataSource_InlineString{ - InlineString: routingField.format, - }, +// filterStateValue sets key from format. An empty factory means the one +// registered under key. +func filterStateValue(key, factory, format string, shared setfilterstatecommonv3.FilterStateValue_SharedWithUpstream) *setfilterstatecommonv3.FilterStateValue { + return &setfilterstatecommonv3.FilterStateValue{ + Key: &setfilterstatecommonv3.FilterStateValue_ObjectKey{ + ObjectKey: key, + }, + FactoryKey: factory, + Value: &setfilterstatecommonv3.FilterStateValue_FormatString{ + FormatString: &corev3.SubstitutionFormatString{ + Format: &corev3.SubstitutionFormatString_TextFormatSource{ + TextFormatSource: &corev3.DataSource{ + Specifier: &corev3.DataSource_InlineString{ + InlineString: format, }, }, }, }, - SharedWithUpstream: setfilterstatecommonv3.FilterStateValue_ONCE, - }) + }, + SharedWithUpstream: shared, } +} + +func setFilterStateHTTPFilter(values ...*setfilterstatecommonv3.FilterStateValue) *hcmv3.HttpFilter { return &hcmv3.HttpFilter{ Name: "envoy.filters.http.set_filter_state", ConfigType: &hcmv3.HttpFilter_TypedConfig{ @@ -989,6 +1174,98 @@ func actorRoutingFilterStateFilter(captureAuthority bool) *hcmv3.HttpFilter { } } +// actorRoutingFilterStateFilter captures the actor routing header so +// main_internal can read it across the CONNECT internal-listener hop. +func actorRoutingFilterStateFilter(captureAuthority bool) *hcmv3.HttpFilter { + once := setfilterstatecommonv3.FilterStateValue_ONCE + values := []*setfilterstatecommonv3.FilterStateValue{ + filterStateValue(extproc.TargetActorFilterStateKey, "envoy.string", "%CEL(request.headers['"+atenet.TargetActorHeader+"'])%", once), + } + if captureAuthority { + values = append(values, + filterStateValue(extproc.ConnectAuthorityFilterStateKey, "envoy.string", "%REQ("+extproc.AuthorityHeader+")%", once)) + } + return setFilterStateHTTPFilter(values...) +} + +// actorTunnelFilterStateFilter copies ext_proc's routing decision into filter +// state. It must run after ext_proc, which overwrites the actor header with the +// checked value. +func actorTunnelFilterStateFilter() *hcmv3.HttpFilter { + worker := "%DYNAMIC_METADATA(" + ingress.OriginalDstMetadataKey + ":" + ingress.OriginalDstAddressKey + ")%" + actor := "%REQ(" + atenet.TargetActorHeader + ")%" + port := "%DYNAMIC_METADATA(" + ingress.OriginalDstMetadataKey + ":" + ingress.OriginalDstPortKey + ")%" + once := setfilterstatecommonv3.FilterStateValue_ONCE + return setFilterStateHTTPFilter( + filterStateValue(tunnelKeyFilterStateKey, "envoy.hashable_string", worker+"|"+actor+"|"+port, once), + filterStateValue(tunnelWorkerFilterStateKey, "envoy.string", worker, once), + filterStateValue(tunnelActorFilterStateKey, "envoy.string", actor, once), + filterStateValue(tunnelPortFilterStateKey, "envoy.string", port, once), + ) +} + +// buildActorConnectListener turns each connection from an actor_tunnel cluster +// into a CONNECT to atunnel, which takes the port from the authority and +// ignores the host. +func buildActorConnectListener() *listenerv3.Listener { + return &listenerv3.Listener{ + Name: ActorConnectName, + ListenerSpecifier: &listenerv3.Listener_InternalListener{ + InternalListener: &listenerv3.Listener_InternalListenerConfig{}, + }, + FilterChains: []*listenerv3.FilterChain{ + { + Filters: []*listenerv3.Filter{ + { + // Set here so the address stays out of the tunnel + // pool's key. + Name: "envoy.filters.network.set_filter_state", + ConfigType: &listenerv3.Filter_TypedConfig{ + TypedConfig: newAny(&setfilterstatenetworkv3.Config{ + OnNewConnection: []*setfilterstatecommonv3.FilterStateValue{ + filterStateValue(extproc.OriginalDstFilterStateKey, "", + "%FILTER_STATE("+tunnelWorkerFilterStateKey+":PLAIN)%", + setfilterstatecommonv3.FilterStateValue_NONE), + }, + }), + }, + }, + { + Name: "envoy.filters.network.tcp_proxy", + ConfigType: &listenerv3.Filter_TypedConfig{ + TypedConfig: newAny(&tcpproxyv3.TcpProxy{ + StatPrefix: ActorConnectName, + ClusterSpecifier: &tcpproxyv3.TcpProxy_Cluster{ + Cluster: ActorConnectName, + }, + TunnelingConfig: &tcpproxyv3.TcpProxy_TunnelingConfig{ + Hostname: "actor:%FILTER_STATE(" + tunnelPortFilterStateKey + ":PLAIN)%", + HeadersToAdd: []*corev3.HeaderValueOption{ + { + Header: &corev3.HeaderValue{ + Key: atenet.TargetActorHeader, + Value: "%FILTER_STATE(" + tunnelActorFilterStateKey + ":PLAIN)%", + }, + AppendAction: corev3.HeaderValueOption_OVERWRITE_IF_EXISTS_OR_ADD, + }, + }, + // Logs why atunnel refused a CONNECT; the + // client only sees a 503. + PropagateResponseHeaders: true, + }, + // Off, so tunnelIdleTimeout and the route + // timeouts always fire first. + IdleTimeout: durationpb.New(0), + AccessLog: stdoutAccessLog(tunnelAccessLogFormat), + }), + }, + }, + }, + }, + }, + } +} + func (x *XdsServer) buildConnectTerminateHCM(statPrefix string) *anypb.Any { routerAny := newAny(&routerv3.Router{}) hcm := newAny(&hcmv3.HttpConnectionManager{ @@ -997,7 +1274,7 @@ func (x *XdsServer) buildConnectTerminateHCM(statPrefix string) *anypb.Any { Tracing: x.buildTracing(), // TODO: this format is not very useful for CONNECT requests. // Need to customize it to surface useful information - AccessLog: stdoutAccessLog(), + AccessLog: stdoutAccessLog(accessLogFormat), RouteSpecifier: &hcmv3.HttpConnectionManager_RouteConfig{ RouteConfig: buildConnectRoutes(), }, @@ -1113,20 +1390,21 @@ func (x *XdsServer) buildHcm(statPrefix string, captureActorRouting bool) *anypb if captureActorRouting { httpFilters = append(httpFilters, actorRoutingFilterStateFilter(false)) } - httpFilters = append(httpFilters, - &hcmv3.HttpFilter{ - Name: httpExtProcFilterName, - ConfigType: &hcmv3.HttpFilter_TypedConfig{ - TypedConfig: extProcConfig, - }, + httpFilters = append(httpFilters, &hcmv3.HttpFilter{ + Name: httpExtProcFilterName, + ConfigType: &hcmv3.HttpFilter_TypedConfig{ + TypedConfig: extProcConfig, }, - &hcmv3.HttpFilter{ - Name: "envoy.filters.http.router", - ConfigType: &hcmv3.HttpFilter_TypedConfig{ - TypedConfig: routerAny, - }, + }) + if x.upstreamConnect { + httpFilters = append(httpFilters, actorTunnelFilterStateFilter()) + } + httpFilters = append(httpFilters, &hcmv3.HttpFilter{ + Name: "envoy.filters.http.router", + ConfigType: &hcmv3.HttpFilter_TypedConfig{ + TypedConfig: routerAny, }, - ) + }) hcm := newAny(&hcmv3.HttpConnectionManager{ StatPrefix: statPrefix, @@ -1135,7 +1413,7 @@ func (x *XdsServer) buildHcm(statPrefix string, captureActorRouting bool) *anypb {UpgradeType: "websocket"}, }, Tracing: x.buildTracing(), - AccessLog: stdoutAccessLog(), + AccessLog: stdoutAccessLog(accessLogFormat), HttpFilters: httpFilters, RouteSpecifier: &hcmv3.HttpConnectionManager_Rds{ Rds: &hcmv3.Rds{ @@ -1498,9 +1776,15 @@ const accessLogFormat = `[%START_TIME%] "%REQ(:METHOD)% %PATH(NQ:ORIG_OR_PATH)% `%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" ` + `"%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"` + "\n" -// stdoutAccessLog returns the access log configuration shared by the router's -// HTTP connection managers. -func stdoutAccessLog() []*accesslogv3.AccessLog { +// tunnelAccessLogFormat logs each tunnel when it closes, with atunnel's CONNECT +// response. +const tunnelAccessLogFormat = `[%START_TIME%] tunnel actor=%FILTER_STATE(` + tunnelActorFilterStateKey + `:PLAIN)% ` + + `port=%FILTER_STATE(` + tunnelPortFilterStateKey + `:PLAIN)% "%UPSTREAM_HOST%" %RESPONSE_FLAGS% ` + + `connect_response=%FILTER_STATE(envoy.tcp_proxy.propagate_response_headers:TYPED)% ` + + `%BYTES_RECEIVED% %BYTES_SENT% %DURATION%` + "\n" + +// stdoutAccessLog returns a stdout access log that writes format. +func stdoutAccessLog(format string) []*accesslogv3.AccessLog { return []*accesslogv3.AccessLog{ { Name: "envoy.access_loggers.stdout", @@ -1510,7 +1794,7 @@ func stdoutAccessLog() []*accesslogv3.AccessLog { LogFormat: &corev3.SubstitutionFormatString{ Format: &corev3.SubstitutionFormatString_TextFormatSource{ TextFormatSource: &corev3.DataSource{ - Specifier: &corev3.DataSource_InlineString{InlineString: accessLogFormat}, + Specifier: &corev3.DataSource_InlineString{InlineString: format}, }, }, }, diff --git a/cmd/atenet/internal/router/xds_upstream_connect_test.go b/cmd/atenet/internal/router/xds_upstream_connect_test.go new file mode 100644 index 0000000000..7bea556e82 --- /dev/null +++ b/cmd/atenet/internal/router/xds_upstream_connect_test.go @@ -0,0 +1,324 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package router + +import ( + "regexp" + "testing" + + clusterv3 "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3" + listenerv3 "github.com/envoyproxy/go-control-plane/envoy/config/listener/v3" + setfilterstatecommonv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/common/set_filter_state/v3" + setfilterstatev3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/http/set_filter_state/v3" + hcmv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/http_connection_manager/v3" + setfilterstatenetworkv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/set_filter_state/v3" + tcpproxyv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/tcp_proxy/v3" + tlsv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/tls/v3" + httpv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/upstreams/http/v3" + cachev3 "github.com/envoyproxy/go-control-plane/pkg/cache/v3" + resourcev3 "github.com/envoyproxy/go-control-plane/pkg/resource/v3" + + "github.com/agent-substrate/substrate/cmd/atenet/internal/router/extproc" + "github.com/agent-substrate/substrate/internal/atenet" + "github.com/agent-substrate/substrate/internal/atunnel" +) + +// upstreamConnectSnapshot also enables the CONNECT listener, so main_internal +// is covered. +func upstreamConnectSnapshot(t *testing.T) *cachev3.Snapshot { + t.Helper() + server := NewXdsServer(18000) + server.SetConfig(8085, 50053, "127.0.0.1") + server.SetConnectPorts(8081, 0) + server.SetUpstreamTls("/run/bundle.pem", "/run/trust.pem", "spiffe://cluster.local/") + server.SetUpstreamConnect(true) + if err := server.UpdateSnapshot(); err != nil { + t.Fatalf("UpdateSnapshot failed: %v", err) + } + res, err := server.snapshot.GetSnapshot(NodeID) + if err != nil { + t.Fatalf("Failed to get snapshot: %v", err) + } + snap := res.(*cachev3.Snapshot) + if err := snap.Consistent(); err != nil { + t.Fatalf("Integrity check failed on snapshot: %v", err) + } + return snap +} + +func TestXdsServer_UpstreamConnectResources(t *testing.T) { + snap := upstreamConnectSnapshot(t) + + clusters := snap.GetResources(resourcev3.ClusterType) + for _, name := range []string{ClusterName, MainInternalName, ActorTunnelHTTP1ClusterName, ActorTunnelHTTP2ClusterName, ActorConnectName} { + if _, ok := clusters[name]; !ok { + t.Errorf("cluster %q missing", name) + } + } + if _, ok := clusters[OriginalDstClusterName]; ok { + t.Errorf("cluster %q built although no route sends to it", OriginalDstClusterName) + } + if len(clusters) != 5 { + t.Errorf("got %d clusters, want 5", len(clusters)) + } + + listeners := snap.GetResources(resourcev3.ListenerType) + l, ok := listeners[ActorConnectName] + if !ok { + t.Fatalf("listener %q missing", ActorConnectName) + } + if l.(*listenerv3.Listener).GetInternalListener() == nil { + t.Errorf("listener %q is not an internal listener", ActorConnectName) + } + + // The tunnel filter copies ext_proc's decision, and the router picks a + // tunnel by it. + for _, name := range []string{IngressHTTPListener, MainInternalName} { + hcm := &hcmv3.HttpConnectionManager{} + if err := listeners[name].(*listenerv3.Listener).GetFilterChains()[0].GetFilters()[0].GetTypedConfig().UnmarshalTo(hcm); err != nil { + t.Fatalf("listener %s: unmarshal HCM: %v", name, err) + } + filters := hcm.GetHttpFilters() + var names []string + for _, f := range filters { + names = append(names, f.GetName()) + } + n := len(filters) + if n < 3 || names[n-3] != httpExtProcFilterName || names[n-2] != "envoy.filters.http.set_filter_state" || names[n-1] != "envoy.filters.http.router" { + t.Fatalf("listener %s HTTP filters = %v, want them to end with ext_proc, set_filter_state, router", name, names) + } + config := &setfilterstatev3.Config{} + if err := filters[n-2].GetTypedConfig().UnmarshalTo(config); err != nil { + t.Fatalf("listener %s: unmarshal set_filter_state: %v", name, err) + } + if got := config.GetOnRequestHeaders()[0].GetObjectKey(); got != tunnelKeyFilterStateKey { + t.Errorf("listener %s: filter after ext_proc sets %q first, want the tunnel filter", name, got) + } + } +} + +// Envoy pools tunnels by the hash of shared hashable values. A non-hashable +// key lets one actor's tunnel serve another, and a second hashable value +// splits the pool, since Envoy hashes them in map order. +func TestActorTunnelFilterStateFilter(t *testing.T) { + config := &setfilterstatev3.Config{} + if err := actorTunnelFilterStateFilter().GetTypedConfig().UnmarshalTo(config); err != nil { + t.Fatalf("unmarshal set_filter_state config: %v", err) + } + + const ( + worker = "%DYNAMIC_METADATA(envoy.filters.listener.original_dst:local)%" + actor = "%REQ(ate-target-actor)%" + port = "%DYNAMIC_METADATA(envoy.filters.listener.original_dst:port)%" + ) + want := map[string]struct{ factory, format string }{ + tunnelKeyFilterStateKey: {"envoy.hashable_string", worker + "|" + actor + "|" + port}, + tunnelWorkerFilterStateKey: {"envoy.string", worker}, + tunnelActorFilterStateKey: {"envoy.string", actor}, + tunnelPortFilterStateKey: {"envoy.string", port}, + } + values := config.GetOnRequestHeaders() + if len(values) != len(want) { + t.Fatalf("filter sets %d values, want %d", len(values), len(want)) + } + for _, v := range values { + w, ok := want[v.GetObjectKey()] + if !ok { + t.Errorf("unexpected value %q", v.GetObjectKey()) + continue + } + if v.GetFactoryKey() != w.factory { + t.Errorf("%s factory = %q, want %q", v.GetObjectKey(), v.GetFactoryKey(), w.factory) + } + if got := v.GetFormatString().GetTextFormatSource().GetInlineString(); got != w.format { + t.Errorf("%s format = %q, want %q", v.GetObjectKey(), got, w.format) + } + if v.GetSharedWithUpstream() != setfilterstatecommonv3.FilterStateValue_ONCE { + t.Errorf("%s shared with upstream = %v, want ONCE", v.GetObjectKey(), v.GetSharedWithUpstream()) + } + } +} + +func TestXdsServer_ActorTunnelRoutes(t *testing.T) { + x := NewXdsServer(0) + x.SetUpstreamConnect(true) + routes := x.buildRoutes().GetVirtualHosts()[0].GetRoutes() + if len(routes) != 2 { + t.Fatalf("got %d routes, want 2", len(routes)) + } + grpc, rest := routes[0], routes[1] + if got := grpc.GetRoute().GetCluster(); got != ActorTunnelHTTP2ClusterName { + t.Errorf("gRPC route sends to %q, want %q", got, ActorTunnelHTTP2ClusterName) + } + if got := rest.GetRoute().GetCluster(); got != ActorTunnelHTTP1ClusterName { + t.Errorf("fallback route sends to %q, want %q", got, ActorTunnelHTTP1ClusterName) + } + if got := rest.GetMatch().GetHeaders(); len(got) != 0 { + t.Errorf("fallback route matches headers %v, want none", got) + } + + // Matches atunnel's isGRPC, minus the protocol check a route can't do. + var method, contentType string + for _, h := range grpc.GetMatch().GetHeaders() { + switch h.GetName() { + case ":method": + method = h.GetStringMatch().GetExact() + case "content-type": + contentType = h.GetStringMatch().GetSafeRegex().GetRegex() + } + } + if method != "POST" { + t.Errorf("gRPC route matches method %q, want POST", method) + } + re, err := regexp.Compile(contentType) + if err != nil { + t.Fatalf("gRPC route content-type regex %q: %v", contentType, err) + } + for ct, isGRPC := range map[string]bool{ + "application/grpc": true, + "application/grpc+proto": true, + "application/grpc;charset=utf-8": true, + "Application/GRPC": true, + "application/grpc-web": false, + "application/json": false, + } { + if re.MatchString(ct) != isGRPC { + t.Errorf("content-type %q matches the gRPC route = %v, want %v", ct, !isGRPC, isGRPC) + } + } + + for _, r := range routes { + if got := r.GetRequestHeadersToRemove(); len(got) != 1 || got[0] != atenet.TargetActorHeader { + t.Errorf("route to %s removes %v, want [%s]: nothing else strips it before the actor", r.GetRoute().GetCluster(), got, atenet.TargetActorHeader) + } + if got := r.GetRequestHeadersToAdd(); len(got) != 0 { + t.Errorf("route to %s adds headers %v, want none: the CONNECT authority carries the port", r.GetRoute().GetCluster(), got) + } + if got := r.GetRoute().GetTimeout().AsDuration(); got != defaultRouteTimeout { + t.Errorf("route to %s timeout = %v, want %v", r.GetRoute().GetCluster(), got, defaultRouteTimeout) + } + if got := r.GetRoute().GetIdleTimeout().AsDuration(); got != x.routeIdleTimeout() { + t.Errorf("route to %s idle timeout = %v, want %v", r.GetRoute().GetCluster(), got, x.routeIdleTimeout()) + } + } +} + +func TestBuildActorConnectListener(t *testing.T) { + filters := buildActorConnectListener().GetFilterChains()[0].GetFilters() + if len(filters) != 2 { + t.Fatalf("got %d network filters, want set_filter_state and tcp_proxy", len(filters)) + } + + setWorker := &setfilterstatenetworkv3.Config{} + if err := filters[0].GetTypedConfig().UnmarshalTo(setWorker); err != nil { + t.Fatalf("unmarshal set_filter_state: %v", err) + } + values := setWorker.GetOnNewConnection() + if len(values) != 1 { + t.Fatalf("set_filter_state sets %d values, want 1", len(values)) + } + if got := values[0].GetObjectKey(); got != extproc.OriginalDstFilterStateKey { + t.Errorf("set_filter_state sets %q, want %q for the ORIGINAL_DST cluster", got, extproc.OriginalDstFilterStateKey) + } + if got, want := values[0].GetFormatString().GetTextFormatSource().GetInlineString(), "%FILTER_STATE(dev.ate.tunnel.worker:PLAIN)%"; got != want { + t.Errorf("worker address format = %q, want %q", got, want) + } + + proxy := &tcpproxyv3.TcpProxy{} + if err := filters[1].GetTypedConfig().UnmarshalTo(proxy); err != nil { + t.Fatalf("unmarshal tcp_proxy: %v", err) + } + if got := proxy.GetCluster(); got != ActorConnectName { + t.Errorf("tcp_proxy cluster = %q, want %q", got, ActorConnectName) + } + tunnel := proxy.GetTunnelingConfig() + if got, want := tunnel.GetHostname(), "actor:%FILTER_STATE(dev.ate.tunnel.port:PLAIN)%"; got != want { + t.Errorf("CONNECT authority = %q, want %q", got, want) + } + headers := tunnel.GetHeadersToAdd() + if len(headers) != 1 || headers[0].GetHeader().GetKey() != atenet.TargetActorHeader || + headers[0].GetHeader().GetValue() != "%FILTER_STATE(dev.ate.tunnel.actor:PLAIN)%" { + t.Errorf("CONNECT headers = %v, want only %s from dev.ate.tunnel.actor", headers, atenet.TargetActorHeader) + } + if idle := proxy.GetIdleTimeout(); idle == nil || idle.AsDuration() != 0 { + t.Errorf("tcp_proxy idle timeout = %v, want 0 (off) so the HTTP side's timeouts fire first", idle) + } +} + +func TestXdsServer_ActorTunnelClusters(t *testing.T) { + protocolOptions := func(t *testing.T, c *clusterv3.Cluster) *httpv3.HttpProtocolOptions { + t.Helper() + opts := &httpv3.HttpProtocolOptions{} + if err := c.GetTypedExtensionProtocolOptions()[httpProtocolOptionsName].UnmarshalTo(opts); err != nil { + t.Fatalf("cluster %s: unmarshal protocol options: %v", c.GetName(), err) + } + return opts + } + checkCircuitBreakers := func(t *testing.T, c *clusterv3.Cluster) { + t.Helper() + limits := c.GetCircuitBreakers().GetThresholds()[0] + for name, got := range map[string]uint32{ + "max_connections": limits.GetMaxConnections().GetValue(), + "max_pending_requests": limits.GetMaxPendingRequests().GetValue(), + "max_requests": limits.GetMaxRequests().GetValue(), + } { + if got != tunnelCircuitBreakerLimit { + t.Errorf("cluster %s %s = %d, want %d", c.GetName(), name, got, tunnelCircuitBreakerLimit) + } + } + } + + for _, http2 := range []bool{false, true} { + c := buildActorTunnelCluster("tunnel", http2) + target := c.GetLoadAssignment().GetEndpoints()[0].GetLbEndpoints()[0].GetEndpoint().GetAddress().GetEnvoyInternalAddress() + if got := target.GetServerListenerName(); got != ActorConnectName { + t.Errorf("http2=%v: tunnel cluster sends to %q, want internal listener %q", http2, got, ActorConnectName) + } + opts := protocolOptions(t, c) + explicit := opts.GetExplicitHttpConfig() + if http2 && explicit.GetHttp2ProtocolOptions() == nil || !http2 && explicit.GetHttpProtocolOptions() == nil { + t.Errorf("http2=%v: tunnel cluster protocol = %v", http2, explicit) + } + if got := opts.GetCommonHttpProtocolOptions().GetIdleTimeout().AsDuration(); got != tunnelIdleTimeout { + t.Errorf("http2=%v: tunnel idle timeout = %v, want %v", http2, got, tunnelIdleTimeout) + } + checkCircuitBreakers(t, c) + } + + x := NewXdsServer(0) + x.SetUpstreamTls("/run/bundle.pem", "/run/trust.pem", "spiffe://cluster.local/") + c := x.buildActorConnectCluster() + if c.GetType() != clusterv3.Cluster_ORIGINAL_DST { + t.Errorf("actor_connect type = %v, want ORIGINAL_DST", c.GetType()) + } + lb := c.GetOriginalDstLbConfig() + if lb.GetMetadataKey() != nil { + t.Error("actor_connect reads a metadata key; the worker address comes from filter state") + } + if got := lb.GetUpstreamPortOverride().GetValue(); got != atunnel.DefaultConnectPort { + t.Errorf("actor_connect port = %d, want atunnel's CONNECT port %d", got, atunnel.DefaultConnectPort) + } + upstreamTLS := &tlsv3.UpstreamTlsContext{} + if err := c.GetTransportSocket().GetTypedConfig().UnmarshalTo(upstreamTLS); err != nil { + t.Fatalf("actor_connect: unmarshal UpstreamTlsContext: %v", err) + } + if len(upstreamTLS.GetCommonTlsContext().GetTlsCertificateSdsSecretConfigs()) == 0 { + t.Error("actor_connect presents no client certificate; atunnel requires mTLS") + } + if got := protocolOptions(t, c).GetExplicitHttpConfig().GetHttp2ProtocolOptions(); got == nil { + t.Error("actor_connect is not HTTP/2; tunnels to one worker would each need a connection") + } + checkCircuitBreakers(t, c) +} diff --git a/cmd/ateom-gvisor/flags.go b/cmd/ateom-gvisor/flags.go new file mode 100644 index 0000000000..58073f577e --- /dev/null +++ b/cmd/ateom-gvisor/flags.go @@ -0,0 +1,212 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "os" + "path/filepath" + "sync" + + "github.com/fsnotify/fsnotify" +) + +// commandFlags holds per-command flags for a single command. +type commandFlags struct { + // GlobalFlags are passed before the command name (e.g. + // "--cpu-num-from-quota", "-allow-connected-on-save"). + GlobalFlags []string `json:"globalFlags,omitempty"` + // SubcommandFlags are passed after the command name (e.g. + // "-background", "-detach", "-force", "-quiet"). + SubcommandFlags []string `json:"subcommandFlags,omitempty"` +} + +// flagConfig defines global and per-command flags passed to runsc. +type flagConfig struct { + // GlobalFlags are global flags passed to every command before the command name. + GlobalFlags []string `json:"globalFlags"` + // Commands maps a command name (e.g. "create", "restore") to its + // per-command flags. + Commands map[string]commandFlags `json:"commands"` +} + +// globalFlags returns the combined global flags for command: top-level +// GlobalFlags followed by the command's own GlobalFlags. +func (c *flagConfig) globalFlags(command string) []string { + cmdFlags := c.Commands[command] + out := make([]string, 0, len(c.GlobalFlags)+len(cmdFlags.GlobalFlags)) + out = append(out, c.GlobalFlags...) + out = append(out, cmdFlags.GlobalFlags...) + return out +} + +// subcommandFlags returns the subcommand flags for command. +func (c *flagConfig) subcommandFlags(command string) []string { + return c.Commands[command].SubcommandFlags +} + +// parseFlagConfig parses and validates a JSON flags configuration. +func parseFlagConfig(data []byte) (*flagConfig, error) { + dec := json.NewDecoder(bytes.NewReader(data)) + dec.DisallowUnknownFields() + + var cfg flagConfig + if err := dec.Decode(&cfg); err != nil { + return nil, fmt.Errorf("invalid flags JSON: %w", err) + } + if dec.More() { + return nil, errors.New("invalid flags JSON: unexpected trailing data") + } + var extra any + if err := dec.Decode(&extra); !errors.Is(err, io.EOF) { + return nil, errors.New("invalid flags JSON: unexpected trailing tokens") + } + return &cfg, nil +} + +// loadFlagsFile reads and parses a JSON flags configuration file. +func loadFlagsFile(path string) (*flagConfig, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("while reading flags config %q: %w", path, err) + } + cfg, err := parseFlagConfig(data) + if err != nil { + return nil, fmt.Errorf("while parsing flags config %q: %w", path, err) + } + return cfg, nil +} + +// flagStore holds the active flagConfig in memory behind a lock and watches +// its backing file to reload changes asynchronously without file I/O on the hot +// path. +type flagStore struct { + path string + + mu sync.RWMutex + cfg *flagConfig + + // onReload is an optional hook called after each reload attempt (for tests). + onReload func() +} + +// newFlagStore creates an empty store if path is empty, or reads path once to +// populate the initial configuration and starts watching it for changes until +// ctx is cancelled. +func newFlagStore(ctx context.Context, path string) (*flagStore, error) { + if path == "" { + return &flagStore{ + cfg: &flagConfig{}, + }, nil + } + + cleanPath := filepath.Clean(path) + cfg, err := loadFlagsFile(cleanPath) + if err != nil { + return nil, err + } + + watcher, err := fsnotify.NewWatcher() + if err != nil { + return nil, fmt.Errorf("while creating fsnotify watcher for %q: %w", cleanPath, err) + } + + // Watch the parent directory so both in-place writes and atomic file + // replacements (renames) are detected. + dir := filepath.Dir(cleanPath) + if err := watcher.Add(dir); err != nil { + _ = watcher.Close() + return nil, fmt.Errorf("while watching directory %q for %q: %w", dir, cleanPath, err) + } + + store := &flagStore{ + path: cleanPath, + cfg: cfg, + } + go store.watchLoop(ctx, watcher) + return store, nil +} + +// get returns the current immutable flagConfig under a read lock. +func (s *flagStore) get() *flagConfig { + s.mu.RLock() + cfg := s.cfg + s.mu.RUnlock() + return cfg +} + +func (s *flagStore) watchLoop(ctx context.Context, watcher *fsnotify.Watcher) { + defer func() { + if err := watcher.Close(); err != nil { + slog.Warn("Failed to close flags config watcher", slog.String("path", s.path), slog.Any("err", err)) + } + }() + for { + select { + case <-ctx.Done(): + return + case event, ok := <-watcher.Events: + if !ok { + return + } + if s.isConfigEvent(event) { + s.reload() + } + case err, ok := <-watcher.Errors: + if !ok { + return + } + slog.Error("Error watching flags configuration file", + slog.String("path", s.path), + slog.Any("err", err)) + } + } +} + +func (s *flagStore) isConfigEvent(event fsnotify.Event) bool { + if filepath.Clean(event.Name) != s.path { + return false + } + return event.Has(fsnotify.Write) || event.Has(fsnotify.Create) +} + +func (s *flagStore) reload() { + defer func() { + if s.onReload != nil { + s.onReload() + } + }() + + cfg, err := loadFlagsFile(s.path) + if err != nil { + slog.Error("Failed to reload flags configuration; ignoring file changes", + slog.String("path", s.path), + slog.Any("err", err)) + return + } + + s.mu.Lock() + s.cfg = cfg + s.mu.Unlock() + slog.Info("Reloaded flags configuration", slog.String("path", s.path)) +} diff --git a/cmd/ateom-gvisor/flags_test.go b/cmd/ateom-gvisor/flags_test.go new file mode 100644 index 0000000000..0d291d90eb --- /dev/null +++ b/cmd/ateom-gvisor/flags_test.go @@ -0,0 +1,217 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "os" + "path/filepath" + "reflect" + "testing" + "time" +) + +func TestFlagsBuildArgs(t *testing.T) { + r := &runsc{ + path: "/usr/bin/runsc", + actorUID: "test-actor-123", + actorDirs: testActorDirs, + flags: mustFlagStore(t), + } + + got := r.buildArgs("start", "app") + want := []string{ + "-global1", + "-global2", + "-start-global", + "-root", "/node/actors/test-actor-123/runsc-state", + "start", + "app", + } + if !reflect.DeepEqual(got, want) { + t.Errorf("buildArgs(start) = %v, want %v", got, want) + } + + got = r.buildArgs("restore", "app", "-bundle", "/bundle") + want = []string{ + "-global1", + "-global2", + "-restore-global", + "-root", "/node/actors/test-actor-123/runsc-state", + "restore", + "-restore-sub1", + "-restore-sub2", + "-bundle", "/bundle", + "app", + } + if !reflect.DeepEqual(got, want) { + t.Errorf("buildArgs(restore) = %v, want %v", got, want) + } + + got = r.buildArgs("list", "") + want = []string{ + "-global1", + "-global2", + "-root", "/node/actors/test-actor-123/runsc-state", + "list", + "-list-sub", + } + if !reflect.DeepEqual(got, want) { + t.Errorf("buildArgs(list) = %v, want %v", got, want) + } +} + +func TestFlagStoreEmptyPath(t *testing.T) { + store, err := newFlagStore(t.Context(), "") + if err != nil { + t.Fatalf("newFlagStore(ctx, \"\") unexpected error: %v", err) + } + if store == nil { + t.Fatal("newFlagStore(ctx, \"\") returned nil store") + } + + r := &runsc{ + path: "/usr/bin/runsc", + actorUID: "test-actor-123", + actorDirs: testActorDirs, + flags: store, + } + + got := r.buildArgs("pause", "contId") + want := []string{ + "-root", "/node/actors/test-actor-123/runsc-state", + "pause", + "contId", + } + if !reflect.DeepEqual(got, want) { + t.Errorf("buildArgs(pause) with empty store = %v, want %v", got, want) + } +} + +func waitForReload(t *testing.T, reloadCh <-chan struct{}) { + t.Helper() + select { + case <-reloadCh: + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for config reload") + } +} + +func TestFlagStoreWatchAndReload(t *testing.T) { + dir := t.TempDir() + cfgPath := filepath.Join(dir, "flags.json") + + initialJSON := `{ + "globalFlags": ["-global1"], + "commands": { + "create": { + "globalFlags": ["-create-global1"] + } + } +}` + if err := os.WriteFile(cfgPath, []byte(initialJSON), 0o600); err != nil { + t.Fatalf("failed to write initial config: %v", err) + } + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + store, err := newFlagStore(ctx, cfgPath) + if err != nil { + t.Fatalf("newFlagStore() unexpected error: %v", err) + } + + r := &runsc{ + path: "/usr/bin/runsc", + actorUID: "test-actor-123", + actorDirs: testActorDirs, + flags: store, + } + + // Verify initial config loaded from JSON file. + gotInitial := r.buildArgs("create", "app") + wantInitial := []string{ + "-global1", + "-create-global1", + "-root", "/node/actors/test-actor-123/runsc-state", + "create", + "app", + } + if !reflect.DeepEqual(gotInitial, wantInitial) { + t.Fatalf("initial buildArgs(create) = %v, want %v", gotInitial, wantInitial) + } + + // Set up reload notification channel for deterministic synchronization. + reloadCh := make(chan struct{}, 16) + store.onReload = func() { + select { + case reloadCh <- struct{}{}: + default: + } + } + + // Update with valid new flags via atomic rename. + updatedJSON := `{ + "globalFlags": ["-global1", "-global2"], + "commands": { + "create": { + "globalFlags": ["-create-global1", "-create-global2"] + }, + "delete": { + "subcommandFlags": ["-delete-sub"] + } + } +}` + tmpPath := filepath.Join(dir, ".flags.json.tmp") + if err := os.WriteFile(tmpPath, []byte(updatedJSON), 0o600); err != nil { + t.Fatalf("failed to write temp config: %v", err) + } + if err := os.Rename(tmpPath, cfgPath); err != nil { + t.Fatalf("failed to rename temp config: %v", err) + } + waitForReload(t, reloadCh) + + gotUpdated := r.buildArgs("create", "app") + wantUpdated := []string{ + "-global1", + "-global2", + "-create-global1", + "-create-global2", + "-root", "/node/actors/test-actor-123/runsc-state", + "create", + "app", + } + if !reflect.DeepEqual(gotUpdated, wantUpdated) { + t.Fatalf("updated buildArgs(create) = %v, want %v", gotUpdated, wantUpdated) + } + + // Write invalid JSON via atomic rename: watcher must ignore invalid update + // and preserve the existing valid configuration struct. + badTmp := filepath.Join(dir, ".bad.tmp") + if err := os.WriteFile(badTmp, []byte(`{"globalFlags": ["broken"`), 0o600); err != nil { + t.Fatalf("failed to write bad temp config: %v", err) + } + if err := os.Rename(badTmp, cfgPath); err != nil { + t.Fatalf("failed to rename bad temp config: %v", err) + } + waitForReload(t, reloadCh) + + gotAfterBad := r.buildArgs("create", "app") + if !reflect.DeepEqual(gotAfterBad, wantUpdated) { + t.Fatalf("after invalid reload buildArgs(create) = %v, want previous valid config %v", gotAfterBad, wantUpdated) + } +} diff --git a/cmd/ateom-gvisor/kodata/internal_flags.json b/cmd/ateom-gvisor/kodata/internal_flags.json new file mode 100644 index 0000000000..89c3244df2 --- /dev/null +++ b/cmd/ateom-gvisor/kodata/internal_flags.json @@ -0,0 +1,38 @@ +{ + "globalFlags": [ + "-log-format", + "json", + "--alsologtostderr" + ], + "commands": { + "create": { + "globalFlags": [ + "--cpu-num-from-quota" + ] + }, + "start": { + "globalFlags": [ + "-allow-connected-on-save" + ] + }, + "restore": { + "globalFlags": [ + "--cpu-num-from-quota" + ], + "subcommandFlags": [ + "-background", + "-detach" + ] + }, + "delete": { + "subcommandFlags": [ + "-force" + ] + }, + "list": { + "subcommandFlags": [ + "-quiet" + ] + } + } +} diff --git a/cmd/ateom-gvisor/main.go b/cmd/ateom-gvisor/main.go index c7215fbeb3..8feeb9b00b 100644 --- a/cmd/ateom-gvisor/main.go +++ b/cmd/ateom-gvisor/main.go @@ -63,12 +63,20 @@ import ( "k8s.io/apimachinery/pkg/util/validation/field" ) +func flagsConfigPath() string { + path := "internal_flags.json" + if koDataPath := os.Getenv("KO_DATA_PATH"); koDataPath != "" { + path = filepath.Join(koDataPath, path) + } + return path +} + var ( podUID = pflag.String("pod-uid", "", "The UID of the current pod") tunnelConfig = ateomtunnel.RegisterFlags(pflag.CommandLine) - readinessListenAddress = pflag.String("readiness-listen-address", "0.0.0.0:8080", "Address for HTTP readiness checks") + readinessListenAddress = pflag.String("readiness-listen-address", "0.0.0.0:8080", "Address for the HTTP /readyz and /healthz probes") maxActors = pflag.Int("max-actors", 1000, "How many actors this worker will host at once") usageSampleInterval = pflag.Duration("usage-sample-interval", time.Minute, "How often the ateom samples the resource usage of its actors. Each sample writes an ate.actor.usage_sampled record per actor, and GetActiveWorkloadStats serves the latest.") @@ -183,15 +191,11 @@ func do(ctx context.Context) error { if err := resources.ValidateWorkerPodUID(*podUID); err != nil { return fmt.Errorf("in resources.ValidateWorkerPodUID: %w", err) } + // atelet owns the directory and the sandbox versions file in it; only the + // socket below is this process's. Both outlive a container restart. if err := os.MkdirAll(ateomDir, 0o700); err != nil { return fmt.Errorf("in os.MkdirAll(%q): %w", ateomDir, err) } - // Clean up the ateom directory during graceful shutdown (#1677). - defer func() { - if err := os.RemoveAll(ateomDir); err != nil { - slog.ErrorContext(ctx, "Failed to remove the ateom directory on shutdown", slog.Any("err", err)) - } - }() // Prepare the pod cgroup so runsc can create per-actor-container leaves under // it with real accounting. @@ -212,13 +216,23 @@ func do(ctx context.Context) error { if err != nil { return fmt.Errorf("while opening unix socket: %w", err) } + defer func() { + if err := os.Remove(sockPath); err != nil && !errors.Is(err, os.ErrNotExist) { + slog.ErrorContext(ctx, "Failed to remove the ateom socket on shutdown", slog.Any("err", err)) + } + }() + + flags, err := newFlagStore(ctx, flagsConfigPath()) + if err != nil { + return err + } actorLogger := actorlog.NewActorLogger(syncedWriter, metadata.OnGCE()) tunnel, err := ateomtunnel.Start(ctx, *tunnelConfig, ateomnet.ActorHTTPUpstream) if err != nil { return err } - ateomService := NewService(tunnel, actorLogger, *maxActors) + ateomService := NewService(tunnel, actorLogger, *maxActors, flags) // The controller sets both from the downward API. pool := ateomstats.Pool{Namespace: os.Getenv("POD_NAMESPACE"), Name: os.Getenv("WORKER_POOL_NAME")} if pool.Namespace == "" || pool.Name == "" { @@ -272,7 +286,7 @@ func do(ctx context.Context) error { } }() - go serverboot.StartReadinessServer(ctx, *readinessListenAddress, readiness) + go serverboot.StartHealthServer(ctx, *readinessListenAddress, readiness) if err := svr.Serve(lis); err != nil { return fmt.Errorf("while serving: %w", err) @@ -331,12 +345,15 @@ type AteomService struct { // stats handlers' lock-free read, the way containerStatsReader does for the // micro-VM runtime. nil means the real read. readSandboxCgroup func(dir string) (cgroupstats.Sample, error) + + // flags holds the watched JSON flag configuration. + flags *flagStore } var _ ateompb.AteomServer = (*AteomService)(nil) // NewService creates a new AteomService. -func NewService(tunnel *ateomtunnel.Tunnel, actorLogger *actorlog.ActorLogger, maxActors int) *AteomService { +func NewService(tunnel *ateomtunnel.Tunnel, actorLogger *actorlog.ActorLogger, maxActors int, flags *flagStore) *AteomService { return &AteomService{ locks: actorlock.New(), inFlight: actorlock.NewInFlight(), @@ -345,6 +362,7 @@ func NewService(tunnel *ateomtunnel.Tunnel, actorLogger *actorlog.ActorLogger, m tunnel: tunnel, actorLogger: actorLogger, cgroupRoot: defaultCgroupRoot, + flags: flags, } } @@ -612,6 +630,7 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload size: sizing.FromLimits(req.GetCpuMilli(), req.GetMemoryBytes()), durableVolumes: durableVolumeNames(req.GetSpec()), containers: req.GetSpec().GetContainers(), + flags: s.flags, } var containersToDelete []string defer func() { @@ -744,6 +763,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec path: req.GetRunscPath(), actorUID: req.GetActorUid(), actorDirs: req.GetActorDirs(), + flags: s.flags, } checkpointPath := req.GetActorDirs().GetCheckpointDir() @@ -961,6 +981,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore size: sizing.FromLimits(req.GetCpuMilli(), req.GetMemoryBytes()), durableVolumes: durableVolumeNames(req.GetSpec()), containers: req.GetSpec().GetContainers(), + flags: s.flags, } var containersToDelete []string defer func() { @@ -1124,6 +1145,7 @@ func (s *AteomService) terminateWorkload(ctx context.Context, actorRef resources path: runscPath, actorUID: actorUID, actorDirs: actorDirs, + flags: s.flags, } // Detached from the caller: a deadline mid-`runsc delete` would leave the diff --git a/cmd/ateom-gvisor/runsc.go b/cmd/ateom-gvisor/runsc.go index 8caf5157f5..e50563bd28 100644 --- a/cmd/ateom-gvisor/runsc.go +++ b/cmd/ateom-gvisor/runsc.go @@ -47,6 +47,30 @@ type runsc struct { durableVolumes []string // containers are the workload's containers, from the request. containers []*ateompb.Container + // flags holds the configured flags; must not be nil. + flags *flagStore +} + +// buildArgs constructs the runsc argument vector for command, appending +// containerName at the end if non-empty: +// +// -root [] +func (r *runsc) buildArgs(command, containerName string, extraArgs ...string) []string { + cfg := r.flags.get() + globalFlags := cfg.globalFlags(command) + subcommandFlags := cfg.subcommandFlags(command) + + args := make([]string, 0, + len(globalFlags)+2+1+len(subcommandFlags)+len(extraArgs)+1) + args = append(args, globalFlags...) + args = append(args, "-root", runscStateDir(r.actorDirs)) + args = append(args, command) + args = append(args, subcommandFlags...) + args = append(args, extraArgs...) + if containerName != "" { + args = append(args, containerName) + } + return args } // durableVolumeNames returns the sorted, deduplicated durable-dir volume names @@ -101,32 +125,17 @@ func (r *runsc) cmdCreate(ctx context.Context, out io.Writer, containerName stri return fmt.Errorf("while writing the OCI spec for %q: %w", containerName, err) } - args := []string{ - "-log-format", "json", - "--alsologtostderr", - // "-debug", - // "-debug-log", filepath.Join(r.actorDirs.GetRootDir(), "runsc-debug-logs", containerName) + "/", - // "-debug-to-user-log", - // "-log-packets", - // "-strace", - "-root", runscStateDir(r.actorDirs), - // Provision the sentry's vCPU count from the cgroup CPU quota written by - // sizing.ApplyToOCISpec, so the sandbox is sized to the pod's limit (runsc - // otherwise sizes to all host CPUs). Global flag: before the subcommand. - "--cpu-num-from-quota", - } + args := make([]string, 0, 4+len(additionalArgs)) args = append(args, - "create", "-bundle", ociBundlePath(r.actorDirs, containerName), "-pid-file", pidFilePath(r.actorDirs, containerName), ) - args = append(args, additionalArgs...) - args = append(args, containerName) // Name of the container + cmd := exec.CommandContext( ctx, r.path, - args..., + r.buildArgs("create", containerName, args...)..., ) cmd.Stdout = out cmd.Stderr = out @@ -142,19 +151,7 @@ func (r *runsc) cmdCreate(ctx context.Context, out io.Writer, containerName stri func (r *runsc) cmdStart(ctx context.Context, out io.Writer, containerName string) error { slog.InfoContext(ctx, "About to run runsc start", slog.String("container", containerName)) - startArgs := []string{ - "-log-format", "json", - "--alsologtostderr", - // "-debug", - // "-debug-log", filepath.Join(r.actorDirs.GetRootDir(), "runsc-debug-logs", containerName) + "/", - // "-debug-to-user-log", - // "-log-packets", - // "-strace", - "-allow-connected-on-save", - "-root", runscStateDir(r.actorDirs), - } - startArgs = append(startArgs, "start", containerName) - cmd := exec.CommandContext(ctx, r.path, startArgs...) + cmd := exec.CommandContext(ctx, r.path, r.buildArgs("start", containerName)...) cmd.Stdout = out cmd.Stderr = out @@ -169,23 +166,11 @@ func (r *runsc) cmdStart(ctx context.Context, out io.Writer, containerName strin // checkpointArgs builds the argv for `runsc checkpoint `. Factored // out so the argument construction can be unit-tested without executing runsc. func (r *runsc) checkpointArgs(containerName, checkpointPath string, fsCheckpointPaths []string) []string { - args := []string{ - "-log-format", "json", - "--alsologtostderr", - // "-debug", - // "-debug-log", filepath.Join(r.actorDirs.GetRootDir(), "runsc-debug-logs", containerName) + "/", - // "-debug-to-user-log", - // "-log-packets", - // "-strace", - "-root", runscStateDir(r.actorDirs), - "checkpoint", - "-image-path", checkpointPath, - } + extra := []string{"-image-path", checkpointPath} if len(fsCheckpointPaths) > 0 { - args = append(args, "-fs-checkpoint-paths", strings.Join(fsCheckpointPaths, ",")) + extra = append(extra, "-fs-checkpoint-paths", strings.Join(fsCheckpointPaths, ",")) } - args = append(args, containerName) - return args + return r.buildArgs("checkpoint", containerName, extra...) } func (r *runsc) cmdCheckpoint(ctx context.Context, containerName, checkpointPath string, fsCheckpointPaths []string) error { @@ -205,25 +190,12 @@ func (r *runsc) cmdCheckpoint(ctx context.Context, containerName, checkpointPath // Factored out so the argument construction can be unit-tested without // executing runsc. func (r *runsc) fsCheckpointArgs(containerName, checkpointPath string, fsCheckpointPaths []string) []string { - args := []string{ - "-log-format", "json", - "--alsologtostderr", - // "-debug", - // "-debug-log", filepath.Join(r.actorDirs.GetRootDir(), "runsc-debug-logs", containerName) + "/", - // "-debug-to-user-log", - // "-log-packets", - // "-strace", - "-root", runscStateDir(r.actorDirs), - "fscheckpoint", - "-image-path", filepath.Join(checkpointPath, resources.FSCheckpointSubdir), - } + extra := make([]string, 0, 2+2*len(fsCheckpointPaths)) + extra = append(extra, "-image-path", filepath.Join(checkpointPath, resources.FSCheckpointSubdir)) for _, ddv := range fsCheckpointPaths { - args = append(args, "-path", ddv) + extra = append(extra, "-path", ddv) } - - // name of the container must be the last parameter. - args = append(args, containerName) - return args + return r.buildArgs("fscheckpoint", containerName, extra...) } func (r *runsc) cmdFsCheckpoint(ctx context.Context, containerName, checkpointPath string, fsCheckpointPaths []string) error { @@ -242,25 +214,11 @@ func (r *runsc) cmdFsCheckpoint(ctx context.Context, containerName, checkpointPa // restoreArgs builds the argv for `runsc restore `. Factored out so // the argument construction can be unit-tested without executing runsc. func (r *runsc) restoreArgs(containerName, checkpointPath string) []string { - return []string{ - "-log-format", "json", - "--alsologtostderr", - // "-debug", - // "-debug-log", filepath.Join(r.actorDirs.GetRootDir(), "runsc-debug-logs", containerName) + "/", - // "-debug-to-user-log", - // "-log-packets", - // "-strace", - "-root", runscStateDir(r.actorDirs), - // Match cmdCreate: size the restored sentry from the cgroup CPU quota. - "--cpu-num-from-quota", - "restore", + return r.buildArgs("restore", containerName, "-bundle", ociBundlePath(r.actorDirs, containerName), "-image-path", checkpointPath, "-pid-file", pidFilePath(r.actorDirs, containerName), - "-background", - "-detach", - containerName, - } + ) } // We take a checkpoint only of the root container of the sandbox, but we need @@ -285,13 +243,7 @@ func (r *runsc) cmdDelete(ctx context.Context, containerName string) error { cmd := exec.CommandContext( ctx, r.path, - "-log-format", "json", - "--alsologtostderr", - // "-debug", - "-root", runscStateDir(r.actorDirs), - "delete", - "-force", - containerName, + r.buildArgs("delete", containerName)..., ) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr @@ -306,11 +258,7 @@ func (r *runsc) cmdState(ctx context.Context, containerName string) error { cmd := exec.CommandContext( ctx, r.path, - "-log-format", "json", - "--alsologtostderr", - "-root", runscStateDir(r.actorDirs), - "state", - containerName, + r.buildArgs("state", containerName)..., ) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr @@ -325,11 +273,7 @@ func (r *runsc) cmdList(ctx context.Context) ([]string, error) { cmd := exec.CommandContext( ctx, r.path, - "-log-format", "json", - "--alsologtostderr", - "-root", runscStateDir(r.actorDirs), - "list", - "-quiet", + r.buildArgs("list", "")..., ) var out bytes.Buffer cmd.Stdout = &out @@ -340,25 +284,12 @@ func (r *runsc) cmdList(ctx context.Context) ([]string, error) { return strings.Fields(out.String()), nil } -// killArgs builds the argv for `runsc kill `. Factored out -// so the argument construction can be unit-tested without executing runsc. -func (r *runsc) killArgs(containerName, signal string) []string { - return []string{ - "-log-format", "json", - "--alsologtostderr", - "-root", runscStateDir(r.actorDirs), - "kill", - containerName, - signal, - } -} - // cmdKill sends signal to the given container's process(es) inside the gVisor // sandbox. Used during graceful shutdown to propagate SIGTERM to the actor. func (r *runsc) cmdKill(ctx context.Context, containerName, signal string) error { slog.InfoContext(ctx, "About to run runsc kill", slog.String("container", containerName), slog.String("signal", signal)) - cmd := exec.CommandContext(ctx, r.path, r.killArgs(containerName, signal)...) + cmd := exec.CommandContext(ctx, r.path, r.buildArgs("kill", "", containerName, signal)...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := reaper.RunCommand(cmd); err != nil { @@ -367,18 +298,6 @@ func (r *runsc) cmdKill(ctx context.Context, containerName, signal string) error return nil } -// waitArgs builds the argv for `runsc wait `. Factored out so the -// argument construction can be unit-tested without executing runsc. -func (r *runsc) waitArgs(containerName string) []string { - return []string{ - "-log-format", "json", - "--alsologtostderr", - "-root", runscStateDir(r.actorDirs), - "wait", - containerName, - } -} - // cmdWait blocks until the given container's process exits. Used during // graceful shutdown to confirm the actor has stopped before ateom exits. // @@ -388,7 +307,7 @@ func (r *runsc) waitArgs(containerName string) []string { func (r *runsc) cmdWait(ctx context.Context, containerName string) error { slog.InfoContext(ctx, "About to run runsc wait", slog.String("container", containerName)) - cmd := exec.CommandContext(ctx, r.path, r.waitArgs(containerName)...) + cmd := exec.CommandContext(ctx, r.path, r.buildArgs("wait", containerName)...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := cmd.Run(); err != nil { diff --git a/cmd/ateom-gvisor/runsc_test.go b/cmd/ateom-gvisor/runsc_test.go index 1588366cfa..3695aa8a80 100644 --- a/cmd/ateom-gvisor/runsc_test.go +++ b/cmd/ateom-gvisor/runsc_test.go @@ -83,46 +83,97 @@ func TestWriteSpec(t *testing.T) { } } -func TestKillArgs(t *testing.T) { - r := &runsc{ - path: "/usr/bin/runsc", - actorUID: "test-actor-123", - actorDirs: testActorDirs, +func mustFlagStore(t *testing.T) *flagStore { + t.Helper() + cfgPath := filepath.Join(t.TempDir(), "flags.json") + const testFlagsJSON = `{ + "globalFlags": [ + "-global1", + "-global2" + ], + "commands": { + "restore": { + "globalFlags": ["-restore-global"], + "subcommandFlags": ["-restore-sub1", "-restore-sub2"] + }, + "start": { + "globalFlags": ["-start-global"] + }, + "list": { + "subcommandFlags": ["-list-sub"] + } + } +}` + if err := os.WriteFile(cfgPath, []byte(testFlagsJSON), 0o600); err != nil { + t.Fatalf("failed to write test flags.json: %v", err) } - - got := r.killArgs("my-container", "SIGTERM") - want := []string{ - "-log-format", "json", - "--alsologtostderr", - "-root", "/node/actors/test-actor-123/runsc-state", - "kill", - "my-container", - "SIGTERM", - } - - if !reflect.DeepEqual(got, want) { - t.Errorf("killArgs() = %v, want %v", got, want) + store, err := newFlagStore(t.Context(), cfgPath) + if err != nil { + t.Fatalf("newFlagStore(%s) failed: %v", cfgPath, err) } + return store } -func TestWaitArgs(t *testing.T) { +func TestBuildArgs(t *testing.T) { r := &runsc{ path: "/usr/bin/runsc", actorUID: "test-actor-123", actorDirs: testActorDirs, + flags: mustFlagStore(t), } - got := r.waitArgs("my-container") - want := []string{ - "-log-format", "json", - "--alsologtostderr", - "-root", "/node/actors/test-actor-123/runsc-state", - "wait", - "my-container", - } - - if !reflect.DeepEqual(got, want) { - t.Errorf("waitArgs() = %v, want %v", got, want) + for _, tc := range []struct { + name string + command string + containerName string + extraArgs []string + want []string + }{ + { + name: "kill", + command: "kill", + containerName: "", + extraArgs: []string{"my-container", "SIGTERM"}, + want: []string{ + "-global1", + "-global2", + "-root", "/node/actors/test-actor-123/runsc-state", + "kill", + "my-container", + "SIGTERM", + }, + }, + { + name: "wait", + command: "wait", + containerName: "my-container", + want: []string{ + "-global1", + "-global2", + "-root", "/node/actors/test-actor-123/runsc-state", + "wait", + "my-container", + }, + }, + { + name: "list", + command: "list", + containerName: "", + want: []string{ + "-global1", + "-global2", + "-root", "/node/actors/test-actor-123/runsc-state", + "list", + "-list-sub", + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + got := r.buildArgs(tc.command, tc.containerName, tc.extraArgs...) + if !reflect.DeepEqual(got, tc.want) { + t.Errorf("buildArgs(%q, %q, %v) = %v, want %v", tc.command, tc.containerName, tc.extraArgs, got, tc.want) + } + }) } } @@ -131,13 +182,14 @@ func TestCheckpointArgs(t *testing.T) { path: "/usr/bin/runsc", actorUID: "test-actor-123", actorDirs: testActorDirs, + flags: mustFlagStore(t), } checkpointDir := "/node/actors/test-actor-123/checkpoints/snap-1" got := r.checkpointArgs(ocispec.PauseContainer, checkpointDir, []string{"app:/var/data", "app:/var/cache"}) want := []string{ - "-log-format", "json", - "--alsologtostderr", + "-global1", + "-global2", "-root", "/node/actors/test-actor-123/runsc-state", "checkpoint", "-image-path", checkpointDir, @@ -155,13 +207,14 @@ func TestFsCheckpointArgs(t *testing.T) { path: "/usr/bin/runsc", actorUID: "test-actor-123", actorDirs: testActorDirs, + flags: mustFlagStore(t), } checkpointDir := "/node/actors/test-actor-123/checkpoints/snap-1" got := r.fsCheckpointArgs(ocispec.PauseContainer, checkpointDir, []string{"app:/var/data", "app:/var/cache"}) want := []string{ - "-log-format", "json", - "--alsologtostderr", + "-global1", + "-global2", "-root", "/node/actors/test-actor-123/runsc-state", "fscheckpoint", "-image-path", filepath.Join(checkpointDir, "fs"), @@ -180,21 +233,22 @@ func TestRestoreArgs(t *testing.T) { path: "/usr/bin/runsc", actorUID: "test-actor-123", actorDirs: testActorDirs, + flags: mustFlagStore(t), } checkpointDir := "/node/actors/test-actor-123/checkpoints/snap-1" got := r.restoreArgs(ocispec.PauseContainer, checkpointDir) want := []string{ - "-log-format", "json", - "--alsologtostderr", + "-global1", + "-global2", + "-restore-global", "-root", "/node/actors/test-actor-123/runsc-state", - "--cpu-num-from-quota", "restore", + "-restore-sub1", + "-restore-sub2", "-bundle", "/node/actors/test-actor-123/bundle/" + ocispec.PauseContainer, "-image-path", checkpointDir, "-pid-file", "/node/actors/test-actor-123/pidfiles/" + ocispec.PauseContainer + ".pid", - "-background", - "-detach", ocispec.PauseContainer, } diff --git a/cmd/ateom-gvisor/stats_test.go b/cmd/ateom-gvisor/stats_test.go index 42f80e92da..c1f086a1fe 100644 --- a/cmd/ateom-gvisor/stats_test.go +++ b/cmd/ateom-gvisor/stats_test.go @@ -661,11 +661,12 @@ func TestDeadSandboxCheckDuringDrain(t *testing.T) { } // TestDrainNeverKillsPauseContainer: gracefulShutdown kills the names -// containerNames takes from the spec. atelet rejects a spec that names the -// pause container, so the drain cannot stop the sandbox itself. +// containerNames takes from the spec. atelet requires container names to be +// DNS-1123 labels (k8s-short-name) and the pause container's name is not one, +// so no spec can name it and the drain cannot stop the sandbox itself. func TestDrainNeverKillsPauseContainer(t *testing.T) { - if err := resources.ValidateContainerNames([]string{ocispec.PauseContainer}); err == nil { - t.Fatalf("ValidateContainerNames accepted %q; a spec could then put it on the drain's kill list", ocispec.PauseContainer) + if resources.IsValidResourceName(ocispec.PauseContainer) { + t.Fatalf("%q is a valid container name; a spec could then put it on the drain's kill list", ocispec.PauseContainer) } spec := []*ateompb.Container{{Name: "app"}, {Name: "sidecar"}} for _, name := range containerNames(spec) { diff --git a/cmd/ateom-microvm/internal/ch/createvm.go b/cmd/ateom-microvm/internal/ch/createvm.go index 51d0ea4a35..466e45afd1 100644 --- a/cmd/ateom-microvm/internal/ch/createvm.go +++ b/cmd/ateom-microvm/internal/ch/createvm.go @@ -26,7 +26,7 @@ type VmConfig struct { Cpus CpusConfig `json:"cpus"` Memory MemoryConfig `json:"memory"` Payload PayloadConfig `json:"payload"` - Disks []DiskConfig `json:"disks,omitempty"` + Pmem []PmemConfig `json:"pmem,omitempty"` Fs []FsConfig `json:"fs,omitempty"` Rng *RngConfig `json:"rng,omitempty"` Serial *ConsoleConfig `json:"serial,omitempty"` @@ -66,22 +66,19 @@ type MemoryConfig struct { } // PayloadConfig points at the guest kernel + its cmdline (initramfs/firmware -// unused: the kata guest boots from a virtio-blk image disk, root=/dev/vda1). +// unused: the kata guest boots from its image on virtio-pmem). type PayloadConfig struct { Kernel string `json:"kernel"` Cmdline string `json:"cmdline"` } -// DiskConfig is one virtio-blk disk. The only disk is the kata guest image -// (/dev/vda, read-only); the actor rootfs is an overlay served over virtio-fs, not a -// disk. NumQueues/QueueSize mirror kata's clh (num_queues = vcpus, queue_size = 1024). -type DiskConfig struct { - Path string `json:"path"` - Readonly bool `json:"readonly"` - Direct bool `json:"direct"` - NumQueues int32 `json:"num_queues,omitempty"` - QueueSize int32 `json:"queue_size,omitempty"` - ImageType string `json:"image_type,omitempty"` +// PmemConfig configures a virtio-pmem device (/dev/pmem). +type PmemConfig struct { + ID string `json:"id,omitempty"` + File string `json:"file"` + // If true, maps File with MAP_PRIVATE so guest writes land in a private + // in-memory copy. + DiscardWrites bool `json:"discard_writes,omitempty"` } // RngConfig sets the entropy source (kata uses /dev/urandom). diff --git a/cmd/ateom-microvm/internal/kata/overlay_wait_test.go b/cmd/ateom-microvm/internal/kata/overlay_wait_test.go index 80eee7aaed..b53fcfb8f9 100644 --- a/cmd/ateom-microvm/internal/kata/overlay_wait_test.go +++ b/cmd/ateom-microvm/internal/kata/overlay_wait_test.go @@ -25,29 +25,6 @@ import ( "time" ) -// TestWaitForSocketReturnsPromptly pins the reason this helper exists: it sits on -// the restore path, so what it costs is decided by how often it looks, not by how -// long virtiofsd takes to bind. A socket appearing after 5ms should not cost a -// 50ms tick. -func TestWaitForSocketReturnsPromptly(t *testing.T) { - path := filepath.Join(t.TempDir(), "vfsd.sock") - go func() { - time.Sleep(5 * time.Millisecond) - f, err := os.Create(path) - if err == nil { - _ = f.Close() - } - }() - - start := time.Now() - if err := waitForSocket(context.Background(), path, 5*time.Second); err != nil { - t.Fatalf("waitForSocket: %v", err) - } - if elapsed := time.Since(start); elapsed > 40*time.Millisecond { - t.Errorf("waited %s for a socket that appeared after 5ms; the poll interval dominates", elapsed) - } -} - func TestWaitForSocketAlreadyPresent(t *testing.T) { path := filepath.Join(t.TempDir(), "vfsd.sock") f, err := os.Create(path) diff --git a/cmd/ateom-microvm/main.go b/cmd/ateom-microvm/main.go index 36bde56802..5e9edefe13 100644 --- a/cmd/ateom-microvm/main.go +++ b/cmd/ateom-microvm/main.go @@ -24,6 +24,7 @@ package main import ( "context" + "errors" "fmt" "log/slog" "net" @@ -74,7 +75,7 @@ var ( tunnelConfig = ateomtunnel.RegisterFlags(pflag.CommandLine) - readinessListenAddress = pflag.String("readiness-listen-address", "0.0.0.0:8080", "Address for HTTP readiness checks") + readinessListenAddress = pflag.String("readiness-listen-address", "0.0.0.0:8080", "Address for the HTTP /readyz and /healthz probes") maxActors = pflag.Int("max-actors", 1000, "How many actors this worker will host at once") usageSampleInterval = pflag.Duration("usage-sample-interval", time.Minute, "How often the ateom samples the resource usage of its actors. Each sample writes an ate.actor.usage_sampled record per actor, and GetActiveWorkloadStats serves the latest.") ) @@ -175,15 +176,11 @@ func do(ctx context.Context) error { if err := resources.ValidateWorkerPodUID(*podUID); err != nil { return fmt.Errorf("in resources.ValidateWorkerPodUID: %w", err) } + // atelet owns the directory and the sandbox versions file in it; only the + // socket below is this process's. Both outlive a container restart. if err := os.MkdirAll(ateomDir, 0o700); err != nil { return fmt.Errorf("in os.MkdirAll(%q): %w", ateomDir, err) } - // Clean up the ateom directory during graceful shutdown (#1677). - defer func() { - if err := os.RemoveAll(ateomDir); err != nil { - slog.ErrorContext(ctx, "Failed to remove the ateom directory on shutdown", slog.Any("err", err)) - } - }() // Reap children reparented to us: the detached cloud-hypervisor VMM and // virtiofsd. Synchronous subprocesses (mount, umount, cp, ...) instead go @@ -214,6 +211,11 @@ func do(ctx context.Context) error { if err != nil { return fmt.Errorf("while opening unix socket: %w", err) } + defer func() { + if err := os.Remove(sockPath); err != nil && !errors.Is(err, os.ErrNotExist) { + slog.ErrorContext(ctx, "Failed to remove the ateom socket on shutdown", slog.Any("err", err)) + } + }() // Forward the actor container's stdout/stderr to the worker pod's stdout as // JSON with ate.dev/* labels (logging parity with ateom-gvisor). It shares @@ -290,7 +292,7 @@ func do(ctx context.Context) error { } }() - go serverboot.StartReadinessServer(ctx, *readinessListenAddress, readiness) + go serverboot.StartHealthServer(ctx, *readinessListenAddress, readiness) slog.InfoContext(ctx, "ateom-microvm serving", slog.String("socket", sockPath)) if err := svr.Serve(lis); err != nil { diff --git a/cmd/ateom-microvm/restore.go b/cmd/ateom-microvm/restore.go index caae2478bf..560cae3eb1 100644 --- a/cmd/ateom-microvm/restore.go +++ b/cmd/ateom-microvm/restore.go @@ -366,8 +366,7 @@ func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootPar restoredNets = append(restoredNets, rn) } - // Relaunch CH and restore with the tap FDs attached (SCM_RIGHTS). CH reopens - // /dev/vda (image) + each /dev/vd{b+i} (actor rootfs) from the snapshot config paths. + // Relaunch CH and restore with the tap FDs attached (SCM_RIGHTS). apiSocket := filepath.Join(kata.VMDir(actorUID), "clh-api-restore.sock") tTap := time.Now() chCmd, client, err := ch.LaunchVMM(ctx, ch.LaunchVMMOptions{ @@ -401,6 +400,17 @@ func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootPar return fmt.Errorf("while restoring VM with net FDs: %w", err) } tVMRestore := time.Now() + // An eager restore has read the whole snapshot into guest memory, and nothing + // merges against it afterwards, so drop the staged memory image and evict its + // page cache before resuming the VM so guest RAM and snapshot page cache do + // not overlap during Resume and wakeup probe gating. + // + // Dropping ahead of Resume means a Resume, agent dial, or wakeup failure leaves + // restore-state empty; atelet re-downloads with O_TRUNC and resetActorDirs + // wipes the directory on retry, so this costs a re-download on retry rather + // than correctness. + maybeDropStagedMemoryImage(ctx, restoreDir, memMode, preserveRestoreDir) + dropActorSnapshotCacheAsync(restoreDir, memMode == ch.MemRestoreEager) if err := client.Resume(ctx); err != nil { return fmt.Errorf("while resuming restored guest: %w", err) } @@ -504,9 +514,6 @@ func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootPar } s.setRunningVM(actorUID, ra) - // After the last error return, so a failed restore stays retryable. - maybeDropStagedMemoryImage(ctx, restoreDir, memMode, preserveRestoreDir) - // Publish the guest to GetWorkloadStats, past the last error return above // for the same reason as in coldBootActor. Same client the forwarding above // reads over. @@ -522,26 +529,21 @@ func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootPar return nil } -// maybeDropStagedMemoryImage deletes memory-ranges after an eager restore (it is -// fully in guest memory and nothing merges against it), unless restoreDir is a -// preserved snapshot. +// maybeDropStagedMemoryImage deletes the staged memory-ranges file and +// already-extracted tar archives after an eager restore (it is fully in guest +// memory and nothing merges against it), unless restoreDir is a preserved +// snapshot. func maybeDropStagedMemoryImage(ctx context.Context, restoreDir, memMode string, preserveRestoreDir bool) { if memMode != ch.MemRestoreEager || preserveRestoreDir { return } - staged := filepath.Join(restoreDir, "memory-ranges") - if err := os.Remove(staged); err != nil && !os.IsNotExist(err) { - // Not fatal: it only costs disk until the actor is torn down. - slog.WarnContext(ctx, "could not drop the staged memory image", "error", err) - } else { - slog.InfoContext(ctx, "dropped the staged memory image (eager restore needs no merge base)") - } + dropStagedMemoryImage(ctx, restoreDir) } // rewriteSnapshotSocketPaths repoints the snapshot config.json's per-VMDir paths from // the source actor's VMDir to the restoring actor's: the hybrid-vsock socket, the // File serial console, and each virtio-fs socket, so the sockets/files we create are -// the ones CH reopens. The kernel and /dev/vda kata image are content-addressed static +// the ones CH reopens. The kernel and the kata image are content-addressed static // files with identical paths on every node, so they need no rewrite, and the overlay // has no per-actor disk to repoint. func rewriteSnapshotSocketPaths(snapshotDir, id string) error { diff --git a/cmd/ateom-microvm/run.go b/cmd/ateom-microvm/run.go index 1bdbfe186f..9dd7a68875 100644 --- a/cmd/ateom-microvm/run.go +++ b/cmd/ateom-microvm/run.go @@ -529,7 +529,7 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re } }() - // Assemble the CH VmConfig (kata-compatible cmdline, RO kata image on /dev/vda + + // Assemble the CH VmConfig (kata-compatible cmdline, RO kata image on /dev/pmem0 + // the virtio-fs device; no actor virtio-blk disks — rootfs writes land in the // host-side overlay upper through the shared mount). The console log is also read // on a failed agent dial below, so keep it here. @@ -792,25 +792,7 @@ func initParams(agentInit bool) string { "systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket" } -// buildVMConfig assembles the cloud-hypervisor VmConfig. The console is arch-specific: -// ttyAMA0 on arm64, ttyS0 on amd64. /dev/vda is the RO guest image; the actor rootfs's RO -// lower is the virtio-fs device on PCI segment 1 (hence num_pci_segments=2), with no -// actor disks. -// -// init=kataAgentPath boots the kata agent as PID 1 instead of systemd. The agent detects -// that it is PID 1 and does the init work itself: it mounts /proc, /sys, devtmpfs /dev, -// /dev/shm, /dev/pts, tmpfs /run and the cgroup hierarchy, then serves ttrpc over vsock. -// Nothing else in the guest image is ours to run — the workload is a container the agent -// starts — so systemd only cost us. Measured on the counter demo, dropping it took the -// guest's boot-time reads from this disk from 58.6MiB to 35.0MiB, the snapshot from 145MiB -// to 106.6MiB at the same guest RAM, and a cold boot from 15.9s to 10.3s: the agent is -// PID 1 rather than a unit systemd reaches several seconds in, so ateom stops waiting for -// it (the dial phase goes 10.4s -> 4.7s). -// -// Dropping systemd also drops chronyd (kata-containers.target wants it), which is what -// used to repair the guest clock after a resume. That is safe only from cloud-hypervisor -// v53, which advances the guest clock across a restore itself; on v52 a restored guest -// stays frozen at the instant it was snapshotted. +// buildVMConfig assembles the cloud-hypervisor VmConfig. // // The disk-backed rootfs upper share (see rootfsupper.go) is always present. // @@ -823,7 +805,7 @@ func initParams(agentInit bool) string { // map, CPU features and ACPI lines never reach the log. guestDebug adds the UART back // with earlycon (and pays the ~800ms) for diagnosing a guest that dies before then. func buildVMConfig(id, kernel, image, kparams, consoleLog string, memMiB, vcpus int, agentInit, debug bool) ch.VmConfig { - cmdline := "root=/dev/vda1 rootflags=data=ordered,errors=remount-ro ro rootfstype=ext4 " + + cmdline := "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 " + "panic=1 no_timer_check noreplace-smp console=hvc0 " + initParams(agentInit) if kparams != "" { @@ -841,9 +823,9 @@ func buildVMConfig(id, kernel, image, kparams, consoleLog string, memMiB, vcpus Cpus: ch.CpusConfig{BootVcpus: int32(vcpus), MaxVcpus: int32(vcpus)}, Memory: ch.MemoryConfig{Size: int64(memMiB) * 1024 * 1024, Shared: true}, Payload: ch.PayloadConfig{Kernel: kernel, Cmdline: cmdline}, - Disks: []ch.DiskConfig{ - {Path: image, Readonly: true, ImageType: "Raw", NumQueues: int32(vcpus), QueueSize: 1024}, - }, + // The guest image is a virtio-pmem device (/dev/pmem0), mounted read-only + // with DAX, so its file data stays in the host page cache instead of being copied into guest RAM. + Pmem: []ch.PmemConfig{{ID: "guest-image", File: image, DiscardWrites: true}}, Fs: buildFsConfigs(id), Platform: &ch.PlatformConfig{NumPciSegments: 2}, Rng: &ch.RngConfig{Src: "/dev/urandom"}, diff --git a/cmd/ateom-microvm/snapshotcache.go b/cmd/ateom-microvm/snapshotcache.go new file mode 100644 index 0000000000..6dbc711082 --- /dev/null +++ b/cmd/ateom-microvm/snapshotcache.go @@ -0,0 +1,61 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "log/slog" + "strings" + + "github.com/agent-substrate/substrate/internal/snapshotcache" +) + +const memoryRangesFile = "memory-ranges" + +func isSnapshotDataFile(name string, includeMemory bool) bool { + if includeMemory && name == memoryRangesFile { + return true + } + return strings.HasSuffix(name, ".tar") +} + +// dropStagedMemoryImage drops the staged memory-ranges file and already-extracted +// tar archives in a scratch restoreDir once an eager ("Copy") restore has +// finished reading guest memory. +func dropStagedMemoryImage(ctx context.Context, restoreDir string) { + targets := snapshotcache.FindFiles(restoreDir, func(name string) bool { + return isSnapshotDataFile(name, true) + }) + if len(targets) == 0 { + return + } + snapshotcache.DropFiles(targets) + slog.InfoContext(ctx, "dropped the staged memory image (eager restore needs no merge base)") +} + +// dropActorSnapshotCacheAsync pins the actor's snapshot files in restoreDir +// while the caller holds the actor lock, then evicts their clean pages from the +// host page cache in the background. +func dropActorSnapshotCacheAsync(restoreDir string, includeMemory bool) { + targets := snapshotcache.FindFiles(restoreDir, func(name string) bool { + return isSnapshotDataFile(name, includeMemory) + }) + if len(targets) == 0 { + return + } + go snapshotcache.EvictWithRetries(targets) +} diff --git a/cmd/ateom-microvm/snapshotcache_test.go b/cmd/ateom-microvm/snapshotcache_test.go new file mode 100644 index 0000000000..d262fafdfc --- /dev/null +++ b/cmd/ateom-microvm/snapshotcache_test.go @@ -0,0 +1,112 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/ch" + "github.com/agent-substrate/substrate/internal/snapshotcache" +) + +const ( + testRestoreStateDir = "restore-state" + testRootfsUpperTar = "rootfs-upper-app.tar" + testDurableDirTar = "durable-dir-data.tar" +) + +func TestDropStagedMemoryImage(t *testing.T) { + restoreDir := filepath.Join(t.TempDir(), testRestoreStateDir) + if err := os.MkdirAll(restoreDir, 0o700); err != nil { + t.Fatal(err) + } + stagedFiles := []string{memoryRangesFile, testRootfsUpperTar, testDurableDirTar} + for _, name := range stagedFiles { + if err := os.WriteFile(filepath.Join(restoreDir, name), []byte("staged data"), 0o600); err != nil { + t.Fatal(err) + } + } + cfgPath := filepath.Join(restoreDir, "config.json") + if err := os.WriteFile(cfgPath, []byte("{}"), 0o600); err != nil { + t.Fatal(err) + } + + dropStagedMemoryImage(context.Background(), restoreDir) + + for _, name := range stagedFiles { + if _, err := os.Stat(filepath.Join(restoreDir, name)); !os.IsNotExist(err) { + t.Fatalf("staged %s still exists after dropStagedMemoryImage: err=%v", name, err) + } + } + if _, err := os.Stat(cfgPath); err != nil { + t.Fatalf("config.json should be preserved: %v", err) + } +} + +func TestMaybeDropStagedMemoryImage_PreservedDirNotUnlinked(t *testing.T) { + restoreDir := filepath.Join(t.TempDir(), testRestoreStateDir) + if err := os.MkdirAll(restoreDir, 0o700); err != nil { + t.Fatal(err) + } + memFile := filepath.Join(restoreDir, memoryRangesFile) + want := "preserved snapshot memory" + if err := os.WriteFile(memFile, []byte(want), 0o600); err != nil { + t.Fatal(err) + } + + maybeDropStagedMemoryImage(context.Background(), restoreDir, ch.MemRestoreEager, true) + + got, err := os.ReadFile(memFile) + if err != nil { + t.Fatalf("preserved memory-ranges missing: %v", err) + } + if string(got) != want { + t.Fatalf("preserved memory-ranges = %q, want %q", string(got), want) + } +} + +func TestIsSnapshotDataFile(t *testing.T) { + restoreDir := filepath.Join(t.TempDir(), testRestoreStateDir) + if err := os.MkdirAll(restoreDir, 0o700); err != nil { + t.Fatal(err) + } + + for _, name := range []string{memoryRangesFile, testRootfsUpperTar, "config.json"} { + if err := os.WriteFile(filepath.Join(restoreDir, name), []byte("data"), 0o600); err != nil { + t.Fatal(err) + } + } + + // Eager mode (includeMemory=true): both memory-ranges and .tar are matched. + targetsEager := snapshotcache.FindFiles(restoreDir, func(name string) bool { + return isSnapshotDataFile(name, true) + }) + if len(targetsEager) != 2 { + t.Fatalf("FindFiles(includeMemory=true) returned %d targets, want 2", len(targetsEager)) + } + + // OnDemand mode (includeMemory=false): memory-ranges is excluded, only .tar is matched. + targetsOnDemand := snapshotcache.FindFiles(restoreDir, func(name string) bool { + return isSnapshotDataFile(name, false) + }) + if len(targetsOnDemand) != 1 || filepath.Base(targetsOnDemand[0].Path) != testRootfsUpperTar { + t.Fatalf("FindFiles(includeMemory=false) returned %+v, want only %s", targetsOnDemand, testRootfsUpperTar) + } +} diff --git a/cmd/benchmarking/boomer-worker/main.go b/cmd/benchmarking/boomer-worker/main.go index b7e48a7ec0..d4e1acbbf4 100644 --- a/cmd/benchmarking/boomer-worker/main.go +++ b/cmd/benchmarking/boomer-worker/main.go @@ -38,6 +38,7 @@ import ( // Register user classes via init(): "github.com/agent-substrate/substrate/internal/benchmarking/boomer/agentsession" _ "github.com/agent-substrate/substrate/internal/benchmarking/boomer/glutton" + _ "github.com/agent-substrate/substrate/internal/benchmarking/boomer/resumepaused" _ "github.com/agent-substrate/substrate/internal/benchmarking/boomer/sweperf" ) @@ -47,7 +48,7 @@ func main() { routerURL = flag.String("router-url", "http://atenet-router.ate-system.svc.cluster.local", "atenet HTTP router base URL (no trailing slash).") atespace = flag.String("atespace", "benchmark", "Atespace every actor this worker creates lives in. Ensured (CreateAtespace, AlreadyExists is ok) at startup.") promAddr = flag.String("prometheus-addr", ":8001", "Address for the Prometheus /metrics endpoint.") - configJSON = flag.String("config-json", "", "Initial dynconfig as a JSON object (keys: trace_probability, min_wait_time, max_wait_time, min_live_time, max_live_time in seconds, durdir_file_size_bytes, resume_mode, lifecycle_mode, durdir_read_mode, durdir_template, mem_target, mem_churn, mem_read, cpu_cores, cpu_duty_cycle, max_pings_per_wake). Unset fields keep their built-in defaults.") + configJSON = flag.String("config-json", "", "Initial dynconfig as a JSON object (keys: trace_probability, min_wait_time, max_wait_time, min_live_time, max_live_time in seconds, durdir_file_size_bytes, resume_mode, lifecycle_mode, durdir_read_mode, durdir_template, mem_target, mem_churn, mem_read, cpu_cores, cpu_duty_cycle, max_pings_per_wake, target_rps, total_actors). Unset fields keep their built-in defaults.") masterWebPort = flag.Int("master-web-port", 0, "If non-zero, fetch dynconfig from http://{master-host}:{master-web-port}/boomer-config on each spawn message. Exits if the first fetch fails; later failures keep the last fetched values. {master-host} comes from boomer's existing --master-host flag.") configPollInterval = flag.Duration("config-poll-interval", 10*time.Second, "With --master-web-port, also fetch dynconfig on this interval. A spawn message comes only when the number of users or the spawn rate changes, thus a load shape that changes the sample rate alone needs this. Zero stops the polling.") userClass = flag.String("user-class", "glutton", fmt.Sprintf("Locust user class to run, lowercase; one of %s.", strings.Join(userclass.Names(), "|"))) diff --git a/cmd/benchmarking/glutton/main.go b/cmd/benchmarking/glutton/main.go index 901d175789..8f5d8fd09c 100644 --- a/cmd/benchmarking/glutton/main.go +++ b/cmd/benchmarking/glutton/main.go @@ -93,10 +93,7 @@ func main() { serverboot.Fatal(ctx, "Failed to start listener", fmt.Errorf("%s: %w", *listenAddr, err)) } - go serverboot.StartMetricsServer(ctx, serverboot.MetricsServerOptions{ - Addr: *metricsListenAddr, - Readiness: &serverboot.Readiness{}, - }) + go serverboot.StartMetricsServer(ctx, *metricsListenAddr) slog.InfoContext(ctx, "glutton starting", slog.String("listen-addr", *listenAddr), diff --git a/cmd/benchmarking/isolate/fake-atelet/config.go b/cmd/benchmarking/isolate/fake-atelet/config.go index 9271002f44..1cb0dfa967 100644 --- a/cmd/benchmarking/isolate/fake-atelet/config.go +++ b/cmd/benchmarking/isolate/fake-atelet/config.go @@ -16,12 +16,52 @@ package main import ( "fmt" + "strings" "time" + + "github.com/spf13/pflag" ) +// tuningFlags are the flags a FAKE_ATELET_ environment variable can +// set, as from the fake-atelet-tuning ConfigMap a benchmark writes: the +// installer chooses the fake data plane, and the benchmark tunes it. +var tuningFlags = []string{ + "delay", + "delay-run", + "delay-restore", + "delay-checkpoint", + "delay-promote-snapshot", + "delay-terminate", + "mint-actor-certificate", +} + +// tuningEnv is the environment variable that sets the tuning flag name. +func tuningEnv(name string) string { + return "FAKE_ATELET_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_")) +} + +// applyTuningEnv sets each tuning flag the command line left unset from its +// environment variable, read through lookup. A flag on the command line +// wins. +func applyTuningEnv(fs *pflag.FlagSet, lookup func(string) (string, bool)) error { + for _, name := range tuningFlags { + if fs.Changed(name) { + continue + } + value, ok := lookup(tuningEnv(name)) + if !ok { + continue + } + if err := fs.Set(name, value); err != nil { + return fmt.Errorf("%s=%q: %w", tuningEnv(name), value, err) + } + } + return nil +} + // resolveDelays applies each per-call override over the default delay. A // negative override means "use the default". -func resolveDelays(def, run, restore, checkpoint, uploadPausedCheckpoint, terminate time.Duration) (delays, error) { +func resolveDelays(def, run, restore, checkpoint, promoteSnapshot, terminate time.Duration) (delays, error) { if def < 0 { return delays{}, fmt.Errorf("delay must not be negative, got %v", def) } @@ -32,10 +72,10 @@ func resolveDelays(def, run, restore, checkpoint, uploadPausedCheckpoint, termin return d } return delays{ - run: pick(run), - restore: pick(restore), - checkpoint: pick(checkpoint), - uploadPausedCheckpoint: pick(uploadPausedCheckpoint), - terminate: pick(terminate), + run: pick(run), + restore: pick(restore), + checkpoint: pick(checkpoint), + promoteSnapshot: pick(promoteSnapshot), + terminate: pick(terminate), }, nil } diff --git a/cmd/benchmarking/isolate/fake-atelet/config_test.go b/cmd/benchmarking/isolate/fake-atelet/config_test.go index fc10f212fd..771eadb1d0 100644 --- a/cmd/benchmarking/isolate/fake-atelet/config_test.go +++ b/cmd/benchmarking/isolate/fake-atelet/config_test.go @@ -15,8 +15,13 @@ package main import ( + "os" + "regexp" + "slices" "testing" "time" + + "github.com/spf13/pflag" ) func TestResolveDelays(t *testing.T) { @@ -24,7 +29,7 @@ func TestResolveDelays(t *testing.T) { if err != nil { t.Fatalf("resolveDelays: %v", err) } - want := delays{run: time.Second, restore: 2 * time.Second, checkpoint: 0, uploadPausedCheckpoint: time.Second, terminate: time.Second} + want := delays{run: time.Second, restore: 2 * time.Second, checkpoint: 0, promoteSnapshot: time.Second, terminate: time.Second} if got != want { t.Errorf("resolveDelays = %+v, want %+v", got, want) } @@ -32,3 +37,86 @@ func TestResolveDelays(t *testing.T) { t.Error("resolveDelays accepted a negative default") } } + +// A benchmark tunes fake-atelet through FAKE_ATELET_* variables; a flag on +// the command line wins over its variable, and a bad value is an error. +func TestApplyTuningEnv(t *testing.T) { + newFlags := func() (*pflag.FlagSet, *time.Duration, *time.Duration, *bool) { + fs := pflag.NewFlagSet("fake-atelet", pflag.ContinueOnError) + d := fs.Duration("delay", 0, "") + run := fs.Duration("delay-run", -1, "") + for _, name := range []string{"delay-restore", "delay-checkpoint", "delay-promote-snapshot", "delay-terminate"} { + fs.Duration(name, -1, "") + } + mint := fs.Bool("mint-actor-certificate", true, "") + fs.String("port", "", "") + return fs, d, run, mint + } + env := func(vars map[string]string) func(string) (string, bool) { + return func(k string) (string, bool) { v, ok := vars[k]; return v, ok } + } + + fs, d, run, mint := newFlags() + if err := fs.Parse([]string{"--delay-run=3s"}); err != nil { + t.Fatal(err) + } + err := applyTuningEnv(fs, env(map[string]string{ + "FAKE_ATELET_DELAY": "500ms", + "FAKE_ATELET_DELAY_RUN": "9s", + "FAKE_ATELET_MINT_ACTOR_CERTIFICATE": "false", + // Not a tuning flag, so not read. + "FAKE_ATELET_PORT": "1", + })) + if err != nil { + t.Fatalf("applyTuningEnv() = %v", err) + } + if *d != 500*time.Millisecond || *run != 3*time.Second || *mint { + t.Errorf("delay, delay-run, mint = %v, %v, %v, want 500ms from the environment, 3s from the flag, false", *d, *run, *mint) + } + if fs.Changed("port") { + t.Error("FAKE_ATELET_PORT set --port, which is not a tuning flag") + } + + fs, _, _, _ = newFlags() + if err := applyTuningEnv(fs, env(map[string]string{"FAKE_ATELET_DELAY": "fast"})); err == nil { + t.Error("applyTuningEnv() accepted FAKE_ATELET_DELAY=fast") + } +} + +// Every tuning flag is one fake-atelet registers, so applyTuningEnv can set +// it rather than failing startup. +func TestTuningFlagsRegistered(t *testing.T) { + for _, name := range tuningFlags { + if pflag.CommandLine.Lookup(name) == nil { + t.Errorf("tuning flag %q is not a fake-atelet flag", name) + } + } +} + +// The benchmark automation's isolate keys set exactly the tuning variables. +// fake-atelet ignores a variable it does not know, so a key left behind by a +// rename would tune nothing without an error. +func TestIsolateTuningMatchesAutomation(t *testing.T) { + const automation = "../../../../benchmarking/automation/util.py" + src, err := os.ReadFile(automation) + if err != nil { + t.Fatalf("reading %s: %v", automation, err) + } + block := regexp.MustCompile(`(?s)ISOLATE_TUNING = \{(.*?)\}`).FindSubmatch(src) + if block == nil { + t.Fatalf("%s has no ISOLATE_TUNING map", automation) + } + var got []string + for _, m := range regexp.MustCompile(`"(FAKE_ATELET_[A-Z_]+)"`).FindAllSubmatch(block[1], -1) { + got = append(got, string(m[1])) + } + var want []string + for _, name := range tuningFlags { + want = append(want, tuningEnv(name)) + } + slices.Sort(got) + slices.Sort(want) + if !slices.Equal(got, want) { + t.Errorf("%s ISOLATE_TUNING sets %v, want fake-atelet's tuning variables %v", automation, got, want) + } +} diff --git a/cmd/benchmarking/isolate/fake-atelet/herder.go b/cmd/benchmarking/isolate/fake-atelet/herder.go index 42c2d8ad66..48a34b4fa7 100644 --- a/cmd/benchmarking/isolate/fake-atelet/herder.go +++ b/cmd/benchmarking/isolate/fake-atelet/herder.go @@ -50,15 +50,16 @@ type actorCertMinter interface { // delays is how long each AteomHerder call takes before it succeeds: the // data plane's share of it, not counting an actor certificate mint. type delays struct { - run, restore, checkpoint, uploadPausedCheckpoint, terminate time.Duration + run, restore, checkpoint, promoteSnapshot, terminate time.Duration } // herder answers every AteomHerder call with success after its delay, without // running or saving any workload. Actors it "runs" do not exist. // // The one side effect it keeps is a placeholder object under each snapshot -// URI it is asked to write: ate-api-server copies an actor's snapshot when it -// creates a tag, golden tags included, and refuses to copy an empty one. +// URI a call uploads to before it returns: ate-api-server copies an actor's +// snapshot when it creates a tag, golden tags included, and refuses to copy +// an empty one. // // The one call back it keeps is the actor certificate mint: before ateom // starts a workload with tunneled egress it has atelet mint the actor's @@ -170,8 +171,11 @@ func (h *herder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (*at func (h *herder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest) (*ateletpb.CheckpointResponse, error) { start := time.Now() - if req.GetType() == ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL { - if err := h.writePlaceholder(ctx, start, h.delays.checkpoint, req.GetExternalConfig().GetSnapshotUri()); err != nil { + // An immediate upload puts the snapshot in object storage before + // Checkpoint returns. The fake does not run deferred uploads: the + // actor's PromoteSnapshot writes it instead. + if req.GetUploadConfig().GetMode() == ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE { + if err := h.writePlaceholder(ctx, start, h.delays.checkpoint, req.GetSnapshot().GetObject().GetSnapshotUri()); err != nil { return nil, err } } @@ -181,15 +185,15 @@ func (h *herder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest return &ateletpb.CheckpointResponse{}, nil } -func (h *herder) UploadPausedCheckpoint(ctx context.Context, req *ateletpb.UploadPausedCheckpointRequest) (*ateletpb.UploadPausedCheckpointResponse, error) { +func (h *herder) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnapshotRequest) (*ateletpb.PromoteSnapshotResponse, error) { start := time.Now() - if err := h.writePlaceholder(ctx, start, h.delays.uploadPausedCheckpoint, req.GetDestinationSnapshotUri()); err != nil { + if err := h.writePlaceholder(ctx, start, h.delays.promoteSnapshot, req.GetSnapshot().GetObject().GetSnapshotUri()); err != nil { return nil, err } - if err := wait(ctx, start, h.delays.uploadPausedCheckpoint); err != nil { + if err := wait(ctx, start, h.delays.promoteSnapshot); err != nil { return nil, err } - return &ateletpb.UploadPausedCheckpointResponse{}, nil + return &ateletpb.PromoteSnapshotResponse{}, nil } func (h *herder) Terminate(ctx context.Context, _ *ateletpb.TerminateRequest) (*ateletpb.TerminateResponse, error) { diff --git a/cmd/benchmarking/isolate/fake-atelet/herder_test.go b/cmd/benchmarking/isolate/fake-atelet/herder_test.go index 4c90f158f3..8e8ac9018d 100644 --- a/cmd/benchmarking/isolate/fake-atelet/herder_test.go +++ b/cmd/benchmarking/isolate/fake-atelet/herder_test.go @@ -66,11 +66,12 @@ func (s *recordingStorage) objects() []string { } func uniformDelays(d time.Duration) delays { - return delays{run: d, restore: d, checkpoint: d, uploadPausedCheckpoint: d, terminate: d} + return delays{run: d, restore: d, checkpoint: d, promoteSnapshot: d, terminate: d} } // herderCalls invokes every AteomHerder method the herder implements. The -// checkpoint calls ask for an external snapshot, so each writes a placeholder. +// Checkpoint and PromoteSnapshot calls upload a snapshot, so each writes a +// placeholder. func herderCalls(h *herder) map[string]func(context.Context) error { return map[string]func(context.Context) error{ "Run": func(ctx context.Context) error { @@ -85,8 +86,12 @@ func herderCalls(h *herder) map[string]func(context.Context) error { _, err := h.Checkpoint(ctx, externalCheckpoint(testSnapshotURI)) return err }, - "UploadPausedCheckpoint": func(ctx context.Context) error { - _, err := h.UploadPausedCheckpoint(ctx, &ateletpb.UploadPausedCheckpointRequest{DestinationSnapshotUri: testSnapshotURI}) + "PromoteSnapshot": func(ctx context.Context) error { + _, err := h.PromoteSnapshot(ctx, &ateletpb.PromoteSnapshotRequest{ + Snapshot: &ateletpb.Snapshot{ + Object: &ateletpb.ObjectStorage{SnapshotUri: testSnapshotURI}, + }, + }) return err }, "Terminate": func(ctx context.Context) error { @@ -96,10 +101,17 @@ func herderCalls(h *herder) map[string]func(context.Context) error { } } +// externalCheckpoint is a Checkpoint that uploads to uri before it returns. func externalCheckpoint(uri string) *ateletpb.CheckpointRequest { + return checkpointWithMode(uri, ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE) +} + +func checkpointWithMode(uri string, mode ateletpb.UploadMode) *ateletpb.CheckpointRequest { return &ateletpb.CheckpointRequest{ - Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL, - Config: &ateletpb.CheckpointRequest_ExternalConfig{ExternalConfig: &ateletpb.ExternalCheckpointConfiguration{SnapshotUri: uri}}, + Snapshot: &ateletpb.Snapshot{ + Object: &ateletpb.ObjectStorage{SnapshotUri: uri}, + }, + UploadConfig: &ateletpb.UploadConfig{Mode: mode}, } } @@ -151,11 +163,11 @@ func TestHerderUsesEachCallsOwnDelay(t *testing.T) { } // ate-api-server copies an actor's snapshot when it creates a tag and refuses -// to copy an empty one, so both calls that would upload a snapshot must leave -// an object under its URI. +// to copy an empty one, so calls that upload a snapshot before they return +// must leave an object under its URI. func TestHerderWritesPlaceholderUnderSnapshotURI(t *testing.T) { const want = "bench-bucket/benchmark-workloads/sleep/atespaces/team-a/actors/0f9c1d2e-3b4a-4c5d-8e6f-7a8b9c0d1e2f/snapshots/snap-1/" + placeholderObject - for _, name := range []string{"Checkpoint", "UploadPausedCheckpoint"} { + for _, name := range []string{"Checkpoint", "PromoteSnapshot"} { t.Run(name, func(t *testing.T) { storage := &recordingStorage{} if err := herderCalls(&herder{storage: storage})[name](context.Background()); err != nil { @@ -168,14 +180,19 @@ func TestHerderWritesPlaceholderUnderSnapshotURI(t *testing.T) { } } +// A deferred upload happens after Checkpoint returns, which the fake does not +// model, so only an immediate one writes a placeholder. func TestHerderWritesNothingForLocalCheckpoint(t *testing.T) { storage := &recordingStorage{} h := &herder{storage: storage} - if _, err := h.Checkpoint(context.Background(), &ateletpb.CheckpointRequest{Type: ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL}); err != nil { + if _, err := h.Checkpoint(context.Background(), &ateletpb.CheckpointRequest{}); err != nil { t.Fatalf("Checkpoint: %v", err) } + if _, err := h.Checkpoint(context.Background(), checkpointWithMode(testSnapshotURI, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED)); err != nil { + t.Fatalf("Checkpoint with a deferred upload: %v", err) + } for name, call := range herderCalls(h) { - if name == "Checkpoint" || name == "UploadPausedCheckpoint" { + if name == "Checkpoint" || name == "PromoteSnapshot" { continue } if err := call(context.Background()); err != nil { @@ -183,7 +200,7 @@ func TestHerderWritesNothingForLocalCheckpoint(t *testing.T) { } } if got := storage.objects(); len(got) != 0 { - t.Errorf("wrote %v, want nothing outside external checkpoints", got) + t.Errorf("wrote %v, want nothing outside immediate uploads and promotions", got) } } @@ -195,6 +212,9 @@ func TestHerderFailsWhenPlaceholderCannotBeWritten(t *testing.T) { if _, err := h.Checkpoint(context.Background(), externalCheckpoint("not a snapshot uri")); err == nil { t.Error("Checkpoint succeeded with an unparseable snapshot URI") } + if err := herderCalls(h)["PromoteSnapshot"](context.Background()); err == nil { + t.Error("PromoteSnapshot succeeded with no placeholder written; a later tag of this snapshot would fail") + } } // The write runs inside the delay, so it does not add to it. diff --git a/cmd/benchmarking/isolate/fake-atelet/main.go b/cmd/benchmarking/isolate/fake-atelet/main.go index 609629ff02..1a1cda1631 100644 --- a/cmd/benchmarking/isolate/fake-atelet/main.go +++ b/cmd/benchmarking/isolate/fake-atelet/main.go @@ -13,7 +13,7 @@ // limitations under the License. // Command fake-atelet stands in for atelet in control-plane benchmarks. It -// runs as atelet's DaemonSet would, on the benchmark nodes only, so +// runs in atelet's DaemonSet, installed by ate-setup --fake-data-plane, so // ate-api-server dials it as the node's atelet. It answers every AteomHerder // call with success after a delay, without running or saving any workload, // and relays fake-workersync's capacity reports for the fake Workers on its @@ -57,8 +57,8 @@ import ( var ( port = pflag.Int("port", atelet.DefaultPort, "Port to serve AteomHerder on. ate-api-server dials atelet.DefaultPort.") relayPort = pflag.Int("relay-port", fakeworker.DefaultRelayPort, "Port to serve fake-workersync's capacity reports on.") - relayAllowedID = pflag.String("relay-allowed-spiffe-id", installdefaults.SPIFFEID("benchmark-workloads", "fake-workersync"), "The only SPIFFE ID allowed to send capacity reports.") - healthListenAddr = pflag.String("health-listen-addr", ":9090", "Address to serve /healthz and /readyz on.") + relayAllowedID = pflag.String("relay-allowed-spiffe-id", installdefaults.SPIFFEID("ate-system", "fake-workersync"), "The only SPIFFE ID allowed to send capacity reports.") + healthListenAddr = pflag.String("health-listen-addr", ":9091", "Address to serve /healthz and /readyz on.") grpcServerCredBundle = pflag.String("grpc-server-cred-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "Credential bundle presented as the serving certificate and the ate-api-server client certificate.") clientCACerts = pflag.String("client-ca-certs", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify client certificates.") @@ -66,13 +66,13 @@ var ( ateapiCAFile = pflag.String("ateapi-ca-file", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify ate-api-server.") ateapiServerName = pflag.String("ateapi-server-name", "api.ate-system.svc", "DNS name expected on the ate-api-server certificate.") - delay = pflag.Duration("delay", 0, "How long each AteomHerder call takes before it succeeds, not counting the actor certificate mint on Run and Restore.") - delayRun = pflag.Duration("delay-run", -1, "Override --delay for Run. Negative uses --delay.") - delayRestore = pflag.Duration("delay-restore", -1, "Override --delay for Restore. Negative uses --delay.") - delayCheckpoint = pflag.Duration("delay-checkpoint", -1, "Override --delay for Checkpoint. Negative uses --delay.") - delayUploadPausedCheckpoint = pflag.Duration("delay-upload-paused-checkpoint", -1, "Override --delay for UploadPausedCheckpoint. Negative uses --delay.") - delayTerminate = pflag.Duration("delay-terminate", -1, "Override --delay for Terminate. Negative uses --delay.") - mintActorCertificate = pflag.Bool("mint-actor-certificate", true, "On each Run and Restore that names an egress gateway, mint the actor's certificate from ate-api-server before the delay, as ateom does before starting the workload.") + delay = pflag.Duration("delay", 0, "How long each AteomHerder call takes before it succeeds, not counting the actor certificate mint on Run and Restore.") + delayRun = pflag.Duration("delay-run", -1, "Override --delay for Run. Negative uses --delay.") + delayRestore = pflag.Duration("delay-restore", -1, "Override --delay for Restore. Negative uses --delay.") + delayCheckpoint = pflag.Duration("delay-checkpoint", -1, "Override --delay for Checkpoint. Negative uses --delay.") + delayPromoteSnapshot = pflag.Duration("delay-promote-snapshot", -1, "Override --delay for PromoteSnapshot. Negative uses --delay.") + delayTerminate = pflag.Duration("delay-terminate", -1, "Override --delay for Terminate. Negative uses --delay.") + mintActorCertificate = pflag.Bool("mint-actor-certificate", true, "On each Run and Restore that names an egress gateway, mint the actor's certificate from ate-api-server before the delay, as ateom does before starting the workload.") showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") @@ -90,7 +90,10 @@ func main() { if err := serverboot.SetLogLevel(*logLevelFlag); err != nil { serverboot.Fatal(ctx, "Invalid --log-level", err) } - callDelays, err := resolveDelays(*delay, *delayRun, *delayRestore, *delayCheckpoint, *delayUploadPausedCheckpoint, *delayTerminate) + if err := applyTuningEnv(pflag.CommandLine, os.LookupEnv); err != nil { + serverboot.Fatal(ctx, "Invalid tuning environment", err) + } + callDelays, err := resolveDelays(*delay, *delayRun, *delayRestore, *delayCheckpoint, *delayPromoteSnapshot, *delayTerminate) if err != nil { serverboot.Fatal(ctx, "Invalid --delay", err) } diff --git a/cmd/benchmarking/isolate/fake-atelet/relay_test.go b/cmd/benchmarking/isolate/fake-atelet/relay_test.go index e6b3658dbf..2d5afcace6 100644 --- a/cmd/benchmarking/isolate/fake-atelet/relay_test.go +++ b/cmd/benchmarking/isolate/fake-atelet/relay_test.go @@ -96,7 +96,7 @@ func TestRelayServesOnlyCapacityReports(t *testing.T) { // The relay admits fake-workersync and no one else: any other workload with a // pod certificate could otherwise set any Worker's capacity. func TestRelayTLSAdmitsOnlyTheAllowedID(t *testing.T) { - const allowed = "spiffe://cluster.local/ns/benchmark-workloads/sa/fake-workersync" + const allowed = "spiffe://cluster.local/ns/ate-system/sa/fake-workersync" ca := fakeworkertest.NewCA(t) serverCfg, err := relayTLSConfig(ca.Issue(t, "spiffe://cluster.local/ns/ate-system/sa/atelet"), ca.TrustBundle, allowed) if err != nil { @@ -138,7 +138,7 @@ func TestRelayTLSAdmitsOnlyTheAllowedID(t *testing.T) { // A pod-identity CA rotation during a long run must not cut fake-workersync // off: the relay reads the trust bundle at each handshake. func TestRelayTLSFollowsATrustBundleRotation(t *testing.T) { - const allowed = "spiffe://cluster.local/ns/benchmark-workloads/sa/fake-workersync" + const allowed = "spiffe://cluster.local/ns/ate-system/sa/fake-workersync" ca := fakeworkertest.NewCA(t) serverCfg, err := relayTLSConfig(ca.Issue(t, "spiffe://cluster.local/ns/ate-system/sa/atelet"), ca.TrustBundle, allowed) if err != nil { diff --git a/cmd/benchmarking/isolate/fake-workersync/controller.go b/cmd/benchmarking/isolate/fake-workersync/controller.go index b646fe1faf..55efbe09a2 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller.go @@ -61,7 +61,7 @@ type capacityRelay interface { Prune(live map[string]string) } -// node is a benchmark node and what it can allocate. +// node is a fake-atelet node and what it can allocate. type node struct { name string allocatable corev1.ResourceList @@ -71,7 +71,7 @@ type node struct { type cluster interface { // Pools returns every WorkerPool. Pools() ([]*atev1alpha1.WorkerPool, error) - // Nodes returns the benchmark nodes, sorted by name. + // Nodes returns the nodes that run a fake-atelet pod, sorted by name. Nodes(ctx context.Context) ([]node, error) // Relays returns the fake-atelet relay address on each node that has one. Relays(ctx context.Context) (map[string]string, error) @@ -94,7 +94,7 @@ type fakeWorker struct { node string // created is set once ate-api-server has the Worker. A Worker is tracked // from before its create, so a create the server commits but the caller - // sees fail is still retired, or deleted at shutdown. + // sees fail is still retired, or deleted by --cleanup. created bool // labels and sandboxClass are what the Worker is registered with. labels map[string]string @@ -141,13 +141,52 @@ func newController(client workerClient, relay capacityRelay, cl cluster, run str return &controller{client: client, relay: relay, cluster: cl, run: run, concurrency: concurrency, drainGrace: drainGrace, now: time.Now, newUID: uuid.NewString, workers: map[string]*fakeWorker{}} } +// adoptUntilServed runs adopt until it succeeds, waiting between tries from +// initial up to maxBackoff. fake-workersync is installed in the same apply as +// ate-api-server, so it retries the errors of a server that does not serve +// yet; any other error, or the end of ctx, ends it with that error. +func (c *controller) adoptUntilServed(ctx context.Context, initial, maxBackoff time.Duration) error { + backoff := initial + for { + err := c.adopt(ctx) + if err == nil || ctx.Err() != nil { + return err + } + switch status.Code(err) { + case codes.Unavailable, codes.DeadlineExceeded: + default: + return err + } + slog.WarnContext(ctx, "Failed to load registered fake Workers; retrying", slog.Any("err", err), slog.Duration("backoff", backoff)) + select { + case <-ctx.Done(): + return err + case <-time.After(backoff): + } + backoff = min(2*backoff, maxBackoff) + } +} + +// nodes returns the nodes the tracked Workers are on. +func (c *controller) nodes() []string { + c.mu.Lock() + defer c.mu.Unlock() + var out []string + for _, w := range c.workers { + if w.node != "" && !slices.Contains(out, w.node) { + out = append(out, w.node) + } + } + return out +} + // adopt loads the fake Workers of this run that are already registered, as // after a restart, so they are reconciled rather than created again. func (c *controller) adopt(ctx context.Context) error { prefix := fakeworker.Prefix(c.run) var token string for { - page, err := c.client.ListWorkers(ctx, &ateapipb.ListWorkersRequest{PageSize: 1000, PageToken: token}) + page, err := c.client.ListWorkers(ctx, &ateapipb.ListWorkersRequest{Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: token}}) if err != nil { return fmt.Errorf("while listing Workers: %w", err) } @@ -248,12 +287,12 @@ func (c *controller) reconcile(ctx context.Context) error { // A real pool replaces every worker pod when its sandbox class, limits or // actor count change, and each new pod reports its capacity once, so its // fake Workers are replaced (see fits). So is a Worker whose node has left - // the benchmark set, as a pod is when its node goes away; with no - // benchmark node listed, Workers are left where they are, since nothing + // the fake-atelet set, as a pod is when its node goes away; with no + // fake-atelet node listed, Workers are left where they are, since nothing // could replace them. A replaced pod gets a new Worker // in the same pass, while the old one drains, as a Deployment creates a // replacement pod without waiting for the old one to terminate. - onBenchmarkNode := func(w *fakeWorker) bool { + onFakeAteletNode := func(w *fakeWorker) bool { _, ok := allocatable[w.node] return ok || len(nodes) == 0 } @@ -266,7 +305,7 @@ func (c *controller) reconcile(ctx context.Context) error { continue } d, ok := desired[w.pod] - if ok && !w.draining && onBenchmarkNode(w) && fits(w, d.pool, allocatable) && current[w.pod] == nil { + if ok && !w.draining && onFakeAteletNode(w) && fits(w, d.pool, allocatable) && current[w.pod] == nil { current[w.pod] = w continue } @@ -299,7 +338,7 @@ func (c *controller) reconcile(ctx context.Context) error { _ = g.Wait() for _, wp := range pools { - if err := c.syncStatus(ctx, wp, onBenchmarkNode, allocatable); err != nil { + if err := c.syncStatus(ctx, wp, onFakeAteletNode, allocatable); err != nil { errs = append(errs, err) } } @@ -317,7 +356,7 @@ func (c *controller) reconcile(ctx context.Context) error { func (c *controller) ensure(ctx context.Context, pod string, d desiredWorker, w *fakeWorker, nodes []string, allocatable map[string]corev1.ResourceList, relays map[string]string) error { if w == nil { if len(nodes) == 0 { - return errors.New("no benchmark node to place fake Workers on") + return errors.New("no fake-atelet node to place fake Workers on") } w = &fakeWorker{ name: c.newUID(), @@ -348,7 +387,7 @@ func (c *controller) ensure(ctx context.Context, pod string, d desiredWorker, w if fixed == nil { alloc, ok := allocatable[w.node] if !ok { - return fmt.Errorf("node %s of Worker %s is not a benchmark node", w.node, name) + return fmt.Errorf("node %s of Worker %s is not a fake-atelet node", w.node, name) } fixed = t.capacity(alloc) } @@ -470,7 +509,7 @@ func (c *controller) retire(ctx context.Context, name string) error { w.drainedAt = c.now() c.mu.Unlock() } - assigned, err := c.client.ListWorkerActorAssignments(ctx, &ateapipb.ListWorkerActorAssignmentsRequest{Worker: ref, PageSize: 1}) + assigned, err := c.client.ListWorkerActorAssignments(ctx, &ateapipb.ListWorkerActorAssignmentsRequest{Worker: ref, Options: &ateapipb.ListOptions{PageSize: 1}}) if err != nil { if status.Code(err) == codes.NotFound { c.forget(name) @@ -497,16 +536,16 @@ func (c *controller) forget(name string) { // syncStatus writes the pool's status as the WorkerPool controller does from // its Deployment: a replica is a registered Worker, ready once its capacity // is reported, and the selector is the one worker pods would carry. -func (c *controller) syncStatus(ctx context.Context, wp *atev1alpha1.WorkerPool, onBenchmarkNode func(*fakeWorker) bool, allocatable map[string]corev1.ResourceList) error { +func (c *controller) syncStatus(ctx context.Context, wp *atev1alpha1.WorkerPool, onFakeAteletNode func(*fakeWorker) bool, allocatable map[string]corev1.ResourceList) error { want := atev1alpha1.WorkerPoolStatus{ Selector: labels.SelectorFromSet(labels.Set{fakeworker.WorkerPoolLabel: wp.Name}).String(), } c.mu.Lock() for _, w := range c.workers { // A Worker its pool no longer wants, made from an older template, or - // off the benchmark nodes is no replica even before its drain lands. + // off the fake-atelet nodes is no replica even before its drain lands. if w.namespace != wp.Namespace || w.pool != wp.Name || !w.created || w.draining || - w.index >= int(wp.Spec.Replicas) || !onBenchmarkNode(w) || !fits(w, wp, allocatable) { + w.index >= int(wp.Spec.Replicas) || !onFakeAteletNode(w) || !fits(w, wp, allocatable) { continue } want.Replicas++ @@ -526,9 +565,8 @@ func (c *controller) syncStatus(ctx context.Context, wp *atev1alpha1.WorkerPool, return nil } -// deleteAll deletes every fake Worker this run registered, as at shutdown, -// after the benchmark has deleted its actors. It keeps going past a failure -// and reports every one. +// deleteAll deletes every fake Worker this run registered, for --cleanup. It +// keeps going past a failure and reports every one. func (c *controller) deleteAll(ctx context.Context) error { c.mu.Lock() names := slices.Sorted(maps.Keys(c.workers)) diff --git a/cmd/benchmarking/isolate/fake-workersync/controller_test.go b/cmd/benchmarking/isolate/fake-workersync/controller_test.go index cf9493ed22..9b046bdb98 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller_test.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller_test.go @@ -54,6 +54,11 @@ type fakeControl struct { lostReply error // drainErr, when set, fails every DrainWorker. drainErr error + // listFailures fails that many ListWorkers calls with listCode before any + // succeeds; Unavailable, as while ate-api-server does not serve yet, when + // listCode is unset. + listFailures int + listCode codes.Code } func newFakeControl() *fakeControl { @@ -139,6 +144,13 @@ func (f *fakeControl) UpdateWorker(_ context.Context, in *ateapipb.UpdateWorkerR func (f *fakeControl) ListWorkers(_ context.Context, _ *ateapipb.ListWorkersRequest, _ ...grpc.CallOption) (*ateapipb.ListWorkersResponse, error) { f.mu.Lock() defer f.mu.Unlock() + if f.listFailures > 0 { + f.listFailures-- + if f.listCode != codes.OK { + return nil, status.Error(f.listCode, "refused") + } + return nil, status.Error(codes.Unavailable, "no ready endpoints") + } return &ateapipb.ListWorkersResponse{Workers: slices.Collect(maps.Values(f.workers))}, nil } @@ -527,7 +539,7 @@ func TestWorkerOnADepartedNodeIsReplaced(t *testing.T) { } // An empty node list replaces nothing: there is nowhere to put a replacement. -func TestNoBenchmarkNodeLeavesWorkersInPlace(t *testing.T) { +func TestNoFakeAteletNodeLeavesWorkersInPlace(t *testing.T) { c, ctl, _, cl := newTestController(pool("bench", 2, nil)) if err := reconcile(t, c); err != nil { t.Fatalf("reconcile: %v", err) @@ -996,6 +1008,58 @@ func TestAdoptAfterRestart(t *testing.T) { } } +// Installed with ate-api-server, fake-workersync starts before the server +// serves, and keeps trying to adopt rather than exiting. +func TestAdoptUntilServedRetries(t *testing.T) { + c, ctl, rel, cl := newTestController(pool("bench", 2, nil)) + if err := reconcile(t, c); err != nil { + t.Fatalf("reconcile: %v", err) + } + ctl.listFailures = 3 + + restarted := newController(ctl, rel, cl, testRun, 4, testDrainGrace) + if err := restarted.adoptUntilServed(context.Background(), time.Millisecond, 2*time.Millisecond); err != nil { + t.Fatalf("adoptUntilServed: %v", err) + } + if got := len(restarted.workers); got != 2 { + t.Errorf("adopted %d Workers, want 2", got) + } + if ctl.listFailures != 0 { + t.Errorf("%d ListWorkers failures left, want every one retried", ctl.listFailures) + } + // Each adopted Worker's node, so a restart keeps them in the node set. + if nodes := restarted.nodes(); len(nodes) == 0 { + t.Error("nodes() after adopt is empty, want the adopted Workers' nodes") + } +} + +// An error retrying cannot cure, such as a refused identity, ends adopt at +// once, so the pod fails visibly instead of looking healthy. +func TestAdoptUntilServedFailsOnAPermanentError(t *testing.T) { + c, ctl, _, _ := newTestController() + ctl.listFailures, ctl.listCode = 1<<30, codes.PermissionDenied + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + err := c.adoptUntilServed(ctx, time.Millisecond, 5*time.Millisecond) + if status.Code(err) != codes.PermissionDenied { + t.Fatalf("adoptUntilServed() = %v, want PermissionDenied", err) + } + if ctx.Err() != nil { + t.Error("adoptUntilServed retried a permanent error until its context ended") + } +} + +// A shutdown during the retries ends them with the last error. +func TestAdoptUntilServedStopsWithItsContext(t *testing.T) { + c, ctl, _, _ := newTestController() + ctl.listFailures = 1 << 30 + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + if err := c.adoptUntilServed(ctx, time.Millisecond, 5*time.Millisecond); err == nil { + t.Error("adoptUntilServed() = nil after its context ended, want the last error") + } +} + // A limits edit made while fake-workersync was down still replaces the // pool's Workers: an adopted Worker whose reported capacity is not what the // pool now gives is from an older template. diff --git a/cmd/benchmarking/isolate/fake-workersync/main.go b/cmd/benchmarking/isolate/fake-workersync/main.go index fa4a8ca762..915b0e3503 100644 --- a/cmd/benchmarking/isolate/fake-workersync/main.go +++ b/cmd/benchmarking/isolate/fake-workersync/main.go @@ -15,10 +15,12 @@ // Command fake-workersync stands in for atecontroller's WorkerPool // controller and worker syncer in control-plane benchmarks. It honors every // WorkerPool's spec.replicas with fake Workers that no pod backs, placed -// round robin on the nodes carrying --node-label, and has each Worker's +// round robin on the nodes that run a fake-atelet pod, and has each Worker's // capacity reported through the fake-atelet on its node, as ateom reports // through atelet. It writes each pool's status, so the scale subresource and -// `kubectl get workerpool` work, and deletes its Workers when it shuts down. +// `kubectl get workerpool` work. It leaves its Workers registered when it +// shuts down and adopts them when it starts, so a restart keeps every Actor +// placed; a deleted pool's Workers are retired while it runs. // // atecontroller must not run alongside it: its WorkerPool controller would // create real worker pods for the same pools, and its worker syncer deletes @@ -28,12 +30,17 @@ package main import ( "cmp" "context" + "errors" "fmt" "log/slog" + "maps" "net" + "net/http" "os/signal" "slices" "strconv" + "sync" + "sync/atomic" "syscall" "time" @@ -56,20 +63,20 @@ import ( ) var ( - apiEndpoint = pflag.String("api-endpoint", "k8s:///api.ate-system.svc.cluster.local:443", "ate-api-server gRPC dial target.") - run = pflag.String("run", "bench", "Run prefix of the fake Worker pod names.") - nodeLabel = pflag.String("node-label", "ate.dev/fake-data-plane=true", "Label selector for the benchmark nodes that run fake-atelet.") - ateletNS = pflag.String("atelet-namespace", "ate-system", "Namespace of the fake-atelet pods.") - ateletLabel = pflag.String("fake-atelet-label", "ate.dev/fake-atelet=true", "Label selector for the fake-atelet pods.") - relayPort = pflag.Int("relay-port", fakeworker.DefaultRelayPort, "Port fake-atelet serves capacity reports on.") - ateletID = pflag.String("atelet-spiffe-id", installdefaults.SPIFFEID("ate-system", "atelet"), "SPIFFE ID the fake-atelet relay must present.") - clientBundle = pflag.String("client-cred-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "Credential bundle presented to the relay.") - trustBundle = pflag.String("trust-bundle", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "Pod-identity trust bundle the relay's certificate must chain to.") - resync = pflag.Duration("resync", 10*time.Second, "Reconcile at least this often, besides on every WorkerPool change.") - concurrency = pflag.Int("concurrency", 32, "Worker calls in flight.") - cleanup = pflag.Bool("cleanup", false, "Delete this run's registered fake Workers, then exit.") - drainGrace = pflag.Duration("drain-grace", time.Minute, "How long a draining fake Worker waits for its Actors to leave before it is deleted, releasing them. Stands in for the time ateom takes to stop a terminating pod's Actors.") - deleteTimeout = pflag.Duration("delete-timeout", 5*time.Minute, "How long shutdown spends deleting the fake Workers.") + apiEndpoint = pflag.String("api-endpoint", "k8s:///api.ate-system.svc.cluster.local:443", "ate-api-server gRPC dial target.") + run = pflag.String("run", "bench", "Run prefix of the fake Worker pod names.") + ateletNS = pflag.String("atelet-namespace", "ate-system", "Namespace of the fake-atelet pods.") + ateletLabel = pflag.String("fake-atelet-label", "ate.dev/fake-atelet=true", "Label selector for the fake-atelet pods.") + relayPort = pflag.Int("relay-port", fakeworker.DefaultRelayPort, "Port fake-atelet serves capacity reports on.") + ateletID = pflag.String("atelet-spiffe-id", installdefaults.SPIFFEID("ate-system", "atelet"), "SPIFFE ID the fake-atelet relay must present.") + clientBundle = pflag.String("client-cred-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "Credential bundle presented to the relay.") + trustBundle = pflag.String("trust-bundle", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "Pod-identity trust bundle the relay's certificate must chain to.") + resync = pflag.Duration("resync", 10*time.Second, "Reconcile at least this often, besides on every WorkerPool change.") + concurrency = pflag.Int("concurrency", 32, "Worker calls in flight.") + cleanup = pflag.Bool("cleanup", false, "Delete this run's registered fake Workers, then exit. Stop the serving fake-workersync first, as by scaling its Deployment to zero: it would go on counting the deleted Workers until it restarts.") + nodeGrace = pflag.Duration("node-grace", 5*time.Minute, "How long a node keeps its fake Workers with no fake-atelet pod, as while its fake-atelet is replaced.") + healthAddr = pflag.String("health-listen-addr", ":9091", "Address to serve /healthz and /readyz on.") + drainGrace = pflag.Duration("drain-grace", time.Minute, "How long a draining fake Worker waits for its Actors to leave before it is deleted, releasing them. Stands in for the time ateom takes to stop a terminating pod's Actors.") showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") @@ -93,6 +100,9 @@ func main() { if *concurrency < 1 { serverboot.Fatal(ctx, "Invalid --concurrency", fmt.Errorf("must be at least 1, got %d", *concurrency)) } + if *nodeGrace < 0 { + serverboot.Fatal(ctx, "Invalid --node-grace", fmt.Errorf("must not be negative, got %s", *nodeGrace)) + } conn, control, err := boomerutil.DialControl(*apiEndpoint) if err != nil { @@ -138,16 +148,37 @@ func main() { kc: kc, ac: ac, pools: poolInformer.Lister(), - nodeLabel: *nodeLabel, ateletNS: *ateletNS, ateletLabel: *ateletLabel, relayPort: *relayPort, + nodeGrace: *nodeGrace, + now: time.Now, } c := newController(control, relay, cl, *run, *concurrency, *drainGrace) - if err := c.adopt(ctx); err != nil { + // --cleanup is a one-off run that nothing probes, so it serves no health + // endpoints. + var ready atomic.Bool + if !*cleanup { + health := &http.Server{Addr: *healthAddr, Handler: healthHandler(&ready), ReadHeaderTimeout: 10 * time.Second} + go func() { + if err := health.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + serverboot.Fatal(ctx, "Failed to serve health endpoints", err) + } + }() + defer func() { _ = health.Shutdown(context.Background()) }() + } + + if err := c.adoptUntilServed(ctx, time.Second, 30*time.Second); err != nil { + if ctx.Err() != nil { + return + } serverboot.Fatal(ctx, "Failed to load registered fake Workers", err) } + ready.Store(true) + // A node whose fake-atelet is being replaced has no pod right now; its + // adopted Workers keep it in the set. + cl.remember(c.nodes()) if *cleanup { if err := c.deleteAll(ctx); err != nil { serverboot.Fatal(ctx, "Failed to delete fake Workers", err) @@ -165,13 +196,6 @@ func main() { } select { case <-ctx.Done(): - // Shutdown runs after the benchmark has deleted its actors - // (teardown order), so no Actor is left on a deleted Worker. - delCtx, cancel := context.WithTimeout(context.Background(), *deleteTimeout) - defer cancel() - if err := c.deleteAll(delCtx); err != nil { - slog.ErrorContext(delCtx, "Failed to delete every fake Worker; run with --cleanup, or let the next atecontroller's startup sweep remove the rest", slog.Any("err", err)) - } return case <-changed: case <-ticker.C: @@ -179,29 +203,96 @@ func main() { } } +// healthHandler serves /healthz, 200 while the process runs, and /readyz, +// 200 once ready is set: after the registered fake Workers are adopted, so a +// rollout waits until ate-api-server has answered. +func healthHandler(ready *atomic.Bool) http.Handler { + mux := http.NewServeMux() + mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) }) + mux.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) { + if !ready.Load() { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + w.WriteHeader(http.StatusOK) + }) + return mux +} + // kubeCluster is the controller's view of Kubernetes. type kubeCluster struct { kc kubernetes.Interface ac versioned.Interface pools atelisters.WorkerPoolLister - nodeLabel string ateletNS string ateletLabel string relayPort int + + // nodeGrace is how long a node stays in the set with no fake-atelet pod. + nodeGrace time.Duration + // now is the clock nodeGrace is measured on. + now func() time.Time + + mu sync.Mutex + // seen holds when each node last had a fake-atelet pod scheduled to it. + seen map[string]time.Time } func (k *kubeCluster) Pools() ([]*atev1alpha1.WorkerPool, error) { return k.pools.List(labels.Everything()) } +// remember adds nodes to the set Nodes draws on, as if a fake-atelet pod had +// just been scheduled to each. +func (k *kubeCluster) remember(nodes []string) { + k.mu.Lock() + defer k.mu.Unlock() + if k.seen == nil { + k.seen = map[string]time.Time{} + } + now := k.now() + for _, n := range nodes { + k.seen[n] = now + } +} + +// Nodes returns the nodes that still exist and have had a fake-atelet pod +// scheduled to them, ready or not, within nodeGrace. A node stays in the set +// while its fake-atelet is replaced, as a DaemonSet rollout does, so its +// Workers wait for the relay rather than moving; one that stops running +// fake-atelet for good leaves it, and its Workers move. func (k *kubeCluster) Nodes(ctx context.Context) ([]node, error) { - list, err := k.kc.CoreV1().Nodes().List(ctx, metav1.ListOptions{LabelSelector: k.nodeLabel}) + pods, err := k.kc.CoreV1().Pods(k.ateletNS).List(ctx, metav1.ListOptions{LabelSelector: k.ateletLabel}) + if err != nil { + return nil, err + } + list, err := k.kc.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) if err != nil { return nil, err } - nodes := make([]node, 0, len(list.Items)) + k.mu.Lock() + defer k.mu.Unlock() + if k.seen == nil { + k.seen = map[string]time.Time{} + } + now := k.now() + for _, p := range pods.Items { + if p.Spec.NodeName != "" { + k.seen[p.Spec.NodeName] = now + } + } + exists := map[string]bool{} for _, n := range list.Items { - nodes = append(nodes, node{name: n.Name, allocatable: n.Status.Allocatable}) + exists[n.Name] = true + } + maps.DeleteFunc(k.seen, func(name string, at time.Time) bool { + return !exists[name] || now.Sub(at) > k.nodeGrace + }) + nodes := make([]node, 0, len(k.seen)) + for _, n := range list.Items { + if _, ok := k.seen[n.Name]; ok { + nodes = append(nodes, node{name: n.Name, allocatable: n.Status.Allocatable}) + } } slices.SortFunc(nodes, func(a, b node) int { return cmp.Compare(a.name, b.name) }) return nodes, nil diff --git a/cmd/benchmarking/isolate/fake-workersync/main_test.go b/cmd/benchmarking/isolate/fake-workersync/main_test.go new file mode 100644 index 0000000000..c3d96543d1 --- /dev/null +++ b/cmd/benchmarking/isolate/fake-workersync/main_test.go @@ -0,0 +1,178 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "net/http" + "net/http/httptest" + "slices" + "sync/atomic" + "testing" + "time" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" +) + +var fakeAteletLabels = map[string]string{"app": "atelet", "ate.dev/fake-atelet": "true"} + +func testNode(name string) *corev1.Node { + return &corev1.Node{ + ObjectMeta: metav1.ObjectMeta{Name: name}, + Status: corev1.NodeStatus{Allocatable: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("8Gi")}}, + } +} + +func testPod(name, namespace, nodeName string, labels map[string]string) *corev1.Pod { + return &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace, Labels: labels}, + Spec: corev1.PodSpec{NodeName: nodeName}, + } +} + +// testKubeCluster returns a kubeCluster on kc with a five-minute node grace, +// and a function that advances its clock. +func testKubeCluster(kc *fake.Clientset) (*kubeCluster, func(time.Duration)) { + now := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC) + cl := &kubeCluster{ + kc: kc, ateletNS: "ate-system", ateletLabel: "ate.dev/fake-atelet=true", + nodeGrace: 5 * time.Minute, now: func() time.Time { return now }, + } + return cl, func(d time.Duration) { now = now.Add(d) } +} + +func nodeNames(t *testing.T, cl *kubeCluster) []string { + t.Helper() + nodes, err := cl.Nodes(t.Context()) + if err != nil { + t.Fatalf("Nodes() error = %v", err) + } + var names []string + for _, n := range nodes { + names = append(names, n.name) + if n.allocatable.Memory().IsZero() { + t.Errorf("node %s has no allocatable memory", n.name) + } + } + return names +} + +// fake-workersync is not ready until it has adopted the registered Workers, +// so the install's rollout wait covers ate-api-server answering it. +func TestHealthHandlerReadyAfterAdopt(t *testing.T) { + var ready atomic.Bool + h := healthHandler(&ready) + get := func(path string) int { + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, path, nil)) + return rec.Code + } + if got := get("/healthz"); got != http.StatusOK { + t.Errorf("/healthz = %d, want 200", got) + } + if got := get("/readyz"); got != http.StatusServiceUnavailable { + t.Errorf("/readyz before adopt = %d, want 503", got) + } + ready.Store(true) + if got := get("/readyz"); got != http.StatusOK { + t.Errorf("/readyz after adopt = %d, want 200", got) + } +} + +// Workers go only to nodes that run a fake-atelet pod: a node without one, +// such as a tainted control plane node, has no relay to report through. +func TestKubeClusterNodesAreTheFakeAteletNodes(t *testing.T) { + kc := fake.NewClientset( + testNode("node-b"), testNode("node-a"), testNode("control-plane"), testNode("unscheduled"), testNode("other-ns"), + testPod("fake-atelet-b", "ate-system", "node-b", fakeAteletLabels), + testPod("fake-atelet-a", "ate-system", "node-a", fakeAteletLabels), + testPod("fake-atelet-pending", "ate-system", "", fakeAteletLabels), + testPod("not-fake", "ate-system", "control-plane", map[string]string{"app": "atenet-router"}), + testPod("fake-atelet-elsewhere", "default", "other-ns", fakeAteletLabels), + ) + cl, _ := testKubeCluster(kc) + + if got, want := nodeNames(t, cl), []string{"node-a", "node-b"}; !slices.Equal(got, want) { + t.Errorf("Nodes() = %v, want %v", got, want) + } +} + +// After a restart, a node whose fake-atelet is being replaced has no pod; the +// adopted Workers' nodes keep it in the set. +func TestKubeClusterRemembersAdoptedNodes(t *testing.T) { + kc := fake.NewClientset( + testNode("node-a"), testNode("node-b"), testNode("gone-later"), + testPod("fake-atelet-a", "ate-system", "node-a", fakeAteletLabels), + ) + cl, _ := testKubeCluster(kc) + cl.remember([]string{"node-b", "missing"}) + if got, want := nodeNames(t, cl), []string{"node-a", "node-b"}; !slices.Equal(got, want) { + t.Errorf("Nodes() = %v, want %v", got, want) + } +} + +// A DaemonSet rollout leaves a node without a fake-atelet pod for a moment; +// the node stays, so its Workers are not moved. A deleted node goes. +func TestKubeClusterNodesSurviveAFakeAteletReplacement(t *testing.T) { + kc := fake.NewClientset( + testNode("node-a"), testNode("node-b"), + testPod("fake-atelet-a", "ate-system", "node-a", fakeAteletLabels), + testPod("fake-atelet-b", "ate-system", "node-b", fakeAteletLabels), + ) + cl, _ := testKubeCluster(kc) + if got, want := nodeNames(t, cl), []string{"node-a", "node-b"}; !slices.Equal(got, want) { + t.Fatalf("Nodes() = %v, want %v", got, want) + } + + if err := kc.CoreV1().Pods("ate-system").Delete(t.Context(), "fake-atelet-a", metav1.DeleteOptions{}); err != nil { + t.Fatal(err) + } + if got, want := nodeNames(t, cl), []string{"node-a", "node-b"}; !slices.Equal(got, want) { + t.Errorf("Nodes() during node-a's fake-atelet replacement = %v, want %v", got, want) + } + + if err := kc.CoreV1().Nodes().Delete(t.Context(), "node-b", metav1.DeleteOptions{}); err != nil { + t.Fatal(err) + } + if got, want := nodeNames(t, cl), []string{"node-a"}; !slices.Equal(got, want) { + t.Errorf("Nodes() after node-b is deleted = %v, want %v", got, want) + } +} + +// A node that stops running fake-atelet for good leaves the set once the +// grace has passed, so its Workers move to a node with a relay. +func TestKubeClusterNodesExpireWithoutAFakeAtelet(t *testing.T) { + kc := fake.NewClientset( + testNode("node-a"), testNode("node-b"), + testPod("fake-atelet-a", "ate-system", "node-a", fakeAteletLabels), + testPod("fake-atelet-b", "ate-system", "node-b", fakeAteletLabels), + ) + cl, advance := testKubeCluster(kc) + nodeNames(t, cl) + if err := kc.CoreV1().Pods("ate-system").Delete(t.Context(), "fake-atelet-b", metav1.DeleteOptions{}); err != nil { + t.Fatal(err) + } + + advance(4 * time.Minute) + if got, want := nodeNames(t, cl), []string{"node-a", "node-b"}; !slices.Equal(got, want) { + t.Errorf("Nodes() within the grace = %v, want %v", got, want) + } + advance(2 * time.Minute) + if got, want := nodeNames(t, cl), []string{"node-a"}; !slices.Equal(got, want) { + t.Errorf("Nodes() past the grace = %v, want %v", got, want) + } +} diff --git a/cmd/benchmarking/isolate/fake-workersync/relayclient_test.go b/cmd/benchmarking/isolate/fake-workersync/relayclient_test.go index 72c17a79e0..f0bdffc756 100644 --- a/cmd/benchmarking/isolate/fake-workersync/relayclient_test.go +++ b/cmd/benchmarking/isolate/fake-workersync/relayclient_test.go @@ -39,7 +39,7 @@ import ( const ( testAteletID = "spiffe://cluster.local/ns/ate-system/sa/atelet" - testWorkerSyncID = "spiffe://cluster.local/ns/benchmark-workloads/sa/fake-workersync" + testWorkerSyncID = "spiffe://cluster.local/ns/ate-system/sa/fake-workersync" ) // stubRelay is a stand-in fake-atelet relay: it records each report and diff --git a/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go b/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go index bc00b4a3f6..5df7beff47 100644 --- a/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go +++ b/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go @@ -109,7 +109,7 @@ func TestValidateRun(t *testing.T) { } func TestVerifyPeerID(t *testing.T) { - const want = "spiffe://cluster.local/ns/benchmark-workloads/sa/fake-workersync" + const want = "spiffe://cluster.local/ns/ate-system/sa/fake-workersync" state := func(path string) tls.ConnectionState { cert := &x509.Certificate{} if path != "" { @@ -117,7 +117,7 @@ func TestVerifyPeerID(t *testing.T) { } return tls.ConnectionState{PeerCertificates: []*x509.Certificate{cert}} } - if err := VerifyPeerID(state("/ns/benchmark-workloads/sa/fake-workersync"), want); err != nil { + if err := VerifyPeerID(state("/ns/ate-system/sa/fake-workersync"), want); err != nil { t.Errorf("VerifyPeerID(matching) = %v", err) } if err := VerifyPeerID(state("/ns/ate-system/sa/atelet"), want); err == nil { diff --git a/cmd/credential-provider/kubernetes-secrets/main.go b/cmd/credential-provider/kubernetes-secrets/main.go index 9ccf2c099c..53c7416d88 100644 --- a/cmd/credential-provider/kubernetes-secrets/main.go +++ b/cmd/credential-provider/kubernetes-secrets/main.go @@ -52,7 +52,8 @@ const serviceName = "credprovider" var ( listenAddr = pflag.String("listen-address", ":50051", "gRPC listen address") - metricsAddr = pflag.String("metrics-address", ":9090", "Prometheus/health HTTP listen address") + metricsAddr = pflag.String("metrics-address", ":9090", "Prometheus /metrics HTTP listen address") + healthAddr = pflag.String("health-address", ":9091", "/readyz and /healthz HTTP listen address") statusAddr = pflag.String("status-address", ":4040", "/statusz HTTP listen address; empty disables the page") serverBundle = pflag.String("server-cred-bundle", "", "credential bundle (PEM key+chain) presented for serving TLS (required)") clientCAFile = pflag.String("client-ca-file", "", "CA bundle that caller (injector) client certificates must chain to (required)") @@ -90,12 +91,7 @@ func run(ctx context.Context) error { } defer serverboot.ShutdownProvider("MeterProvider", mp.Shutdown) - readiness := &serverboot.Readiness{} - go serverboot.StartMetricsServer(ctx, serverboot.MetricsServerOptions{ - Addr: *metricsAddr, - Readiness: readiness, - EnableHealthz: true, - }) + go serverboot.StartMetricsServer(ctx, *metricsAddr) client, err := newKubeClient() if err != nil { @@ -131,6 +127,11 @@ func run(ctx context.Context) error { return fmt.Errorf("listen on %s: %w", *listenAddr, err) } + // The health server starts after the gRPC listener opens, so /readyz does + // not report 200 before the pod can accept calls. + readiness := &serverboot.Readiness{} + go serverboot.StartHealthServer(ctx, *healthAddr, readiness) + shutdownCtx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM) defer stop() diff --git a/cmd/credential-provider/kubernetes-secrets/manifests_test.go b/cmd/credential-provider/kubernetes-secrets/manifests_test.go index f5153fabb9..ea99acd9b9 100644 --- a/cmd/credential-provider/kubernetes-secrets/manifests_test.go +++ b/cmd/credential-provider/kubernetes-secrets/manifests_test.go @@ -161,7 +161,7 @@ func TestProviderManifests(t *testing.T) { } args := strings.Join(container.Args, " ") for _, required := range []string{ - "--listen-address=:50051", "--metrics-address=:9090", + "--listen-address=:50051", "--metrics-address=:9090", "--health-address=:9091", "--injector-identity=spiffe://cluster.local/ns/" + tc.namespace + "/sa/" + tc.prefix + "atenet-egress", "--server-cred-bundle=/run/servicedns.podcert.ate.dev/credential-bundle.pem", "--client-ca-file=/run/podidentity.podcert.ate.dev/trust-bundle.pem", @@ -173,9 +173,15 @@ func TestProviderManifests(t *testing.T) { t.Errorf("provider missing %s", required) } } - if container.ReadinessProbe == nil || container.ReadinessProbe.HTTPGet.Port.StrVal != "metrics" { + if container.ReadinessProbe == nil || container.ReadinessProbe.HTTPGet == nil || + container.ReadinessProbe.HTTPGet.Port.StrVal != "health" || container.ReadinessProbe.HTTPGet.Path != "/readyz" { t.Fatal("missing dedicated readiness probe") } + if !slices.ContainsFunc(container.Ports, func(port corev1.ContainerPort) bool { + return port.Name == "health" && port.ContainerPort == 9091 + }) { + t.Fatal("missing health listener port") + } } case "Service": if doc.Metadata.Name != tc.prefix+"k8s-credential-provider" { diff --git a/cmd/dataplane/envoy/Dockerfile b/cmd/dataplane/envoy/Dockerfile index 9de411563c..f133873e9e 100644 --- a/cmd/dataplane/envoy/Dockerfile +++ b/cmd/dataplane/envoy/Dockerfile @@ -34,7 +34,7 @@ RUN cargo build --release --locked # Pin the multi-platform image index, not one platform's manifest, so that the # --platform the installer passes (the host architecture on kind, linux/amd64 # otherwise) selects a matching Envoy binary. -FROM envoyproxy/envoy:v1.39.1@sha256:57e14a549d7bd43c8d3f6d03e8cfa653e037d4b38e133acd9b54f38c524401b4 +FROM envoyproxy/envoy:v1.39.4@sha256:74bd4eb5629708d3fadca1a5e6b85d208f338d7bb54e06b69565905c5294cc01 # Copy the built dynamic module shared library from the builder stage COPY --from=builder /usr/src/dynamic-modules/egress-policy/target/release/libenvoy_substrate_egress_policy.so /usr/local/lib/libenvoy_substrate_egress_policy.so diff --git a/cmd/dataplane/envoy/README.md b/cmd/dataplane/envoy/README.md index 23727118fc..a3ffb8602e 100644 --- a/cmd/dataplane/envoy/README.md +++ b/cmd/dataplane/envoy/README.md @@ -13,7 +13,7 @@ cmd/dataplane/envoy/ - **`Dockerfile`**: Multi-stage container build that: 1. Compiles the Rust dynamic module (`envoy-substrate-egress-policy`) into a shared library (`libenvoy_substrate_egress_policy.so`) in a `rust:bookworm` builder stage. - 2. Packages the compiled `.so` into the `envoyproxy/envoy:v1.39-latest` runtime image under `/usr/local/lib/libenvoy_substrate_egress_policy.so` and sets `ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/usr/local/lib`. + 2. Packages the compiled `.so` into the `envoyproxy/envoy:v1.39.4` runtime image under `/usr/local/lib/libenvoy_substrate_egress_policy.so` and sets `ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/usr/local/lib`. - **`dynamic-modules/egress-policy/`**: A Rust crate using the Envoy Dynamic Modules SDK (`envoy-proxy-dynamic-modules-rust-sdk`) that implements a custom Envoy listener filter for Substrate egress policy evaluation. See [`dynamic-modules/egress-policy/README.md`](dynamic-modules/egress-policy/README.md) for module-specific build, test, and Envoy configuration details. ## Building and Deployment diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index e9ab9dbb8a..ad484fddca 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -69,6 +69,9 @@ These flags can be appended to any command: | `--token-file` | | Path to a bearer token for ate-api authentication, or `-` for stdin | Kubernetes ServiceAccount token | | `--output` | `-o` | Output format (`table`, `json`, `yaml`) | `table` | | `--trace` | | Enable on-demand tracing for the request | `false` | +| `--verbose` | `-v` | Increase output verbosity; may be repeated or stacked (e.g., `-v`, `-vv`, `-vvvvv`) | `0` | + +Pass `--version` (`kubectl ate --version`) to print the CLI version and exit. --- @@ -109,7 +112,7 @@ kubectl ate get workers -l | `ATESPACE` | The atespace the actor belongs to. Part of the actor's identity; folded into the storage key as `actor::`. | | `NAME` | The actor's name. User-provided for application actors; UUID for the golden actor that each template materialises while building its golden tag. | | `TEMPLATE` | The `ActorTemplate` the actor was created from, displayed as `/`. | -| `STATE` | Current lifecycle state (`ACTOR_STATE_RESUMING`, `ACTOR_STATE_RUNNING`, `ACTOR_STATE_SUSPENDING`, `ACTOR_STATE_SUSPENDED`, `ACTOR_STATE_PAUSING`, `ACTOR_STATE_PAUSED`, `ACTOR_STATE_CRASHED`, `ACTOR_STATE_DELETING`, `ACTOR_STATE_REVERTING`). | +| `STATE` | Current lifecycle state (`ACTOR_STATE_RESUMING`, `ACTOR_STATE_RUNNING`, `ACTOR_STATE_SUSPENDING`, `ACTOR_STATE_SUSPENDED`, `ACTOR_STATE_CRASHED`, `ACTOR_STATE_DELETING`, `ACTOR_STATE_REVERTING`). | | `WORKER POD` | The worker pod (namespace/name) currently hosting the actor. Empty while suspended. | | `WORKER IP` | The pod IP of that worker. Empty while suspended. | | `VERSION` | Monotonic integer that increments on every state transition (resume / suspend / checkpoint). Useful for distinguishing snapshots. | @@ -208,7 +211,7 @@ kubectl ate revert actor my-actor -a # Delete an actor (by default, requires the actor to be SUSPENDED or CRASHED). kubectl ate delete actor my-actor -a -# Delete an actor from any state (e.g. RUNNING, PAUSED), terminating workloads and detaching volumes. +# Delete an actor from any state (e.g. RUNNING), terminating workloads and detaching volumes. kubectl ate delete actor my-actor -a --any-state ``` diff --git a/cmd/kubectl-ate/internal/cmd/actor.go b/cmd/kubectl-ate/internal/cmd/actor.go index 5cb3bacebf..7be822eb0d 100644 --- a/cmd/kubectl-ate/internal/cmd/actor.go +++ b/cmd/kubectl-ate/internal/cmd/actor.go @@ -29,7 +29,6 @@ import ( "github.com/agent-substrate/substrate/cmd/kubectl-ate/internal/printer" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/spf13/cobra" @@ -84,7 +83,7 @@ var getActorsCmd = &cobra.Command{ } ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -110,9 +109,8 @@ var getActorsCmd = &cobra.Command{ pageToken := "" for { resp, err := apiClient.ListActors(ctx, &ateapipb.ListActorsRequest{ - PageSize: 1000, - PageToken: pageToken, - Atespace: getActorAtespaceFlag, + Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: pageToken}, + Atespace: getActorAtespaceFlag, }) if err != nil { return fmt.Errorf("failed to list actors: %w", err) @@ -140,7 +138,7 @@ var createActorCmd = &cobra.Command{ } ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -161,7 +159,7 @@ var deleteActorCmd = &cobra.Command{ Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - c, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + c, err := newAPIClient(cmd) if err != nil { return err } @@ -187,7 +185,7 @@ var pauseActorCmd = &cobra.Command{ Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -211,7 +209,7 @@ var resumeActorCmd = &cobra.Command{ Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -235,7 +233,7 @@ var suspendActorCmd = &cobra.Command{ Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -366,7 +364,7 @@ func (r *LogsActorRunner) runOneShot(ctx context.Context) error { scanner.Buffer(buf, 1024*1024) // Support up to 1MB lines for scanner.Scan() { line := scanner.Text() - filterAndDisplayLogLine(line, r.filter, r.stdout) + filterAndDisplayLogLine(line, r.filter, r.stdout, r.stderr) } if err := scanner.Err(); err != nil { return fmt.Errorf("error reading log stream: %w", err) @@ -379,6 +377,8 @@ func (r *LogsActorRunner) runFollow(ctx context.Context) error { // lastSeenTime is the kubelet's timestamp on the last line read from // lastWorkerPod, the resume cursor for a reconnect to it. var lastSeenTime time.Time + var waitingLogged bool + var lastWaitingState ateapipb.ActorState for { select { @@ -392,6 +392,7 @@ func (r *LogsActorRunner) runFollow(ctx context.Context) error { if status.Code(err) == codes.NotFound { return fmt.Errorf("actor %s not found: %w", r.actorRef, err) } + vlogf(r.stderr, vBasic, "Failed to get actor %s: %v; retrying in %s", r.actorRef, err, r.pollInterval) select { case <-ctx.Done(): return ctx.Err() @@ -402,8 +403,14 @@ func (r *LogsActorRunner) runFollow(ctx context.Context) error { podName := actor.GetStatus().GetWorkerAssignment().GetWorkerPod() namespace := actor.GetStatus().GetWorkerAssignment().GetWorkerNamespace() + state := actor.GetStatus().GetState() - if podName == "" || namespace == "" || actor.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING { + if podName == "" || namespace == "" || state != ateapipb.ActorState_ACTOR_STATE_RUNNING { + if !waitingLogged || state != lastWaitingState { + vlogf(r.stderr, vBasic, "Actor %s is not running (state=%s); waiting for worker assignment...", r.actorRef, state) + waitingLogged = true + lastWaitingState = state + } select { case <-ctx.Done(): return ctx.Err() @@ -411,6 +418,7 @@ func (r *LogsActorRunner) runFollow(ctx context.Context) error { continue } } + waitingLogged = false // The actor resumed on another worker. switchedPod := podName != lastWorkerPod @@ -436,6 +444,7 @@ func (r *LogsActorRunner) runFollow(ctx context.Context) error { stream, err := r.streamer.StreamLogs(streamCtx, namespace, podName, opts) if err != nil { streamCancel() + vlogf(r.stderr, vBasic, "Failed to stream logs from pod %s/%s: %v; retrying in %s", namespace, podName, err, r.pollInterval) select { case <-ctx.Done(): return ctx.Err() @@ -461,8 +470,10 @@ func (r *LogsActorRunner) runFollow(ctx context.Context) error { continue } lastSeenTime = stamp + } else { + vlogf(r.stderr, vTrace, "Log line had no kubelet timestamp prefix") } - filterAndDisplayLogLine(line, r.filter, r.stdout) + filterAndDisplayLogLine(line, r.filter, r.stdout, r.stderr) } scanErr := scanner.Err() stream.Close() @@ -508,13 +519,19 @@ func (r *LogsActorRunner) startMigrationMonitor( return case <-ticker.C: resp, err := r.apiClient.GetActor(ctx, &ateapipb.GetActorRequest{Actor: r.actorRef.ToObjectRef()}) - if err == nil { - act := resp - if act.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING || act.GetStatus().GetWorkerAssignment().GetWorkerPod() != currentPod { - // Actor suspended or migrated! Cancel stream context to reconnect. - cancel() - return + if err != nil { + if ctx.Err() == nil { + vlogf(r.stderr, vBasic, "Migration monitor failed to get actor %s: %v", r.actorRef, err) } + continue + } + act := resp + if act.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING || act.GetStatus().GetWorkerAssignment().GetWorkerPod() != currentPod { + // Actor suspended or migrated! Cancel stream context to reconnect. + vlogf(r.stderr, vBasic, "Actor %s left pod %s (state=%s, pod=%s); closing current log stream", + r.actorRef, currentPod, act.GetStatus().GetState(), act.GetStatus().GetWorkerAssignment().GetWorkerPod()) + cancel() + return } } } @@ -548,12 +565,12 @@ func runLogsActor(cmd *cobra.Command, args []string) error { return err } - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } - config, err := ateclient.LoadKubeConfig(kubeconfig, k8sContext) + config, err := loadKubeConfig(cmd.ErrOrStderr(), kubeconfig, k8sContext) if err != nil { apiClient.Close() return fmt.Errorf("while loading kubeconfig: %w", err) @@ -645,7 +662,7 @@ var containerNameKeyQuoted = `"` + string(ateattr.ActorContainerNameKey) + `"` // filterAndDisplayLogLine writes line to w if filter selects it, and // reports whether it did. -func filterAndDisplayLogLine(line string, filter logLineFilter, w io.Writer) bool { +func filterAndDisplayLogLine(line string, filter logLineFilter, w, stderr io.Writer) bool { // Under lifecycle, nearly every line is a container's; a busy actor // writes many, so skip them before decoding. if filter.source == logSourceLifecycle && strings.Contains(line, containerNameKeyQuoted) { @@ -656,6 +673,7 @@ func filterAndDisplayLogLine(line string, filter logLineFilter, w io.Writer) boo dec := json.NewDecoder(strings.NewReader(line)) dec.UseNumber() if err := dec.Decode(&m); err != nil { + vlogf(stderr, vTrace, "Skipping non-JSON log line: %v", err) return false } diff --git a/cmd/kubectl-ate/internal/cmd/actor_test.go b/cmd/kubectl-ate/internal/cmd/actor_test.go index 2c3f36cf4a..a34b982773 100644 --- a/cmd/kubectl-ate/internal/cmd/actor_test.go +++ b/cmd/kubectl-ate/internal/cmd/actor_test.go @@ -318,17 +318,21 @@ func TestFilterAndDisplayLogLine(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - var buf bytes.Buffer - matched := filterAndDisplayLogLine(tc.line, logLineFilter{target: tc.target, source: logSource(tc.source), container: tc.container}, &buf) + var outBuf bytes.Buffer + var errBuf bytes.Buffer + matched := filterAndDisplayLogLine(tc.line, logLineFilter{target: tc.target, source: logSource(tc.source), container: tc.container}, &outBuf, &errBuf) if matched != tc.wantMatched { t.Errorf("got matched = %v, want %v", matched, tc.wantMatched) } - gotOutput := strings.TrimSpace(buf.String()) + gotOutput := strings.TrimSpace(outBuf.String()) if gotOutput != tc.wantOutput { t.Errorf("got output %q, want %q", gotOutput, tc.wantOutput) } + if errBuf.Len() != 0 { + t.Errorf("got unexpected error output %q", errBuf.String()) + } }) } } diff --git a/cmd/kubectl-ate/internal/cmd/actortemplate.go b/cmd/kubectl-ate/internal/cmd/actortemplate.go index 98b19f76d7..2c4205ea0f 100644 --- a/cmd/kubectl-ate/internal/cmd/actortemplate.go +++ b/cmd/kubectl-ate/internal/cmd/actortemplate.go @@ -20,7 +20,6 @@ import ( "os" "github.com/agent-substrate/substrate/cmd/kubectl-ate/internal/printer" - "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/spf13/cobra" "google.golang.org/protobuf/encoding/protojson" @@ -61,7 +60,7 @@ var getActorTemplatesCmd = &cobra.Command{ } ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -88,9 +87,8 @@ var getActorTemplatesCmd = &cobra.Command{ pageToken := "" for { resp, err := apiClient.ListActorTemplates(ctx, &ateapipb.ListActorTemplatesRequest{ - PageSize: 100, - PageToken: pageToken, - Atespace: getActorTemplateAtespaceFlag, + Options: &ateapipb.ListOptions{PageSize: 100, PageToken: pageToken}, + Atespace: getActorTemplateAtespaceFlag, }) if err != nil { return fmt.Errorf("failed to list actor templates: %w", err) @@ -131,7 +129,7 @@ Actor templates are immutable: there is no update; delete and recreate to change } ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -154,7 +152,7 @@ The server also deletes the template's golden actor and golden snapshot.`, Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - c, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + c, err := newAPIClient(cmd) if err != nil { return err } diff --git a/cmd/kubectl-ate/internal/cmd/actortemplate_test.go b/cmd/kubectl-ate/internal/cmd/actortemplate_test.go index b4ffe73c56..2ea0c8b234 100644 --- a/cmd/kubectl-ate/internal/cmd/actortemplate_test.go +++ b/cmd/kubectl-ate/internal/cmd/actortemplate_test.go @@ -80,8 +80,8 @@ func TestActorTemplateFromManifest(t *testing.T) { Name: "counter", Image: "ko://github.com/agent-substrate/substrate/demos/counter", Command: []string{"/ko-app/counter", "--extra-port=9090"}, - WakeupProbe: &ateapipb.ContainerWakeupProbe{ - HttpGet: &ateapipb.HTTPGetAction{Path: "/readyz", Port: 80}, + WakeupProbe: &ateapipb.WakeupProbe{ + HttpGet: &ateapipb.HttpGetAction{Path: "/readyz", Port: 80}, }, VolumeMounts: []*ateapipb.VolumeMount{{Name: "data", MountPath: "/home/counter"}}, }}, diff --git a/cmd/kubectl-ate/internal/cmd/admin.go b/cmd/kubectl-ate/internal/cmd/admin.go index ab27bcfbb6..af30acbdc6 100644 --- a/cmd/kubectl-ate/internal/cmd/admin.go +++ b/cmd/kubectl-ate/internal/cmd/admin.go @@ -21,7 +21,6 @@ import ( "text/tabwriter" "time" - "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/internal/localca" "github.com/agent-substrate/substrate/internal/localjwtauthority" "github.com/spf13/cobra" @@ -54,7 +53,7 @@ var makeCaPoolCmd = &cobra.Command{ RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - kc, err := newKubeClient() + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { return err } @@ -125,7 +124,7 @@ var makeJwtPoolCmd = &cobra.Command{ RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - kc, err := newKubeClient() + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { return err } @@ -149,7 +148,7 @@ var listJwtKeysCmd = &cobra.Command{ Use: "list-jwt-keys", Short: "List the keys in a JWT authority pool secret", RunE: func(cmd *cobra.Command, args []string) error { - kc, err := newKubeClient() + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { return err } @@ -171,7 +170,7 @@ The key is published to relying parties but signs nothing until activate-jwt-key makes it the signing key. Activate it only once relying parties have refetched the key set.`, RunE: func(cmd *cobra.Command, args []string) error { - kc, err := newKubeClient() + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { return err } @@ -180,7 +179,7 @@ the key set.`, if err != nil { return fmt.Errorf("while generating JWT authority: %w", err) } - if err := updateJWTPool(cmd.Context(), kc.CoreV1().Secrets(poolSecretNamespaceFlag), poolSecretNameFlag, func(pool *localjwtauthority.ConcretePool) error { + if err := updateJWTPool(cmd.Context(), cmd.ErrOrStderr(), kc.CoreV1().Secrets(poolSecretNamespaceFlag), poolSecretNameFlag, func(pool *localjwtauthority.ConcretePool) error { return pool.AddAuthority(authority) }); err != nil { return err @@ -195,12 +194,12 @@ var activateJwtKeyCmd = &cobra.Command{ Use: "activate-jwt-key", Short: "Make a key in a JWT authority pool secret the one that signs", RunE: func(cmd *cobra.Command, args []string) error { - kc, err := newKubeClient() + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { return err } - if err := updateJWTPool(cmd.Context(), kc.CoreV1().Secrets(poolSecretNamespaceFlag), poolSecretNameFlag, func(pool *localjwtauthority.ConcretePool) error { + if err := updateJWTPool(cmd.Context(), cmd.ErrOrStderr(), kc.CoreV1().Secrets(poolSecretNamespaceFlag), poolSecretNameFlag, func(pool *localjwtauthority.ConcretePool) error { return pool.Activate(jwtKeyIDFlag) }); err != nil { return err @@ -219,12 +218,12 @@ var removeJwtKeyCmd = &cobra.Command{ Tokens the key signed stop verifying once relying parties refetch the key set, so remove it only after the last of them has expired.`, RunE: func(cmd *cobra.Command, args []string) error { - kc, err := newKubeClient() + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { return err } - if err := updateJWTPool(cmd.Context(), kc.CoreV1().Secrets(poolSecretNamespaceFlag), poolSecretNameFlag, func(pool *localjwtauthority.ConcretePool) error { + if err := updateJWTPool(cmd.Context(), cmd.ErrOrStderr(), kc.CoreV1().Secrets(poolSecretNamespaceFlag), poolSecretNameFlag, func(pool *localjwtauthority.ConcretePool) error { return pool.RemoveAuthority(jwtKeyIDFlag) }); err != nil { return err @@ -238,8 +237,13 @@ so remove it only after the last of them has expired.`, // updateJWTPool applies change to the pool in the named secret and writes it // back. The write is conditional on the secret's resourceVersion, and a // conflict reruns change against the current pool. -func updateJWTPool(ctx context.Context, secrets typedcorev1.SecretInterface, name string, change func(*localjwtauthority.ConcretePool) error) error { +func updateJWTPool(ctx context.Context, errOut io.Writer, secrets typedcorev1.SecretInterface, name string, change func(*localjwtauthority.ConcretePool) error) error { + attempt := 0 return retry.RetryOnConflict(retry.DefaultRetry, func() error { + attempt++ + if attempt > 1 { + vlogf(errOut, vBasic, "Conflict updating secret %q (attempt %d); retrying...", name, attempt) + } secret, pool, err := getJWTPool(ctx, secrets, name) if err != nil { return err @@ -294,8 +298,8 @@ func printJWTKeys(out io.Writer, pool *localjwtauthority.ConcretePool) error { } // newKubeClient builds a client from the --kubeconfig and --context flags. -func newKubeClient() (kubernetes.Interface, error) { - kconfig, err := ateclient.LoadKubeConfig(kubeconfig, k8sContext) +func newKubeClient(errOut io.Writer) (kubernetes.Interface, error) { + kconfig, err := loadKubeConfig(errOut, kubeconfig, k8sContext) if err != nil { return nil, fmt.Errorf("while reading kubeconfig: %w", err) } diff --git a/cmd/kubectl-ate/internal/cmd/admin_test.go b/cmd/kubectl-ate/internal/cmd/admin_test.go index 143adab735..a7a5fe681e 100644 --- a/cmd/kubectl-ate/internal/cmd/admin_test.go +++ b/cmd/kubectl-ate/internal/cmd/admin_test.go @@ -16,6 +16,7 @@ package cmd import ( "context" + "io" "strings" "testing" @@ -128,7 +129,7 @@ func TestUpdateJWTPoolRotation(t *testing.T) { {"activate", func(p *localjwtauthority.ConcretePool) error { return p.Activate("next") }, []string{"old", "next"}, "next"}, {"remove", func(p *localjwtauthority.ConcretePool) error { return p.RemoveAuthority("old") }, []string{"next"}, "next"}, } { - if err := updateJWTPool(ctx, secrets, "actor-id-jwt-pool", step.change); err != nil { + if err := updateJWTPool(ctx, io.Discard, secrets, "actor-id-jwt-pool", step.change); err != nil { t.Fatalf("%s: %v", step.name, err) } ids, active := poolState(t, kc) @@ -139,6 +140,7 @@ func TestUpdateJWTPoolRotation(t *testing.T) { } func TestUpdateJWTPoolRetriesConflict(t *testing.T) { + setVerbosity(t, vBasic) kc := newPoolClientset(t) conflicted := false kc.PrependReactor("update", "secrets", func(k8stesting.Action) (bool, runtime.Object, error) { @@ -175,7 +177,8 @@ func TestUpdateJWTPoolRetriesConflict(t *testing.T) { if err != nil { t.Fatal(err) } - if err := updateJWTPool(context.Background(), kc.CoreV1().Secrets("ate-system"), "actor-id-jwt-pool", func(p *localjwtauthority.ConcretePool) error { + var errOut strings.Builder + if err := updateJWTPool(context.Background(), &errOut, kc.CoreV1().Secrets("ate-system"), "actor-id-jwt-pool", func(p *localjwtauthority.ConcretePool) error { return p.AddAuthority(next) }); err != nil { t.Fatalf("updateJWTPool: %v", err) @@ -184,6 +187,10 @@ func TestUpdateJWTPoolRetriesConflict(t *testing.T) { if diff := cmp.Diff([]string{"old", "other", "next"}, ids); diff != "" { t.Errorf("key IDs (-want +got):\n%s", diff) } + wantLog := "Conflict updating secret \"actor-id-jwt-pool\" (attempt 2); retrying...\n" + if got := errOut.String(); got != wantLog { + t.Errorf("stderr = %q, want %q", got, wantLog) + } } func TestUpdateJWTPoolLeavesSecretOnError(t *testing.T) { @@ -202,7 +209,7 @@ func TestUpdateJWTPoolLeavesSecretOnError(t *testing.T) { if tc.secret != nil { kc = fake.NewClientset(tc.secret) } - err := updateJWTPool(context.Background(), kc.CoreV1().Secrets("ate-system"), "actor-id-jwt-pool", func(p *localjwtauthority.ConcretePool) error { + err := updateJWTPool(context.Background(), io.Discard, kc.CoreV1().Secrets("ate-system"), "actor-id-jwt-pool", func(p *localjwtauthority.ConcretePool) error { return p.RemoveAuthority("old") }) if err == nil { @@ -219,7 +226,7 @@ func TestUpdateJWTPoolLeavesSecretOnError(t *testing.T) { func TestUpdateJWTPoolMissingSecret(t *testing.T) { kc := fake.NewClientset() - err := updateJWTPool(context.Background(), kc.CoreV1().Secrets("ate-system"), "actor-id-jwt-pool", func(*localjwtauthority.ConcretePool) error { + err := updateJWTPool(context.Background(), io.Discard, kc.CoreV1().Secrets("ate-system"), "actor-id-jwt-pool", func(*localjwtauthority.ConcretePool) error { t.Error("change ran without a pool") return nil }) @@ -235,7 +242,7 @@ func TestListJWTKeys(t *testing.T) { if err != nil { t.Fatal(err) } - if err := updateJWTPool(context.Background(), secrets, "actor-id-jwt-pool", func(p *localjwtauthority.ConcretePool) error { + if err := updateJWTPool(context.Background(), io.Discard, secrets, "actor-id-jwt-pool", func(p *localjwtauthority.ConcretePool) error { return p.AddAuthority(next) }); err != nil { t.Fatal(err) diff --git a/cmd/kubectl-ate/internal/cmd/atespace.go b/cmd/kubectl-ate/internal/cmd/atespace.go index 37cedad0ff..0427c02dbf 100644 --- a/cmd/kubectl-ate/internal/cmd/atespace.go +++ b/cmd/kubectl-ate/internal/cmd/atespace.go @@ -18,7 +18,6 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/kubectl-ate/internal/printer" - "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/spf13/cobra" ) @@ -29,7 +28,7 @@ var getAtespacesCmd = &cobra.Command{ Short: "List all atespaces or get one or more atespaces", RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -54,8 +53,7 @@ var getAtespacesCmd = &cobra.Command{ pageToken := "" for { resp, err := apiClient.ListAtespaces(ctx, &ateapipb.ListAtespacesRequest{ - PageSize: 1000, - PageToken: pageToken, + Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: pageToken}, }) if err != nil { return fmt.Errorf("failed to list atespaces: %w", err) @@ -77,7 +75,7 @@ var createAtespaceCmd = &cobra.Command{ Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -104,7 +102,7 @@ var deleteAtespaceCmd = &cobra.Command{ Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } diff --git a/cmd/kubectl-ate/internal/cmd/egress_policy.go b/cmd/kubectl-ate/internal/cmd/egress_policy.go index 0657125407..b80171dcc1 100644 --- a/cmd/kubectl-ate/internal/cmd/egress_policy.go +++ b/cmd/kubectl-ate/internal/cmd/egress_policy.go @@ -22,7 +22,6 @@ import ( "io" "github.com/agent-substrate/substrate/cmd/kubectl-ate/internal/printer" - "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/spf13/cobra" @@ -213,7 +212,7 @@ func (r *getEgressPolicyRunner) Run(ctx context.Context) error { func runGetEgressPolicy(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -261,7 +260,7 @@ func runCreateEgressPolicy(cmd *cobra.Command, args []string) error { } ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -337,7 +336,7 @@ func runUpdateEgressPolicy(cmd *cobra.Command, args []string) error { } ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -420,7 +419,7 @@ func runDeleteEgressPolicy(cmd *cobra.Command, args []string) error { return err } - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } diff --git a/cmd/kubectl-ate/internal/cmd/egress_policy_test.go b/cmd/kubectl-ate/internal/cmd/egress_policy_test.go index d736701d1d..6871aa919a 100644 --- a/cmd/kubectl-ate/internal/cmd/egress_policy_test.go +++ b/cmd/kubectl-ate/internal/cmd/egress_policy_test.go @@ -39,9 +39,9 @@ func TestEgressPolicyFromManifest(t *testing.T) { t.Parallel() fullMetadata := &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default"} - httpRules := []*ateapipb.EgressRule{{Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}}}} - tlsRules := []*ateapipb.EgressRule{{TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: []string{"*.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}} - anyRules := []*ateapipb.EgressRule{{Http: &ateapipb.HTTPRule{Hostnames: []string{"*"}}}} + httpRules := []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}}}} + tlsRules := []*ateapipb.EgressRule{{TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: []string{"*.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}}}} + anyRules := []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{Hostnames: []string{"*"}}}} tests := []struct { name string @@ -283,7 +283,7 @@ rules: func TestOverrideEgressPolicyMetadata(t *testing.T) { t.Parallel() - rules := []*ateapipb.EgressRule{{Http: &ateapipb.HTTPRule{Hostnames: []string{"*"}}}} + rules := []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{Hostnames: []string{"*"}}}} filled := &ateapipb.EgressPolicy{Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default"}, Rules: rules} pinned := &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default", Uid: "3f2b1c0e-8d5a-4b6e-9c1d-2a7e4f6b8c0d", Version: 2} @@ -365,11 +365,11 @@ func TestEgressPolicyManifest_RoundTrip(t *testing.T) { UpdateTime: timestamppb.New(time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)), }, Rules: []*ateapipb.EgressRule{ - {Http: &ateapipb.HTTPRule{Hostnames: []string{"*.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80, 8080}}}}, - {Https: &ateapipb.HTTPSRule{Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{ + {Http: &ateapipb.HttpRule{Hostnames: []string{"*.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{80, 8080}}}}, + {Https: &ateapipb.HttpsRule{Hostnames: []string{"api.example.com"}, Effects: &ateapipb.HttpRuleEffects{ ReplaceHeaders: []*ateapipb.CredentialHeader{{Header: "authorization", Prefix: "Bearer ", CredentialUri: "ate-secret://k8s/default/token"}}, }}}, - {TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, + {TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, }, } @@ -496,7 +496,7 @@ func TestGetEgressPolicyRunner_Run(t *testing.T) { Version: 1, CreateTime: timestamppb.New(now.Add(-5 * time.Minute)), // table row prints AGE 5m }, - Rules: []*ateapipb.EgressRule{{Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}}}}, + Rules: []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}}}}, } tests := []struct { @@ -623,7 +623,7 @@ func TestCreateEgressPolicyRunner_Run(t *testing.T) { pinTime(t, now) actor := &ateapipb.ObjectRef{Atespace: "team-a", Name: "c1"} - rules := []*ateapipb.EgressRule{{Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}}}} + rules := []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}}}} // A manifest cloned from another actor still carries that actor's // server-managed fields; the CLI sends them as is and the server scrubs them. manifest := &ateapipb.EgressPolicy{ @@ -739,8 +739,8 @@ func TestUpdateEgressPolicyRunner_Run(t *testing.T) { actor := &ateapipb.ObjectRef{Atespace: "team-a", Name: "c1"} const uid = "3f2b1c0e-8d5a-4b6e-9c1d-2a7e4f6b8c0d" rules := []*ateapipb.EgressRule{ - {Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}}}, - {Https: &ateapipb.HTTPSRule{Hostnames: []string{"www.example.com"}}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}}}, + {Https: &ateapipb.HttpsRule{Hostnames: []string{"www.example.com"}}}, } // The manifest is what `get -o yaml` printed, edited: it still carries the // uid, version, and timestamps of the policy being replaced. @@ -967,7 +967,7 @@ func TestDeleteEgressPolicyRunner_Run(t *testing.T) { actor := &ateapipb.ObjectRef{Atespace: "team-a", Name: "c1"} deleted := &ateapipb.EgressPolicy{ Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default", Version: 1}, - Rules: []*ateapipb.EgressRule{{Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}}}}, + Rules: []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}}}}, } unguarded := &ateapipb.DeleteActorEgressPolicyRequest{Actor: actor, Name: resources.SingletonName} wantActorReq := &ateapipb.GetActorRequest{Actor: actor} @@ -1103,19 +1103,19 @@ func TestEgressPolicyFromManifest_NighthawkEgress(t *testing.T) { }, Rules: []*ateapipb.EgressRule{ { - Http: &ateapipb.HTTPRule{ + Http: &ateapipb.HttpRule{ Hostnames: []string{"*"}, Ports: &ateapipb.Ports{Numbers: []int32{80}}, }, }, { - Https: &ateapipb.HTTPSRule{ + Https: &ateapipb.HttpsRule{ Hostnames: []string{"*"}, Ports: &ateapipb.Ports{Numbers: []int32{443}}, }, }, { - TlsPassthrough: &ateapipb.TLSPassthroughRule{ + TlsPassthrough: &ateapipb.TlsPassthroughRule{ Hostnames: []string{"*"}, Ports: &ateapipb.Ports{Numbers: []int32{8443}}, }, diff --git a/cmd/kubectl-ate/internal/cmd/revert_actor.go b/cmd/kubectl-ate/internal/cmd/revert_actor.go index 79c7df933b..c419330bf4 100644 --- a/cmd/kubectl-ate/internal/cmd/revert_actor.go +++ b/cmd/kubectl-ate/internal/cmd/revert_actor.go @@ -18,7 +18,6 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/kubectl-ate/internal/printer" - "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/spf13/cobra" @@ -36,7 +35,7 @@ from it. Only crashed, running, or paused actors can be reverted.`, Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } diff --git a/cmd/kubectl-ate/internal/cmd/root.go b/cmd/kubectl-ate/internal/cmd/root.go index 744d44b7be..592e466a84 100644 --- a/cmd/kubectl-ate/internal/cmd/root.go +++ b/cmd/kubectl-ate/internal/cmd/root.go @@ -30,6 +30,7 @@ var ( tokenFile string outputFmt string traceEnabled bool + verbosity int ) var rootCmd = &cobra.Command{ @@ -60,4 +61,9 @@ func init() { rootCmd.PersistentFlags().StringVar(&tokenFile, "token-file", "", "Path to a bearer token for ate-api authentication, or - to read it from stdin. Defaults to a Kubernetes ServiceAccount token.") rootCmd.PersistentFlags().StringVarP(&outputFmt, "output", "o", "table", "Output format. One of: table|json|yaml") rootCmd.PersistentFlags().BoolVar(&traceEnabled, "trace", false, "Enable tracing for the request") + rootCmd.PersistentFlags().CountVarP(&verbosity, "verbose", "v", "Increase output verbosity; may be repeated (e.g. -v, -vv, -vvvvv)") + + // Register --version without a -v shorthand so -v stays free for --verbose. + rootCmd.Flags().Bool("version", false, "version for "+rootCmd.DisplayName()) + _ = rootCmd.Flags().SetAnnotation("version", cobra.FlagSetByCobraAnnotation, []string{"true"}) } diff --git a/cmd/kubectl-ate/internal/cmd/root_test.go b/cmd/kubectl-ate/internal/cmd/root_test.go new file mode 100644 index 0000000000..336d1962c6 --- /dev/null +++ b/cmd/kubectl-ate/internal/cmd/root_test.go @@ -0,0 +1,103 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cmd + +import ( + "bytes" + "strings" + "testing" +) + +func resetRootFlags(t *testing.T) { + t.Helper() + rootCmd.SetArgs(nil) + rootCmd.SetOut(nil) + rootCmd.SetErr(nil) + if f := rootCmd.Flags().Lookup("version"); f != nil { + _ = f.Value.Set("false") + f.Changed = false + } + if f := rootCmd.PersistentFlags().Lookup("verbose"); f != nil { + _ = f.Value.Set("0") + f.Changed = false + } + verbosity = 0 +} + +func TestVersionFlag(t *testing.T) { + t.Cleanup(func() { resetRootFlags(t) }) + + var out, errOut bytes.Buffer + rootCmd.SetOut(&out) + rootCmd.SetErr(&errOut) + rootCmd.SetArgs([]string{"--version"}) + + if err := rootCmd.Execute(); err != nil { + t.Fatalf("Execute(--version) unexpected error: %v", err) + } + if !strings.HasPrefix(out.String(), "kubectl-ate version ") { + t.Errorf("Execute(--version) output = %q, want prefix %q", out.String(), "kubectl-ate version ") + } + + if f := rootCmd.Flags().Lookup("version"); f != nil && f.Shorthand != "" { + t.Errorf("--version shorthand = %q, want empty", f.Shorthand) + } + + resetRootFlags(t) + out.Reset() + errOut.Reset() + rootCmd.SetOut(&out) + rootCmd.SetErr(&errOut) + rootCmd.SetArgs([]string{"-v"}) + if err := rootCmd.Execute(); err != nil { + t.Fatalf("Execute(-v) unexpected error: %v", err) + } + if strings.HasPrefix(out.String(), "kubectl-ate version ") { + t.Errorf("Execute(-v) printed version %q, want help output", out.String()) + } +} + +func TestVerboseFlagCounting(t *testing.T) { + t.Cleanup(func() { resetRootFlags(t) }) + + tests := []struct { + name string + args []string + want int + }{ + {name: "default", args: nil, want: 0}, + {name: "single short", args: []string{"-v"}, want: 1}, + {name: "stacked short 2", args: []string{"-vv"}, want: 2}, + {name: "stacked short 3", args: []string{"-vvv"}, want: 3}, + {name: "stacked short 5", args: []string{"-vvvvv"}, want: 5}, + {name: "repeated short", args: []string{"-v", "-v", "-v"}, want: 3}, + {name: "single long", args: []string{"--verbose"}, want: 1}, + {name: "repeated long", args: []string{"--verbose", "--verbose"}, want: 2}, + {name: "mixed short and long", args: []string{"-vv", "--verbose"}, want: 3}, + {name: "explicit value", args: []string{"--verbose=4"}, want: 4}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + resetRootFlags(t) + if err := rootCmd.PersistentFlags().Parse(tc.args); err != nil { + t.Fatalf("Parse(%v) unexpected error: %v", tc.args, err) + } + if verbosity != tc.want { + t.Errorf("verbosity = %d, want %d", verbosity, tc.want) + } + }) + } +} diff --git a/cmd/kubectl-ate/internal/cmd/tag.go b/cmd/kubectl-ate/internal/cmd/tag.go index 00270a0fc9..cd3337320b 100644 --- a/cmd/kubectl-ate/internal/cmd/tag.go +++ b/cmd/kubectl-ate/internal/cmd/tag.go @@ -20,7 +20,6 @@ import ( "strings" "github.com/agent-substrate/substrate/cmd/kubectl-ate/internal/printer" - "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/spf13/cobra" "google.golang.org/grpc" @@ -53,7 +52,7 @@ var getTagsCmd = &cobra.Command{ } ctx := cmd.Context() - client, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + client, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -79,7 +78,7 @@ var getTagsCmd = &cobra.Command{ var tags []*ateapipb.Tag pageToken := "" for { - resp, err := client.ListTags(ctx, &ateapipb.ListTagsRequest{Atespace: getTagAtespaceFlag, PageSize: 1000, PageToken: pageToken}) + resp, err := client.ListTags(ctx, &ateapipb.ListTagsRequest{Atespace: getTagAtespaceFlag, Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: pageToken}}) if err != nil { return fmt.Errorf("failed to list tags: %w", err) } @@ -106,7 +105,7 @@ var createTagCmd = &cobra.Command{ return err } ctx := cmd.Context() - client, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + client, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -136,7 +135,7 @@ var updateTagCmd = &cobra.Command{ return err } ctx := cmd.Context() - client, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + client, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -157,7 +156,7 @@ var deleteTagCmd = &cobra.Command{ Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - client, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + client, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } diff --git a/cmd/kubectl-ate/internal/cmd/verbose.go b/cmd/kubectl-ate/internal/cmd/verbose.go new file mode 100644 index 0000000000..0a718a6218 --- /dev/null +++ b/cmd/kubectl-ate/internal/cmd/verbose.go @@ -0,0 +1,145 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cmd + +import ( + "context" + "fmt" + "io" + "os" + "strings" + "sync" + "time" + + "github.com/agent-substrate/substrate/internal/ateclient" + "github.com/spf13/cobra" + "google.golang.org/grpc" + "google.golang.org/protobuf/encoding/protojson" + "google.golang.org/protobuf/proto" + "k8s.io/client-go/rest" +) + +// Verbosity levels for -v / --verbose. +const ( + vBasic = 1 // -v: high-level info, non-fatal errors, retries, and waits + vFlow = 2 // -vv: step-by-step execution flow + vTrace = 5 // -vvvvv: execution debugging and tracing +) + +// slowOpThreshold is the duration above which an operation is reported at +// vBasic (-v) even if its normal completion log is at vFlow (-vv). +const slowOpThreshold = 2 * time.Second + +var vlock sync.Mutex + +// vlogf writes a formatted diagnostic line to w (or os.Stderr if w is nil) +// when the configured verbosity is at least level. +func vlogf(w io.Writer, level int, format string, args ...any) { + if verbosity < level { + return + } + if w == nil { + w = os.Stderr + } + vlock.Lock() + defer vlock.Unlock() + fmt.Fprintf(w, strings.TrimSuffix(format, "\n")+"\n", args...) +} + +// loadKubeConfig loads the kubeconfig and logs the resolved API server +// when verbosity is enabled. +func loadKubeConfig(errOut io.Writer, kubeconfigPath, contextName string) (*rest.Config, error) { + vlogf(errOut, vFlow, "Loading kubeconfig (path=%q, context=%q)", kubeconfigPath, contextName) + cfg, err := ateclient.LoadKubeConfig(kubeconfigPath, contextName) + if err != nil { + return nil, err + } + vlogf(errOut, vTrace, "Loaded kubeconfig for Kubernetes API server %s", cfg.Host) + return cfg, nil +} + +// newAPIClient connects to ate-api-server with a verbosity-aware gRPC unary +// interceptor attached. +func newAPIClient(cmd *cobra.Command) (*ateclient.Client, error) { + ctx := cmd.Context() + errOut := cmd.ErrOrStderr() + + if verbosity >= vFlow { + vlogf(errOut, vFlow, "Connecting to ate-api-server (endpoint=%q, kubeconfig=%q, context=%q, token-file=%q, trace=%t)", + endpoint, kubeconfig, k8sContext, tokenFile, traceEnabled) + } else if endpoint != "" { + vlogf(errOut, vBasic, "Connecting to ate-api-server at %s", endpoint) + } else { + vlogf(errOut, vBasic, "Connecting to ate-api-server via port-forward") + } + + start := time.Now() + cli, err := ateclient.NewClient( + ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled, + grpc.WithChainUnaryInterceptor(newLoggingUnaryInterceptor(errOut)), + ) + elapsed := time.Since(start).Round(time.Millisecond) + if err != nil { + vlogf(errOut, vBasic, "Connection to ate-api-server failed after %s: %v", elapsed, err) + return nil, err + } + + if elapsed >= slowOpThreshold { + vlogf(errOut, vBasic, "Connected to ate-api-server in %s", elapsed) + } else { + vlogf(errOut, vFlow, "Connected to ate-api-server in %s", elapsed) + } + + return cli, nil +} + +func formatProtoCompact(v any) string { + msg, ok := v.(proto.Message) + if !ok || msg == nil { + return "" + } + b, err := (protojson.MarshalOptions{Multiline: false}).Marshal(msg) + if err != nil { + return fmt.Sprintf("", err) + } + return string(b) +} + +// newLoggingUnaryInterceptor returns a gRPC unary client interceptor that logs +// RPC calls, durations, errors, and (at vTrace) request/response payloads. +func newLoggingUnaryInterceptor(errOut io.Writer) grpc.UnaryClientInterceptor { + return func(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error { + if verbosity >= vTrace { + vlogf(errOut, vTrace, "gRPC %s request: %s", method, formatProtoCompact(req)) + } else { + vlogf(errOut, vFlow, "gRPC %s", method) + } + start := time.Now() + err := invoker(ctx, method, req, reply, cc, opts...) + elapsed := time.Since(start).Round(time.Millisecond) + if err != nil { + vlogf(errOut, vBasic, "gRPC %s failed after %s: %v", method, elapsed, err) + return err + } + if verbosity >= vTrace { + vlogf(errOut, vTrace, "gRPC %s response (%s): %s", method, elapsed, formatProtoCompact(reply)) + } else if elapsed >= slowOpThreshold { + vlogf(errOut, vBasic, "gRPC %s completed in %s", method, elapsed) + } else { + vlogf(errOut, vFlow, "gRPC %s completed in %s", method, elapsed) + } + return nil + } +} diff --git a/cmd/kubectl-ate/internal/cmd/verbose_test.go b/cmd/kubectl-ate/internal/cmd/verbose_test.go new file mode 100644 index 0000000000..a39d87f38b --- /dev/null +++ b/cmd/kubectl-ate/internal/cmd/verbose_test.go @@ -0,0 +1,289 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cmd + +import ( + "bytes" + "context" + "errors" + "io" + "strings" + "sync" + "testing" + "time" + + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/grpc" + "google.golang.org/protobuf/proto" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + metricsv1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1" +) + +func setVerbosity(t *testing.T, v int) { + t.Helper() + prev := verbosity + verbosity = v + t.Cleanup(func() { verbosity = prev }) +} + +func TestVlogfLevels(t *testing.T) { + tests := []struct { + level int + wantBasic bool + wantFlow bool + wantTrace bool + }{ + {level: 0, wantBasic: false, wantFlow: false, wantTrace: false}, + {level: vBasic, wantBasic: true, wantFlow: false, wantTrace: false}, + {level: vFlow, wantBasic: true, wantFlow: true, wantTrace: false}, + {level: vTrace, wantBasic: true, wantFlow: true, wantTrace: true}, + } + + for _, tc := range tests { + setVerbosity(t, tc.level) + var buf bytes.Buffer + vlogf(&buf, vBasic, "basic-msg") + vlogf(&buf, vFlow, "flow-msg") + vlogf(&buf, vTrace, "trace-msg") + got := buf.String() + + if strings.Contains(got, "basic-msg") != tc.wantBasic { + t.Errorf("verbosity=%d: basic-msg present=%t, want %t (got %q)", tc.level, !tc.wantBasic, tc.wantBasic, got) + } + if strings.Contains(got, "flow-msg") != tc.wantFlow { + t.Errorf("verbosity=%d: flow-msg present=%t, want %t (got %q)", tc.level, !tc.wantFlow, tc.wantFlow, got) + } + if strings.Contains(got, "trace-msg") != tc.wantTrace { + t.Errorf("verbosity=%d: trace-msg present=%t, want %t (got %q)", tc.level, !tc.wantTrace, tc.wantTrace, got) + } + } +} + +func TestLoggingUnaryInterceptor(t *testing.T) { + req := &ateapipb.GetActorRequest{Actor: &ateapipb.ObjectRef{Atespace: "demo", Name: "act-1"}} + wantActor := &ateapipb.Actor{Metadata: &ateapipb.ResourceMetadata{Atespace: "demo", Name: "act-1"}} + const method = ateapipb.Control_GetActor_FullMethodName + + okInvoker := func(_ context.Context, _ string, _, reply any, _ *grpc.ClientConn, _ ...grpc.CallOption) error { + proto.Merge(reply.(proto.Message), wantActor) + return nil + } + errInvoker := func(_ context.Context, _ string, _, _ any, _ *grpc.ClientConn, _ ...grpc.CallOption) error { + return errors.New("boom") + } + + t.Run("vBasic silent on fast success, logs failure", func(t *testing.T) { + setVerbosity(t, vBasic) + var errOut bytes.Buffer + ic := newLoggingUnaryInterceptor(&errOut) + + reply := &ateapipb.Actor{} + if err := ic(context.Background(), method, req, reply, nil, okInvoker); err != nil { + t.Fatalf("interceptor unexpected error: %v", err) + } + if errOut.Len() != 0 { + t.Errorf("vBasic fast success wrote %q, want empty", errOut.String()) + } + + _ = ic(context.Background(), method, req, reply, nil, errInvoker) + if !strings.Contains(errOut.String(), "gRPC /ateapi.Control/GetActor failed") { + t.Errorf("vBasic failure stderr = %q, want 'gRPC /ateapi.Control/GetActor failed'", errOut.String()) + } + }) + + t.Run("vFlow logs RPC call and completion", func(t *testing.T) { + setVerbosity(t, vFlow) + var errOut bytes.Buffer + ic := newLoggingUnaryInterceptor(&errOut) + + reply := &ateapipb.Actor{} + if err := ic(context.Background(), method, req, reply, nil, okInvoker); err != nil { + t.Fatalf("interceptor unexpected error: %v", err) + } + got := errOut.String() + if !strings.Contains(got, "gRPC /ateapi.Control/GetActor\n") || !strings.Contains(got, "gRPC /ateapi.Control/GetActor completed in ") { + t.Errorf("vFlow stderr = %q, want call and completion lines", got) + } + }) + + t.Run("vTrace logs request and response payloads", func(t *testing.T) { + setVerbosity(t, vTrace) + var errOut bytes.Buffer + ic := newLoggingUnaryInterceptor(&errOut) + + reply := &ateapipb.Actor{} + if err := ic(context.Background(), method, req, reply, nil, okInvoker); err != nil { + t.Fatalf("interceptor unexpected error: %v", err) + } + got := errOut.String() + if !strings.Contains(got, "gRPC /ateapi.Control/GetActor request:") || !strings.Contains(got, `"act-1"`) || !strings.Contains(got, "gRPC /ateapi.Control/GetActor response") { + t.Errorf("vTrace stderr = %q, want request and response payloads", got) + } + }) +} + +func TestLogsActorRunner_FollowVerboseSurfacesDelaysAndErrors(t *testing.T) { + setVerbosity(t, vBasic) + + var mu sync.Mutex + var getCalls int + mockAPI := &mockAteAPIClient{ + GetActorFunc: func(ctx context.Context, in *ateapipb.GetActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) { + mu.Lock() + defer mu.Unlock() + getCalls++ + switch getCalls { + case 1: + return nil, errors.New("transient apiserver error") + case 2: + return &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Name: "act-1"}, + Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, + }, nil + default: + return &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Name: "act-1"}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + WorkerAssignment: &ateapipb.WorkerAssignment{ + WorkerPod: "pod-1", + WorkerNamespace: "ns-1", + }, + }, + }, nil + } + }, + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + var streamCalls int + mockStreamer := &mockPodLogsStreamer{ + StreamLogsFunc: func(streamCtx context.Context, ns, name string, opts *corev1.PodLogOptions) (io.ReadCloser, error) { + mu.Lock() + streamCalls++ + call := streamCalls + mu.Unlock() + if call == 1 { + return nil, errors.New("kubelet stream unavailable") + } + cancel() + return io.NopCloser(strings.NewReader("")), nil + }, + } + + var stdout, stderr bytes.Buffer + runner := &LogsActorRunner{ + apiClient: mockAPI, + actorRef: resources.ActorRef{Atespace: "space-1", Name: "act-1"}, + filter: logLineFilter{target: resources.ActorRef{Atespace: "space-1", Name: "act-1"}, source: logSourceAll}, + streamer: mockStreamer, + stdout: &stdout, + stderr: &stderr, + follow: true, + pollInterval: 1 * time.Millisecond, + reconnectInterval: 1 * time.Millisecond, + tickerInterval: 50 * time.Millisecond, + } + + _ = runner.Run(ctx) + gotStderr := stderr.String() + + for _, want := range []string{ + "Failed to get actor space-1/act-1: transient apiserver error", + "Actor space-1/act-1 is not running (state=ACTOR_STATE_SUSPENDED); waiting for worker assignment...", + "Failed to stream logs from pod ns-1/pod-1: kubelet stream unavailable", + } { + if !strings.Contains(gotStderr, want) { + t.Errorf("stderr = %q, want substring %q", gotStderr, want) + } + } +} + +func TestTopWorkersRunner_VerboseSurfacesMetricsIssues(t *testing.T) { + workers := []*ateapipb.Worker{ + { + Metadata: &ateapipb.ResourceMetadata{Name: "w-1"}, + WorkerNamespace: "ns-1", + WorkerPool: "pool-1", + WorkerPod: "pod-1", + }, + { + Metadata: &ateapipb.ResourceMetadata{Name: "w-2"}, + WorkerNamespace: "ns-1", + WorkerPool: "pool-1", + WorkerPod: "pod-2", + }, + } + + t.Run("vBasic surfaces ListPodMetrics failure", func(t *testing.T) { + setVerbosity(t, vBasic) + var out, errOut bytes.Buffer + runner := &TopWorkersRunner{ + workerLister: &mockWorkerLister{workers: workers}, + podMetricsLister: &mockPodMetricsLister{err: errors.New("metrics API down")}, + outputFmt: "table", + out: &out, + stderr: &errOut, + } + if err := runner.Run(context.Background()); err != nil { + t.Fatalf("Run() error: %v", err) + } + if !strings.Contains(errOut.String(), "Failed to list pod metrics (CPU/memory will be unavailable): metrics API down") { + t.Errorf("stderr = %q, want ListPodMetrics error", errOut.String()) + } + }) + + t.Run("vFlow surfaces missing pod metrics and container fallback", func(t *testing.T) { + setVerbosity(t, vFlow) + var out, errOut bytes.Buffer + runner := &TopWorkersRunner{ + workerLister: &mockWorkerLister{workers: workers}, + podMetricsLister: &mockPodMetricsLister{metrics: []metricsv1beta1.PodMetrics{ + { + ObjectMeta: metav1.ObjectMeta{Namespace: "ns-1", Name: "pod-1"}, + Containers: []metricsv1beta1.ContainerMetrics{ + { + Name: "sidecar", + Usage: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("100m"), + corev1.ResourceMemory: resource.MustParse("32Mi"), + }, + }, + }, + }, + }}, + outputFmt: "table", + out: &out, + stderr: &errOut, + } + if err := runner.Run(context.Background()); err != nil { + t.Fatalf("Run() error: %v", err) + } + got := errOut.String() + for _, want := range []string{ + `Container "ateom" not found in pod metrics for ns-1/pod-1; summing all 1 container(s)`, + "No pod metrics returned for worker pod ns-1/pod-2", + } { + if !strings.Contains(got, want) { + t.Errorf("stderr = %q, want substring %q", got, want) + } + } + }) +} diff --git a/cmd/kubectl-ate/internal/cmd/worker.go b/cmd/kubectl-ate/internal/cmd/worker.go index 14798778ae..e06d82d5b4 100644 --- a/cmd/kubectl-ate/internal/cmd/worker.go +++ b/cmd/kubectl-ate/internal/cmd/worker.go @@ -82,9 +82,8 @@ func workersHostingAtespace(ctx context.Context, lister ActorLister, atespace st pageToken := "" for { resp, err := lister.ListActors(ctx, &ateapipb.ListActorsRequest{ - PageSize: 1000, - PageToken: pageToken, - Atespace: atespace, + Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: pageToken}, + Atespace: atespace, }) if err != nil { return nil, fmt.Errorf("failed to list actors in atespace %q: %w", atespace, err) @@ -107,8 +106,7 @@ func listAllWorkers(ctx context.Context, lister WorkerLister) ([]*ateapipb.Worke pageToken := "" for { resp, err := lister.ListWorkers(ctx, &ateapipb.ListWorkersRequest{ - PageSize: 1000, - PageToken: pageToken, + Options: &ateapipb.ListOptions{PageSize: 1000, PageToken: pageToken}, }) if err != nil { return nil, fmt.Errorf("failed to list workers: %w", err) @@ -208,8 +206,7 @@ func (r *GetWorkersRunner) Run(ctx context.Context) error { } func runGetWorkers(cmd *cobra.Command, args []string) error { - ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -227,7 +224,7 @@ func runGetWorkers(cmd *cobra.Command, args []string) error { outputFmt: outputFmt, out: cmd.OutOrStdout(), } - return runner.Run(ctx) + return runner.Run(cmd.Context()) } // PodMetricsLister abstracts fetching Kubernetes pod metrics. @@ -258,6 +255,7 @@ type TopWorkersRunner struct { sandboxClass string outputFmt string out io.Writer + stderr io.Writer } func (r *TopWorkersRunner) Run(ctx context.Context) error { @@ -275,6 +273,7 @@ func (r *TopWorkersRunner) Run(ctx context.Context) error { if r.podMetricsLister != nil { podMetricsList, err := r.podMetricsLister.ListPodMetrics(ctx, r.namespace, metav1.ListOptions{}) if err != nil { + vlogf(r.stderr, vBasic, "Failed to list pod metrics (CPU/memory will be unavailable): %v", err) metricsUnavailable = true } else { for _, pm := range podMetricsList { @@ -298,7 +297,9 @@ func (r *TopWorkersRunner) Run(ctx context.Context) error { if !metricsUnavailable { key := ns + "/" + podName if pm, ok := metricsMap[key]; ok { - cpuStr, memStr = extractContainerUsage(pm) + cpuStr, memStr = extractContainerUsage(pm, r.stderr) + } else { + vlogf(r.stderr, vBasic, "No pod metrics returned for worker pod %s", key) } } @@ -316,7 +317,7 @@ func (r *TopWorkersRunner) Run(ctx context.Context) error { return printer.PrintWorkerTopTo(r.out, items, r.outputFmt) } -func extractContainerUsage(pm metricsv1beta1.PodMetrics) (string, string) { +func extractContainerUsage(pm metricsv1beta1.PodMetrics, stderr io.Writer) (string, string) { var cpuQuant, memQuant *resource.Quantity for _, c := range pm.Containers { if c.Name == "ateom" { @@ -328,6 +329,8 @@ func extractContainerUsage(pm metricsv1beta1.PodMetrics) (string, string) { } } if cpuQuant == nil && len(pm.Containers) > 0 { + vlogf(stderr, vFlow, "Container \"ateom\" not found in pod metrics for %s/%s; summing all %d container(s)", + pm.Namespace, pm.Name, len(pm.Containers)) var totalCPU, totalMem resource.Quantity for _, c := range pm.Containers { totalCPU.Add(c.Usage[corev1.ResourceCPU]) @@ -347,8 +350,7 @@ func extractContainerUsage(pm metricsv1beta1.PodMetrics) (string, string) { } func runTopWorkers(cmd *cobra.Command, args []string) error { - ctx := cmd.Context() - apiClient, err := ateclient.NewClient(ctx, kubeconfig, k8sContext, endpoint, tokenFile, traceEnabled) + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) } @@ -356,7 +358,9 @@ func runTopWorkers(cmd *cobra.Command, args []string) error { var metricsLister PodMetricsLister metricsClient, err := ateclient.NewMetricsClientset(kubeconfig, k8sContext) - if err == nil { + if err != nil { + vlogf(cmd.ErrOrStderr(), vBasic, "Failed to create Kubernetes metrics client (CPU/memory will be unavailable): %v", err) + } else { metricsLister = &k8sPodMetricsLister{metricsClient: metricsClient} } @@ -370,9 +374,10 @@ func runTopWorkers(cmd *cobra.Command, args []string) error { sandboxClass: topWorkerClassFlag, outputFmt: outputFmt, out: cmd.OutOrStdout(), + stderr: cmd.ErrOrStderr(), } - return runner.Run(ctx) + return runner.Run(cmd.Context()) } func init() { diff --git a/cmd/kubectl-ate/internal/printer/egress_policy_test.go b/cmd/kubectl-ate/internal/printer/egress_policy_test.go index a209bf5cba..58c1c2b97d 100644 --- a/cmd/kubectl-ate/internal/printer/egress_policy_test.go +++ b/cmd/kubectl-ate/internal/printer/egress_policy_test.go @@ -37,8 +37,8 @@ func TestPrintEgressPolicyTo(t *testing.T) { CreateTime: timestamppb.New(now.Add(-5 * time.Minute)), }, Rules: []*ateapipb.EgressRule{ - {Http: &ateapipb.HTTPRule{Hostnames: []string{"api.example.com"}}}, - {Https: &ateapipb.HTTPSRule{Hostnames: []string{"www.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{8443}}}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"api.example.com"}}}, + {Https: &ateapipb.HttpsRule{Hostnames: []string{"www.example.com"}, Ports: &ateapipb.Ports{Numbers: []int32{8443}}}}, }, } empty := &ateapipb.EgressPolicy{ diff --git a/cmd/objectstore-plugin/main_test.go b/cmd/objectstore-plugin/main_test.go index 07b9cd1f9b..a939cc614e 100644 --- a/cmd/objectstore-plugin/main_test.go +++ b/cmd/objectstore-plugin/main_test.go @@ -58,7 +58,7 @@ func TestInstallStagingCapacityFitsVolume(t *testing.T) { if err := yaml.Unmarshal([]byte(doc), &obj); err != nil { t.Fatal(err) } - if obj.Kind == "DaemonSet" && strings.HasPrefix(obj.Name, "atelet-") { + if obj.Kind == "DaemonSet" && obj.Name == "atelet" { ds = &obj } } diff --git a/demos/claude-code-multiplex/README.md b/demos/claude-code-multiplex/README.md index 7bef61dbb3..23cbd0ddb8 100644 --- a/demos/claude-code-multiplex/README.md +++ b/demos/claude-code-multiplex/README.md @@ -28,7 +28,7 @@ This guide assumes you know Kubernetes and the general shape of agent runtimes ( - A Kubernetes cluster with **Agent Substrate** installed (`./hack/install-ate.sh` from this repo's root). - `kubectl` configured against that cluster (the dashboard uses the operator's kubeconfig via [`client-go`](https://github.com/kubernetes/client-go) for pod-log reads). -- Network reach to the substrate **api** gRPC service (`api.ate-system:443`). When running the dashboard from outside the cluster, port-forward it in a separate terminal (same convention as [`demos/sandbox/README.md`](../sandbox/README.md#3-port-forward-services)) and keep it running for the lifetime of the demo: +- Network reach to the substrate **api** gRPC service (`api.ate-system:443`). If you run the dashboard as a pod in the cluster, label its namespace `ate.dev/api-access=true`, because ate-system's NetworkPolicies admit API clients only from labeled namespaces (see [`ate-system-networkpolicies.yaml`](../../manifests/ate-install/ate-system-networkpolicies.yaml)). When running the dashboard from outside the cluster, port-forward it in a separate terminal (same convention as [`demos/sandbox/README.md`](../sandbox/README.md#3-port-forward-services)) and keep it running for the lifetime of the demo: ```bash # Terminal 1: api port-forward kubectl port-forward svc/api 8080:443 -n ate-system @@ -105,6 +105,18 @@ Click "Give a task". The UI picks a random idle agent and creates a task for it. With three agents and two pods, the third agent stays suspended (state snapshotted) until a pod opens up. +## Settings + +The installer reads this setting in addition to those in +[docs/operator-install.md](../../docs/operator-install.md#settings), through the same +environment variable and configuration document channels. + +| Key | Environment variable | Flag | Default | Applies to | +|------------------------|----------------------|-------------|------------|------------| +| `demo.anthropicAPIKey` | `ANTHROPIC_API_KEY` | — (no flag) | — (secret) | (global) | + +It is a secret, so it has no flag and the run report prints `` for it. + ## Upstream blockers worked around for this demo This demo currently applies workarounds at runtime for two Substrate issues. See the linked issue threads for details and workarounds. diff --git a/demos/counter/README.md b/demos/counter/README.md index f69d21ac05..ee31c67d90 100644 --- a/demos/counter/README.md +++ b/demos/counter/README.md @@ -180,6 +180,17 @@ it, resume it (even on a different worker), and confirm the count continues — the actor's counter lives in guest RAM, so a continuing count proves the guest-memory snapshot survived the round trip. +## Settings + +The installer reads these settings in addition to those in +[docs/operator-install.md](../../docs/operator-install.md#settings), through the same +flag, environment variable, and configuration document channels. + +| Key | Environment variable | Flag | Default | Applies to | +|-----------------------------------|-----------------------------------------|--------------------------|------------|-----------------------| +| `demo.counter.storageClass` | `STORAGE_CLASS` | `--storage-class` | `standard` | `deploy demo counter` | +| `demo.counter.withExternalVolume` | `ATE_DEMO_COUNTER_WITH_EXTERNAL_VOLUME` | `--with-external-volume` | `false` | `deploy demo counter` | + ## How to Uninstall To remove the counter demo resources from your cluster, run: diff --git a/demos/counter/counter-microvm.yaml.tmpl b/demos/counter/counter-microvm.yaml.tmpl index aa68bdd1f0..08e1048abb 100644 --- a/demos/counter/counter-microvm.yaml.tmpl +++ b/demos/counter/counter-microvm.yaml.tmpl @@ -45,8 +45,6 @@ spec: name: microvm workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-microvm template: - nodeSelector: - ate.dev/substrate-version: "${SUBSTRATE_VERSION}" # Per-worker resources: these size the worker POD and advertise its # scheduling capacity (the per-actor ceiling). The micro-VM sandbox itself is # sized by the ActorTemplate's resources in counter-microvm-template.yaml.tmpl, diff --git a/demos/counter/counter.yaml.tmpl b/demos/counter/counter.yaml.tmpl index dea8cb2541..4cc0ea0516 100644 --- a/demos/counter/counter.yaml.tmpl +++ b/demos/counter/counter.yaml.tmpl @@ -44,8 +44,6 @@ spec: sandboxClasses: - name: gvisor template: - nodeSelector: - ate.dev/substrate-version: "${SUBSTRATE_VERSION}" # Per-worker resources: these size the worker POD and advertise its # scheduling capacity (the per-actor ceiling). The sandbox itself is sized by # the ActorTemplate's resources in counter-template.yaml.tmpl, not by these diff --git a/demos/egress/main.go b/demos/egress/main.go index dcbc72daa1..da40483663 100644 --- a/demos/egress/main.go +++ b/demos/egress/main.go @@ -18,6 +18,8 @@ package main import ( + "bufio" + "bytes" "context" "crypto/tls" "crypto/x509" @@ -31,6 +33,7 @@ import ( "net/url" "os" "strconv" + "sync" "time" "github.com/gorilla/websocket" @@ -50,6 +53,11 @@ const ( maxWebSocketMessages = 16 maxWebSocketMessageSize = 64 << 10 maxWebSocketHandshakeBody = 4 << 10 + muxRequestCount = 250 + muxBodySize = 64 + // Keep connection around longer than ext_proc cache freshness (10s) + postBodyDelay = 15 * time.Second + slowHeaderDelay = 5 * time.Second ) type fetchRequest struct { @@ -197,11 +205,376 @@ func newHandler(client *http.Client) http.Handler { TLS: verifiedTLS, }) }) + mux.HandleFunc("/post", handlePost(postBodyDelay)) + mux.HandleFunc("/slow-headers", handleSlowHeaders(slowHeaderDelay)) + mux.HandleFunc("/mux", handleMux) mux.HandleFunc("/websocket", handleWebSocket) mux.HandleFunc("/grpc", handleGRPC) return mux } +func handleSlowHeaders(headerDelay time.Duration) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + w.Header().Set("Allow", http.MethodPost) + writeJSON(w, http.StatusMethodNotAllowed, fetchResponse{Error: "method must be POST"}) + return + } + + var input fetchRequest + decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxRequestBody)) + if err := decoder.Decode(&input); err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: fmt.Sprintf("invalid JSON payload: %v", err)}) + return + } + if err := validateURL(input.URL); err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: err.Error()}) + return + } + parsed, err := url.Parse(input.URL) + if err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: fmt.Sprintf("invalid URL: %v", err)}) + return + } + if parsed.Scheme != "http" { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: "URL scheme must be http"}) + return + } + + targetAddr := parsed.Host + if parsed.Port() == "" { + targetAddr = net.JoinHostPort(parsed.Hostname(), "80") + } + + ctx, cancel := context.WithTimeout(r.Context(), requestTimeout+headerDelay) + defer cancel() + + var dialer net.Dialer + conn, err := dialer.DialContext(ctx, "tcp", targetAddr) + if err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + defer conn.Close() + + if deadline, ok := ctx.Deadline(); ok { + if err := conn.SetDeadline(deadline); err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + } + + // Send request line and headers without the final "\r\n" that terminates the header block. + reqHeader := fmt.Sprintf( + "GET %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n", + parsed.RequestURI(), parsed.Host, + ) + if _, err := io.WriteString(conn, reqHeader); err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + + select { + case <-time.After(headerDelay): + case <-ctx.Done(): + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", ctx.Err())}) + return + } + + if _, err := io.WriteString(conn, "\r\n"); err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + + resp, err := http.ReadResponse(bufio.NewReader(conn), &http.Request{Method: http.MethodGet}) + if err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("reading response: %v", err)}) + return + } + defer resp.Body.Close() + + respBody, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBody)) + if err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("reading response: %v", err)}) + return + } + writeJSON(w, resp.StatusCode, fetchResponse{ + StatusCode: resp.StatusCode, + Body: string(respBody), + Protocol: resp.Proto, + TLS: false, + }) + } +} + +func handlePost(bodyDelay time.Duration) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + w.Header().Set("Allow", http.MethodPost) + writeJSON(w, http.StatusMethodNotAllowed, fetchResponse{Error: "method must be POST"}) + return + } + + var input fetchRequest + decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxRequestBody)) + if err := decoder.Decode(&input); err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: fmt.Sprintf("invalid JSON payload: %v", err)}) + return + } + if err := validateURL(input.URL); err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: err.Error()}) + return + } + parsed, err := url.Parse(input.URL) + if err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: fmt.Sprintf("invalid URL: %v", err)}) + return + } + + var tlsCfg *tls.Config + if parsed.Scheme == "https" { + tlsCfg = &tls.Config{ + ServerName: parsed.Hostname(), + MinVersion: tls.VersionTLS12, + } + if input.RootCA != "" { + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM([]byte(input.RootCA)) { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: "rootCA does not contain a valid certificate"}) + return + } + tlsCfg.RootCAs = roots + } + } + + targetAddr := parsed.Host + if parsed.Port() == "" { + defaultPort := "80" + if parsed.Scheme == "https" { + defaultPort = "443" + } + targetAddr = net.JoinHostPort(parsed.Hostname(), defaultPort) + } + + ctx, cancel := context.WithTimeout(r.Context(), requestTimeout+bodyDelay) + defer cancel() + + var dialer net.Dialer + conn, err := dialer.DialContext(ctx, "tcp", targetAddr) + if err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + defer conn.Close() + + if deadline, ok := ctx.Deadline(); ok { + if err := conn.SetDeadline(deadline); err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + } + + if tlsCfg != nil { + tlsConn := tls.Client(conn, tlsCfg) + if err := tlsConn.HandshakeContext(ctx); err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + conn = tlsConn + } + + reqHeader := fmt.Sprintf( + "POST %s HTTP/1.1\r\nHost: %s\r\nContent-Length: %d\r\nContent-Type: application/octet-stream\r\nConnection: close\r\n\r\n", + parsed.RequestURI(), parsed.Host, muxBodySize, + ) + if _, err := io.WriteString(conn, reqHeader); err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + + select { + case <-time.After(bodyDelay): + case <-ctx.Done(): + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", ctx.Err())}) + return + } + + body := bytes.Repeat([]byte("a"), muxBodySize) + if _, err := conn.Write(body); err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request failed: %v", err)}) + return + } + + resp, err := http.ReadResponse(bufio.NewReader(conn), &http.Request{Method: http.MethodPost}) + if err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("reading response: %v", err)}) + return + } + defer resp.Body.Close() + + respBody, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBody)) + if err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("reading response: %v", err)}) + return + } + writeJSON(w, resp.StatusCode, fetchResponse{ + StatusCode: resp.StatusCode, + Body: string(respBody), + Protocol: resp.Proto, + TLS: tlsCfg != nil, + }) + } +} + +// This handler is for testing mutiplexing of actor `muxRequestCount` connections over H/2 +// CONNECT tunnels between an atunnel and egress gateway. This handler: +// 1. opens `muxRequestCount` connections to the origin +// 2. sends POST request with headers, but not bodies on all of them. +// 3. once headers were sent, it sends bodies +// 4. waits for all 200 responses. +// Tests can then validate the expected number of H/2 connections on egress gateways. +func handleMux(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + w.Header().Set("Allow", http.MethodPost) + writeJSON(w, http.StatusMethodNotAllowed, fetchResponse{Error: "method must be POST"}) + return + } + + var input fetchRequest + decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxRequestBody)) + if err := decoder.Decode(&input); err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: fmt.Sprintf("invalid JSON payload: %v", err)}) + return + } + if err := validateURL(input.URL); err != nil { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: err.Error()}) + return + } + parsed, err := url.Parse(input.URL) + if err != nil || parsed.Scheme != "http" { + writeJSON(w, http.StatusBadRequest, fetchResponse{Error: "URL scheme must be http"}) + return + } + + targetAddr := parsed.Host + if parsed.Port() == "" { + targetAddr = net.JoinHostPort(parsed.Hostname(), "80") + } + requestURI := parsed.RequestURI() + + ctx, cancel := context.WithTimeout(r.Context(), requestTimeout) + defer cancel() + + var headersWg sync.WaitGroup + headersWg.Add(muxRequestCount) + allHeadersSent := make(chan struct{}) + go func() { + headersWg.Wait() + close(allHeadersSent) + }() + + type muxOutcome struct { + status int + body string + err error + } + outcomes := make([]muxOutcome, muxRequestCount) + + var doneWg sync.WaitGroup + doneWg.Add(muxRequestCount) + for i := range muxRequestCount { + go func(idx int) { + defer doneWg.Done() + status, body, reqErr := sendMuxRequest(ctx, cancel, targetAddr, parsed.Host, requestURI, &headersWg, allHeadersSent) + outcomes[idx] = muxOutcome{status: status, body: body, err: reqErr} + }(i) + } + doneWg.Wait() + + for i, out := range outcomes { + if out.err != nil && !errors.Is(out.err, context.Canceled) { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request %d failed: %v", i, out.err)}) + return + } + if out.err == nil && out.status != http.StatusOK { + writeJSON(w, out.status, fetchResponse{ + StatusCode: out.status, + Body: out.body, + Error: fmt.Sprintf("request %d returned status %d, want 200", i, out.status), + }) + return + } + } + for i, out := range outcomes { + if out.err != nil { + writeJSON(w, http.StatusBadGateway, fetchResponse{Error: fmt.Sprintf("request %d failed: %v", i, out.err)}) + return + } + } + + writeJSON(w, http.StatusOK, fetchResponse{StatusCode: http.StatusOK}) +} + +func sendMuxRequest(ctx context.Context, cancel context.CancelFunc, targetAddr, host, requestURI string, headersWg *sync.WaitGroup, allHeadersSent <-chan struct{}) (int, string, error) { + headerDone := false + defer func() { + if !headerDone { + cancel() + headersWg.Done() + } + }() + + var dialer net.Dialer + conn, err := dialer.DialContext(ctx, "tcp", targetAddr) + if err != nil { + return 0, "", fmt.Errorf("dialing %s: %w", targetAddr, err) + } + defer conn.Close() + + if deadline, ok := ctx.Deadline(); ok { + if err := conn.SetDeadline(deadline); err != nil { + return 0, "", fmt.Errorf("setting deadline: %w", err) + } + } + + reqHeader := fmt.Sprintf( + "POST %s HTTP/1.1\r\nHost: %s\r\nContent-Length: %d\r\nContent-Type: application/octet-stream\r\nConnection: close\r\n\r\n", + requestURI, host, muxBodySize, + ) + if _, err := io.WriteString(conn, reqHeader); err != nil { + return 0, "", fmt.Errorf("writing request headers: %w", err) + } + + headerDone = true + headersWg.Done() + + select { + case <-allHeadersSent: + case <-ctx.Done(): + return 0, "", ctx.Err() + } + if err := ctx.Err(); err != nil { + return 0, "", err + } + + body := bytes.Repeat([]byte("a"), muxBodySize) + if _, err := conn.Write(body); err != nil { + return 0, "", fmt.Errorf("writing request body: %w", err) + } + + resp, err := http.ReadResponse(bufio.NewReader(conn), &http.Request{Method: http.MethodPost}) + if err != nil { + return 0, "", fmt.Errorf("reading response: %w", err) + } + defer resp.Body.Close() + + respBody, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBody)) + if err != nil { + return 0, "", fmt.Errorf("reading response body: %w", err) + } + return resp.StatusCode, string(respBody), nil +} + // fetchClient constructs a per-request client only when the request asks for // request-scoped TLS or protocol behavior. The default path retains the // caller's client, including its transport and redirect policy. diff --git a/demos/egress/main_test.go b/demos/egress/main_test.go index c2182c471c..82c1dc72d8 100644 --- a/demos/egress/main_test.go +++ b/demos/egress/main_test.go @@ -26,7 +26,9 @@ import ( "net/http" "net/http/httptest" "strings" + "sync/atomic" "testing" + "time" "github.com/gorilla/websocket" "google.golang.org/grpc" @@ -636,3 +638,215 @@ func TestGRPCInvalidRequests(t *testing.T) { }) } } + +func TestMux(t *testing.T) { + var completed atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + t.Errorf("method = %s, want POST", r.Method) + } + if r.URL.Path != "/fetch" { + t.Errorf("path = %s, want /fetch", r.URL.Path) + } + body, err := io.ReadAll(r.Body) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + if len(body) != muxBodySize { + t.Errorf("len(body) = %d, want %d", len(body), muxBodySize) + } + completed.Add(1) + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + payload, err := json.Marshal(fetchRequest{URL: srv.URL + "/fetch"}) + if err != nil { + t.Fatal(err) + } + recorder := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodPost, "/mux", strings.NewReader(string(payload))) + newHandler(http.DefaultClient).ServeHTTP(recorder, request) + + if recorder.Code != http.StatusOK { + t.Fatalf("status = %d, want %d; body = %s", recorder.Code, http.StatusOK, recorder.Body.String()) + } + if got := int(completed.Load()); got != muxRequestCount { + t.Errorf("completed = %d, want %d", got, muxRequestCount) + } +} + +func TestMuxNon200Fails(t *testing.T) { + var seen atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = io.Copy(io.Discard, r.Body) + if int(seen.Add(1)) == muxRequestCount { + http.Error(w, "boom", http.StatusInternalServerError) + return + } + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + payload, err := json.Marshal(fetchRequest{URL: srv.URL + "/fetch"}) + if err != nil { + t.Fatal(err) + } + recorder := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodPost, "/mux", strings.NewReader(string(payload))) + newHandler(http.DefaultClient).ServeHTTP(recorder, request) + + if recorder.Code != http.StatusInternalServerError { + t.Fatalf("status = %d, want %d; body = %s", recorder.Code, http.StatusInternalServerError, recorder.Body.String()) + } +} + +func TestMuxInvalidRequests(t *testing.T) { + tests := []struct { + name string + method string + body string + status int + }{ + {name: "method", method: http.MethodGet, body: `{}`, status: http.StatusMethodNotAllowed}, + {name: "malformed JSON", method: http.MethodPost, body: `{`, status: http.StatusBadRequest}, + {name: "missing hostname", method: http.MethodPost, body: `{"url":"http:///path"}`, status: http.StatusBadRequest}, + {name: "https scheme", method: http.MethodPost, body: `{"url":"https://example.com/fetch"}`, status: http.StatusBadRequest}, + } + + handler := newHandler(http.DefaultClient) + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + recorder := httptest.NewRecorder() + request := httptest.NewRequest(test.method, "/mux", strings.NewReader(test.body)) + handler.ServeHTTP(recorder, request) + if recorder.Code != test.status { + t.Errorf("status = %d, want %d; body = %s", recorder.Code, test.status, recorder.Body.String()) + } + }) + } +} + +func TestPost(t *testing.T) { + headersReceived := make(chan struct{}) + var bodyBytes atomic.Int32 + srv, rootCA := startIPTLSServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + t.Errorf("method = %s, want POST", r.Method) + } + if r.URL.Path != "/fetch" { + t.Errorf("path = %s, want /fetch", r.URL.Path) + } + close(headersReceived) + body, err := io.ReadAll(r.Body) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + bodyBytes.Store(int32(len(body))) + w.WriteHeader(http.StatusOK) + })) + + mux := http.NewServeMux() + mux.HandleFunc("/post", handlePost(50*time.Millisecond)) + + done := make(chan *httptest.ResponseRecorder, 1) + go func() { + done <- postHandler(t, mux, "/post", fetchRequest{ + URL: srv.URL + "/fetch", + RootCA: rootCA, + }) + }() + + <-headersReceived + if got := bodyBytes.Load(); got != 0 { + t.Fatalf("body bytes immediately after headers = %d, want 0", got) + } + + rec := <-done + if rec.Code != http.StatusOK { + t.Fatalf("/post status = %d, want %d; body = %s", rec.Code, http.StatusOK, rec.Body.String()) + } + if got := int(bodyBytes.Load()); got != muxBodySize { + t.Errorf("body bytes after /post completed = %d, want %d", got, muxBodySize) + } +} + +func TestPostInvalidRequests(t *testing.T) { + handler := newHandler(http.DefaultClient) + + t.Run("method not allowed", func(t *testing.T) { + recorder := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodGet, "/post", nil) + handler.ServeHTTP(recorder, request) + if recorder.Code != http.StatusMethodNotAllowed { + t.Errorf("status = %d, want %d", recorder.Code, http.StatusMethodNotAllowed) + } + }) + t.Run("invalid rootCA", func(t *testing.T) { + recorder := postHandler(t, handler, "/post", fetchRequest{ + URL: "https://example.com/fetch", + RootCA: "not-a-cert", + }) + if recorder.Code != http.StatusBadRequest { + t.Errorf("status = %d, want %d", recorder.Code, http.StatusBadRequest) + } + }) +} + +func TestSlowHeaders(t *testing.T) { + headersCompleted := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + t.Errorf("method = %s, want GET", r.Method) + } + close(headersCompleted) + w.WriteHeader(http.StatusOK) + _, _ = io.WriteString(w, "ok") + })) + defer srv.Close() + + mux := http.NewServeMux() + mux.HandleFunc("/slow-headers", handleSlowHeaders(80*time.Millisecond)) + + done := make(chan *httptest.ResponseRecorder, 1) + go func() { + done <- postHandler(t, mux, "/slow-headers", fetchRequest{ + URL: srv.URL + "/", + }) + }() + + select { + case <-headersCompleted: + t.Fatal("headers completed before headerDelay elapsed") + case <-time.After(20 * time.Millisecond): + } + + <-headersCompleted + rec := <-done + if rec.Code != http.StatusOK { + t.Fatalf("/slow-headers status = %d, want %d; body = %s", rec.Code, http.StatusOK, rec.Body.String()) + } +} + +func TestSlowHeadersInvalidRequests(t *testing.T) { + handler := newHandler(http.DefaultClient) + + t.Run("method not allowed", func(t *testing.T) { + recorder := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodGet, "/slow-headers", nil) + handler.ServeHTTP(recorder, request) + if recorder.Code != http.StatusMethodNotAllowed { + t.Errorf("status = %d, want %d", recorder.Code, http.StatusMethodNotAllowed) + } + }) + t.Run("https rejected", func(t *testing.T) { + recorder := postHandler(t, handler, "/slow-headers", fetchRequest{ + URL: "https://example.net/", + }) + if recorder.Code != http.StatusBadRequest { + t.Errorf("status = %d, want %d", recorder.Code, http.StatusBadRequest) + } + }) +} diff --git a/docs/api-guide.md b/docs/api-guide.md index d87e300aa6..f744b12472 100644 --- a/docs/api-guide.md +++ b/docs/api-guide.md @@ -37,29 +37,6 @@ syntax. metadata; they do not affect actor scheduling. Actor selectors match `WorkerPool.metadata.labels`, not `WorkerPool.spec.template.labels`. -#### Pin pools to the installed substrate version (`template.nodeSelector`) - -The installation labels every node with the substrate build version it deployed. -Set the same label as a `nodeSelector` on every pool, so its workers only run -on nodes of that version: - -```yaml -spec: - template: - nodeSelector: - ate.dev/substrate-version: "" -``` - -Read the installed version off the atelet DaemonSet: - -```bash -kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version -``` - -The pin is critical to make a rolling upgrade possible. An upgrade moves nodes to -the new version one at a time, deleting each node's old worker pods once it -moves. A pinned pool cannot put those pods back on a moved node, so the old -version drains away node by node. An unpinned pool breaks this constraints. #### Worker Capacity (`spec.template.resources`) Setting `resources.limits` (CPU and Memory) on a `WorkerPool` establishes each worker pod's **capacity** — the envelope its actor sandboxes share, taken from the `ateom` container's limits. The scheduler only places an actor on a worker whose remaining capacity is `>=` the actor's declared resource limits (see [Sandbox Right-Sizing](#sandbox-right-sizing-resources) on the `ActorTemplate`). @@ -253,7 +230,7 @@ Each entry in `containers` describes one process to run in the actor's sandbox. | `command` | `[]string` | Optional. Entrypoint array. If unset, the image's `ENTRYPOINT` is used. If set, it replaces **both** the image's `ENTRYPOINT` and `CMD`. | | `args` | `[]string` | Optional. Arguments to the entrypoint. If unset, the image's `CMD` is used (unless `command` is set, which discards the image's `CMD`). If set, it replaces the image's `CMD`. | | `env` | `[]EnvVar` | Optional. Literal `value` entries. | -| `wakeupProbe` | `ContainerWakeupProbe` | Optional. HTTP wakeup probe — see [Container Wakeup Probe](#container-wakeup-probe-wakeupprobe). | +| `wakeupProbe` | `WakeupProbe` | Optional. HTTP wakeup probe — see [Container Wakeup Probe](#container-wakeup-probe-wakeupprobe). | | `volumeMounts` | `[]VolumeMount` | Optional. Mounts a `volumes` entry (e.g. `durableDir`) into this container. | | `securityContext` | `SecurityContext` | Optional. Security settings for the container process — see [Container Capabilities](#container-capabilities-securitycontextcapabilities). | | `resources` | `ContainerResources` | Optional. Compute limits for this container, enforced inside the actor's sandbox. Only `limits` is supported, and only `cpu` and `memory`. See [Per-container limits](#per-container-limits). | @@ -404,7 +381,9 @@ a `Snapshot` with the following fields: pause completes and cleared once those files are deleted from the node (when the actor is next paused or suspended, or is reverted). A resume restores from the files without deleting them, so an actor resumed from pause still reports - where they are. + where they are. A paused actor is an `ACTOR_STATE_SUSPENDED` actor whose + latest snapshot has `locality` and no `durableSnapshot`; it is pinned to that + node (`status.assignedNode`) until it is suspended or reverted. - `durableSnapshot`: The copy in object storage, with its URI in `object.snapshotUri` and a `status` that is `PENDING` while the copy is being written and `COMPLETED` once it can be restored from. A snapshot that exists @@ -622,7 +601,7 @@ Deletion always runs before the database reference is dropped, and a failure fai #### `RevertActor` Discards an actor's live or crashed execution and transitions it to `ACTOR_STATE_SUSPENDED` at its last completed external snapshot (`status.snapshots`). * **Request:** `RevertActorRequest` - * `actor`: `ObjectRef` of the actor to revert. Accepted from `ACTOR_STATE_RUNNING`, `ACTOR_STATE_PAUSED`, and `ACTOR_STATE_CRASHED` (plus `ACTOR_STATE_REVERTING` for idempotent retries). Calling `RevertActor` on an already `ACTOR_STATE_SUSPENDED` actor returns `FAILED_PRECONDITION`. + * `actor`: `ObjectRef` of the actor to revert. Accepted from `ACTOR_STATE_RUNNING` and `ACTOR_STATE_CRASHED` (plus `ACTOR_STATE_REVERTING` for idempotent retries), and from a paused actor: `ACTOR_STATE_SUSPENDED` whose latest snapshot is node-local. Calling `RevertActor` on an `ACTOR_STATE_SUSPENDED` actor with no node-local snapshot returns `FAILED_PRECONDITION`. * **Response:** `RevertActorResponse` containing the reverted `Actor` in `ACTOR_STATE_SUSPENDED`. * Reverting terminates any bound worker sandbox, clears node-local pause checkpoints, and garbage-collects any partial external snapshot left by an interrupted suspend while preserving the last committed durable snapshot in `status.snapshots`. * External volumes are not reverted. Their contents are never part of a snapshot, so a reverted actor comes back with its memory and root filesystem rewound but its volumes exactly as the discarded execution left them. @@ -632,7 +611,7 @@ Removes an actor from the registry and cleans up associated resources. * **Request:** `DeleteActorRequest` * `actor`: `ObjectRef` of the actor to delete. * `options`: (Optional) `DeleteOptions`. `uid` and `version` are preconditions checked against the actor as the caller last read it: a mismatch returns `ABORTED`, an omitted guard is skipped. They are checked before the workflow starts, so a guarded delete that fails part-way leaves the actor `ACTOR_STATE_DELETING` at a higher version. Retry it with the `uid` guard alone, or re-read first. - * `any_state`: (Optional) If `true`, allows deleting the actor from any state (e.g. `RUNNING`, `PAUSED`), terminating active workloads, detaching volumes, and releasing worker allocations. By default (`false`), only actors in `ACTOR_STATE_SUSPENDED` or `ACTOR_STATE_CRASHED` (or already `ACTOR_STATE_DELETING`) can be deleted. + * `any_state`: (Optional) If `true`, allows deleting the actor from any state (e.g. `RUNNING`), terminating active workloads, detaching volumes, and releasing worker allocations. By default (`false`), only actors in `ACTOR_STATE_SUSPENDED` (paused ones included) or `ACTOR_STATE_CRASHED` (or already `ACTOR_STATE_DELETING`) can be deleted. * **Response:** the deleted `Actor`, as it was immediately before removal. * Deleting an actor also deletes the external snapshot it owns, along with one an interrupted suspend left behind. Snapshots it only borrows from a tag are left alone, and its tags are unaffected — they hold their own copies. diff --git a/docs/api-style-guide.md b/docs/api-style-guide.md index 6809490e11..64e887174e 100644 --- a/docs/api-style-guide.md +++ b/docs/api-style-guide.md @@ -177,7 +177,7 @@ Rules: ### 3.2 List -*Follows [AIP-132](https://google.aip.dev/132).* +*Adapted from [AIP-132](https://google.aip.dev/132).* ```proto rpc ListActors(ListActorsRequest) returns (ListActorsResponse) {} @@ -186,14 +186,19 @@ message ListActorsRequest { // The atespace to list actors from. string atespace = 1; - // Maximum number of actors to return. The server may return fewer. + // Pagination controls. Reused across every ListRequest. + ListOptions options = 2; +} + +message ListOptions { + // Maximum number of resources to return. The server may return fewer. // If unspecified, defaults to a server-chosen value. // The maximum value is 1000; values above 1000 are coerced to 1000. - int32 page_size = 2; + int32 page_size = 1; - // Pagination token from a previous ListActors response. + // Pagination token from a previous response of the same List method. // Omit or leave empty for the first request. - string page_token = 3; + string page_token = 2; } message ListActorsResponse { @@ -208,12 +213,15 @@ message ListActorsResponse { Rules: - RPC name **must** begin with `List` followed by the **plural** resource name. - Both the request and response message names **must** match the RPC name with `Request`/`Response` suffixes. (Unlike Get/Create/Update, List responses are not the resource itself.) +- `page_size` and `page_token` **must** be carried in a `ListOptions` field named `options`. Further non-resource "control" fields belong in `ListOptions`, not as loose top-level fields on the request. - `next_page_token` **must** be present on every List response message. It **must** be empty when there are no further pages. - The repeated resource field **must** use the plural form of the resource name (e.g., `actors`, not `actor`). - If a user provides a `page_size` above the maximum, coerce it silently. If a user provides a negative value, return `INVALID_ARGUMENT`. - Sorting and filtering as specified in AIP-132 are not supported. - Clients must iterate over all pages until an empty `next_page_token` is returned. Clients should not assume that an empty result (i.e. len(actors) == 0) means the end of the stream. +**Divergence from AIP-132:** AIP-132 puts `page_size` and `page_token` at the top level of the request. Substrate groups them in a shared `ListOptions` message, following the `DeleteOptions` pattern (section #3.5), so every List method carries the same controls with the same validation. + ### 3.3 Create *Adapted from [AIP-133](https://google.aip.dev/133).* @@ -357,6 +365,7 @@ Rules: - Use standard abbreviations where well-established: `config`, `spec`, `id`, `info`, `stats`. - Adjectives come before the noun: `suspended_actors`, not `actors_suspended`. - Avoid prepositions in field names: `error_reason`, not `reason_for_error`. +- `google.protobuf.Timestamp` fields **must** end in `_time`, following [AIP-142](https://google.aip.dev/142): use `create_time`, `expire_time`, or `crash_time`, not `created_at` or `expires_at`. ### 5.1 Enum naming diff --git a/docs/architecture.md b/docs/architecture.md index 83eeaa90f3..f313ab117d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -352,7 +352,8 @@ Handles actor-aware routing and automatic re-animation. authenticated TLS tunnel to the worker's `atunnel` listener on port 443. `atunnel`, hosted by `ateom`, forwards the request to the active Actor over its private veth interface. Worker pod port 80 is not a direct Actor ingress - path. + path. With `--upstream-connect`, the router instead reaches `atunnel` + through mTLS `CONNECT` tunnels on port 8443. * **Arbitrary-Port Ingress**: A client reaches a port on the Actor other than its default (80) with an HTTP `CONNECT` request naming the port in the @@ -420,13 +421,19 @@ stateDiagram-v2 RESUMING --> RUNNING : restore / boot complete RUNNING --> SUSPENDING : SuspendActor SUSPENDING --> SUSPENDED : checkpoint complete - RUNNING --> PAUSING : PauseActor - PAUSING --> PAUSED : node-local checkpoint complete - PAUSED --> RESUMING : ResumeActor (pinned to the snapshot's node) - PAUSED --> SUSPENDING : SuspendActor (uploads the node-local snapshot) + RUNNING --> SUSPENDING : PauseActor + SUSPENDED --> SUSPENDING : SuspendActor (paused: uploads the node-local snapshot) SUSPENDED --> [*] : DeleteActor ``` +A pause and a suspend both pass through `SUSPENDING`, and a paused actor and +a suspended actor are both `SUSPENDED`. They differ only in the shape of +`status.snapshots`: after a pause the latest snapshot has a +`locality` (the node holding its files) and no `durableSnapshot`, and +`status.assignedNode` pins the actor to that node, so `ResumeActor` restores +it there from the local files. After a suspend the latest snapshot has a +`COMPLETED` `durableSnapshot` and the actor is pinned to no node. + ### Phase 1: Creation (`CreateActor`) A user or framework calls `CreateActor` with a unique ID and a reference to an @@ -485,9 +492,9 @@ tags cannot be deleted until they are. ### Phase 4: Recovery (`RevertActor`) and Deletion (`DeleteActor`) -An actor in `ACTOR_STATE_RUNNING`, `ACTOR_STATE_PAUSED`, or `ACTOR_STATE_CRASHED` can be reverted back to `ACTOR_STATE_SUSPENDED` via `RevertActor`. Reverting terminates any active sandbox, discards any local pause checkpoint or partial in-progress snapshot, and preserves the actor's last completed external snapshot so a subsequent `ResumeActor` restores from that checkpoint. +An actor in `ACTOR_STATE_RUNNING` or `ACTOR_STATE_CRASHED`, or a paused actor (`ACTOR_STATE_SUSPENDED` with a node-local latest snapshot), can be reverted back to `ACTOR_STATE_SUSPENDED` at its durable snapshot via `RevertActor`. Reverting terminates any active sandbox, discards any local pause checkpoint or partial in-progress snapshot, and preserves the actor's last completed external snapshot so a subsequent `ResumeActor` restores from that checkpoint. -By default, only actors in `ACTOR_STATE_SUSPENDED` or `ACTOR_STATE_CRASHED` state can be deleted from the Control Plane. With the `any_state` flag enabled, an actor in any state (such as `ACTOR_STATE_RUNNING` or `ACTOR_STATE_PAUSED`) can be deleted directly; the workflow terminates the running containers on the worker, detaches mounted volumes, and frees the worker assignment before deleting the record. +By default, only actors in `ACTOR_STATE_SUSPENDED` or `ACTOR_STATE_CRASHED` state can be deleted from the Control Plane. With the `any_state` flag enabled, an actor in any state (such as `ACTOR_STATE_RUNNING`) can be deleted directly; the workflow terminates the running containers on the worker, detaches mounted volumes, and frees the worker assignment before deleting the record. After deletion, the state of the actor (i.e., memory+disk snapshots) is garbage collected. diff --git a/docs/dev/best-practices/metrics.md b/docs/dev/best-practices/metrics.md index d39232f572..9990b2a7e0 100644 --- a/docs/dev/best-practices/metrics.md +++ b/docs/dev/best-practices/metrics.md @@ -278,9 +278,9 @@ component name as the scope (`"atelet"`, `"ateapi"`, `"atecontroller"`): The meter provider itself is set up once per binary by `serverboot.InitMetrics` (Prometheus reader plus OTLP push), `serverboot.InitMetricsBridged` (OTLP push with a foreign Prometheus registry -bridged onto it, or that registry alone under `OTEL_METRICS_EXPORTER=none`; -atecontroller), or `serverboot.InitMetricsPushOnlyVia` (OTLP push only, over -the atelet relay when given one; ateom). +bridged onto it, or that registry alone under `OTEL_METRICS_EXPORTER=prometheus`, +and whether to serve it; atecontroller), or `serverboot.InitMetricsPushOnlyVia` +(OTLP push only, over the atelet relay when given one; ateom). A new component calls one of these and defers `ShutdownProvider`; a new package inside an existing component adds nothing there. diff --git a/docs/dev/suspend-resume-phase-breakdown.md b/docs/dev/suspend-resume-phase-breakdown.md index 8cc72e3b6a..c4c3247c4e 100644 --- a/docs/dev/suspend-resume-phase-breakdown.md +++ b/docs/dev/suspend-resume-phase-breakdown.md @@ -24,9 +24,10 @@ the report. or the [GKE Quickstart](../../README.md#gke-quickstart-development). For GKE the scripts below read `.ate-dev-env.sh` from the repository root (`collect_logs.sh` sources it itself; set `NO_DEV_ENV` to opt out). -* The **microVM** sandbox class, if you want the inner (ateom) phases: see - [microvm-local.md](microvm-local.md). ateom-gvisor writes no breakdown - record, so on a gVisor pool the report shows the atelet layer only. +* A **micro-VM** or **gVisor** pool: both ateoms write the inner records, + so the report shows both layers for either class (a worker image that + predates the gVisor records shows the atelet layer only on a gVisor pool). + For micro-VM see [microvm-local.md](microvm-local.md). * `kubectl`, `kubectl-ate` (`go install ./cmd/kubectl-ate`), and `python3` (standard library only). @@ -92,6 +93,12 @@ atespace produce one record pair each. ## 3. Collect the node logs +(A headless `runner.py` run does this step and the next one itself, and +writes the percentiles into its `stats.jsonl`; see +[benchmarking/analysis/README.md](../../benchmarking/analysis/README.md#automated-runs). +The rest of this section is the by-hand path for a run driven from the web +UI, as above.) + Collect **while the pods still exist**. `benchmarking/automation/ orchestrator.py` deletes the workload and ate-system pods right after each test and does not run this step yet, so on a cluster you manage yourself this @@ -153,9 +160,9 @@ phases scale with dirty memory. ## Troubleshooting -* **Only atelet rows, no ateom rows.** The pool is gVisor (ateom-gvisor emits - no record), or the worker image predates the records. Check - `kubectl get workerpools -A` and step 1's grep. +* **Only atelet rows, no ateom rows.** The worker image predates the ateom + records (check step 1's grep against a worker pod), or the ateom records + fell outside the collection window. * **Counts differ between layers.** The collection window cut through a cycle, or a worker pod restarted and took its log with it (the script dumps a restarted container's previous log as `.previous.log`). On GKE the diff --git a/docs/metrics/registry/metrics.yaml b/docs/metrics/registry/metrics.yaml index b74d03a2c5..36892ee8a1 100644 --- a/docs/metrics/registry/metrics.yaml +++ b/docs/metrics/registry/metrics.yaml @@ -103,19 +103,15 @@ groups: - id: suspending stability: development value: suspending - brief: ateapi is writing the durable snapshot. + brief: > + ateapi is writing a snapshot: durable for a suspend, + node-local for a pause. - id: suspended stability: development value: suspended - brief: The actor has a durable snapshot and holds no worker. - - id: pausing - stability: development - value: pausing - brief: ateapi is writing the node-local snapshot. - - id: paused - stability: development - value: paused - brief: The actor has a node-local snapshot. + brief: > + The actor holds no worker. Its latest snapshot is durable, or + node-local after a pause. - id: crashed stability: development value: crashed @@ -124,6 +120,10 @@ groups: stability: development value: deleting brief: ateapi is removing the actor. + - id: reverting + stability: development + value: reverting + brief: ateapi is discarding the execution of the actor and returning it to the suspended state. - id: deleted stability: development value: deleted @@ -162,11 +162,14 @@ groups: - id: revert stability: development value: revert - brief: ateapi discarded the actor's execution and returned it to its existing snapshot. + brief: ateapi discarded the execution of the actor and returned it to the suspended state. - id: unknown stability: development value: unknown - brief: The operation is not in this list. This is the default value. + brief: > + No lifecycle operation is running, such as when a running or + paused actor crashes because its worker is lost, or the + operation is not in this list. This is the default value. - id: registry.ate.template type: attribute_group @@ -589,7 +592,13 @@ groups: - id: final stability: development value: final - brief: The last sample of an activation, written when a checkpoint or a terminate tears it down. It is read just before the teardown; when that read fails or times out, it repeats the newest sample. + brief: > + The last sample of an activation, written when a checkpoint, a + terminate, a graceful shutdown, or a re-host on ateom-microvm + ends it. When the sandbox is still running, ateom reads the + stats just before the teardown; when no read runs, or the read + fails or times out, it repeats the newest measured sample, or + writes a pending record when there is none. - id: ate.stats.memory.usage stability: development type: int @@ -689,7 +698,28 @@ groups: stability: stable type: string brief: The gRPC status that the server returned. - examples: ["OK", "ResourceExhausted"] + examples: ["OK", "RESOURCE_EXHAUSTED"] + - id: rpc.system.name + stability: stable + type: string + brief: The RPC system. otelgrpc always sets the value grpc. + examples: ["grpc"] + - id: server.address + stability: stable + type: string + brief: > + The address of the server that a gRPC client called. otelgrpc reads it + from the remote address of the connection, so it holds an IP address + and not the name in the dial target. For a unix socket it holds the + path of the socket. atelet calls each ateom on a socket path that holds + the UID of the worker pod, so on atelet this key has one value for + each worker pod. + examples: ["10.8.2.14", "/var/lib/ate/ateoms//ateom.sock"] + - id: server.port + stability: stable + type: int + brief: The TCP port of the server that a gRPC client called. + examples: [8085] # --------------------------------------------------------------------------- # Metrics. ateapi @@ -781,8 +811,8 @@ groups: note: > ateapi does not record a resume of an actor that is already in operation. Thus this histogram counts the true activations and not the traffic of the - router. The set of labels changes with the operation. A resume has the - most labels. A delete has only the operation and error.type. + router. The set of labels changes with the operation. Each attribute + below gives the condition for its key. annotations: substrate: emitted_by: [ateapi] @@ -794,24 +824,37 @@ groups: - ref: ate.actor.operation.name requirement_level: required - ref: ate.template.atespace - requirement_level: required + requirement_level: + conditionally_required: > + The operation is a create, or ateapi loaded the actor. Thus this key + is absent when a delete fails or when the load step fails. For a + resume, a suspend and a pause, the load step also reads the + ActorTemplate. - ref: ate.template.name - requirement_level: required + requirement_level: + conditionally_required: > + The operation is a create, or ateapi loaded the actor. Thus this key + is absent when a delete fails or when the load step fails. For a + resume, a suspend and a pause, the load step also reads the + ActorTemplate. - ref: ate.workerpool.namespace requirement_level: - conditionally_required: The actor has a worker. + conditionally_required: The operation is not a create or a delete, and the actor has a worker. - ref: ate.workerpool.name requirement_level: - conditionally_required: The actor has a worker. + conditionally_required: The operation is not a create or a delete, and the actor has a worker. - ref: ate.sandbox.class requirement_level: - conditionally_required: ateapi found the ActorTemplate. + conditionally_required: > + The operation is resume, suspend, pause or revert, and ateapi found + the ActorTemplate. A revert continues when the ActorTemplate no + longer exists, and then omits this key. - ref: ate.snapshot.kind requirement_level: - conditionally_required: The operation is a resume. A suspend and a pause do not restore, thus this key is absent. + conditionally_required: The operation is a resume and the snapshot source is known. Other operations do not restore, thus this key is absent. - ref: ate.snapshot.fidelity requirement_level: - conditionally_required: The fidelity is known. + conditionally_required: The operation is resume, suspend or pause, and the fidelity is known. - ref: error.type requirement_level: conditionally_required: The operation failed. If this key is absent, the operation was correct. @@ -1247,17 +1290,21 @@ groups: brief: The time, the rate and the errors of each gRPC method on the server. note: > The otelgrpc interceptor sends this instrument. The code of Substrate does - not make it. This instrument is the only instrumentation of ateom. ateom - has it on the gRPC interface that atelet calls. + not make it. This instrument is the only metric of ateom. ateom has it on + the gRPC interface that atelet calls. credprovider is the + kubernetes-secrets credential provider in + cmd/credential-provider/kubernetes-secrets. annotations: substrate: - emitted_by: [ateapi, atelet, atenet-router, ateom-gvisor, ateom-microvm] + emitted_by: [ateapi, atelet, atenet-router, ateom-gvisor, ateom-microvm, credprovider] golden_signals: [latency, traffic, errors] source: external instrumentation: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc code_anchor: cmd/atenet/internal/router/xds.go cuj: Which RPC is slow or fails, and on which component? attributes: + - ref: rpc.system.name + requirement_level: required - ref: rpc.method requirement_level: required - ref: rpc.response.status_code @@ -1281,13 +1328,21 @@ groups: golden_signals: [latency, traffic, errors] source: external instrumentation: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc - code_anchor: internal/ateclient/builder.go + code_anchor: cmd/ateapi/internal/controlapi/dialer.go cuj: A component waits on another component. Which call, and for how long? attributes: + - ref: rpc.system.name + requirement_level: required - ref: rpc.method requirement_level: required - ref: rpc.response.status_code requirement_level: required + - ref: server.address + requirement_level: + conditionally_required: The client sent the request headers on a connection. + - ref: server.port + requirement_level: + conditionally_required: The client called the server over TCP. A call on a unix socket has no port. # --------------------------------------------------------------------------- # The metrics that Substrate exports but does not define # diff --git a/docs/metrics/substrate.yaml b/docs/metrics/substrate.yaml index ffef2b775c..3313e3782f 100644 --- a/docs/metrics/substrate.yaml +++ b/docs/metrics/substrate.yaml @@ -26,8 +26,9 @@ registry_version: 1 # The version of the upstream semantic conventions that Substrate borrows from. -# Substrate takes error.type, file.name and the rpc.* attributes from it. This -# is the only version that the code imports under cmd/, internal/ and pkg/. +# Substrate takes error.type, file.name, the rpc.* attributes, server.address +# and server.port from it. This is the only version that the code imports under +# cmd/, internal/ and pkg/. # It is not the version of this registry: manifest.yaml holds that. upstream_semconv_version: 1.40.0 @@ -124,9 +125,9 @@ bridged_metric_families: # ----------------------------------------------------------------------------- # The rules that keep the number of series small # -# No rule is enforced at this time. This PR adds the registry only. Each rule -# says what could enforce it, so that a reader knows the difference between a -# rule that a machine holds and a rule that a person holds. +# No rule is enforced at this time. Each rule says what could enforce it, so +# that a reader knows the difference between a rule that a machine holds and a +# rule that a person holds. # ----------------------------------------------------------------------------- cardinality_rules: - id: no-actor-identity @@ -228,6 +229,11 @@ blind_spots: - area: podcertcontroller code: cmd/podcertcontroller brief: This component has no metrics. + - area: GCP Secret Manager credential provider + code: internal/plugins/gcp-secret-manager + brief: > + This plugin has no metrics. It does not install the otelgrpc server + handler. Thus rpc.server.call.duration has no credprovider series for it. - area: golden snapshot pipeline code: cmd/ateapi/internal/controlapi/template_reconciler.go brief: The template reconciler in ate-api has no instruments. Thus nothing measures a build of a golden image. diff --git a/docs/network-egress.md b/docs/network-egress.md index 741d69674e..950749ce6c 100644 --- a/docs/network-egress.md +++ b/docs/network-egress.md @@ -10,6 +10,8 @@ Agent Substrate enforces that all outbound traffic from an actor, except traffic At present, actor TCP traffic exits the sandbox through [`atunnel`](./glossary.md#components) and is forwarded on to the PEP, except traffic to destination port 53; traffic using TCP or UDP destination port 53 is allowed to bypass the PEP for DNS, while all other protocols are blocked. `atunnel` is configured with the PEP's network address and port at startup and uses that address to open an mTLS HTTP/1.1 CONNECT tunnel to the PEP. The CONNECT request target and `Host` header are both set to the original destination IP and port that `atunnel` derived from the actor's TCP connection. Actor networking is currently IPv4-only. +Before traffic reaches the egress PEP, ateom rejects packets destined for the IPv4 link-local range `169.254.0.0/16` or the IPv6 link-local range `fe80::/10`. The internal actor network `169.254.17.0/30` is exempt because it carries traffic for the actor gateway, DNS relay, ingress, and wakeup probes. Localhost traffic remains inside the actor sandbox and does not traverse the PEP. Access to non-link-local destinations remains controlled by substrate egress policy and, at the `WorkerPool` boundary, Kubernetes NetworkPolicy. + ## Trust Boundary In substrate, we consider everything originating from the actor to be untrusted. The egress PEP MUST NOT trust any decision or attestation coming from the actor itself; it MUST instead trust only information carried through substrate-controlled channels (e.g. the actor certificate and the CONNECT request produced by atunnel). For example, the PEP MUST NOT treat an actor-provided HTTP hostname or TLS SNI as proof that the original destination has that name. The PEP MAY use the hostname as a policy input, but when authorization is based on that hostname, it MUST route the traffic to the authorized hostname rather than use the hostname to authorize an arbitrary IP address from the CONNECT request. diff --git a/docs/observability.md b/docs/observability.md index 9a27cf3196..c451ebf633 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -264,7 +264,7 @@ Agent Substrate emits foundational OpenTelemetry system and server metrics to mo | Metric | Emitted by | Type | Measures | |--------|------------|------|----------| -| `rpc.server.call.duration` | ateapi & atelet (gRPC servers, via `otelgrpc`) | histogram | per-method gRPC latency, request rate, and errors (labels `rpc.method`, `rpc.response.status_code`) | +| `rpc.server.call.duration` | ateapi, atelet, atenet-router, ateom-gvisor, ateom-microvm & credprovider (gRPC servers, via `otelgrpc`) | histogram | per-method gRPC latency, request rate, and errors (labels `rpc.method`, `rpc.response.status_code`) | | `ate.actor.crashes` | ateapi | counter | Number of times actors transitioned to `ACTOR_STATE_CRASHED` (labels `ate.actor.operation.name`, `ate.template.atespace`, `ate.template.name`, `ate.workerpool.namespace`, `ate.workerpool.name`, `ate.sandbox.class`) | | `atenet.router.route.duration` | atenet-router | histogram | Substrate E2E — Envoy receiving a request to Envoy forwarding it to the resolved worker, excluding actor compute and the response (labels `ate.template.atespace`, `ate.template.name`, `ate.router.outcome`, `ate.router.resume`) | | `atelet.snapshot.size` | atelet | histogram | uncompressed allocated size in bytes of each snapshot image written during checkpoint (`st_blocks * 512`, excluding sparse holes in micro-VM `memory-ranges`) (labels `file.name`, `ate.template.atespace`, `ate.template.name`) | @@ -273,7 +273,7 @@ Agent Substrate emits foundational OpenTelemetry system and server metrics to mo | `ate.workerpool.ready_workers` | atecontroller | up/down counter | number of worker pods currently ready for a WorkerPool, from `status.readyReplicas` (labels `ate.workerpool.namespace`, `ate.workerpool.name`) | | `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance | -| `ate.actor.lifecycle.operation.duration` | ateapi | histogram | how long each actor operation (create/resume/suspend/pause/delete/revert) takes and whether it failed (`error.type` present = failure, absent = success); labeled by operation, template, pool (`ate.workerpool.namespace` + `ate.workerpool.name`), sandbox class, and snapshot kind and scope on resume; already-running resume no-ops are not recorded so the histogram tracks actual activations, not router traffic | +| `ate.actor.lifecycle.operation.duration` | ateapi | histogram | how long each actor operation (create/resume/suspend/pause/delete/revert) takes and whether it failed (`error.type` present = failure, absent = success); labeled by operation, template, pool (`ate.workerpool.namespace` + `ate.workerpool.name`), sandbox class, snapshot kind on resume, and snapshot fidelity on resume, suspend, and pause; already-running resume no-ops are not recorded so the histogram tracks actual activations, not router traffic | | `ate.scheduler.assignment.duration` | ateapi | histogram | time of one attempt to assign a worker to an actor, with one record for each resume (ateapi retries an attempt that loses a race for a worker and records only the last attempt), with the outcome (`assigned` / `no_capacity` / `error`), the assigned pool (`ate.workerpool.namespace` + `ate.workerpool.name`) and sandbox class to catch scheduling latency and capacity starvation problems | | `ate.actor.restore.duration` | atelet | histogram | how long each phase of a restore takes on the worker node, which is where cold-start latency actually goes once ateapi hands off (labels `ate.snapshot.phase`, `ate.snapshot.kind`, `ate.snapshot.fidelity`, `ate.template.atespace`, `ate.template.name`, `ate.sandbox.class`) | | `ate.actor.checkpoint.duration` | atelet | histogram | the same phase breakdown for writing a snapshot, so a slow suspend can be attributed to ateom or to the upload (same labels as the restore histogram) | @@ -344,7 +344,7 @@ Weaver permits only `groups` and `imports` at the top level of a registry file, ### Bridged controller-runtime metrics (atecontroller) -atecontroller bridges controller-runtime's private Prometheus registry, which the manager serves on an unscraped `:8080`, onto its OTLP reader. So `controller_runtime_*`, `workqueue_*`, `certwatcher_*`, `rest_client_*`, `leader_election_*`, `go_*`, and `process_*` reach the collector too, keeping their Prometheus names because they are upstream instruments and renaming them would break existing controller-runtime dashboards. +atecontroller bridges controller-runtime's private Prometheus registry, which the manager serves on `:8080` only while `OTEL_METRICS_EXPORTER` leaves it on (see [Scraping instead of pushing](#scraping-instead-of-pushing)), onto its OTLP reader. So `controller_runtime_*`, `workqueue_*`, `certwatcher_*`, `rest_client_*`, `leader_election_*`, `go_*`, and `process_*` reach the collector too, keeping their Prometheus names because they are upstream instruments and renaming them would break existing controller-runtime dashboards. These can be used to answer whether the controller is keeping up, e.g. rising `workqueue_depth` or `workqueue_queue_duration_seconds` means reconciles are falling behind, and `controller_runtime_reconcile_errors_total` says which controller. @@ -356,7 +356,9 @@ A queue that has never processed an item bridges as an exponential histogram wit ### Scraping instead of pushing -ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=none` on the components, so each series reaches the backend once. With `none` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. atecontroller then registers its instruments (`ate.workerpool.*`) on controller-runtime's registry, so the manager's `:8080` serves them next to the controller-runtime families. ateom serves no endpoint of its own, so leave the variable unset on the worker pods, or it exports no metrics at all. The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. `otlp` (the default) keeps the push, and `prometheus` or `none` stops it. `/metrics` stays on for each value. ateom serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`. +ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=prometheus` on the components, so each series reaches the backend once. With `prometheus` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. `none` stops the push too. For now `/metrics` stays on for every value, including `otlp` and `none`. This is temporary until the probes move off `:9090`; `otlp` and `none` will then turn it off ([#2067](https://github.com/agent-substrate/substrate/issues/2067)). The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. Unset is `otlp,prometheus`, or `otlp` alone on ateom. `otlp` keeps the push, and `prometheus` or `none` stops it. ateom serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`. + +atecontroller's `:8080` listener follows the variable. When the variable is unset or `otlp,prometheus`, atecontroller pushes every metric over OTLP, and `:8080` serves only the controller-runtime families ([#2402](https://github.com/agent-substrate/substrate/issues/2402)). With `otlp`, atecontroller pushes every metric and closes `:8080`. With `prometheus`, atecontroller does not push, and `:8080` serves the controller-runtime families and the `ate.workerpool.*` instruments. With `none`, atecontroller does not push and closes `:8080`. ### Local Metrics with Prometheus (Kind Cluster) diff --git a/docs/operator-install.md b/docs/operator-install.md index 6f7bf5fb57..51ef67c42a 100644 --- a/docs/operator-install.md +++ b/docs/operator-install.md @@ -264,6 +264,12 @@ changes it. Defaults are also shown by `ate-setup --help` and by `--help` on the owning command. A setting with no default is unset unless something supplies it. +Settings for the demos, benchmarks, and e2e tests are listed with them: the +[counter](../demos/counter/README.md#settings) and +[claude-code-multiplex](../demos/claude-code-multiplex/README.md#settings) +demos, the [benchmarks](../benchmarking/README.md#settings), and the +[e2e tests](../internal/e2e/README.md#settings). + | Key | Environment variable | Flag | Default | Applies to | |----------------------------------------------|-------------------------------------------------|----------------------------------------------------|-----------------------|------------------------------------------| | `actorJWT.algorithm` | `ACTOR_JWT_ALGORITHM` | `--actor-jwt-algorithm` | `ES256` | (global) | @@ -284,18 +290,12 @@ command. A setting with no default is unset unless something supplies it. | `atenet.dataplane` | `ATE_ATENET_DATAPLANE` | `--atenet-dataplane` | `envoy` | (global) | | `atenet.egress.additionalExtprocService` | `ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE` | `--experimental-additional-egress-extproc-service` | — | (global) | | `atenet.egress.credentialProvider` | `ATE_CREDENTIAL_PROVIDER` | `--credential-provider` | — | (global) | -| `benchmark.actorMemory` | `BENCHMARK_ACTOR_MEMORY` | `--benchmark-actor-memory` | — | (global) | -| `benchmark.sandboxClass` | `ATE_BENCHMARK_SANDBOX_CLASS` | `--sandbox-class` | `gvisor` | `deploy benchmarks`, `delete benchmarks` | -| `benchmark.workerCount` | `ATE_BENCHMARK_WORKER_COUNT` | `--worker-count` | `1` | `deploy benchmarks`, `delete benchmarks` | | `clusterSize` | `ATE_INSTALL_CLUSTER_SIZE` | `--cluster-size` | `size0` | (global) | | `context` | `KUBECTL_CONTEXT` | `--context` | — | (global) | | `cordonControlPlane` | `ATE_INSTALL_CORDON_CONTROL_PLANE` | `--cordon-control-plane` | `false` | (global) | | `csi.setup` | `SETUP_CSI` | `--setup-csi` | `none` | `deploy ate-system` | -| `demo.anthropicAPIKey` | `ANTHROPIC_API_KEY` | — (no flag) | — (secret) | (global) | -| `demo.counter.storageClass` | `STORAGE_CLASS` | `--storage-class` | `standard` | `deploy demo counter` | -| `demo.counter.withExternalVolume` | `ATE_DEMO_COUNTER_WITH_EXTERNAL_VOLUME` | `--with-external-volume` | `false` | `deploy demo counter` | | `docker.buildFlags` | `DOCKER_BUILD_FLAGS` | `--docker-build-flags` | — | (global) | -| `e2e.envoyConcurrency` | `E2E_ATE_ENVOY_CONCURRENCY` | `--e2e-envoy-concurrency` | `0` | (global) | +| `fakeDataPlane.enabled` | `ATE_INSTALL_FAKE_DATA_PLANE` | `--fake-data-plane` | `false` | (global) | | `gcp.clusterLocation` | `CLUSTER_LOCATION` | `--gcp-cluster-location` | — | (global) | | `gcp.clusterName` | `CLUSTER_NAME` | `--gcp-cluster-name` | — | (global) | | `gcp.projectID` | `PROJECT_ID` | `--gcp-project-id` | — | (global) | diff --git a/docs/security.md b/docs/security.md index 168bf42535..0221bd35d5 100644 --- a/docs/security.md +++ b/docs/security.md @@ -72,7 +72,7 @@ Other forms of tenancy: | Principle | Description | Status | | :--- | :--- | :--- | -| **No Access by Default** | Sandboxed actors and worker nodes launch with default-deny networking and have zero network or API access to control plane services (`ate-api-server`, Kubernetes API servers, PostgreSQL state store). DNS is currently allowed outbound by default, work on that is in progress. | Met (Loopback, private netns)
In-Progress (DNS filtering, see [#2075](https://github.com/agent-substrate/substrate/issues/2075)) | +| **No Access by Default** | Sandboxed actors and worker nodes launch with default-deny networking and have zero network or API access to control plane services (`ate-api-server`, Kubernetes API servers, PostgreSQL state store). DNS is currently allowed outbound by default, work on that is in progress. Every control plane pod in `ate-system` also denies [ingress by default](../manifests/ate-install/ate-system-networkpolicies.yaml), admitting only the flows the control plane needs. | Met (Loopback, private netns)
In-Progress (DNS filtering, see [#2075](https://github.com/agent-substrate/substrate/issues/2075)) | | **Authenticated and Policy-Enforced Ingress** | Inbound invocations to actors terminate at authenticated edge routers (`atenet-router`), which authenticate callers and verify invocation policies before triggering on-demand actor cold-start or resume. | Met (atenet-router)
In-Progress (authentication) | | **Fail-Closed Policy-Enforced Egress Gateway** | Outbound traffic is funneled through the multi-tenant egress proxy (`atenet-egress`), which validates actor identity, enforces destination host allowlists, executes TLS inspection or prompt filtering, and dynamically injects target credentials. If an egress policy enforcement point (PEP) is unconfigured/fails egress is dropped. Unsupported protocols are dropped, see [Egress Policy Table](https://docs.google.com/document/d/1XLCo9SMROddQzwsyEhdBNWuLYFtWWZWfYvGlIO1BcUI/edit?resourcekey=0-st2SqKp-O68nv5LeIUfA-g&tab=t.0#bookmark=id.pjnob1gppauk) for specifics. | Met (atenet-egress)
In-Progress (fail closed, UDP currently passthrough) | diff --git a/docs/upgrade.md b/docs/upgrade.md index 6fc20f5fad..10bcb912bd 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -63,6 +63,18 @@ Every item has to hold. Nothing in the roll stops you if one does not. kubectl get ds -n ate-system -l app=atelet ``` +- [ ] If you are upgrading across the release that added the ate-system + NetworkPolicies, every namespace whose pods call the Substrate API + directly (not through `kubectl ate`) is labeled + `ate.dev/api-access=true`. The new manifests deny the API to every + other namespace as soon as they are applied. Worker pods are never + admitted, labeled or not. See + [`ate-system-networkpolicies.yaml`](../manifests/ate-install/ate-system-networkpolicies.yaml). + + ```bash + kubectl label namespace ate.dev/api-access=true + ``` + - [ ] Every serving pool is pinned to `$OLD_VERSION`. An unpinned pool cannot take part in the roll. diff --git a/go.mod b/go.mod index 44eef6b558..0c6d2892c3 100644 --- a/go.mod +++ b/go.mod @@ -65,6 +65,7 @@ require ( golang.org/x/net v0.60.0 golang.org/x/sync v0.23.0 golang.org/x/sys v0.48.0 + golang.org/x/time v0.15.0 google.golang.org/api v0.274.0 google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d google.golang.org/grpc v1.83.2 @@ -245,7 +246,6 @@ require ( golang.org/x/oauth2 v0.36.0 // indirect golang.org/x/term v0.46.0 // indirect golang.org/x/text v0.42.0 // indirect - golang.org/x/time v0.15.0 // indirect gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect gonum.org/v1/gonum v0.17.0 // indirect google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect diff --git a/hack/create-kind-cluster.sh b/hack/create-kind-cluster.sh index 280d9d135d..9d5a0e42e5 100755 --- a/hack/create-kind-cluster.sh +++ b/hack/create-kind-cluster.sh @@ -171,6 +171,16 @@ if [[ "${IP_FAMILY}" == "ipv6" ]]; then fi fi +# 2.4 Every sandbox gets a link-local address, and the kernel's neighbor table +# ceiling is global across network namespaces: at its default of 1024 entries a +# host with about a thousand sandboxes overflows it ("neighbour: arp_cache: +# neighbor table overflow!" in dmesg) and connections to sandboxes time out while +# the host sits idle. Only Linux hosts expose the sysctl; warn, do not change it. +if thresh3="$(sysctl -n net.ipv4.neigh.default.gc_thresh3 2>/dev/null)" && [ "${thresh3}" -le 1024 ]; then + echo "warning: net.ipv4.neigh.default.gc_thresh3=${thresh3}; above about 1000 sandboxes on this host set" >&2 + echo " net.ipv4.neigh.default.gc_thresh1=4096 gc_thresh2=16384 gc_thresh3=32768 (sysctl -w, as root)" >&2 +fi + # 2.5 Enable Proxy ARP/NDP on kind nodes for gVisor loopback pod-to-pod networking echo "Enabling Proxy ARP/NDP on kind nodes..." for node in $("${ROOT}"/hack/kind.sh get nodes --name "${KIND_CLUSTER_NAME}"); do diff --git a/hack/delete-kind-cluster.sh b/hack/delete-kind-cluster.sh index fe4aec4ff3..30ce29cf37 100755 --- a/hack/delete-kind-cluster.sh +++ b/hack/delete-kind-cluster.sh @@ -30,6 +30,22 @@ if [ "$#" -gt 0 ]; then esac fi +# Unmount NFS volumes (e.g. from --setup-csi=nfs) before deleting. The NFS +# server runs in the cluster, and once it is gone the node's hard mounts retry +# forever, leaving the node stuck and impossible to remove. So stop everything +# but the server and unmount while it is still up. +for node in $("${ROOT}"/hack/kind.sh get nodes --name "${KIND_CLUSTER_NAME}" 2>/dev/null); do + docker exec "${node}" sh -c ' + grep -qE "^[^ ]+ [^ ]+ nfs4? " /proc/mounts || exit 0 + systemctl stop kubelet + server="$(crictl pods -q --label app=nfs-server)" + for pod in $(crictl pods -q); do + case "${server}" in *"${pod}"*) ;; *) crictl stopp "${pod}" >/dev/null ;; esac + done + awk "\$3 ~ /^nfs4?\$/ {print \$2}" /proc/mounts | sort -r | xargs -r umount -l + ' || true +done + echo "Deleting kind cluster '${KIND_CLUSTER_NAME}'..." "${ROOT}"/hack/kind.sh delete cluster --name "${KIND_CLUSTER_NAME}" diff --git a/hack/install-ate.sh b/hack/install-ate.sh index 529464f6ed..4031b8bdaf 100755 --- a/hack/install-ate.sh +++ b/hack/install-ate.sh @@ -100,6 +100,9 @@ usage() { echo " --cordon-control-plane Keep the control plane off the worker nodes: assumes a small shared pool labeled and" echo " tainted ate.dev/workloadType=ate-control-plane:NoSchedule, and a one-node pool" echo " labeled and tainted ate.dev/workloadType=ate-postgres:NoSchedule for postgres alone" + echo " --fake-data-plane Install fake-atelet in place of atelet and fake-workersync in place of" + echo " ate-controller, for control-plane benchmarks. Actors do not exist; never" + echo " use it on a cluster that serves real actors" echo " --rollout-timeout DURATION Per-workload readiness wait timeout, kubectl-style Go duration (default: 60s)" echo " --otlp-endpoint URL Send all control plane telemetry to URL, not to the cluster default (see benchmarking/telemetry/README.md)" echo "" @@ -303,6 +306,7 @@ for ((i = 0; i < ${#prescan_args[@]}; i++)); do GLOBAL_FLAGS+=("--cluster-size=${prescan_args[$((i + 1))]}") ;; --cordon-control-plane|--cordon-control-plane=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; + --fake-data-plane|--fake-data-plane=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; --rollout-timeout=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; --rollout-timeout) if (( i + 1 >= ${#prescan_args[@]} )); then @@ -365,6 +369,7 @@ while [[ "$#" -gt 0 ]]; do --benchmark-worker-count|--benchmark-sandbox-class|--benchmark-actor-memory) shift ;; --atenet-dataplane=*|--podcert-workers-per-signer=*|--rollout-timeout=*|--otlp-endpoint=*) ;; --cluster-size=*|--cordon-control-plane|--cordon-control-plane=*) ;; + --fake-data-plane|--fake-data-plane=*) ;; --experimental-additional-egress-extproc-service=*) ;; --credential-provider=*) ;; --benchmark-worker-count=*|--benchmark-sandbox-class=*|--benchmark-actor-memory=*) ;; diff --git a/hack/microvm-assets/README.md b/hack/microvm-assets/README.md index d4b9f26af8..3fe5b35534 100644 --- a/hack/microvm-assets/README.md +++ b/hack/microvm-assets/README.md @@ -42,7 +42,7 @@ it as a device, which is what places micro-VM workers there. 4. **Apply the demo + drive it:** ```sh - BUCKET_NAME=ate-snapshots SUBSTRATE_VERSION="$(git describe --tags --always --dirty)" envsubst < demos/counter/counter-microvm.yaml.tmpl | kubectl apply -f - # the pool pins workers to nodes labeled with this version + BUCKET_NAME=ate-snapshots envsubst < demos/counter/counter-microvm.yaml.tmpl | kubectl apply -f - ``` Create an actor from `counter-microvm`, hit the in-RAM counter to increment it, suspend (checkpoint), resume on a different worker pod, and confirm the count continues — proving the diff --git a/hack/microvm-assets/assemble.sh b/hack/microvm-assets/assemble.sh index e3456a7f96..7ca4eca5a4 100755 --- a/hack/microvm-assets/assemble.sh +++ b/hack/microvm-assets/assemble.sh @@ -99,6 +99,13 @@ tar --zstd -xf kata-static.tar.zst -C kata KROOT="kata/opt/kata" cp "$(readlink -f "${KROOT}/share/kata-containers/vmlinux.container")" "${OUT}/vmlinux" +# rootfs.img is attached over virtio-pmem and mounted with DAX (cmd/ateom-microvm/run.go). +# The image must meet the following requirements: +# - a size that is a multiple of 2 MiB (cloud-hypervisor) +# - partition 1 aligned to the guest page size +# - a filesystem block size equal to the guest page size +# The kata-containers.img used here is built for this, and vmlinux.container uses 4 KiB pages. +# See https://github.com/kata-containers/kata-containers/blob/main/tools/osbuilder/image-builder/image_builder.sh cp "$(readlink -f "${KROOT}/share/kata-containers/kata-containers.img")" "${OUT}/rootfs.img" # Statically linked, so it runs as-is outside the kata layout it is packaged for. cp "${KROOT}/libexec/virtiofsd" "${OUT}/virtiofsd" diff --git a/hack/render-manifests.sh b/hack/render-manifests.sh index 8215fcb31a..27df6e9553 100755 --- a/hack/render-manifests.sh +++ b/hack/render-manifests.sh @@ -33,6 +33,7 @@ PRESERVED_FILES=( ate-controller.yaml ate-otel-config.yaml ate-system-namespace.yaml + ate-system-networkpolicies.yaml atelet.yaml atenet-egress.yaml atenet-router.yaml @@ -149,6 +150,7 @@ find "${OUT_DIR}" -maxdepth 1 -type f -name '*.yaml' \ ! -name 'ate-controller.yaml' \ ! -name 'ate-otel-config.yaml' \ ! -name 'ate-system-namespace.yaml' \ + ! -name 'ate-system-networkpolicies.yaml' \ ! -name 'atelet.yaml' \ ! -name 'atenet-egress.yaml' \ ! -name 'atenet-router.yaml' \ diff --git a/hack/setup-csi-hostpath-kind.sh b/hack/setup-csi-hostpath-kind.sh index 514d9ae64c..f0a0a6e95b 100755 --- a/hack/setup-csi-hostpath-kind.sh +++ b/hack/setup-csi-hostpath-kind.sh @@ -134,8 +134,7 @@ spec: EOF # 9. Restart atelet to recreate image cache directories if they were wiped. -# atelet DaemonSet names carry a build-version suffix, so select by label. echo "Restarting atelet DaemonSet (if present)..." -kubectl rollout restart daemonset -l app=atelet -n ate-system >/dev/null 2>&1 || true +kubectl rollout restart daemonset/atelet -n ate-system >/dev/null 2>&1 || true echo "CSI Hostpath setup complete!" diff --git a/hack/setup-csi-nfs-kind.sh b/hack/setup-csi-nfs-kind.sh index 68a41e011b..eeec5a61ac 100755 --- a/hack/setup-csi-nfs-kind.sh +++ b/hack/setup-csi-nfs-kind.sh @@ -119,8 +119,7 @@ spec: EOF # 9. Restart atelet to ensure it reconnects to the new CSI socket. -# atelet DaemonSet names carry a build-version suffix, so select by label. echo "Restarting atelet DaemonSet (if present)..." -kubectl rollout restart daemonset -l app=atelet -n ate-system >/dev/null 2>&1 || true +kubectl rollout restart daemonset/atelet -n ate-system >/dev/null 2>&1 || true echo "CSI NFS setup complete!" diff --git a/hack/setup-e2e-egress-tls-kind.sh b/hack/setup-e2e-egress-tls-kind.sh index 47c36a160d..ebfc6906d4 100755 --- a/hack/setup-e2e-egress-tls-kind.sh +++ b/hack/setup-e2e-egress-tls-kind.sh @@ -42,7 +42,7 @@ fi # this E2E test setup; production keeps the gateway's normal root behavior. root_image="${image}" if [[ "${gateway}" == agentgateway ]]; then - root_image="envoyproxy/envoy:v1.39-latest@sha256:57e14a549d7bd43c8d3f6d03e8cfa653e037d4b38e133acd9b54f38c524401b4" + root_image="envoyproxy/envoy:v1.39.4@sha256:74bd4eb5629708d3fadca1a5e6b85d208f338d7bb54e06b69565905c5294cc01" fi # Use the helper image's public roots on every Pod start. The separate init diff --git a/hack/tools/code-generator/go.mod b/hack/tools/code-generator/go.mod index 2ffce8047e..a6fa2a4ae5 100644 --- a/hack/tools/code-generator/go.mod +++ b/hack/tools/code-generator/go.mod @@ -3,44 +3,25 @@ module github.com/agent-substrate/substrate/hack/tools/code-generator go 1.27.0 require ( - github.com/emicklei/go-restful/v3 v3.13.0 // indirect github.com/fxamacker/cbor/v2 v2.9.2 // indirect github.com/go-logr/logr v1.4.4 // indirect - github.com/go-openapi/jsonpointer v1.0.0 // indirect - github.com/go-openapi/jsonreference v1.0.0 // indirect - github.com/go-openapi/swag v0.28.0 // indirect - github.com/go-openapi/swag/cmdutils v0.28.0 // indirect - github.com/go-openapi/swag/conv v0.28.0 // indirect - github.com/go-openapi/swag/fileutils v0.28.0 // indirect - github.com/go-openapi/swag/jsonutils v0.28.0 // indirect - github.com/go-openapi/swag/loading v0.28.0 // indirect - github.com/go-openapi/swag/mangling v0.28.0 // indirect - github.com/go-openapi/swag/netutils v0.28.0 // indirect - github.com/go-openapi/swag/pools v0.28.0 // indirect - github.com/go-openapi/swag/stringutils v0.28.0 // indirect - github.com/go-openapi/swag/typeutils v0.28.0 // indirect - github.com/go-openapi/swag/yamlutils v0.28.0 // indirect - github.com/google/gnostic-models v0.7.1 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect github.com/spf13/pflag v1.0.10 // indirect github.com/x448/float16 v0.8.4 // indirect go.yaml.in/yaml/v2 v2.4.4 // indirect - go.yaml.in/yaml/v3 v3.0.5 // indirect - golang.org/x/mod v0.40.0 // indirect - golang.org/x/net v0.58.0 // indirect - golang.org/x/sync v0.22.0 // indirect - golang.org/x/text v0.41.0 // indirect - golang.org/x/tools v0.49.0 // indirect - google.golang.org/protobuf v1.36.12 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/net v0.61.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/tools v0.52.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect k8s.io/apimachinery v0.36.0-alpha.2.0.20260904054801-5a60cf1c6b81 // indirect k8s.io/code-generator v0.30.0-alpha.3.0.20260904060942-0cddac4383e4 // indirect k8s.io/gengo/v2 v2.0.0-20260408192533-25e2208e0dc3 // indirect k8s.io/klog/v2 v2.140.0 // indirect k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad // indirect - k8s.io/streaming v0.37.0 // indirect k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3 // indirect sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect sigs.k8s.io/randfill v1.0.0 // indirect diff --git a/hack/tools/code-generator/go.sum b/hack/tools/code-generator/go.sum index e2b65bfe38..0821f0432b 100644 --- a/hack/tools/code-generator/go.sum +++ b/hack/tools/code-generator/go.sum @@ -1,43 +1,11 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/emicklei/go-restful/v3 v3.13.0 h1:C4Bl2xDndpU6nJ4bc1jXd+uTmYPVUwkD6bFY/oTyCes= -github.com/emicklei/go-restful/v3 v3.13.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/fxamacker/cbor/v2 v2.9.2 h1:X4Ksno9+x3cz0TZv69ec1hxP/+tymuR8PXQJyDwfh78= github.com/fxamacker/cbor/v2 v2.9.2/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= -github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= -github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= -github.com/go-openapi/jsonpointer v1.0.0 h1:kR9tHqY0CtZaOPVFm622dPVNhrvYpwr4uCxgL3h1H8s= -github.com/go-openapi/jsonpointer v1.0.0/go.mod h1:Z3rw7dWu1p9IgitXCFamSlA5lmDiklEB6vkaxcNZW5Y= -github.com/go-openapi/jsonreference v1.0.0 h1:jlmTr6torcd1YgDQvSfNmRtKzYDO4FGBkrAdlAVWnpY= -github.com/go-openapi/jsonreference v1.0.0/go.mod h1:jtwdyGbJk0Xhe5Y+rwtglQP6Sb1WZST4rT32LWB+sv0= -github.com/go-openapi/swag v0.28.0 h1:xkgbOSKj6DZziNpyqRRAOt3GJGtgjgsd2RoyT30VWuw= -github.com/go-openapi/swag v0.28.0/go.mod h1:4qYnT3Cqr1p1VknOdPo70evN4rgQnAg6jwApHyxSGIg= -github.com/go-openapi/swag/cmdutils v0.28.0 h1:7TOeNtkYru1SG8Y34tDh9WBbLsMqGnptuxWiHREPZ4Q= -github.com/go-openapi/swag/cmdutils v0.28.0/go.mod h1:Sm1MVFMkF6guJJ+pQqHnQA3N0j9qALV3NxzDSv6bETM= -github.com/go-openapi/swag/conv v0.28.0 h1:GtqqbyFe7vR5Y7ehxG9W6/OvrSFdf1OLeTGp40TqxH8= -github.com/go-openapi/swag/conv v0.28.0/go.mod h1:mbUE+mzctnhxi864m0Q07SpN8OowD9JhxmxuYvZZD/k= -github.com/go-openapi/swag/fileutils v0.28.0 h1:Z04XWQD7R8Eq+7GnOrjovBxPPmZzsS4gt2H2GPGIViU= -github.com/go-openapi/swag/fileutils v0.28.0/go.mod h1:VvJFZLTZS0AI854gEQz5tk7dBESdLjiNUMSZ/th2ry8= -github.com/go-openapi/swag/jsonutils v0.28.0 h1:YIch6FwO7RXzeAnbO8Tu7dWBZeUEH+4nA0HXltVTnv4= -github.com/go-openapi/swag/jsonutils v0.28.0/go.mod h1:CYM3WlTUcagR2ZoHdz54di/cbBqt82tuxuXgAjxw+mg= -github.com/go-openapi/swag/loading v0.28.0 h1:td8QZdZC9MIYGGSnSPKShKiK22I2tU5UQvuUhIBPRLU= -github.com/go-openapi/swag/loading v0.28.0/go.mod h1:rXB0QiQX5mMveXEA7ouM4KiiM9jVJe4K6BVbwhD1M4k= -github.com/go-openapi/swag/mangling v0.28.0 h1:pH8eyeNO9SLYsTMWJrurnNfKmDa28XrlA+HePVD53VM= -github.com/go-openapi/swag/mangling v0.28.0/go.mod h1:jtBE2+V+3pILxOR7Vgce+Cwp6A2PgZbvVqfNntbVs0w= -github.com/go-openapi/swag/netutils v0.28.0 h1:YXN6TALEi2pzts8/8GNm6T61HTAZsieukGZidap989k= -github.com/go-openapi/swag/netutils v0.28.0/go.mod h1:J+WYyFMLtvtCGqa6jLv+YNUmIKI3ZRQRrvfNDMoQoEQ= -github.com/go-openapi/swag/pools v0.28.0 h1:HPMZWSAfce3rdVTFcjFiCIBtDg9h4x2QlRrHipwhxeU= -github.com/go-openapi/swag/pools v0.28.0/go.mod h1:kVQefhSK5RWuRe7BXsL8htgBPAMpN7HDGpGEknqugeE= -github.com/go-openapi/swag/stringutils v0.28.0 h1:ixsc9iYgDPubHL/8nSkbnryEHpD2VRlBMLKpQyPXcDU= -github.com/go-openapi/swag/stringutils v0.28.0/go.mod h1:lzRN95CxXmA03XcDWHLOb6nOMcxCqR5rGY0lOgsfRoM= -github.com/go-openapi/swag/typeutils v0.28.0 h1:nRBKSBXjDgf01VDPB3fWeD9nQuhCOVeIYAkUx2tbkyY= -github.com/go-openapi/swag/typeutils v0.28.0/go.mod h1:Srm0xFNRZ1Y+vCxJclo5qzx8aj+1pAKda/YfFPrG0dQ= -github.com/go-openapi/swag/yamlutils v0.28.0 h1:TV3JXH6DS46KUroDtMLAYHGkdWf5VDq3wVWFirmzROY= -github.com/go-openapi/swag/yamlutils v0.28.0/go.mod h1:x0q/yndZHEgk9Rx3DyDqzFUmHy55KTvIZldvF2dTJXs= -github.com/google/gnostic-models v0.7.1 h1:SisTfuFKJSKM5CPZkffwi6coztzzeYUhc3v4yxLWH8c= -github.com/google/gnostic-models v0.7.1/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7OUGxBlw57miDrQ= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= @@ -50,64 +18,42 @@ github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjY github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee h1:W5t00kpgFdJifH4BDsTlE89Zl93FEloxaWZfGcifgq8= github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY= -github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= +github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= -golang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA= -golang.org/x/mod v0.29.0/go.mod h1:NyhrlYXJ2H4eJiRy/WDBO6HMqZQ6q9nk4JzS3NuCK+w= -golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs= -golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE= -golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= -golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= -golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I= -golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= -golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= -golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= -golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM= -golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM= -golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= -golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= -golang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ= -golang.org/x/tools v0.38.0/go.mod h1:yEsQ/d/YK8cjh0L6rZlY8tgtlKiBNTL14pGDJPJpYQs= -golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI= -golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo= -google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= -google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= k8s.io/apimachinery v0.36.0-alpha.2.0.20260904054801-5a60cf1c6b81 h1:RwCPehCTEZyBA64K7xaHTyXwIwsoBdsLmq21B0P/VcY= k8s.io/apimachinery v0.36.0-alpha.2.0.20260904054801-5a60cf1c6b81/go.mod h1:1Um7tr4O65Pd822NFLn+MI08uCzUIPahh2lCPsWzNgk= -k8s.io/apimachinery v0.37.0-rc.0 h1:z92lapcEJUiMb38pzUIp81kEXT6lIWXhs6auvm8+/s4= -k8s.io/apimachinery v0.37.0-rc.0/go.mod h1:mhq6CPCzI6XJNHSiek+w7Ws9/rP9qL5s+7aBrh5ODSI= k8s.io/code-generator v0.30.0-alpha.3.0.20260904060942-0cddac4383e4 h1:WgL8fyPUJi6VbagUkeOhvYYKCIB/ChMCtIalZ9TBh1Q= k8s.io/code-generator v0.30.0-alpha.3.0.20260904060942-0cddac4383e4/go.mod h1:AA8MJmGgNcxTDTeJgSh2SFpNtAqq9KvjpYooMOcZmDk= -k8s.io/code-generator v0.35.0 h1:TvrtfKYZTm9oDF2z+veFKSCcgZE3Igv0svY+ehCmjHQ= -k8s.io/code-generator v0.35.0/go.mod h1:iS1gvVf3c/T71N5DOGYO+Gt3PdJ6B9LYSvIyQ4FHzgc= -k8s.io/code-generator v0.37.0-rc.0 h1:d0i65nsz5LS2HUl3CgJcjxFr3IvLBGs82IZth2UuQEk= -k8s.io/code-generator v0.37.0-rc.0/go.mod h1:gwljmdmKalNet8U3+3tu8dyV0+etqGStU2uXJy+NgD8= -k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b h1:gMplByicHV/TJBizHd9aVEsTYoJBnnUAT5MHlTkbjhQ= -k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b/go.mod h1:CgujABENc3KuTrcsdpGmrrASjtQsWCT7R99mEV4U/fM= k8s.io/gengo/v2 v2.0.0-20260408192533-25e2208e0dc3 h1:3L6PNkMLXkU/pz3jWzaaIUz0Rs2V9h+5O51AeRC7poc= k8s.io/gengo/v2 v2.0.0-20260408192533-25e2208e0dc3/go.mod h1:yvyl3l9E+UxlqOMUULdKTAYB0rEhsmjr7+2Vb/1pCSo= -k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk= -k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= k8s.io/klog/v2 v2.140.0 h1:Tf+J3AH7xnUzZyVVXhTgGhEKnFqye14aadWv7bzXdzc= k8s.io/klog/v2 v2.140.0/go.mod h1:o+/RWfJ6PwpnFn7OyAG3QnO47BFsymfEfrz6XyYSSp0= k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad h1:oXImqH8mQNk7PmvzKhmN3ddJoY6OnyM225MXwGHPm0A= k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad/go.mod h1:0/mqHCVhlumdJ3BhCfnjSZQE037nAhNodh1/hK0T8/I= -k8s.io/streaming v0.37.0-rc.0 h1:zDBjrKCnSLQyznohS8DyT1vmiD1s7XEcj5B9kyfN1WQ= -k8s.io/streaming v0.37.0-rc.0/go.mod h1:APlJR26ZWRcVy5bIEj0QRrKUXROtBHPcxl2NT7EAzPU= -k8s.io/streaming v0.37.0 h1:iPBUZLZiKt5bV+lxJurASMOV07VuBhNpiwJt2//AWrM= -k8s.io/streaming v0.37.0/go.mod h1:APlJR26ZWRcVy5bIEj0QRrKUXROtBHPcxl2NT7EAzPU= k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3 h1:jVkFFVfXdXP74B/zbO3hM3hpSFD0xvhQ5U686DPurkE= k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3/go.mod h1:M2s5JB1lIYP3jzZdorPLHXIPJzt9vv2muW5a6L9DtNM= sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 h1:IpInykpT6ceI+QxKBbEflcR5EXP7sU1kvOlxwZh5txg= diff --git a/hack/tools/controller-gen/go.mod b/hack/tools/controller-gen/go.mod index 1772888e8a..977f2766bf 100644 --- a/hack/tools/controller-gen/go.mod +++ b/hack/tools/controller-gen/go.mod @@ -24,12 +24,12 @@ require ( github.com/x448/float16 v0.8.4 // indirect go.yaml.in/yaml/v2 v2.4.3 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect - golang.org/x/mod v0.35.0 // indirect - golang.org/x/net v0.55.0 // indirect - golang.org/x/sync v0.20.0 // indirect - golang.org/x/sys v0.45.0 // indirect - golang.org/x/text v0.37.0 // indirect - golang.org/x/tools v0.44.0 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/net v0.61.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/tools v0.52.0 // indirect google.golang.org/protobuf v1.36.8 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/hack/tools/controller-gen/go.sum b/hack/tools/controller-gen/go.sum index 42aba42267..00fbaaf79b 100644 --- a/hack/tools/controller-gen/go.sum +++ b/hack/tools/controller-gen/go.sum @@ -145,26 +145,26 @@ go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8= golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY= -golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= -golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= -golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= -golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI= golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU= -golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= -golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= -golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= -golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4= -golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk= -golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= -golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM= +golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY= golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= -golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= -golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM= golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY= golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM= diff --git a/hack/tools/go-licenses/go.mod b/hack/tools/go-licenses/go.mod index f0482391d2..432b1753f3 100644 --- a/hack/tools/go-licenses/go.mod +++ b/hack/tools/go-licenses/go.mod @@ -14,12 +14,12 @@ require ( github.com/spf13/cobra v1.7.0 // indirect github.com/spf13/pflag v1.0.5 // indirect go.opencensus.io v0.24.0 // indirect - golang.org/x/mod v0.35.0 // indirect - golang.org/x/net v0.55.0 // indirect - golang.org/x/sync v0.20.0 // indirect - golang.org/x/sys v0.45.0 // indirect - golang.org/x/text v0.37.0 // indirect - golang.org/x/tools v0.44.0 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/net v0.61.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/tools v0.52.0 // indirect k8s.io/klog/v2 v2.90.1 // indirect ) diff --git a/hack/tools/go-licenses/go.sum b/hack/tools/go-licenses/go.sum index d6038883ed..28369d2a31 100644 --- a/hack/tools/go-licenses/go.sum +++ b/hack/tools/go-licenses/go.sum @@ -81,39 +81,39 @@ golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= -golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= -golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= -golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= -golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= -golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= -golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= diff --git a/hack/tools/golangci-lint/go.mod b/hack/tools/golangci-lint/go.mod index c545ab22c8..2593d4f0b0 100644 --- a/hack/tools/golangci-lint/go.mod +++ b/hack/tools/golangci-lint/go.mod @@ -204,11 +204,11 @@ require ( go.uber.org/zap v1.27.0 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect golang.org/x/exp/typeparams v0.0.0-20260811152304-ee035b5b010f // indirect - golang.org/x/mod v0.40.0 // indirect - golang.org/x/sync v0.22.0 // indirect - golang.org/x/sys v0.47.0 // indirect - golang.org/x/text v0.40.0 // indirect - golang.org/x/tools v0.49.0 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/tools v0.52.0 // indirect google.golang.org/protobuf v1.36.11 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/hack/tools/golangci-lint/go.sum b/hack/tools/golangci-lint/go.sum index 67cd6a0cff..fd75e99e95 100644 --- a/hack/tools/golangci-lint/go.sum +++ b/hack/tools/golangci-lint/go.sum @@ -689,8 +689,8 @@ golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs= -golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -725,8 +725,8 @@ golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= -golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -746,8 +746,8 @@ golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= -golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -792,8 +792,8 @@ golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= -golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= @@ -805,8 +805,8 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs= -golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -857,8 +857,8 @@ golang.org/x/tools v0.1.1-0.20210302220138-2ac05c832e1a/go.mod h1:9bzcO0MWcOuT0t golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI= -golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM= golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY= golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM= diff --git a/hack/tools/gotestsum/go.mod b/hack/tools/gotestsum/go.mod index 47a5c0ceba..cf0ca9cfd1 100644 --- a/hack/tools/gotestsum/go.mod +++ b/hack/tools/gotestsum/go.mod @@ -12,11 +12,11 @@ require ( github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect github.com/mattn/go-colorable v0.1.13 // indirect github.com/mattn/go-isatty v0.0.20 // indirect - golang.org/x/mod v0.27.0 // indirect - golang.org/x/sync v0.17.0 // indirect - golang.org/x/sys v0.36.0 // indirect - golang.org/x/term v0.35.0 // indirect - golang.org/x/text v0.17.0 // indirect - golang.org/x/tools v0.36.0 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/term v0.47.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/tools v0.52.0 // indirect gotest.tools/gotestsum v1.13.0 // indirect ) diff --git a/hack/tools/gotestsum/go.sum b/hack/tools/gotestsum/go.sum index 9d4a067145..482ec2295b 100644 --- a/hack/tools/gotestsum/go.sum +++ b/hack/tools/gotestsum/go.sum @@ -17,20 +17,20 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/rogpeppe/go-internal v1.11.0 h1:cWPaGQEPrBb5/AsnsZesgZZ9yb1OQ+GOISoDNXVBh4M= github.com/rogpeppe/go-internal v1.11.0/go.mod h1:ddIwULY96R17DhadqLgMfk9H9tvdUzkipdSkR5nkCZA= -golang.org/x/mod v0.27.0 h1:kb+q2PyFnEADO2IEF935ehFUXlWiNjJWtRNgBLSfbxQ= -golang.org/x/mod v0.27.0/go.mod h1:rWI627Fq0DEoudcK+MBkNkCe0EetEaDSwJJkCcjpazc= -golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug= -golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k= -golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= -golang.org/x/term v0.35.0 h1:bZBVKBudEyhRcajGcNc3jIfWPqV4y/Kt2XcoigOWtDQ= -golang.org/x/term v0.35.0/go.mod h1:TPGtkTLesOwf2DE8CgVYiZinHAOuy5AYUYT1lENIZnA= -golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc= -golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= -golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg= -golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM= +golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= gotest.tools/gotestsum v1.13.0 h1:+Lh454O9mu9AMG1APV4o0y7oDYKyik/3kBOiCqiEpRo= gotest.tools/gotestsum v1.13.0/go.mod h1:7f0NS5hFb0dWr4NtcsAsF0y1kzjEFfAil0HiBQJE03Q= gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q= diff --git a/hack/tools/kind/go.mod b/hack/tools/kind/go.mod index 1343de508d..9100fa9ad0 100644 --- a/hack/tools/kind/go.mod +++ b/hack/tools/kind/go.mod @@ -15,7 +15,7 @@ require ( github.com/spf13/cobra v1.8.0 // indirect github.com/spf13/pflag v1.0.5 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect - golang.org/x/sys v0.6.0 // indirect + golang.org/x/sys v0.49.0 // indirect sigs.k8s.io/kind v0.33.0 // indirect sigs.k8s.io/yaml v1.4.0 // indirect ) diff --git a/hack/tools/kind/go.sum b/hack/tools/kind/go.sum index 73d1641ca0..af4c849a30 100644 --- a/hack/tools/kind/go.sum +++ b/hack/tools/kind/go.sum @@ -30,8 +30,9 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= -golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b h1:QRR6H1YWRnHb4Y/HeNFCTJLFVxaq6wH4YuVdsUOr75U= gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/hack/tools/ko/go.mod b/hack/tools/ko/go.mod index e0cc73266c..bc6d2f33ec 100644 --- a/hack/tools/ko/go.mod +++ b/hack/tools/ko/go.mod @@ -140,15 +140,15 @@ require ( go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect go.yaml.in/yaml/v4 v4.0.0-rc.6 // indirect - golang.org/x/crypto v0.55.0 // indirect - golang.org/x/mod v0.38.0 // indirect - golang.org/x/net v0.58.0 // indirect - golang.org/x/oauth2 v0.36.0 // indirect - golang.org/x/sync v0.22.0 // indirect - golang.org/x/sys v0.47.0 // indirect - golang.org/x/term v0.45.0 // indirect - golang.org/x/text v0.41.0 // indirect - golang.org/x/tools v0.48.0 // indirect + golang.org/x/crypto v0.58.0 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/net v0.61.0 // indirect + golang.org/x/oauth2 v0.37.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/term v0.47.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/tools v0.52.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect google.golang.org/grpc v1.83.2 // indirect diff --git a/hack/tools/ko/go.sum b/hack/tools/ko/go.sum index 86ab6c74e5..4f80ab511d 100644 --- a/hack/tools/ko/go.sum +++ b/hack/tools/ko/go.sum @@ -483,27 +483,27 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= -golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M= -golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis= +golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54= +golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk= -golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= -golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= -golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= -golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= -golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= +golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98= +golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= -golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -516,15 +516,15 @@ golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= -golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0= -golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0= -golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w= +golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM= +golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= @@ -532,16 +532,16 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= -golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE= -golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= diff --git a/hack/tools/kube-api-linter/go.mod b/hack/tools/kube-api-linter/go.mod index c8ccbc74dd..24e6a02533 100644 --- a/hack/tools/kube-api-linter/go.mod +++ b/hack/tools/kube-api-linter/go.mod @@ -199,11 +199,11 @@ require ( go.uber.org/multierr v1.10.0 // indirect go.uber.org/zap v1.27.0 // indirect golang.org/x/exp/typeparams v0.0.0-20250911091902-df9299821621 // indirect - golang.org/x/mod v0.39.0 // indirect - golang.org/x/sync v0.22.0 // indirect - golang.org/x/sys v0.47.0 // indirect - golang.org/x/text v0.29.0 // indirect - golang.org/x/tools v0.49.0 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/text v0.43.0 // indirect + golang.org/x/tools v0.52.0 // indirect google.golang.org/protobuf v1.36.6 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/hack/tools/kube-api-linter/go.sum b/hack/tools/kube-api-linter/go.sum index 42fbbe46da..fcca1574bd 100644 --- a/hack/tools/kube-api-linter/go.sum +++ b/hack/tools/kube-api-linter/go.sum @@ -687,8 +687,8 @@ golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= -golang.org/x/mod v0.39.0 h1:UF5zwQdCRRUpHfyPwr7d4UrGiVeldIsogtzWVnczL74= -golang.org/x/mod v0.39.0/go.mod h1:bvIbwjQ0HUFFf5AKukeeYQG4ZBUG9yxQbR9aEweIwYY= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -727,8 +727,8 @@ golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk= golang.org/x/net v0.16.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE= -golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= -golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -750,8 +750,8 @@ golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= golang.org/x/sync v0.4.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= -golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= -golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -803,8 +803,8 @@ golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= -golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= @@ -821,8 +821,8 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= -golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk= -golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -876,8 +876,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58= golang.org/x/tools v0.14.0/go.mod h1:uYBEerGOWcJyEORxN+Ek8+TT266gXkNlHdJBwexUsBg= -golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI= -golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM= golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY= golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM= diff --git a/hack/tools/setup-envtest/go.mod b/hack/tools/setup-envtest/go.mod index 7d3df4e35f..e80f2207cd 100644 --- a/hack/tools/setup-envtest/go.mod +++ b/hack/tools/setup-envtest/go.mod @@ -10,7 +10,7 @@ require ( go.uber.org/multierr v1.10.0 // indirect go.uber.org/zap v1.27.0 // indirect go.yaml.in/yaml/v2 v2.4.3 // indirect - golang.org/x/text v0.33.0 // indirect + golang.org/x/text v0.43.0 // indirect sigs.k8s.io/controller-runtime/tools/setup-envtest v0.0.0-20260405152528-6210f847b2c1 // indirect sigs.k8s.io/yaml v1.6.0 // indirect ) diff --git a/hack/tools/setup-envtest/go.sum b/hack/tools/setup-envtest/go.sum index 39bc915854..f5e331412a 100644 --- a/hack/tools/setup-envtest/go.sum +++ b/hack/tools/setup-envtest/go.sum @@ -34,18 +34,18 @@ go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0= go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8= go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= -golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c= -golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU= +golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= +golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o= golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8= -golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4= -golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ= golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= -golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE= -golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8= -golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA= -golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc= +golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= +golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= +golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI= +golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA= google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af h1:+5/Sw3GsDNlEmu7TfklWKPdQ0Ykja5VEmq2i817+jbI= google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= diff --git a/hack/verify-atenet-drain.sh b/hack/verify-atenet-drain.sh index 0f9368de6e..f7dabd00bd 100755 --- a/hack/verify-atenet-drain.sh +++ b/hack/verify-atenet-drain.sh @@ -55,7 +55,7 @@ SUFFIX="$(date +%s)" ACTOR_BUSY="busy-${SUFFIX}" # occupies the only worker ACTOR_PARKED="parked-${SUFFIX}" # its request parks, then survives the drain LOCAL_HTTP_PORT="${LOCAL_HTTP_PORT:-18080}" -LOCAL_METRICS_PORT="${LOCAL_METRICS_PORT:-19090}" +LOCAL_HEALTH_PORT="${LOCAL_HEALTH_PORT:-19091}" # Xs must be the trailing characters: BSD (macOS) mktemp rejects suffixes. LOG_FILE="$(mktemp /tmp/atenet-drain-check-log.XXXXXX)" CURL_OUT="$(mktemp /tmp/atenet-drain-check-out.XXXXXX)" @@ -101,10 +101,10 @@ run_kubectl_ate get actor-template "${DEMO_POOL}" -a "${ATESPACE}" -o json 2>/de if run_kubectl_ate get workers 2>/dev/null | awk 'NR>1 && $4=="ASSIGNED"' | grep -q .; then fail "workers on the ${DEMO_POOL} pool are ASSIGNED; the check scales the pool to 1 and would crash running actors — suspend them first" fi -for port in "${LOCAL_HTTP_PORT}" "${LOCAL_METRICS_PORT}"; do +for port in "${LOCAL_HTTP_PORT}" "${LOCAL_HEALTH_PORT}"; do if (exec 3<>"/dev/tcp/127.0.0.1/${port}") 2>/dev/null; then exec 3>&- 3<&- - fail "local port ${port} is already in use (a stale port-forward?); free it or set LOCAL_HTTP_PORT/LOCAL_METRICS_PORT" + fail "local port ${port} is already in use (a stale port-forward?); free it or set LOCAL_HTTP_PORT/LOCAL_HEALTH_PORT" fi done @@ -137,12 +137,12 @@ run_kubectl logs -n "${ROUTER_NS}" "${POD}" -c atenet-router -f > "${LOG_FILE}" BG_PIDS+=($!) run_kubectl port-forward -n "${ROUTER_NS}" svc/atenet-router "${LOCAL_HTTP_PORT}:80" >/dev/null 2>&1 & BG_PIDS+=($!) -run_kubectl port-forward -n "${ROUTER_NS}" "${POD}" "${LOCAL_METRICS_PORT}:9090" >/dev/null 2>&1 & +run_kubectl port-forward -n "${ROUTER_NS}" "${POD}" "${LOCAL_HEALTH_PORT}:9091" >/dev/null 2>&1 & BG_PIDS+=($!) sleep 3 -readyz() { curl -s -o /dev/null -w '%{http_code}' "localhost:${LOCAL_METRICS_PORT}/readyz"; } -healthz() { curl -s -o /dev/null -w '%{http_code}' "localhost:${LOCAL_METRICS_PORT}/healthz"; } +readyz() { curl -s -o /dev/null -w '%{http_code}' "localhost:${LOCAL_HEALTH_PORT}/readyz"; } +healthz() { curl -s -o /dev/null -w '%{http_code}' "localhost:${LOCAL_HEALTH_PORT}/healthz"; } # The port-forwards need a moment to come up; retry before judging. for i in $(seq 1 20); do diff --git a/internal/ateattr/ateattr.go b/internal/ateattr/ateattr.go index 3d3531b864..da02cbb244 100644 --- a/internal/ateattr/ateattr.go +++ b/internal/ateattr/ateattr.go @@ -106,8 +106,6 @@ const ( ActorStateRunning = "running" ActorStateSuspending = "suspending" ActorStateSuspended = "suspended" - ActorStatePausing = "pausing" - ActorStatePaused = "paused" ActorStateCrashed = "crashed" ActorStateDeleting = "deleting" ActorStateReverting = "reverting" @@ -127,10 +125,6 @@ func ActorStateValue(state ateapipb.ActorState) string { return ActorStateSuspending case ateapipb.ActorState_ACTOR_STATE_SUSPENDED: return ActorStateSuspended - case ateapipb.ActorState_ACTOR_STATE_PAUSING: - return ActorStatePausing - case ateapipb.ActorState_ACTOR_STATE_PAUSED: - return ActorStatePaused case ateapipb.ActorState_ACTOR_STATE_CRASHED: return ActorStateCrashed case ateapipb.ActorState_ACTOR_STATE_DELETING: diff --git a/internal/ateattr/ateattr_test.go b/internal/ateattr/ateattr_test.go index f539f03fc9..e4fe0f8baf 100644 --- a/internal/ateattr/ateattr_test.go +++ b/internal/ateattr/ateattr_test.go @@ -603,8 +603,6 @@ func TestActorStateValues(t *testing.T) { {ActorStateRunning, "running"}, {ActorStateSuspending, "suspending"}, {ActorStateSuspended, "suspended"}, - {ActorStatePausing, "pausing"}, - {ActorStatePaused, "paused"}, {ActorStateCrashed, "crashed"}, {ActorStateDeleting, "deleting"}, {ActorStateDeleted, "deleted"}, @@ -631,8 +629,6 @@ func TestActorStateValue(t *testing.T) { {name: "running", state: ateapipb.ActorState_ACTOR_STATE_RUNNING, want: ActorStateRunning}, {name: "suspending", state: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, want: ActorStateSuspending}, {name: "suspended", state: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, want: ActorStateSuspended}, - {name: "pausing", state: ateapipb.ActorState_ACTOR_STATE_PAUSING, want: ActorStatePausing}, - {name: "paused", state: ateapipb.ActorState_ACTOR_STATE_PAUSED, want: ActorStatePaused}, {name: "crashed", state: ateapipb.ActorState_ACTOR_STATE_CRASHED, want: ActorStateCrashed}, {name: "deleting", state: ateapipb.ActorState_ACTOR_STATE_DELETING, want: ActorStateDeleting}, {name: "reverting", state: ateapipb.ActorState_ACTOR_STATE_REVERTING, want: ActorStateReverting}, @@ -665,8 +661,6 @@ func TestActorStateValuesMirrorActorState(t *testing.T) { ActorStateRunning: true, ActorStateSuspending: true, ActorStateSuspended: true, - ActorStatePausing: true, - ActorStatePaused: true, ActorStateCrashed: true, ActorStateDeleting: true, ActorStateReverting: true, diff --git a/internal/ateclient/builder.go b/internal/ateclient/builder.go index acac831ea0..c8a05b3d69 100644 --- a/internal/ateclient/builder.go +++ b/internal/ateclient/builder.go @@ -128,7 +128,7 @@ func (c *Client) Close() { // NewClient creates a new Ate API client. If endpoint is empty, it automatically // port-forwards to the ate-api-server pod in substrate's system namespace. -func NewClient(ctx context.Context, kubeconfigPath, k8sContext, endpoint, tokenFile string, traceEnabled bool) (*Client, error) { +func NewClient(ctx context.Context, kubeconfigPath, k8sContext, endpoint, tokenFile string, traceEnabled bool, extraOpts ...grpc.DialOption) (*Client, error) { tp, err := initTracing(ctx, traceEnabled) if err != nil { return nil, fmt.Errorf("failed to initialize tracing: %w", err) @@ -136,9 +136,9 @@ func NewClient(ctx context.Context, kubeconfigPath, k8sContext, endpoint, tokenF var cli *Client if endpoint != "" { - cli, err = dialDirect(ctx, kubeconfigPath, k8sContext, endpoint, tokenFile, traceEnabled) + cli, err = dialDirect(ctx, kubeconfigPath, k8sContext, endpoint, tokenFile, traceEnabled, extraOpts...) } else { - cli, err = dialPortForward(ctx, kubeconfigPath, k8sContext, tokenFile, traceEnabled) + cli, err = dialPortForward(ctx, kubeconfigPath, k8sContext, tokenFile, traceEnabled, extraOpts...) } if err != nil { @@ -152,7 +152,7 @@ func NewClient(ctx context.Context, kubeconfigPath, k8sContext, endpoint, tokenF return cli, nil } -func dialDirect(ctx context.Context, kubeconfigPath, k8sContext, endpoint, tokenFile string, traceEnabled bool) (*Client, error) { +func dialDirect(ctx context.Context, kubeconfigPath, k8sContext, endpoint, tokenFile string, traceEnabled bool, extraOpts ...grpc.DialOption) (*Client, error) { config, err := LoadKubeConfig(kubeconfigPath, k8sContext) if err != nil { return nil, fmt.Errorf("failed to load kubeconfig: %w", err) @@ -185,8 +185,9 @@ func dialDirect(ctx context.Context, kubeconfigPath, k8sContext, endpoint, token opts = append(opts, tokenOpt) if traceEnabled { - opts = append(opts, grpc.WithUnaryInterceptor(newTraceInterceptor())) + opts = append(opts, grpc.WithChainUnaryInterceptor(newTraceInterceptor())) } + opts = append(opts, extraOpts...) conn, err := grpc.NewClient(endpoint, opts...) if err != nil { @@ -207,7 +208,7 @@ func LoadKubeConfig(kubeconfigPath, k8sContext string) (*rest.Config, error) { return clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, configOverrides).ClientConfig() } -func dialPortForward(ctx context.Context, kubeconfigPath, k8sContext, tokenFile string, traceEnabled bool) (*Client, error) { +func dialPortForward(ctx context.Context, kubeconfigPath, k8sContext, tokenFile string, traceEnabled bool, extraOpts ...grpc.DialOption) (*Client, error) { config, err := LoadKubeConfig(kubeconfigPath, k8sContext) if err != nil { return nil, fmt.Errorf("failed to load kubeconfig: %w", err) @@ -247,8 +248,9 @@ func dialPortForward(ctx context.Context, kubeconfigPath, k8sContext, tokenFile opts = append(opts, tokenOpt) if traceEnabled { - opts = append(opts, grpc.WithUnaryInterceptor(newTraceInterceptor())) + opts = append(opts, grpc.WithChainUnaryInterceptor(newTraceInterceptor())) } + opts = append(opts, extraOpts...) conn, err := grpc.NewClient(localEndpoint, opts...) if err != nil { diff --git a/internal/ateinterceptors/ateinterceptors_test.go b/internal/ateinterceptors/ateinterceptors_test.go index 5f3b848d91..371f04f782 100644 --- a/internal/ateinterceptors/ateinterceptors_test.go +++ b/internal/ateinterceptors/ateinterceptors_test.go @@ -396,9 +396,9 @@ func TestServerUnaryInterceptorResponseLogMasksActorJWT(t *testing.T) { log := captureDefaultLog(t) const token = "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhY3RvciJ9.c2lnbmF0dXJl" - resp := &ateapipb.MintActorJWTResponse{ActorJwt: token} + resp := &ateapipb.MintActorJwtResponse{ActorJwt: token} - got, err := ServerUnaryInterceptor(context.Background(), &ateapipb.MintActorJWTRequest{}, &grpc.UnaryServerInfo{FullMethod: "/ateapi.Control/MintActorJWT"}, func(ctx context.Context, req interface{}) (interface{}, error) { + got, err := ServerUnaryInterceptor(context.Background(), &ateapipb.MintActorJwtRequest{}, &grpc.UnaryServerInfo{FullMethod: "/ateapi.Control/MintActorJwt"}, func(ctx context.Context, req interface{}) (interface{}, error) { return resp, nil }) if err != nil { @@ -412,7 +412,7 @@ func TestServerUnaryInterceptorResponseLogMasksActorJWT(t *testing.T) { if !strings.Contains(gotLog, `"actor_jwt":"`+protoredact.Placeholder+`"`) { t.Fatalf("log does not show the redacted actor_jwt: %s", gotLog) } - if got.(*ateapipb.MintActorJWTResponse).GetActorJwt() != token { + if got.(*ateapipb.MintActorJwtResponse).GetActorJwt() != token { t.Fatalf("interceptor mutated the response returned to the client") } } @@ -445,7 +445,7 @@ func TestInternalServerUnaryInterceptorLogMasksBytesFields(t *testing.T) { func TestServerUnaryInterceptorLogsNilResponseOnHandlerError(t *testing.T) { log := captureDefaultLog(t) - _, err := ServerUnaryInterceptor(context.Background(), &ateapipb.MintActorJWTRequest{}, &grpc.UnaryServerInfo{FullMethod: "/ateapi.Control/MintActorJWT"}, func(ctx context.Context, req interface{}) (interface{}, error) { + _, err := ServerUnaryInterceptor(context.Background(), &ateapipb.MintActorJwtRequest{}, &grpc.UnaryServerInfo{FullMethod: "/ateapi.Control/MintActorJwt"}, func(ctx context.Context, req interface{}) (interface{}, error) { return nil, apierror.PermissionDenied("no") }) if status.Code(err) != codes.PermissionDenied { diff --git a/internal/ateomnet/sandbox.go b/internal/ateomnet/sandbox.go index e3b6f23dea..03f5453699 100644 --- a/internal/ateomnet/sandbox.go +++ b/internal/ateomnet/sandbox.go @@ -23,6 +23,7 @@ import ( "io" "log/slog" "net" + "net/netip" "sync" "github.com/agent-substrate/substrate/internal/ateomnet/dns" @@ -210,13 +211,18 @@ func setupGatewaySide(ctx context.Context, ns netns.Handle, egressPort uint16) e }); err != nil { return err } - return installEgressRedirect(ns, egressPort) + return installEgressRules(ns, egressPort) } -// installEgressRedirect redirects TCP egress to atunnel, excluding the sandbox's -// own /30: that keeps ingress replies and DNS over TCP to the gateway off the -// redirect, so the relay serves them on its own listener. -func installEgressRedirect(ns netns.Handle, egressPort uint16) error { +// installEgressRules rejects link-local traffic before it reaches the egress PEP, +// then redirects other TCP egress to atunnel. The actor's own /30 is excluded +// from both rules so ingress replies and DNS reach their local sockets. +func installEgressRules(ns netns.Handle, egressPort uint16) error { + const ( + icmpDestinationAdministrativelyProhibited = 13 + icmpv6DestinationAdministrativelyProhibited = 1 + ) + if egressPort == 0 { return fmt.Errorf("actornet: atunnel egress port is required") } @@ -226,9 +232,54 @@ func installEgressRedirect(ns netns.Handle, egressPort uint16) error { } defer func() { _ = c.CloseLasting() }() - table := c.AddTable(&nftables.Table{Family: nftables.TableFamilyIPv4, Name: "ateom-actor"}) + var ipv4Table *nftables.Table + for _, family := range []struct { + tableFamily nftables.TableFamily + name string + addressLen uint32 + dstOffset uint32 + rejectCode uint8 + isPrefix func(netip.Prefix) bool + }{ + {nftables.TableFamilyIPv4, "ateom-actor", 4, 16, icmpDestinationAdministrativelyProhibited, func(prefix netip.Prefix) bool { return prefix.Addr().Is4() }}, + {nftables.TableFamilyIPv6, "ateom-actor-v6", 16, 24, icmpv6DestinationAdministrativelyProhibited, func(prefix netip.Prefix) bool { return prefix.Addr().Is6() }}, + } { + table := c.AddTable(&nftables.Table{Family: family.tableFamily, Name: family.name}) + if family.tableFamily == nftables.TableFamilyIPv4 { + ipv4Table = table + } + filter := c.AddChain(&nftables.Chain{ + Name: "disallowed-egress", Table: table, Type: nftables.ChainTypeFilter, + Hooknum: nftables.ChainHookPrerouting, Priority: nftables.ChainPriorityRaw, + }) + if family.tableFamily == nftables.TableFamilyIPv4 { + c.AddRule(&nftables.Rule{Table: table, Chain: filter, Exprs: []expr.Any{ + &expr.Payload{DestRegister: 1, Base: expr.PayloadBaseNetworkHeader, Offset: family.dstOffset, Len: family.addressLen}, + &expr.Bitwise{SourceRegister: 1, DestRegister: 1, Len: 4, Mask: actorSubnetMask, Xor: []byte{0, 0, 0, 0}}, + &expr.Cmp{Op: expr.CmpOpEq, Register: 1, Data: actorSubnetBase}, + &expr.Verdict{Kind: expr.VerdictAccept}, + }}) + } + for _, prefix := range linkLocalEgressCIDRRanges { + if !family.isPrefix(prefix) { + continue + } + mask := net.CIDRMask(prefix.Bits(), int(family.addressLen*8)) + address := prefix.Addr().AsSlice() + expressions := []expr.Any{ + &expr.Payload{DestRegister: 1, Base: expr.PayloadBaseNetworkHeader, Offset: family.dstOffset, Len: family.addressLen}, + } + expressions = append(expressions, + &expr.Bitwise{SourceRegister: 1, DestRegister: 1, Len: family.addressLen, Mask: mask, Xor: make([]byte, family.addressLen)}, + &expr.Cmp{Op: expr.CmpOpEq, Register: 1, Data: address}, + &expr.Reject{Type: unix.NFT_REJECT_ICMP_UNREACH, Code: family.rejectCode}, + ) + c.AddRule(&nftables.Rule{Table: table, Chain: filter, Exprs: expressions}) + } + } + prerouting := c.AddChain(&nftables.Chain{ - Name: "prerouting", Table: table, Type: nftables.ChainTypeNAT, + Name: "prerouting", Table: ipv4Table, Type: nftables.ChainTypeNAT, Hooknum: nftables.ChainHookPrerouting, Priority: nftables.ChainPriorityNATDest, }) @@ -247,7 +298,7 @@ func installEgressRedirect(ns netns.Handle, egressPort uint16) error { &expr.Immediate{Register: 1, Data: binaryutil.BigEndian.PutUint16(egressPort)}, &expr.Redir{RegisterProtoMin: 1}, ) - c.AddRule(&nftables.Rule{Table: table, Chain: prerouting, Exprs: exprs}) + c.AddRule(&nftables.Rule{Table: ipv4Table, Chain: prerouting, Exprs: exprs}) if err := c.Flush(); err != nil { return fmt.Errorf("while installing the actor egress redirect: %w", err) @@ -255,6 +306,14 @@ func installEgressRedirect(ns netns.Handle, egressPort uint16) error { return nil } +// linkLocalEgressCIDRRanges cover all IPv4 and IPv6 link-local space. The +// nftables chain accepts the actor's 169.254.17.0/30 first because that network +// carries actor ingress and DNS relay traffic. +var linkLocalEgressCIDRRanges = []netip.Prefix{ + netip.MustParsePrefix("169.254.0.0/16"), + netip.MustParsePrefix("fe80::/10"), +} + // The actor subnet, in the form the nftables comparison takes. var actorSubnetBase, actorSubnetMask = func() ([]byte, []byte) { _, subnet, err := net.ParseCIDR(ActorVethSubnet) diff --git a/internal/ateomnet/sandbox_atunnel_linux_test.go b/internal/ateomnet/sandbox_atunnel_linux_test.go index e02617a6a0..4b68f8d7ea 100644 --- a/internal/ateomnet/sandbox_atunnel_linux_test.go +++ b/internal/ateomnet/sandbox_atunnel_linux_test.go @@ -96,3 +96,79 @@ func TestSandboxEgressReachesAtunnelOnAnyPort(t *testing.T) { go accept() } } + +func TestSandboxLinkLocalDoesNotReachAtunnel(t *testing.T) { + roottest.Require(t, "creates network namespaces") + ctx := context.Background() + const egressPort = 15001 + + n, err := ateomnet.SetupSandboxNetwork(ctx, ateomnet.SandboxNetworkConfig{ + ActorUID: "99999999-9999-9999-9999-999999999999", + Veth: true, + EgressPort: egressPort, + }) + if err != nil { + t.Fatalf("SetupSandboxNetwork: %v", err) + } + t.Cleanup(func() { ateomnet.CleanupSandboxNetwork(n) }) + + listeners, err := netns.Listen(ctx, n.GatewayNetNS, []uint16{egressPort}) + if err != nil { + t.Fatalf("netns.Listen: %v", err) + } + defer listeners[0].Close() + listener := listeners[0].(*net.TCPListener) + + if err := listener.SetDeadline(time.Now().Add(500 * time.Millisecond)); err != nil { + t.Fatal(err) + } + var linkLocalDialErr error + if err := netns.Do(ctx, n.RuntimeNetNS, func(context.Context) error { + conn, err := net.DialTimeout("tcp", "169.254.169.254:80", 250*time.Millisecond) + linkLocalDialErr = err + if conn != nil { + conn.Close() + } + return nil + }); err != nil { + t.Fatalf("dial to link-local address: %v", err) + } + if linkLocalDialErr == nil { + t.Fatal("dial to link-local address succeeded") + } + if timeout, ok := linkLocalDialErr.(net.Error); ok && timeout.Timeout() { + t.Fatalf("link-local dial timed out instead of being rejected: %v", linkLocalDialErr) + } + if conn, err := listener.Accept(); err == nil { + conn.Close() + t.Fatal("link-local connection reached atunnel") + } else if timeout, ok := err.(net.Error); !ok || !timeout.Timeout() { + t.Fatalf("Accept after link-local dial: %v", err) + } + + if err := listener.SetDeadline(time.Time{}); err != nil { + t.Fatal(err) + } + const allowedDestination = "192.0.2.1:443" + if err := netns.Do(ctx, n.RuntimeNetNS, func(context.Context) error { + conn, err := net.Dial("tcp", allowedDestination) + if err != nil { + return err + } + return conn.Close() + }); err != nil { + t.Fatalf("dial to allowed CIDR: %v", err) + } + conn, err := listener.Accept() + if err != nil { + t.Fatalf("accept allowed connection: %v", err) + } + defer conn.Close() + destination, err := atunnel.TCPOriginalDestination(conn) + if err != nil { + t.Fatalf("TCPOriginalDestination: %v", err) + } + if destination != allowedDestination { + t.Errorf("atunnel saw destination %q, want %q", destination, allowedDestination) + } +} diff --git a/internal/atunnel/client.go b/internal/atunnel/client.go index 1ada528c94..cc6077c839 100644 --- a/internal/atunnel/client.go +++ b/internal/atunnel/client.go @@ -24,10 +24,14 @@ import ( "io" "net" "net/http" + "net/http/httptrace" "net/url" "os" "strings" + "sync" + "sync/atomic" "syscall" + "time" ) // TODO(liorlieberman): support/use CONNECT on Ingress as well. @@ -83,6 +87,11 @@ type Client struct { gatewayAddress string tlsConfig *tls.Config dialContext DialFunc + + mu sync.Mutex + closed bool + h2Conns []*h2ClientConn + dialing *dialCall } // Client implements egressDialer. @@ -119,6 +128,7 @@ func NewClient(cfg ClientConfig, opts ...ClientOption) (*Client, error) { RootCAs: rootCAs, ServerName: cfg.ServerName, GetClientCertificate: cfg.GetClientCertificate, + NextProtos: []string{"h2", "http/1.1"}, }, } for _, opt := range opts { @@ -127,22 +137,280 @@ func NewClient(cfg ClientConfig, opts ...ClientOption) (*Client, error) { return client, nil } -// DialContext opens a CONNECT tunnel to destination. destination becomes the +// Close any pooled HTTP/2 connections to the egress gateway and cancels +// any in-flight connection dial. +func (c *Client) Close() error { + c.mu.Lock() + c.closed = true + conns := c.h2Conns + c.h2Conns = nil + call := c.dialing + c.dialing = nil + c.mu.Unlock() + + if call != nil { + call.cancel() + } + var err error + for _, h2Conn := range conns { + err = errors.Join(err, h2Conn.cc.Close()) + } + return err +} + +// Open a CONNECT tunnel to destination. destination becomes the // request authority, so it must include an explicit port. +// TODO(yanavlasov): update .md file with the description of this machinery. +// TODO(yanavlasov): consider simplifying this code by dropping support for H/1. func (c *Client) DialContext(ctx context.Context, destination string) (net.Conn, error) { if err := validateDestination(destination); err != nil { return nil, err } - rawConn, err := c.dialContext(ctx, "tcp", c.gatewayAddress) + for { + h2Conn, wasVerified, h1Conn, err := c.acquireConn(ctx) + if err != nil { + return nil, err + } + if h1Conn != nil { + return c.connectH1(h1Conn, destination) + } + conn, retry, err := c.connectH2(ctx, h2Conn, wasVerified, destination) + if retry { + continue + } + return conn, err + } +} + +type dialCall struct { + done chan struct{} + + mu sync.Mutex + waiters int + cancel context.CancelFunc + h2Conn *h2ClientConn + h1Conn *tls.Conn + err error +} + +// Open one connection at a time to avoid opening more H/2 connection while one is already being established. +func (c *Client) acquireConn(ctx context.Context) (*h2ClientConn, bool, *tls.Conn, error) { + for { + if err := ctx.Err(); err != nil { + return nil, false, nil, fmt.Errorf("atunnel: connecting to egress gateway: %w", err) + } + + // First check if there is an existing H/2 connection with available stream capacity. + // TODO(yanavlasov): do not use existing H/2 connections if their keys have been rotated. + c.mu.Lock() + if c.closed { + c.mu.Unlock() + return nil, false, nil, fmt.Errorf("atunnel: connecting to egress gateway: %w", net.ErrClosed) + } + keep := c.h2Conns[:0] + var reserved *h2ClientConn + var wasVerified bool + // Loop through established H/2 connections. + // Find one connection with available capacity. + // At the same time, discard these that are closed, or idle and GOAWAY'ed. + for _, h2Conn := range c.h2Conns { + if h2Conn.isClosed() { + continue + } + if reserved == nil { + if err := h2Conn.cc.Reserve(); err == nil { + reserved = h2Conn + wasVerified = h2Conn.verified.Load() + keep = append(keep, h2Conn) + continue + } + // If connection is not yet closed, but has 0 streams and could not reseve a new + // stream, it means it received GOAWAY from the peer. Discard these as well. + if h2Conn.isClosed() || h2Conn.cc.InFlight() == 0 { + continue + } + } + keep = append(keep, h2Conn) + } + c.h2Conns = keep + if reserved != nil { + c.mu.Unlock() + return reserved, wasVerified, nil, nil + } + + // If there are no H/2 connections with idle capacity, start a new connection. + // Dial new connection if there are no in progress connections already. + call := c.dialing + if call == nil { + dialCtx, cancel := context.WithCancel(context.Background()) + call = &dialCall{ + done: make(chan struct{}), + waiters: 1, + cancel: cancel, + } + c.dialing = call + c.mu.Unlock() + go c.runDial(dialCtx, call) + } else { + call.mu.Lock() + call.waiters++ + call.mu.Unlock() + c.mu.Unlock() + } + + select { + case <-call.done: + call.mu.Lock() + call.waiters-- + // If we negotiated CONNECT over H/1 connection, return its TCP stream to the atunnel. + if call.h1Conn != nil { + conn := call.h1Conn + call.h1Conn = nil + call.mu.Unlock() + return nil, false, conn, nil + } + h2Conn := call.h2Conn + err := call.err + call.mu.Unlock() + + if err != nil { + if ctx.Err() != nil { + return nil, false, nil, fmt.Errorf("atunnel: connecting to egress gateway: %w", ctx.Err()) + } + return nil, false, nil, err + } + if h2Conn != nil { + wasVerified := h2Conn.verified.Load() + if err := h2Conn.cc.Reserve(); err == nil { + return h2Conn, wasVerified, nil, nil + } + if !wasVerified && h2Conn.isClosed() { + c.removeH2Conn(h2Conn) + return nil, false, nil, connectExchangeError("reading CONNECT response", h2Conn.connErr(nil)) + } + } + case <-ctx.Done(): + c.mu.Lock() + call.mu.Lock() + call.waiters-- + var orphan *tls.Conn + if call.waiters == 0 { + if c.dialing == call { + c.dialing = nil + } + call.cancel() + if call.h1Conn != nil { + orphan = call.h1Conn + call.h1Conn = nil + } + } + call.mu.Unlock() + c.mu.Unlock() + if orphan != nil { + _ = orphan.Close() + } + return nil, false, nil, fmt.Errorf("atunnel: connecting to egress gateway: %w", ctx.Err()) + } + } +} + +// Dial egress gateway and see what protocol was offered by egress gateway. +func (c *Client) runDial(dialCtx context.Context, call *dialCall) { + defer call.cancel() + + var ( + h2Conn *h2ClientConn + h1Conn *tls.Conn + err error + ) + rawConn, dialErr := c.dialContext(dialCtx, "tcp", c.gatewayAddress) + if dialErr != nil { + err = fmt.Errorf("atunnel: connecting to egress gateway: %w", dialErr) + } else { + tlsConn := tls.Client(rawConn, c.tlsConfig.Clone()) + if hsErr := tlsConn.HandshakeContext(dialCtx); hsErr != nil { + _ = rawConn.Close() + err = fmt.Errorf("%w: %w", ErrGatewayHandshake, hsErr) + } else if tlsConn.ConnectionState().NegotiatedProtocol == "h2" { + h2Conn, err = c.newH2ClientConn(dialCtx, tlsConn) + } else { + h1Conn = tlsConn + } + } + + c.mu.Lock() + call.mu.Lock() + if c.dialing == call { + c.dialing = nil + } + if c.closed || call.waiters == 0 { + if c.closed && err == nil { + call.err = fmt.Errorf("atunnel: connecting to egress gateway: %w", net.ErrClosed) + } else { + call.err = err + } + call.mu.Unlock() + c.mu.Unlock() + if h2Conn != nil { + _ = h2Conn.cc.Close() + } + if h1Conn != nil { + _ = h1Conn.Close() + } + close(call.done) + return + } + if h2Conn != nil { + c.h2Conns = append(c.h2Conns, h2Conn) + call.h2Conn = h2Conn + } + call.h1Conn = h1Conn + call.err = err + close(call.done) + call.mu.Unlock() + c.mu.Unlock() +} + +func (c *Client) newH2ClientConn(ctx context.Context, tlsConn *tls.Conn) (*h2ClientConn, error) { + raw := &trackedTLSConn{Conn: tlsConn} + var protocols http.Protocols + protocols.SetHTTP2(true) + tr := &http.Transport{ + Protocols: &protocols, + DisableCompression: true, + DialTLSContext: func(context.Context, string, string) (net.Conn, error) { + return raw, nil + }, + } + cc, err := tr.NewClientConn(ctx, "https", c.gatewayAddress) if err != nil { - return nil, fmt.Errorf("atunnel: connecting to egress gateway: %w", err) + _ = raw.Close() + connErr := raw.firstErr() + if connErr == nil { + connErr = err + } + return nil, connectExchangeError("writing CONNECT request", connErr) } - tlsConn := tls.Client(rawConn, c.tlsConfig.Clone()) - if err := tlsConn.HandshakeContext(ctx); err != nil { - _ = rawConn.Close() - return nil, fmt.Errorf("%w: %w", ErrGatewayHandshake, err) + return &h2ClientConn{ + cc: cc, + raw: raw, + localAddr: tlsConn.LocalAddr(), + remoteAddr: tlsConn.RemoteAddr(), + }, nil +} + +func (c *Client) removeH2Conn(target *h2ClientConn) { + c.mu.Lock() + defer c.mu.Unlock() + for i, h2Conn := range c.h2Conns { + if h2Conn == target { + c.h2Conns = append(c.h2Conns[:i], c.h2Conns[i+1:]...) + return + } } +} +func (c *Client) connectH1(tlsConn *tls.Conn, destination string) (net.Conn, error) { req := &http.Request{ Method: http.MethodConnect, URL: &url.URL{Host: destination}, @@ -163,16 +431,565 @@ func (c *Client) DialContext(ctx context.Context, destination string) (net.Conn, body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10)) _ = resp.Body.Close() _ = tlsConn.Close() - message := strings.TrimSpace(string(body)) - if message == "" { - message = http.StatusText(resp.StatusCode) - } - return nil, &ConnectRejectedError{StatusCode: resp.StatusCode, Status: resp.Status, Message: message} + return nil, newConnectRejectedError(resp, body) } + // Return wrapped net.Conn including the reader that was used for CONNECT handshake. return &bufferedConn{Conn: tlsConn, reader: reader}, nil } +func (c *Client) connectH2(ctx context.Context, h2Conn *h2ClientConn, wasVerified bool, destination string) (net.Conn, bool, error) { + streamCtx, streamCancel := context.WithCancel(context.Background()) + reqBody := newH2RequestBody() + traceCtx := httptrace.WithClientTrace(streamCtx, &httptrace.ClientTrace{ + WroteRequest: func(info httptrace.WroteRequestInfo) { + reqBody.onWroteRequest(info.Err) + }, + }) + stopCancel := context.AfterFunc(ctx, streamCancel) + + // Roundrip the CONNECT request with provided body writer. + req := (&http.Request{ + Method: http.MethodConnect, + URL: &url.URL{Host: destination}, + Host: destination, + Header: make(http.Header), + Body: reqBody, + ContentLength: -1, + }).WithContext(traceCtx) + + resp, err := h2Conn.cc.RoundTrip(req) + if !stopCancel() { + reqBody.abort() + streamCancel() + if err == nil { + _ = resp.Body.Close() + } + if !wasVerified && h2Conn.isClosed() { + c.removeH2Conn(h2Conn) + _ = h2Conn.cc.Close() + } + return nil, false, connectExchangeError("reading CONNECT response", ctx.Err()) + } + if err != nil { + reqBody.abort() + streamCancel() + if h2Conn.isClosed() { + c.removeH2Conn(h2Conn) + if h2Conn.cc.InFlight() == 0 { + _ = h2Conn.cc.Close() + } + } + if wasVerified && ctx.Err() == nil && (h2Conn.isClosed() || h2Conn.cc.Available() == 0) { + c.removeH2Conn(h2Conn) + return nil, true, nil + } + return nil, false, connectExchangeError("reading CONNECT response", h2Conn.connErr(err)) + } + + h2Conn.verified.Store(true) + if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices { + body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10)) + reqBody.abort() + _ = resp.Body.Close() + streamCancel() + return nil, false, newConnectRejectedError(resp, body) + } + + // Create new stream wrapper with body reader and writer. + // newH2StreamConn implements all net.Conn method redirecting Read/Write calls to + // body reader and writer for the H/2 stream. + return newH2StreamConn(h2Conn, reqBody, resp.Body, streamCancel), false, nil +} + +func newConnectRejectedError(resp *http.Response, body []byte) *ConnectRejectedError { + message := strings.TrimSpace(string(body)) + if message == "" { + message = http.StatusText(resp.StatusCode) + } + return &ConnectRejectedError{ + StatusCode: resp.StatusCode, + Status: resp.Status, + Message: message, + } +} + +type h2ClientConn struct { + // H/2 connection. + cc *http.ClientConn + // TLS connection backing this H/2 connection. + raw *trackedTLSConn + localAddr net.Addr + remoteAddr net.Addr + // Set to true when at least one request had been completed. + // Used to distinguish between connection establishment errors and errors + // on an already working connection. + verified atomic.Bool +} + +func (h *h2ClientConn) isClosed() bool { + return h.cc.Err() != nil || h.raw.firstErr() != nil +} + +func (h *h2ClientConn) connErr(fallback error) error { + if err := h.raw.firstErr(); err != nil { + return err + } + return fallback +} + +type trackedTLSConn struct { + *tls.Conn + mu sync.Mutex + closed bool + err error +} + +func (c *trackedTLSConn) recordErr(err error) { + if err == nil { + return + } + c.mu.Lock() + if !c.closed && c.err == nil { + c.err = err + } + c.mu.Unlock() +} + +func (c *trackedTLSConn) Read(p []byte) (int, error) { + n, err := c.Conn.Read(p) + if err != nil { + c.recordErr(err) + } + return n, err +} + +func (c *trackedTLSConn) Write(p []byte) (int, error) { + n, err := c.Conn.Write(p) + if err != nil { + c.recordErr(err) + } + return n, err +} + +func (c *trackedTLSConn) Close() error { + c.mu.Lock() + c.closed = true + c.mu.Unlock() + if nc := c.Conn.NetConn(); nc != nil { + t := time.AfterFunc(250*time.Millisecond, func() { _ = nc.Close() }) + defer t.Stop() + } + return c.Conn.Close() +} + +func (c *trackedTLSConn) firstErr() error { + c.mu.Lock() + defer c.mu.Unlock() + return c.err +} + +type readSlot struct { + buf []byte + resCh chan readResult +} + +type readResult struct { + n int + err error +} + +type h2RequestBody struct { + readyCh chan readSlot + abortCh chan struct{} + doneCh chan struct{} + + abortOnce sync.Once + doneOnce sync.Once + + errMu sync.Mutex + writeErr error + + // Guarded by h2StreamConn.writeMu. + pendingSlot *readSlot + eofSent bool +} + +func newH2RequestBody() *h2RequestBody { + return &h2RequestBody{ + readyCh: make(chan readSlot), + abortCh: make(chan struct{}), + doneCh: make(chan struct{}), + } +} + +func (b *h2RequestBody) Read(buf []byte) (int, error) { + resCh := make(chan readResult, 1) + select { + case b.readyCh <- readSlot{buf: buf, resCh: resCh}: + case <-b.abortCh: + return 0, io.EOF + } + select { + case res := <-resCh: + return res.n, res.err + case <-b.abortCh: + return 0, io.EOF + } +} + +func (b *h2RequestBody) Close() error { + b.abort() + b.finish(net.ErrClosed) + return nil +} + +func (b *h2RequestBody) onWroteRequest(err error) { + b.finish(err) +} + +func (b *h2RequestBody) finish(err error) { + b.doneOnce.Do(func() { + b.errMu.Lock() + b.writeErr = err + b.errMu.Unlock() + close(b.doneCh) + }) +} + +func (b *h2RequestBody) abort() { + b.abortOnce.Do(func() { + close(b.abortCh) + }) +} + +func (b *h2RequestBody) err() error { + b.errMu.Lock() + defer b.errMu.Unlock() + if b.writeErr != nil { + return b.writeErr + } + return net.ErrClosed +} + +func (b *h2RequestBody) writeError() error { + b.errMu.Lock() + defer b.errMu.Unlock() + return b.writeErr +} + +type readOutcome struct { + buf []byte + err error +} + +type h2StreamConn struct { + h2Conn *h2ClientConn + reqBody *h2RequestBody + respBody io.ReadCloser + cancel context.CancelFunc + + readMu sync.Mutex + readOnce sync.Once + readReqCh chan int + readResCh chan readOutcome + readInFlight bool + readPending []byte + readErr error + + writeMu sync.Mutex + + readDeadline streamDeadline + writeDeadline streamDeadline + + closeOnce sync.Once + closedCh chan struct{} +} + +func newH2StreamConn(h2Conn *h2ClientConn, reqBody *h2RequestBody, respBody io.ReadCloser, cancel context.CancelFunc) *h2StreamConn { + return &h2StreamConn{ + h2Conn: h2Conn, + reqBody: reqBody, + respBody: respBody, + cancel: cancel, + readReqCh: make(chan int, 1), + readResCh: make(chan readOutcome, 1), + readDeadline: makeStreamDeadline(), + writeDeadline: makeStreamDeadline(), + closedCh: make(chan struct{}), + } +} + +func (c *h2StreamConn) startReader() { + c.readOnce.Do(func() { + go func() { + for { + var size int + select { + case size = <-c.readReqCh: + case <-c.closedCh: + return + } + buf := make([]byte, size) + n, err := c.respBody.Read(buf) + c.readResCh <- readOutcome{buf: buf[:n], err: err} + if err != nil { + return + } + } + }() + }) +} + +func (c *h2StreamConn) Read(p []byte) (int, error) { + c.readMu.Lock() + defer c.readMu.Unlock() + + if isClosedChan(c.closedCh) { + return 0, net.ErrClosed + } + if isClosedChan(c.readDeadline.wait()) { + return 0, os.ErrDeadlineExceeded + } + if len(c.readPending) > 0 { + n := copy(p, c.readPending) + c.readPending = c.readPending[n:] + if len(c.readPending) == 0 && c.readErr != nil { + err := c.readErr + c.readErr = nil + return n, err + } + return n, nil + } + if c.readErr != nil { + return 0, c.readErr + } + if len(p) == 0 { + return 0, nil + } + + c.startReader() + if !c.readInFlight { + c.readInFlight = true + c.readReqCh <- len(p) + } + + select { + case out := <-c.readResCh: + c.readInFlight = false + c.readErr = out.err + n := copy(p, out.buf) + c.readPending = out.buf[n:] + if len(c.readPending) > 0 { + return n, nil + } + return n, c.readErr + case <-c.readDeadline.wait(): + return 0, os.ErrDeadlineExceeded + case <-c.closedCh: + return 0, net.ErrClosed + } +} + +func (c *h2StreamConn) Write(p []byte) (int, error) { + c.writeMu.Lock() + defer c.writeMu.Unlock() + + if isClosedChan(c.closedCh) { + return 0, net.ErrClosed + } + if isClosedChan(c.writeDeadline.wait()) { + return 0, os.ErrDeadlineExceeded + } + if c.reqBody.eofSent { + return 0, net.ErrClosed + } + if len(p) == 0 { + return 0, nil + } + + var total int + for len(p) > 0 { + slot, err := c.acquireWriteSlot() + if err != nil { + return total, err + } + n := copy(slot.buf, p) + p = p[n:] + total += n + slot.resCh <- readResult{n: n} + + select { + case nextSlot := <-c.reqBody.readyCh: + c.reqBody.pendingSlot = &nextSlot + case <-c.reqBody.doneCh: + return total, c.reqBody.err() + case <-c.closedCh: + return total, net.ErrClosed + case <-c.writeDeadline.wait(): + return total, os.ErrDeadlineExceeded + } + } + return total, nil +} + +func (c *h2StreamConn) acquireWriteSlot() (readSlot, error) { + if c.reqBody.pendingSlot != nil { + slot := *c.reqBody.pendingSlot + c.reqBody.pendingSlot = nil + return slot, nil + } + select { + case slot := <-c.reqBody.readyCh: + return slot, nil + case <-c.reqBody.doneCh: + return readSlot{}, c.reqBody.err() + case <-c.closedCh: + return readSlot{}, net.ErrClosed + case <-c.writeDeadline.wait(): + return readSlot{}, os.ErrDeadlineExceeded + } +} + +func (c *h2StreamConn) CloseWrite() error { + c.writeMu.Lock() + defer c.writeMu.Unlock() + + if isClosedChan(c.closedCh) { + return net.ErrClosed + } + if c.reqBody.eofSent { + return nil + } + var slot readSlot + if c.reqBody.pendingSlot != nil { + slot = *c.reqBody.pendingSlot + c.reqBody.pendingSlot = nil + } else { + select { + case slot = <-c.reqBody.readyCh: + case <-c.reqBody.doneCh: + c.reqBody.eofSent = true + return c.reqBody.writeError() + case <-c.closedCh: + return net.ErrClosed + } + } + c.reqBody.eofSent = true + slot.resCh <- readResult{n: 0, err: io.EOF} + select { + case <-c.reqBody.doneCh: + return c.reqBody.writeError() + case <-c.closedCh: + return nil + } +} + +func (c *h2StreamConn) Close() error { + var closeErr error + c.closeOnce.Do(func() { + if c.writeMu.TryLock() { + if !c.reqBody.eofSent && c.reqBody.pendingSlot != nil { + slot := *c.reqBody.pendingSlot + c.reqBody.pendingSlot = nil + c.reqBody.eofSent = true + slot.resCh <- readResult{n: 0, err: io.EOF} + select { + case <-c.reqBody.doneCh: + case <-time.After(100 * time.Millisecond): + } + } + c.writeMu.Unlock() + } + close(c.closedCh) + c.reqBody.abort() + closeErr = c.respBody.Close() + c.cancel() + }) + return closeErr +} + +func (c *h2StreamConn) LocalAddr() net.Addr { + return c.h2Conn.localAddr +} + +func (c *h2StreamConn) RemoteAddr() net.Addr { + return c.h2Conn.remoteAddr +} + +func (c *h2StreamConn) SetDeadline(t time.Time) error { + c.readDeadline.set(t) + c.writeDeadline.set(t) + return nil +} + +func (c *h2StreamConn) SetReadDeadline(t time.Time) error { + c.readDeadline.set(t) + return nil +} + +func (c *h2StreamConn) SetWriteDeadline(t time.Time) error { + c.writeDeadline.set(t) + return nil +} + +type streamDeadline struct { + mu sync.Mutex + timer *time.Timer + cancel chan struct{} +} + +func makeStreamDeadline() streamDeadline { + return streamDeadline{cancel: make(chan struct{})} +} + +func (d *streamDeadline) set(t time.Time) { + d.mu.Lock() + defer d.mu.Unlock() + + if d.timer != nil && !d.timer.Stop() { + <-d.cancel + } + d.timer = nil + + closed := isClosedChan(d.cancel) + if t.IsZero() { + if closed { + d.cancel = make(chan struct{}) + } + return + } + + if dur := time.Until(t); dur > 0 { + if closed { + d.cancel = make(chan struct{}) + } + d.timer = time.AfterFunc(dur, func() { + close(d.cancel) + }) + return + } + + if !closed { + close(d.cancel) + } +} + +func (d *streamDeadline) wait() chan struct{} { + d.mu.Lock() + defer d.mu.Unlock() + return d.cancel +} + +func isClosedChan(c <-chan struct{}) bool { + select { + case <-c: + return true + default: + return false + } +} + // connectExchangeError wraps a failure that happened after the TLS handshake // returned but before the gateway answered CONNECT, naming the front door as // the cause when the connection was torn down rather than answered. diff --git a/internal/atunnel/client_test.go b/internal/atunnel/client_test.go index 1d82aee9ef..f7ffb0d4dd 100644 --- a/internal/atunnel/client_test.go +++ b/internal/atunnel/client_test.go @@ -32,6 +32,8 @@ import ( "os" "path/filepath" "strings" + "sync" + "sync/atomic" "syscall" "testing" "time" @@ -91,23 +93,319 @@ func TestClientDialContext(t *testing.T) { } } -func TestClientDialContextRejected(t *testing.T) { +func TestClientDialContextHTTP2(t *testing.T) { ca := newTestCA(t) - gatewayAddress := serveTestConnectGateway(t, ca, func(conn net.Conn, _ *http.Request) { - body := "denied by policy" - _, _ = fmt.Fprintf(conn, "HTTP/1.1 403 Forbidden\r\nContent-Length: %d\r\n\r\n%s", len(body), body) - }) + request := make(chan *http.Request, 1) + gatewayAddress, _ := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + request <- r + flusher, ok := w.(http.Flusher) + if !ok { + t.Error("ResponseWriter does not implement http.Flusher") + return + } + w.WriteHeader(http.StatusOK) + if _, err := io.WriteString(w, "hello"); err != nil { + t.Errorf("writing initial stream payload: %v", err) + return + } + flusher.Flush() + + got, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("reading client stream: %v", err) + return + } + if string(got) != "ping" { + t.Errorf("client stream payload = %q, want ping", got) + return + } + if _, err := io.WriteString(w, "pong"); err != nil { + t.Errorf("writing reply stream payload: %v", err) + return + } + flusher.Flush() + })) + client := newTestClient(t, ca, WithDialer(dialFixedAddress(gatewayAddress))) + + dialCtx, cancelDial := context.WithCancel(context.Background()) + conn, err := client.DialContext(dialCtx, "192.0.2.10:443") + if err != nil { + t.Fatal(err) + } + defer conn.Close() + // Canceling the dial context after DialContext returns must not tear down + // the established tunnel stream. + cancelDial() + + gotRequest := receiveWithin(t, request, "HTTP/2 CONNECT request") + if gotRequest.ProtoMajor != 2 { + t.Errorf("proto = %s, want HTTP/2", gotRequest.Proto) + } + if gotRequest.Method != http.MethodConnect { + t.Errorf("method = %q, want CONNECT", gotRequest.Method) + } + if gotRequest.Host != "192.0.2.10:443" { + t.Errorf("authority = %q, want 192.0.2.10:443", gotRequest.Host) + } + for name := range gotRequest.Header { + if strings.HasPrefix(strings.ToLower(name), "x-ate-") { + t.Errorf("legacy actor-reference header %q was sent", name) + } + } + if got := gotRequest.Header.Get("Authorization"); got != "" { + t.Errorf("Authorization = %q, want empty", got) + } + + hello := make([]byte, len("hello")) + if _, err := io.ReadFull(conn, hello); err != nil { + t.Fatal(err) + } + if string(hello) != "hello" { + t.Errorf("initial payload = %q, want hello", hello) + } + + if _, err := io.WriteString(conn, "ping"); err != nil { + t.Fatal(err) + } + cw, ok := conn.(interface{ CloseWrite() error }) + if !ok { + t.Fatal("conn does not implement CloseWrite") + } + if err := cw.CloseWrite(); err != nil { + t.Fatalf("CloseWrite: %v", err) + } + + reply, err := io.ReadAll(conn) + if err != nil { + t.Fatal(err) + } + if string(reply) != "pong" { + t.Errorf("reply payload = %q, want pong", reply) + } +} + +func TestClientDialContextHTTP2MultiplexesStreams(t *testing.T) { + ca := newTestCA(t) + const streamCount = 4 + allOpen := make(chan struct{}) + var opened atomic.Int32 + + gatewayAddress, _ := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + flusher, ok := w.(http.Flusher) + if !ok { + t.Error("ResponseWriter does not implement http.Flusher") + return + } + w.WriteHeader(http.StatusOK) + flusher.Flush() + + if opened.Add(1) == streamCount { + close(allOpen) + } + <-allOpen + + data, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("reading stream for %s: %v", r.Host, err) + return + } + _, _ = fmt.Fprintf(w, "%s:%s", r.Host, data) + flusher.Flush() + })) + + var tcpDials atomic.Int32 + client := newTestClient(t, ca, WithDialer(func(ctx context.Context, network, _ string) (net.Conn, error) { + tcpDials.Add(1) + return (&net.Dialer{}).DialContext(ctx, network, gatewayAddress) + })) + + var wg sync.WaitGroup + for i := range streamCount { + wg.Add(1) + go func(idx int) { + defer wg.Done() + dest := fmt.Sprintf("192.0.2.%d:443", 10+idx) + conn, err := client.DialContext(context.Background(), dest) + if err != nil { + t.Errorf("DialContext(%s): %v", dest, err) + return + } + defer conn.Close() + + msg := fmt.Sprintf("msg-%d", idx) + if _, err := io.WriteString(conn, msg); err != nil { + t.Errorf("Write(%s): %v", dest, err) + return + } + if cw, ok := conn.(interface{ CloseWrite() error }); ok { + if err := cw.CloseWrite(); err != nil { + t.Errorf("CloseWrite(%s): %v", dest, err) + return + } + } + got, err := io.ReadAll(conn) + if err != nil { + t.Errorf("ReadAll(%s): %v", dest, err) + return + } + want := fmt.Sprintf("%s:%s", dest, msg) + if string(got) != want { + t.Errorf("stream %d got %q, want %q", idx, got, want) + } + }(i) + } + wg.Wait() + + if got := tcpDials.Load(); got != 1 { + t.Errorf("tcpDials = %d, want 1 shared HTTP/2 connection", got) + } +} + +func TestClientDialContextHTTP2ReconnectsAfterConnectionClose(t *testing.T) { + ca := newTestCA(t) + gatewayAddress, closeActiveConns := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + if f, ok := w.(http.Flusher); ok { + f.Flush() + } + _, _ = io.Copy(w, r.Body) + })) + + var tcpDials atomic.Int32 + client := newTestClient(t, ca, WithDialer(func(ctx context.Context, network, _ string) (net.Conn, error) { + tcpDials.Add(1) + return (&net.Dialer{}).DialContext(ctx, network, gatewayAddress) + })) + + conn1, err := client.DialContext(context.Background(), "192.0.2.10:443") + if err != nil { + t.Fatal(err) + } + _ = conn1.Close() + + closeActiveConns() + + conn2, err := client.DialContext(context.Background(), "192.0.2.10:443") + if err != nil { + t.Fatalf("second DialContext after gateway connection closed: %v", err) + } + defer conn2.Close() + + if _, err := io.WriteString(conn2, "after-reconnect"); err != nil { + t.Fatal(err) + } + if cw, ok := conn2.(interface{ CloseWrite() error }); ok { + _ = cw.CloseWrite() + } + got, err := io.ReadAll(conn2) + if err != nil { + t.Fatal(err) + } + if string(got) != "after-reconnect" { + t.Errorf("got %q, want after-reconnect", got) + } + if dials := tcpDials.Load(); dials != 2 { + t.Errorf("tcpDials = %d, want 2", dials) + } +} + +func TestClientDialContextHTTP2Deadlines(t *testing.T) { + ca := newTestCA(t) + releaseServerSend := make(chan struct{}) + gatewayAddress, _ := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + if f, ok := w.(http.Flusher); ok { + f.Flush() + } + <-releaseServerSend + _, _ = io.WriteString(w, "delayed") + })) client := newTestClient(t, ca, WithDialer(dialFixedAddress(gatewayAddress))) - _, err := client.DialContext(context.Background(), "192.0.2.10:443") - if err == nil || !strings.Contains(err.Error(), "denied by policy") { - t.Fatalf("DialContext error = %v, want policy rejection", err) + conn, err := client.DialContext(context.Background(), "192.0.2.10:443") + if err != nil { + t.Fatal(err) + } + defer conn.Close() + + if err := conn.SetReadDeadline(time.Now().Add(20 * time.Millisecond)); err != nil { + t.Fatal(err) + } + buf := make([]byte, len("delayed")) + if _, err := conn.Read(buf); !errors.Is(err, os.ErrDeadlineExceeded) { + t.Fatalf("Read error = %v, want os.ErrDeadlineExceeded", err) + } + + if err := conn.SetReadDeadline(time.Time{}); err != nil { + t.Fatal(err) + } + close(releaseServerSend) + + if _, err := io.ReadFull(conn, buf); err != nil { + t.Fatalf("ReadFull after clearing deadline: %v", err) + } + if string(buf) != "delayed" { + t.Errorf("got %q, want delayed", buf) } } +func TestClientDialContextRejected(t *testing.T) { + t.Run("HTTP/1.1", func(t *testing.T) { + ca := newTestCA(t) + gatewayAddress := serveTestConnectGateway(t, ca, func(conn net.Conn, _ *http.Request) { + body := "denied by policy" + _, _ = fmt.Fprintf(conn, "HTTP/1.1 403 Forbidden\r\nContent-Length: %d\r\n\r\n%s", len(body), body) + }) + client := newTestClient(t, ca, WithDialer(dialFixedAddress(gatewayAddress))) + + _, err := client.DialContext(context.Background(), "192.0.2.10:443") + if err == nil || !strings.Contains(err.Error(), "denied by policy") { + t.Fatalf("DialContext error = %v, want policy rejection", err) + } + }) + + t.Run("HTTP/2", func(t *testing.T) { + ca := newTestCA(t) + gatewayAddress, _ := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Host == "192.0.2.10:443" { + http.Error(w, "denied by policy", http.StatusForbidden) + return + } + w.WriteHeader(http.StatusOK) + if f, ok := w.(http.Flusher); ok { + f.Flush() + } + })) + var tcpDials atomic.Int32 + client := newTestClient(t, ca, WithDialer(func(ctx context.Context, network, _ string) (net.Conn, error) { + tcpDials.Add(1) + return (&net.Dialer{}).DialContext(ctx, network, gatewayAddress) + })) + + _, err := client.DialContext(context.Background(), "192.0.2.10:443") + var rejected *ConnectRejectedError + if !errors.As(err, &rejected) { + t.Fatalf("DialContext error = %v, want *ConnectRejectedError", err) + } + if rejected.StatusCode != http.StatusForbidden || rejected.Message != "denied by policy" { + t.Errorf("ConnectRejectedError = %+v, want 403 denied by policy", rejected) + } + + // A rejected stream must not poison the shared HTTP/2 connection. + conn, err := client.DialContext(context.Background(), "192.0.2.11:443") + if err != nil { + t.Fatalf("subsequent DialContext: %v", err) + } + _ = conn.Close() + if got := tcpDials.Load(); got != 1 { + t.Errorf("tcpDials = %d, want 1", got) + } + }) +} + // TestClientDialContextGatewayRefusesClientCertificate pins the classification -// of a front-door refusal to ErrGatewayHandshake under both TLS versions, which -// is not one behavior but two. +// of a front-door refusal to ErrGatewayHandshake under both TLS versions and +// ALPN negotiations. // // Under 1.2 the server rejects the certificate mid-handshake and // HandshakeContext returns the error. Under 1.3 the client's handshake has @@ -120,16 +418,19 @@ func TestClientDialContextGatewayRefusesClientCertificate(t *testing.T) { for _, tt := range []struct { name string maxTLSVersion uint16 + nextProtos []string }{ - {name: "TLS 1.3 refuses after the client handshake completes", maxTLSVersion: tls.VersionTLS13}, - {name: "TLS 1.2 refuses during the handshake", maxTLSVersion: tls.VersionTLS12}, + {name: "TLS 1.3 HTTP/1.1 refuses after the client handshake completes", maxTLSVersion: tls.VersionTLS13}, + {name: "TLS 1.2 HTTP/1.1 refuses during the handshake", maxTLSVersion: tls.VersionTLS12}, + {name: "TLS 1.3 HTTP/2 refuses after the client handshake completes", maxTLSVersion: tls.VersionTLS13, nextProtos: []string{"h2"}}, + {name: "TLS 1.2 HTTP/2 refuses during the handshake", maxTLSVersion: tls.VersionTLS12, nextProtos: []string{"h2"}}, } { t.Run(tt.name, func(t *testing.T) { ca := newTestCA(t) // The gateway trusts a different CA than the one that issued the // client's certificate, which is the shape of an actor presenting a // podidentity credential to a door that only accepts actor identity. - gatewayAddress := serveTestRefusingGateway(t, ca, newTestCA(t), tt.maxTLSVersion) + gatewayAddress := serveTestRefusingGateway(t, ca, newTestCA(t), tt.maxTLSVersion, tt.nextProtos) client := newTestClient(t, ca, WithDialer(dialFixedAddress(gatewayAddress))) _, err := client.DialContext(context.Background(), "192.0.2.10:443") @@ -155,19 +456,59 @@ func TestClientDialContextGatewayRefusesClientCertificate(t *testing.T) { // and says nothing. There is still nobody but the front door on the other end, // since no CONNECT response means no upstream was ever dialed. func TestClientDialContextGatewayHangsUpBeforeResponding(t *testing.T) { - ca := newTestCA(t) - gatewayAddress := serveTestConnectGateway(t, ca, func(conn net.Conn, _ *http.Request) { - _ = conn.Close() + t.Run("HTTP/1.1", func(t *testing.T) { + ca := newTestCA(t) + gatewayAddress := serveTestConnectGateway(t, ca, func(conn net.Conn, _ *http.Request) { + _ = conn.Close() + }) + client := newTestClient(t, ca, WithDialer(dialFixedAddress(gatewayAddress))) + + _, err := client.DialContext(context.Background(), "192.0.2.10:443") + if err == nil { + t.Fatal("DialContext succeeded against a gateway that hung up") + } + if !errors.Is(err, ErrGatewayHandshake) { + t.Errorf("DialContext error = %v, want it to wrap ErrGatewayHandshake", err) + } }) - client := newTestClient(t, ca, WithDialer(dialFixedAddress(gatewayAddress))) - _, err := client.DialContext(context.Background(), "192.0.2.10:443") - if err == nil { - t.Fatal("DialContext succeeded against a gateway that hung up") - } - if !errors.Is(err, ErrGatewayHandshake) { - t.Errorf("DialContext error = %v, want it to wrap ErrGatewayHandshake", err) - } + t.Run("HTTP/2", func(t *testing.T) { + ca := newTestCA(t) + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = listener.Close() }) + + clientCAs := x509.NewCertPool() + clientCAs.AppendCertsFromPEM(ca.certPEM) + tlsConfig := &tls.Config{ + MinVersion: tls.VersionTLS12, + Certificates: []tls.Certificate{issueDNSCertificate(t, ca, "egress.test")}, + ClientAuth: tls.RequireAndVerifyClientCert, + ClientCAs: clientCAs, + NextProtos: []string{"h2"}, + } + go func() { + conn, err := listener.Accept() + if err != nil { + return + } + defer conn.Close() + tlsConn := tls.Server(conn, tlsConfig) + _ = tlsConn.Handshake() + _ = tlsConn.Close() + }() + + client := newTestClient(t, ca, WithDialer(dialFixedAddress(listener.Addr().String()))) + _, err = client.DialContext(context.Background(), "192.0.2.10:443") + if err == nil { + t.Fatal("DialContext succeeded against an HTTP/2 gateway that hung up") + } + if !errors.Is(err, ErrGatewayHandshake) { + t.Errorf("DialContext error = %v, want it to wrap ErrGatewayHandshake", err) + } + }) } // TestConnectExchangeError is the other half of gatewayHungUp: a failure on @@ -248,6 +589,43 @@ func TestClientDialContextValidatesInput(t *testing.T) { } } +func TestClientClose(t *testing.T) { + ca := newTestCA(t) + gatewayAddr, _ := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + if f, ok := w.(http.Flusher); ok { + f.Flush() + } + _, _ = io.Copy(io.Discard, r.Body) + })) + + connClosed := make(chan struct{}) + client := newTestClient(t, ca, WithDialer(func(ctx context.Context, network, _ string) (net.Conn, error) { + c, err := (&net.Dialer{}).DialContext(ctx, network, gatewayAddr) + if err != nil { + return nil, err + } + return &closeNotifyingConn{Conn: c, closed: connClosed}, nil + })) + + stream, err := client.DialContext(context.Background(), "192.0.2.10:443") + if err != nil { + t.Fatal(err) + } + if err := stream.Close(); err != nil { + t.Fatal(err) + } + + if err := client.Close(); err != nil { + t.Fatalf("Close() = %v", err) + } + receiveWithin(t, connClosed, "idle HTTP/2 connection closed on Client.Close") + + if _, err := client.DialContext(context.Background(), "192.0.2.10:443"); !errors.Is(err, net.ErrClosed) { + t.Fatalf("DialContext after Close = %v, want net.ErrClosed", err) + } +} + // dialFixedAddress ignores the requested address and connects to address, so // tests can point a client at a listener on an ephemeral port. func dialFixedAddress(address string) DialFunc { @@ -316,6 +694,73 @@ func serveTestConnectGateway(t *testing.T, ca *testCA, handle func(net.Conn, *ht return listener.Addr().String() } +func serveTestH2ConnectGateway(t *testing.T, ca *testCA, handler http.Handler) (string, func()) { + t.Helper() + rawListener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + + var connsMu sync.Mutex + var activeConns []net.Conn + trackingListener := &connTrackingListener{ + Listener: rawListener, + onAccept: func(c net.Conn) { + connsMu.Lock() + activeConns = append(activeConns, c) + connsMu.Unlock() + }, + } + + serverCert := issueDNSCertificate(t, ca, "egress.test") + clientCAs := x509.NewCertPool() + clientCAs.AppendCertsFromPEM(ca.certPEM) + + var protocols http.Protocols + protocols.SetHTTP2(true) + server := &http.Server{ + Handler: handler, + Protocols: &protocols, + TLSConfig: &tls.Config{ + MinVersion: tls.VersionTLS12, + Certificates: []tls.Certificate{serverCert}, + ClientAuth: tls.RequireAndVerifyClientCert, + ClientCAs: clientCAs, + NextProtos: []string{"h2"}, + }, + } + tlsListener := tls.NewListener(trackingListener, server.TLSConfig) + t.Cleanup(func() { _ = server.Close() }) + + go func() { + _ = server.Serve(tlsListener) + }() + + closeActive := func() { + connsMu.Lock() + conns := activeConns + activeConns = nil + connsMu.Unlock() + for _, c := range conns { + _ = c.Close() + } + } + return rawListener.Addr().String(), closeActive +} + +type connTrackingListener struct { + net.Listener + onAccept func(net.Conn) +} + +func (l *connTrackingListener) Accept() (net.Conn, error) { + conn, err := l.Listener.Accept() + if err == nil && l.onAccept != nil { + l.onAccept(conn) + } + return conn, err +} + // serveTestRefusingGateway serves a front door that presents a certificate from // serverCA and requires a client certificate from clientCA, so a client holding // one from anywhere else is turned away. maxVersion caps the negotiated TLS @@ -324,7 +769,7 @@ func serveTestConnectGateway(t *testing.T, ca *testCA, handle func(net.Conn, *ht // // It handshakes and closes rather than reading a request: a refused client // never gets to send one, and any error here is the refusal working. -func serveTestRefusingGateway(t *testing.T, serverCA, clientCA *testCA, maxVersion uint16) string { +func serveTestRefusingGateway(t *testing.T, serverCA, clientCA *testCA, maxVersion uint16, nextProtos []string) string { t.Helper() listener, err := net.Listen("tcp", "127.0.0.1:0") if err != nil { @@ -340,6 +785,7 @@ func serveTestRefusingGateway(t *testing.T, serverCA, clientCA *testCA, maxVersi Certificates: []tls.Certificate{issueDNSCertificate(t, serverCA, "egress.test")}, ClientAuth: tls.RequireAndVerifyClientCert, ClientCAs: clientCAs, + NextProtos: nextProtos, } go func() { diff --git a/internal/atunnel/egress.go b/internal/atunnel/egress.go index 511bba5234..634404d13b 100644 --- a/internal/atunnel/egress.go +++ b/internal/atunnel/egress.go @@ -32,6 +32,7 @@ import ( // egressDialer opens an authenticated tunnel to an original destination. type egressDialer interface { DialContext(context.Context, string) (net.Conn, error) + Close() error } type actorCertificateSource interface { @@ -259,12 +260,16 @@ func (e *Egress) Deactivate(ctx context.Context, actorUID string) error { active.wg.Wait() close(done) }() + var err error select { case <-done: - return nil case <-ctx.Done(): - return fmt.Errorf("atunnel: waiting for active egress streams to stop: %w", ctx.Err()) + err = fmt.Errorf("atunnel: waiting for active egress streams to stop: %w", ctx.Err()) + } + if active.dialer != nil { + err = errors.Join(err, active.dialer.Close()) } + return err } func (e *Egress) handle(downstream net.Conn, active *egressActivation) { diff --git a/internal/atunnel/egress_test.go b/internal/atunnel/egress_test.go index dccc2e5292..62fd936106 100644 --- a/internal/atunnel/egress_test.go +++ b/internal/atunnel/egress_test.go @@ -23,6 +23,7 @@ import ( "net" "net/http" "strings" + "sync" "sync/atomic" "syscall" "testing" @@ -485,6 +486,10 @@ func (f egressDialerFunc) DialContext(ctx context.Context, destination string) ( return f(ctx, destination) } +func (f egressDialerFunc) Close() error { + return nil +} + type fakeActorCertificateSource struct { expiresAt time.Time err error @@ -541,3 +546,63 @@ func TestEgressIsArmedPerActor(t *testing.T) { t.Error("actor-uid-2 lost its egress when another actor was torn down") } } + +func TestEgressDeactivateClosesIdleH2Connection(t *testing.T) { + ca := newTestCA(t) + gatewayAddr, _ := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + if f, ok := w.(http.Flusher); ok { + f.Flush() + } + _, _ = io.Copy(io.Discard, r.Body) + })) + + connClosed := make(chan struct{}) + client := newTestClient(t, ca, WithDialer(func(ctx context.Context, network, _ string) (net.Conn, error) { + c, err := (&net.Dialer{}).DialContext(ctx, network, gatewayAddr) + if err != nil { + return nil, err + } + return &closeNotifyingConn{Conn: c, closed: connClosed}, nil + })) + + egress, err := NewEgress(func(net.Conn) (string, error) { return "192.0.2.10:443", nil }) + if err != nil { + t.Fatal(err) + } + if err := egress.Activate(testActorUID, client, fakeActorCertificateSource{expiresAt: time.Now().Add(time.Hour)}, time.Now().Add(time.Hour)); err != nil { + t.Fatal(err) + } + + // Open and close a stream so the Client retains an idle HTTP/2 connection. + stream, err := client.DialContext(context.Background(), "192.0.2.10:443") + if err != nil { + t.Fatal(err) + } + if err := stream.Close(); err != nil { + t.Fatal(err) + } + + select { + case <-connClosed: + t.Fatal("HTTP/2 connection closed while actor was still active") + default: + } + + if err := egress.Deactivate(context.Background(), testActorUID); err != nil { + t.Fatal(err) + } + receiveWithin(t, connClosed, "idle HTTP/2 connection closed on Deactivate") +} + +type closeNotifyingConn struct { + net.Conn + once sync.Once + closed chan struct{} +} + +func (c *closeNotifyingConn) Close() error { + err := c.Conn.Close() + c.once.Do(func() { close(c.closed) }) + return err +} diff --git a/internal/atunnel/ingress.go b/internal/atunnel/ingress.go index 6bcbe20d51..358a348a76 100644 --- a/internal/atunnel/ingress.go +++ b/internal/atunnel/ingress.go @@ -327,6 +327,7 @@ func (s *Server) ServeConnectHTTP(w http.ResponseWriter, r *http.Request) { } active, ctx, release, ok := s.authorize(r) if !ok { + slog.DebugContext(r.Context(), "atunnel CONNECT rejected", slog.Any("actor_header", r.Header.Values(atenet.TargetActorHeader))) s.reject(w) return } @@ -334,10 +335,12 @@ func (s *Server) ServeConnectHTTP(w http.ResponseWriter, r *http.Request) { _, port, err := net.SplitHostPort(r.Host) if err != nil { + slog.DebugContext(r.Context(), "atunnel CONNECT rejected: no port", slog.Any("actor", active.ref), slog.String("authority", r.Host)) http.Error(w, "CONNECT authority must include a port", http.StatusBadRequest) return } if _, ok := ParsePort(port); !ok { + slog.DebugContext(r.Context(), "atunnel CONNECT rejected: invalid port", slog.Any("actor", active.ref), slog.String("authority", r.Host)) http.Error(w, "invalid CONNECT port", http.StatusBadRequest) return } @@ -346,30 +349,32 @@ func (s *Server) ServeConnectHTTP(w http.ResponseWriter, r *http.Request) { defer cancelDial() upstream, err := active.dial(dialCtx, "tcp", net.JoinHostPort(s.upstream.Hostname(), port)) if err != nil { - slog.WarnContext(r.Context(), "atunnel CONNECT upstream failed", slog.Any("actor", active.ref), slog.Any("err", err)) + slog.WarnContext(r.Context(), "atunnel CONNECT upstream failed", slog.Any("actor", active.ref), slog.String("port", port), slog.Any("err", err)) http.Error(w, "bad gateway", http.StatusBadGateway) return } defer upstream.Close() + log := slog.With(slog.Any("actor", active.ref), slog.String("port", port), slog.String("proto", r.Proto)) + log.DebugContext(ctx, "atunnel CONNECT opened") if r.ProtoMajor == 2 { - s.serveH2Connect(w, r, upstream, ctx) + s.serveH2Connect(w, r, upstream, ctx, log) return } - s.serveH1Connect(w, upstream, ctx) + s.serveH1Connect(w, upstream, ctx, log) } -func (s *Server) serveH1Connect(w http.ResponseWriter, upstream net.Conn, ctx context.Context) { +func (s *Server) serveH1Connect(w http.ResponseWriter, upstream net.Conn, ctx context.Context, log *slog.Logger) { hj, ok := w.(http.Hijacker) if !ok { http.Error(w, "CONNECT hijacking unsupported", http.StatusInternalServerError) return } - client, rw, err := hj.Hijack() + downstream, rw, err := hj.Hijack() if err != nil { return } - defer client.Close() + defer downstream.Close() if _, err := rw.WriteString("HTTP/1.1 200 Connection Established\r\n\r\n"); err != nil { return } @@ -377,10 +382,10 @@ func (s *Server) serveH1Connect(w http.ResponseWriter, upstream net.Conn, ctx co return } - relayIngressWithHalfClose(ctx, upstream, rw, client, client) + relayIngressWithHalfClose(ctx, log, upstream, rw, downstream, downstream) } -func (s *Server) serveH2Connect(w http.ResponseWriter, r *http.Request, upstream net.Conn, ctx context.Context) { +func (s *Server) serveH2Connect(w http.ResponseWriter, r *http.Request, upstream net.Conn, ctx context.Context, log *slog.Logger) { // A HTTP/2 CONNECT tunnel is a pair of streams, not a hijackable TCP // socket. Send the response headers before copying so the peer can start // sending DATA frames, then flush each upstream write promptly. @@ -389,32 +394,33 @@ func (s *Server) serveH2Connect(w http.ResponseWriter, r *http.Request, upstream return } - relayIngressWithHalfClose(ctx, upstream, r.Body, flushingWriter{ResponseWriter: w}, r.Body) + relayIngressWithHalfClose(ctx, log, upstream, r.Body, flushingWriter{ResponseWriter: w}, r.Body) } -// relayIngressWithHalfClose copies a CONNECT stream. When the client request -// stream ends, it half-closes the actor connection and continues forwarding -// actor output until that stream ends too. -func relayIngressWithHalfClose(ctx context.Context, upstream net.Conn, clientReader io.Reader, clientWriter io.Writer, clientCloser io.Closer) { - done := make(chan struct{}, 2) +// relayIngressWithHalfClose copies a CONNECT stream between the router +// (downstream) and the actor (upstream). It half-closes the actor connection +// when the router stops sending, and closes the downstream when the actor +// does, so a router that reuses tunnels doesn't send into a dead one. +func relayIngressWithHalfClose(ctx context.Context, log *slog.Logger, upstream net.Conn, downstreamReader io.Reader, downstreamWriter io.Writer, downstreamCloser io.Closer) { + start := time.Now() go func() { - _, _ = io.Copy(upstream, clientReader) + n, err := io.Copy(upstream, downstreamReader) + log.DebugContext(ctx, "atunnel CONNECT router stopped sending", slog.Int64("bytes", n), slog.Any("err", err), slog.Duration("after", time.Since(start))) closeWrite(upstream) - done <- struct{}{} }() + actorDone := make(chan struct{}) go func() { - _, _ = io.Copy(clientWriter, upstream) - done <- struct{}{} + n, err := io.Copy(downstreamWriter, upstream) + log.DebugContext(ctx, "atunnel CONNECT actor stopped sending", slog.Int64("bytes", n), slog.Any("err", err), slog.Duration("after", time.Since(start))) + close(actorDone) }() - for range 2 { - select { - case <-ctx.Done(): - _ = upstream.Close() - _ = clientCloser.Close() - return - case <-done: - } + select { + case <-ctx.Done(): + log.DebugContext(ctx, "atunnel CONNECT canceled", slog.Any("cause", context.Cause(ctx)), slog.Duration("after", time.Since(start))) + _ = upstream.Close() + case <-actorDone: } + _ = downstreamCloser.Close() } type flushingWriter struct { @@ -555,16 +561,20 @@ func (s *Server) authorize(r *http.Request) (*activation, context.Context, func( } active.wg.Add(1) s.mu.Unlock() - requestCtx, cancel := context.WithCancel(r.Context()) - stop := context.AfterFunc(active.ctx, cancel) + requestCtx, cancel := context.WithCancelCause(r.Context()) + stop := context.AfterFunc(active.ctx, func() { cancel(errActorDeactivated) }) release := func() { active.wg.Done() stop() - cancel() + cancel(nil) } return active, requestCtx, release, true } +// errActorDeactivated is the cancel cause of a request whose actor was +// deactivated, which tells a suspend apart from the router hanging up. +var errActorDeactivated = errors.New("atunnel: actor deactivated") + func (s *Server) reject(w http.ResponseWriter) { w.Header().Set(StaleAssignmentHeader, "true") http.Error(w, "misdirected request", http.StatusMisdirectedRequest) diff --git a/internal/atunnel/ingress_test.go b/internal/atunnel/ingress_test.go index 72664cce87..ca25e299e3 100644 --- a/internal/atunnel/ingress_test.go +++ b/internal/atunnel/ingress_test.go @@ -26,6 +26,7 @@ import ( "encoding/pem" "errors" "io" + "log/slog" "math/big" "net" "net/http" @@ -33,6 +34,8 @@ import ( "net/url" "os" "path/filepath" + "strings" + "sync" "sync/atomic" "testing" "time" @@ -150,19 +153,19 @@ func TestRelayIngressWithHalfClose(t *testing.T) { upstream := receiveWithin(t, accepted, "accepted connection") defer upstream.Close() - clientReader, clientInput := io.Pipe() - defer clientReader.Close() - var clientOutput bytes.Buffer + downstreamReader, downstreamInput := io.Pipe() + defer downstreamReader.Close() + var downstreamOutput bytes.Buffer done := make(chan struct{}) go func() { - relayIngressWithHalfClose(context.Background(), upstream, clientReader, &clientOutput, clientReader) + relayIngressWithHalfClose(context.Background(), slog.New(slog.DiscardHandler), upstream, downstreamReader, &downstreamOutput, downstreamReader) close(done) }() - if _, err := io.WriteString(clientInput, "request"); err != nil { + if _, err := io.WriteString(downstreamInput, "request"); err != nil { t.Fatal(err) } - if err := clientInput.Close(); err != nil { + if err := downstreamInput.Close(); err != nil { t.Fatal(err) } if err := actor.SetReadDeadline(time.Now().Add(time.Second)); err != nil { @@ -187,24 +190,64 @@ func TestRelayIngressWithHalfClose(t *testing.T) { case <-time.After(time.Second): t.Fatal("relay did not finish after the actor response ended") } - if got := clientOutput.String(); got != "response" { - t.Errorf("client received %q, want response", got) + if got := downstreamOutput.String(); got != "response" { + t.Errorf("downstream received %q, want response", got) + } +} + +// A router reuses idle tunnels, so an actor closing its connection must end +// the tunnel, or the router's next request hangs until its timeout. +func TestRelayIngressClosesDownstreamWhenActorCloses(t *testing.T) { + upstream, actor := net.Pipe() + defer upstream.Close() + downstreamReader, downstreamInput := io.Pipe() + defer downstreamInput.Close() + var downstreamOutput bytes.Buffer + var logs syncBuffer + log := slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug})) + done := make(chan struct{}) + go func() { + relayIngressWithHalfClose(context.Background(), log, upstream, downstreamReader, &downstreamOutput, downstreamReader) + close(done) + }() + + if _, err := io.WriteString(actor, "response"); err != nil { + t.Fatal(err) + } + if err := actor.Close(); err != nil { + t.Fatal(err) + } + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("relay kept the tunnel open after the actor closed its connection") + } + if got := downstreamOutput.String(); got != "response" { + t.Errorf("downstream received %q, want response", got) + } + if _, err := io.WriteString(downstreamInput, "next request"); err == nil { + t.Fatal("downstream remained open after the actor closed its connection") + } + if want := `msg="atunnel CONNECT actor stopped sending" bytes=8 err=`; !strings.Contains(logs.String(), want) { + t.Errorf("relay logs = %q, want a line containing %q", logs.String(), want) } } func TestRelayIngressCancellationClosesBothSides(t *testing.T) { upstream, actor := net.Pipe() defer actor.Close() - clientReader, clientInput := io.Pipe() - defer clientInput.Close() - ctx, cancel := context.WithCancel(context.Background()) + downstreamReader, downstreamInput := io.Pipe() + defer downstreamInput.Close() + ctx, cancel := context.WithCancelCause(context.Background()) + var logs syncBuffer + log := slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug})) done := make(chan struct{}) go func() { - relayIngressWithHalfClose(ctx, upstream, clientReader, io.Discard, clientReader) + relayIngressWithHalfClose(ctx, log, upstream, downstreamReader, io.Discard, downstreamReader) close(done) }() - cancel() + cancel(errActorDeactivated) select { case <-done: case <-time.After(time.Second): @@ -215,11 +258,32 @@ func TestRelayIngressCancellationClosesBothSides(t *testing.T) { if _, err := actor.Read(make([]byte, 1)); err == nil { t.Fatal("actor connection remained open after relay cancellation") } - if _, err := io.WriteString(clientInput, "request"); err == nil { - t.Fatal("client stream remained open after relay cancellation") + if _, err := io.WriteString(downstreamInput, "request"); err == nil { + t.Fatal("downstream remained open after relay cancellation") + } + if want := `msg="atunnel CONNECT canceled" cause="atunnel: actor deactivated"`; !strings.Contains(logs.String(), want) { + t.Errorf("relay logs = %q, want a line containing %q", logs.String(), want) } } +// syncBuffer collects logs from the relay's goroutines. +type syncBuffer struct { + mu sync.Mutex + buf bytes.Buffer +} + +func (b *syncBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.Write(p) +} + +func (b *syncBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.String() +} + func TestServeHTTP(t *testing.T) { var upstreamHosts []string upstreamURL, err := url.Parse("http://actor.internal:80") @@ -756,9 +820,11 @@ func TestDeactivateCancelsInflightRequest(t *testing.T) { t.Fatal(err) } started := make(chan struct{}) + causes := make(chan error, 1) setActorTransport(t, s, "team-a", "actor-1", roundTripFunc(func(r *http.Request) (*http.Response, error) { close(started) <-r.Context().Done() + causes <- context.Cause(r.Context()) return nil, r.Context().Err() })) @@ -775,6 +841,9 @@ func TestDeactivateCancelsInflightRequest(t *testing.T) { t.Fatal(err) } receiveWithin(t, done, "canceled in-flight request") + if cause := receiveWithin(t, causes, "cancel cause"); !errors.Is(cause, errActorDeactivated) { + t.Errorf("in-flight request canceled with %v, want %v", cause, errActorDeactivated) + } } // testDial is the dialer a test actor is reached through; the sandbox it would diff --git a/internal/benchmarking/boomer/agentsession/scripts/personal-assistant-light.yaml b/internal/benchmarking/boomer/agentsession/scripts/personal-assistant-light.yaml new file mode 100644 index 0000000000..7772e07bbb --- /dev/null +++ b/internal/benchmarking/boomer/agentsession/scripts/personal-assistant-light.yaml @@ -0,0 +1,828 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# personal-assistant with a small memory footprint: the same day, the same +# 61 wakes, the same ops, reads, writes, CPU and dwell per turn, with only +# the RAM arrays scaled down. personal-assistant models a gateway with a +# 1.2 to 1.5 GiB heap; this variant is for runs where the agent's own state +# is tens of MB: many actors per node, cheap pauses, fast restores. +# +# Declared bytes: 28Mi of RAM arrays (gateway_heap 20Mi, browser 4Mi, +# dream_retained 3Mi, catalog_cache 1Mi) and the same files as +# personal-assistant, about 21Mi resident. +# +# Memory image per pause. Under gVisor the image is the workload: about +# 40 MB after gateway start and 80 to 95 MB from mid-morning on, once the +# Go heap has grown to about twice the live arrays. Under micro-VM the +# image is the guest's populated pages: about 100 MB of kernel, agent and +# page cache before the script writes anything, reaching about 200 MB in +# an actor with a 512Mi memory limit and capped near 125 MB with a 256Mi +# limit, plus a rootfs tar of up to 22 MB. +# +# min_actor_memory is 256Mi: the script fits a 128 MiB micro-VM guest. +# Run it like personal-assistant, with a fractional +# --agentsession-think-scale. +name: personal-assistant-light +min_actor_memory: 256Mi +steps: + - name: 0000_gateway_start + agent: "Gateway starts: loads config, skills and the session store, settles its heap" + think: 55m + ops: + - fill_ram: {key: gateway_heap, size: 20Mi} + - write_disk: {key: bootstrap_files, size: 32Ki} + - write_disk: {key: context_window, size: 96Ki} + - write_disk: {key: session_db, size: 6Mi} + - write_disk: {key: memory_notes, size: 8Ki} + - write_disk: {key: memory_index, size: 4Mi} + - burn_cpu: {millis: 3000, parallel: 1} + - ping: {} + - name: 0300_dreaming + agent: "Nightly memory consolidation: rereads the day, rewrites MEMORY.md, reindexes" + think: 3h + ops: + - read_disk_data: {key: memory_index} + - read_disk_digest: {key: session_db} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 15000} + - write_disk: {key: session_wal, size: 1Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 15000} + - write_disk: {key: session_wal, size: 1Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 15000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: memory_notes, size: 32Ki} + - write_disk: {key: memory_index, size: 4Mi} + - fill_ram: {key: dream_retained, size: 3Mi} + - name: 0505_catalog_refresh + agent: Model catalog refresh, no model call + think: 2h5m + ops: + - ingest: {key: model_catalog, size: 2Mi} + - burn_cpu: {millis: 800, parallel: 1} + - fill_ram: {key: catalog_cache, size: 1Mi} + - name: 0730_morning_hello + agent: First message of the day, quick reply + think: 2h25m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 0732_whats_today + agent: Asks what is on today; agent reads memory + think: 2m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - read_disk_data: {key: memory_index} + - burn_cpu: {millis: 1500, parallel: 1} + - write_disk: {key: index_snapshot, size: 3Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 6000} + - write_disk: {key: session_wal, size: 2Mi} + - name: 0735_quick_followup + agent: Short follow-up, quick reply + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 0800_briefing + agent: "Scheduled morning briefing: three web fetches, one message out" + think: 25m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 6000} + - write_disk: {key: session_wal, size: 1Mi} + - ingest: {key: web_fetch, size: 128Ki} + - ingest: {key: web_fetch, size: 128Ki} + - ingest: {key: web_fetch, size: 128Ki} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 800, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 10000} + - write_disk: {key: session_wal, size: 2Mi} + - name: 0830_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 0900_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 0930_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1000_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1015_question + agent: Question with a one-command answer + think: 15m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - burn_cpu: {millis: 1200, parallel: 1} + - write_disk: {key: tool_output, size: 64Ki} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 7000} + - write_disk: {key: session_wal, size: 2Mi} + - churn_ram: {key: gateway_heap, size: 512Ki} + - name: 1017_followup + agent: Follow-up, quick reply + think: 2m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 6000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 1020_remember_this + agent: "Remember-that request: memory file write and reindex" + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: memory_notes, size: 16Ki} + - burn_cpu: {millis: 700, parallel: 1} + - write_disk: {key: memory_index, size: 4Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 2Mi} + - name: 1022_thanks + agent: Closing message, quick reply + think: 2m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 3000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 1030_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 8m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1100_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1105_catalog_refresh + agent: Model catalog refresh, no model call + think: 5m + ops: + - ingest: {key: model_catalog, size: 2Mi} + - burn_cpu: {millis: 800, parallel: 1} + - fill_ram: {key: catalog_cache, size: 1Mi} + - name: 1130_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 25m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1200_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1230_lunch_question + agent: Question needing a web search + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - ingest: {key: web_fetch, size: 256Ki} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 9000} + - write_disk: {key: session_wal, size: 2Mi} + - churn_ram: {key: gateway_heap, size: 512Ki} + - name: 1233_dig_deeper + agent: Asks for more detail on one result + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - ingest: {key: web_fetch, size: 256Ki} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 9000} + - write_disk: {key: session_wal, size: 2Mi} + - churn_ram: {key: gateway_heap, size: 512Ki} + - name: 1236_run_check + agent: Asks the agent to run a check on the machine + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - burn_cpu: {millis: 1200, parallel: 1} + - write_disk: {key: tool_output, size: 64Ki} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 7000} + - write_disk: {key: session_wal, size: 2Mi} + - churn_ram: {key: gateway_heap, size: 512Ki} + - name: 1240_what_did_i_say + agent: "Recall question: memory search" + think: 4m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - read_disk_data: {key: memory_index} + - burn_cpu: {millis: 1500, parallel: 1} + - write_disk: {key: index_snapshot, size: 3Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 6000} + - write_disk: {key: session_wal, size: 2Mi} + - name: 1243_quick_reply + agent: Quick reply + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 1245_bye + agent: Closing message + think: 2m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 3000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 1300_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 15m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1330_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1400_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1430_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1500_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1510_task_request + agent: "Multi-step task: command, file, second command" + think: 10m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - burn_cpu: {millis: 1200, parallel: 1} + - write_disk: {key: tool_output, size: 64Ki} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 7000} + - write_disk: {key: session_wal, size: 2Mi} + - churn_ram: {key: gateway_heap, size: 512Ki} + - burn_cpu: {millis: 900, parallel: 1} + - write_disk: {key: tool_output, size: 128Ki} + - dwell: {millis: 8000} + - write_disk: {key: turn_log, size: 64Ki} + - name: 1514_task_followup + agent: Correction to the task, another command + think: 4m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - burn_cpu: {millis: 1200, parallel: 1} + - write_disk: {key: tool_output, size: 64Ki} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 7000} + - write_disk: {key: session_wal, size: 2Mi} + - churn_ram: {key: gateway_heap, size: 512Ki} + - name: 1518_web_check + agent: Checks a fact on the web + think: 4m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - ingest: {key: web_fetch, size: 256Ki} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 700, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 9000} + - write_disk: {key: session_wal, size: 2Mi} + - churn_ram: {key: gateway_heap, size: 512Ki} + - name: 1521_done + agent: Closing message + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 3000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 1530_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 9m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1600_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1630_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1700_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1705_catalog_refresh + agent: Model catalog refresh, no model call + think: 5m + ops: + - ingest: {key: model_catalog, size: 2Mi} + - burn_cpu: {millis: 800, parallel: 1} + - fill_ram: {key: catalog_cache, size: 1Mi} + - name: 1730_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 25m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1800_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1820_open_site + agent: "First browser use of the day: Chrome launches and stays resident" + think: 20m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - fill_ram: {key: browser, size: 4Mi} + - burn_cpu: {millis: 2500, parallel: 2} + - ingest: {key: page_load, size: 1Mi} + - write_disk: {key: browser_cache, size: 8Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 900, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 8000} + - write_disk: {key: session_wal, size: 3Mi} + - name: 1824_click_through + agent: Browser actions on the page + think: 4m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - burn_cpu: {millis: 600, parallel: 2} + - ingest: {key: page_load, size: 512Ki} + - churn_ram: {key: browser, size: 1Mi} + - write_disk: {key: browser_cache, size: 4Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 800, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 6000} + - write_disk: {key: session_wal, size: 2Mi} + - name: 1828_fill_form + agent: More browser actions + think: 4m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - burn_cpu: {millis: 600, parallel: 2} + - ingest: {key: page_load, size: 512Ki} + - churn_ram: {key: browser, size: 1Mi} + - write_disk: {key: browser_cache, size: 4Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 800, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 6000} + - write_disk: {key: session_wal, size: 2Mi} + - name: 1830_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 2m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 1832_summarize_page + agent: Summarize what was found + think: 2m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 9000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 1836_save_it + agent: "Saves the result: file write and memory note" + think: 4m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 5000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: memory_notes, size: 16Ki} + - burn_cpu: {millis: 700, parallel: 1} + - write_disk: {key: memory_index, size: 4Mi} + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 2Mi} + - name: 1838_bye + agent: Closing message + think: 2m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 3000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 1900_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 22m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 1905_compaction + agent: "Context compaction with a memory flush: rereads the transcript, two long model calls" + think: 5m + ops: + - read_disk_data: {key: session_db} + - burn_cpu: {millis: 1000, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 25000} + - burn_cpu: {millis: 600, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 20000} + - write_disk: {key: memory_notes, size: 24Ki} + - write_disk: {key: session_db, size: 6Mi} + - write_disk: {key: session_wal, size: 4Mi} + - churn_ram: {key: gateway_heap, size: 2Mi} + - name: 1930_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 25m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 2000_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 2030_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 2100_evening_chat + agent: Evening question, quick reply + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 7000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 2103_evening_followup + agent: Follow-up, quick reply + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 6000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 2106_goodnight + agent: Last message of the day + think: 3m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 500, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 3000} + - write_disk: {key: session_wal, size: 1Mi} + - churn_ram: {key: gateway_heap, size: 256Ki} + - name: 2130_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 24m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 2200_heartbeat_and_cron + agent: "Heartbeat turn: rereads context, one model call, usually silent plus an hourly system-event cron" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - write_disk: {key: state_row, size: 64Ki} + - name: 2230_heartbeat + agent: "Heartbeat turn: rereads context, one model call, usually silent" + think: 30m + ops: + - read_disk_digest: {key: bootstrap_files} + - read_disk_digest: {key: session_db} + - burn_cpu: {millis: 400, parallel: 1} + - read_disk_data: {key: context_window} + - dwell: {millis: 4000} + - write_disk: {key: session_wal, size: 1Mi} + - name: 2305_catalog_refresh + agent: Model catalog refresh and the daily version check + think: 35m + ops: + - ingest: {key: model_catalog, size: 2Mi} + - burn_cpu: {millis: 800, parallel: 1} + - fill_ram: {key: catalog_cache, size: 1Mi} diff --git a/internal/benchmarking/boomer/dynconfig/dynconfig.go b/internal/benchmarking/boomer/dynconfig/dynconfig.go index 6280dddd9d..8abf1fadee 100644 --- a/internal/benchmarking/boomer/dynconfig/dynconfig.go +++ b/internal/benchmarking/boomer/dynconfig/dynconfig.go @@ -85,6 +85,8 @@ type Config struct { TotalActors int // spawn batch size; 0 keeps --total-actors SpawnConcurrency int // actors the spawn batch creates concurrently; 0 keeps --spawn-concurrency ActorDeadline time.Duration // per-actor timeout in the spawn batch; 0 keeps --actor-deadline + + TargetRPS float64 // requests per second the whole worker paces to, for user classes that pace; 0 means unpaced } // Holder lets readers Load() the current Config and writers Store() a new @@ -143,6 +145,8 @@ type payload struct { TotalActors *float64 `json:"total_actors"` SpawnConcurrency *float64 `json:"spawn_concurrency"` ActorDeadline *float64 `json:"actor_deadline"` + + TargetRPS *float64 `json:"target_rps"` } // Parse decodes a JSON blob (typically from a CLI flag) and merges its @@ -254,6 +258,9 @@ func (c Config) Validate() error { if c.ActorDeadline < 0 { return fmt.Errorf("actor_deadline cannot be negative: %v", c.ActorDeadline) } + if c.TargetRPS < 0 { + return fmt.Errorf("target_rps cannot be negative: %v", c.TargetRPS) + } // MaxPingsPerWake < 1 is treated as 1 at read time (see iterate() in // glutton/lifecycle.go), so Config's zero value stays usable — no // validate rejection here. @@ -347,6 +354,9 @@ func (p payload) merge(current Config) Config { if p.ActorDeadline != nil { out.ActorDeadline = time.Duration(*p.ActorDeadline * float64(time.Second)) } + if p.TargetRPS != nil { + out.TargetRPS = *p.TargetRPS + } return out } @@ -431,6 +441,7 @@ func StartPoll( slog.Int("total_actors", next.TotalActors), slog.Int("spawn_concurrency", next.SpawnConcurrency), slog.Duration("actor_deadline", next.ActorDeadline), + slog.Float64("target_rps", next.TargetRPS), ) } } @@ -486,6 +497,7 @@ func SubscribeSpawn(url string, holder *Holder, sampler ProbabilityUpdater, fetc slog.Int("total_actors", next.TotalActors), slog.Int("spawn_concurrency", next.SpawnConcurrency), slog.Duration("actor_deadline", next.ActorDeadline), + slog.Float64("target_rps", next.TargetRPS), ) }) } diff --git a/internal/benchmarking/boomer/dynconfig/dynconfig_test.go b/internal/benchmarking/boomer/dynconfig/dynconfig_test.go index 2f1390d888..8479594179 100644 --- a/internal/benchmarking/boomer/dynconfig/dynconfig_test.go +++ b/internal/benchmarking/boomer/dynconfig/dynconfig_test.go @@ -49,7 +49,8 @@ func TestParseValid(t *testing.T) { "agentsession_script_file": "/etc/agentsession/script.yaml", "total_actors": 50, "spawn_concurrency": 5, - "actor_deadline": 60.0 + "actor_deadline": 60.0, + "target_rps": 2500.5 }`) cfg, err := Parse(jsonBlob, Config{}) @@ -120,6 +121,9 @@ func TestParseValid(t *testing.T) { if cfg.ActorDeadline != 60*time.Second { t.Errorf("ActorDeadline: got %v, want 60s", cfg.ActorDeadline) } + if cfg.TargetRPS != 2500.5 { + t.Errorf("TargetRPS: got %v, want 2500.5", cfg.TargetRPS) + } } func TestParseInvalidValues(t *testing.T) { @@ -199,6 +203,10 @@ func TestParseInvalidValues(t *testing.T) { name: "negative sweperf num cycles", json: `{"sweperf_num_cycles": -1}`, }, + { + name: "negative target rps", + json: `{"target_rps": -1}`, + }, { name: "negative sweperf poll interval", json: `{"sweperf_poll_interval_ms": -1}`, diff --git a/internal/benchmarking/boomer/metrics/prom.go b/internal/benchmarking/boomer/metrics/prom.go index c4f7b099fb..34b811a8f2 100644 --- a/internal/benchmarking/boomer/metrics/prom.go +++ b/internal/benchmarking/boomer/metrics/prom.go @@ -86,18 +86,31 @@ func Serve(ctx context.Context, addr string) error { // RecordSuccess reports a successful request to both boomer (→ locust master // via ZMQ) and the local Prometheus surface. func RecordSuccess(method, name, userClass string, latency time.Duration, responseBytes int64) { - ms := latency.Milliseconds() - boomer.RecordSuccess(method, name, ms, responseBytes) - requestsTotal.WithLabelValues(method, name, "success", userClass).Inc() - requestDuration.WithLabelValues(method, name, "success", userClass).Observe(float64(ms)) + boomer.RecordSuccess(method, name, latency.Milliseconds(), responseBytes) + RecordPromSuccess(method, name, userClass, latency) } // RecordFailure reports a failed request to both boomer and Prometheus. func RecordFailure(method, name, userClass string, latency time.Duration, errMsg string) { - ms := latency.Milliseconds() - boomer.RecordFailure(method, name, ms, errMsg) + boomer.RecordFailure(method, name, latency.Milliseconds(), errMsg) + RecordPromFailure(method, name, userClass, latency) +} + +// RecordPromSuccess reports a successful request to Prometheus only. Use it +// once boomer.Run has returned: boomer may have stopped draining its stats +// channel by then, and a send to it blocks once the channel's buffer fills. +func RecordPromSuccess(method, name, userClass string, latency time.Duration) { + ms := float64(latency.Milliseconds()) + requestsTotal.WithLabelValues(method, name, "success", userClass).Inc() + requestDuration.WithLabelValues(method, name, "success", userClass).Observe(ms) +} + +// RecordPromFailure reports a failed request to Prometheus only, as +// RecordPromSuccess does a successful one. +func RecordPromFailure(method, name, userClass string, latency time.Duration) { + ms := float64(latency.Milliseconds()) requestsTotal.WithLabelValues(method, name, "failure", userClass).Inc() - requestDuration.WithLabelValues(method, name, "failure", userClass).Observe(float64(ms)) + requestDuration.WithLabelValues(method, name, "failure", userClass).Observe(ms) } // UpdateUsers shifts the active-users gauge for a class by delta (positive on diff --git a/internal/benchmarking/boomer/resumepaused/resumepaused.go b/internal/benchmarking/boomer/resumepaused/resumepaused.go new file mode 100644 index 0000000000..4967d63544 --- /dev/null +++ b/internal/benchmarking/boomer/resumepaused/resumepaused.go @@ -0,0 +1,446 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package resumepaused drives ate-api-server's cold resume path: every +// ResumeActor targets a paused actor, so ateapi takes the actor lease, binds +// a worker, and waits on an atelet restore. Each resume is followed by a +// pause that makes the actor cold again. Run on the fake data plane +// (benchmarking/workloads/deploy.sh --fake-data-plane), it measures what a +// cold resume costs ateapi and Postgres without the router or the restore in +// the way. The pauses run at the resume rate, so a capacity it finds is one +// of resume and pause cycles, not of resumes alone. It sends the actors +// themselves no requests, since none exist there. +package resumepaused + +import ( + "context" + "errors" + "fmt" + "log/slog" + "maps" + "os" + "slices" + "sync" + "sync/atomic" + "time" + + "github.com/agent-substrate/substrate/internal/benchmarking/boomer/boomerutil" + bmetrics "github.com/agent-substrate/substrate/internal/benchmarking/boomer/metrics" + "github.com/agent-substrate/substrate/internal/benchmarking/boomer/userclass" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/google/uuid" + "go.opentelemetry.io/otel" + "golang.org/x/time/rate" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +const userClass = "ResumePausedUser" + +func init() { + userclass.Add(userclass.Entry{ + Name: "resumepaused", + LocustFile: "resumepaused.py", + UserClass: userClass, + Init: initResumePaused, + }) +} + +// initResumePaused ensures the atespace and returns the task. Each user +// creates and warms its own actor on its first iteration. +func initResumePaused(cfg *userclass.Config) (taskFn func(), shutdown func(context.Context)) { + if cfg.Tracer == nil { + cfg.Tracer = otel.Tracer("substrate-boomer/resumepaused") + } + rt := newRuntime(cfg) + if err := rt.start(context.Background()); err != nil { + slog.Error("fatal: resumepaused", slog.String("err", err.Error())) + os.Exit(1) + } + return rt.iterate, rt.shutdown +} + +type runtime struct { + cfg *userclass.Config + // users maps a boomer goroutine to the actor it cycles. Boomer gives a + // task no user identity, so the goroutine stands in for the user. Go + // never reuses a goroutine ID, so the entry of a user boomer stopped is + // never looked up again; its actor stays paused until shutdown. + users sync.Map // goroutineID -> actor name + // actorsMu guards actors, every actor that may exist, for shutdown. + actorsMu sync.Mutex + actors map[string]struct{} + // limiter paces the resumes of every user. Its burst of 1 spaces them + // evenly: Wait hands each caller the next slot rather than releasing a + // second's budget at once. + limiter *rate.Limiter + // retryBackoff spaces the retries of a delete. + retryBackoff time.Duration + // setupRetryWait is how long a user waits after its actor failed to + // warm, so a failing setup does not loop on boomer's zero-delay + // re-entry. + setupRetryWait time.Duration + // cycleMu guards closing and every cycles.Add, so no cycle or setup + // starts once shutdown has begun waiting for the running ones. + cycleMu sync.Mutex + closing bool + cycles sync.WaitGroup + // cycleCtx carries every cycle's and setup's RPCs; shutdown cancels it + // when the running ones outlast their share of its budget. + cycleCtx context.Context + cancelCycles context.CancelFunc + // stopped is set when shutdown begins, after boomer.Run has returned. + // Boomer may have stopped draining its stats channel by then, so a + // sample sent to it could block for good once the buffer fills: from + // then on, samples go to Prometheus only. + stopped atomic.Bool + // recordSuccess and recordFailure send a sample to both locust and + // Prometheus while boomer runs. + recordSuccess func(method, name, userClass string, latency time.Duration, responseBytes int64) + recordFailure func(method, name, userClass string, latency time.Duration, errMsg string) +} + +func newRuntime(cfg *userclass.Config) *runtime { + cycleCtx, cancelCycles := context.WithCancel(context.Background()) + return &runtime{ + cfg: cfg, + actors: map[string]struct{}{}, + limiter: rate.NewLimiter(rate.Inf, 1), + retryBackoff: time.Second, + setupRetryWait: time.Second, + cycleCtx: cycleCtx, + cancelCycles: cancelCycles, + recordSuccess: bmetrics.RecordSuccess, + recordFailure: bmetrics.RecordFailure, + } +} + +// start ensures the atespace every actor is created in. +func (r *runtime) start(ctx context.Context) error { + if err := r.ensureAtespace(ctx); err != nil { + return fmt.Errorf("while ensuring atespace %s: %w", r.cfg.Atespace, err) + } + return nil +} + +// iterate is the boomer task. A user's first iteration creates its actor and +// makes it cold; every later one runs a measured cycle on that actor. +func (r *runtime) iterate() { + gid := boomerutil.GoroutineID() + if v, ok := r.users.Load(gid); ok { + r.cycle(v.(string)) + return + } + name, ok := r.setUp() + if !ok { + time.Sleep(r.setupRetryWait) + return + } + r.users.Store(gid, name) +} + +// setUp creates one actor and makes it cold. It is not paced, so its calls +// land in the step that added the user, recorded under their own names. An +// actor that fails to warm is deleted, and the user tries again on its next +// iteration. +func (r *runtime) setUp() (string, bool) { + if !r.startCycle() { + return "", false + } + defer r.cycles.Done() + ctx := r.cycleCtx + name := "rp-" + uuid.NewString() + if err := r.create(ctx, name); err != nil { + return "", false + } + r.track(name) + if err := r.warm(ctx, name); err != nil { + slog.Warn("resumepaused: actor did not warm; the user will retry", + slog.String("actor", name), slog.String("err", err.Error())) + if r.delete(ctx, name) == nil { + r.untrack(name) + } + return "", false + } + return name, true +} + +func (r *runtime) track(name string) { + r.actorsMu.Lock() + defer r.actorsMu.Unlock() + r.actors[name] = struct{}{} +} + +func (r *runtime) untrack(name string) { + r.actorsMu.Lock() + defer r.actorsMu.Unlock() + delete(r.actors, name) +} + +// The template every actor is created from. +const ( + templateName = "glutton" + templateAtespace = "benchmark-workloads" +) + +// create creates one actor from the template. +func (r *runtime) create(ctx context.Context, name string) error { + return r.call(ctx, "CreateActor", func(ctx context.Context) error { + _, err := r.cfg.APIStub.CreateActor(ctx, &ateapipb.CreateActorRequest{ + Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: r.cfg.Atespace, Name: name}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: templateAtespace, Name: templateName}, + }, + }) + return err + }) +} + +// warm makes a new actor cold: a first resume, then a pause. +func (r *runtime) warm(ctx context.Context, name string) error { + if err := r.resume(ctx, "ResumeActorWarmup", name); err != nil { + return err + } + return r.pause(ctx, "Warmup", name) +} + +// cycle is one paced cold resume of the user's actor, then the pause that +// makes it cold again. Only the resume is paced, so the target rate is cold +// resumes per second. +func (r *runtime) cycle(name string) { + ctx := r.cycleCtx + r.pace(ctx) + if !r.startCycle() { + return + } + defer r.cycles.Done() + err := r.resume(ctx, "ResumeActorCold", name) + if err == nil { + err = r.pause(ctx, "", name) + } + if err != nil { + // The actor may have been left running. Pausing it again makes it + // cold for the next cycle. + r.recover(ctx, name) + } +} + +// recover pauses an actor after a failed cycle, so it is cold for the next +// one. It is load on the server like any other call, so it is recorded, as +// PauseActorRecovery to keep it apart from the measured pauses. An actor +// already paused fails it with FailedPrecondition; the actor is cold, which +// is all recovery wants, so that counts as a success. +func (r *runtime) recover(ctx context.Context, name string) { + _ = r.call(ctx, "PauseActorRecovery", func(ctx context.Context) error { + _, err := r.cfg.APIStub.PauseActor(ctx, &ateapipb.PauseActorRequest{ + Actor: r.ref(name), + }) + if status.Code(err) == codes.FailedPrecondition { + return nil + } + return err + }) +} + +// startCycle registers a cycle or setup with shutdown, or reports false once +// shutdown has begun. +func (r *runtime) startCycle() bool { + r.cycleMu.Lock() + defer r.cycleMu.Unlock() + if r.closing { + return false + } + r.cycles.Add(1) + return true +} + +// pace blocks until the limiter grants the next resume slot, applying the +// current TargetRPS first. Zero means unpaced. +func (r *runtime) pace(ctx context.Context) { + want := rate.Inf + if target := r.cfg.Dyn.Load().TargetRPS; target > 0 { + want = rate.Limit(target) + } + if r.limiter.Limit() != want { + r.limiter.SetLimit(want) + } + _ = r.limiter.Wait(ctx) +} + +// errAlreadyRunning marks a ResumeActor that found its actor running, so no +// cold resume happened and the sample must not count as one. +var errAlreadyRunning = errors.New("actor was already running: not a cold resume") + +// resume calls ResumeActor. A response that reports no resume happened fails +// the sample with errAlreadyRunning. +func (r *runtime) resume(ctx context.Context, metricName, name string) error { + return r.call(ctx, metricName, func(ctx context.Context) error { + resp, err := r.cfg.APIStub.ResumeActor(ctx, &ateapipb.ResumeActorRequest{ + Actor: r.ref(name), + }) + if err == nil && !resp.GetResumed() { + return errAlreadyRunning + } + return err + }) +} + +// pause calls PauseActor. suffix keeps warm-up calls apart from the measured +// ones. +func (r *runtime) pause(ctx context.Context, suffix, name string) error { + return r.call(ctx, "PauseActor"+suffix, func(ctx context.Context) error { + _, err := r.cfg.APIStub.PauseActor(ctx, &ateapipb.PauseActorRequest{ + Actor: r.ref(name), + }) + return err + }) +} + +// shutdown waits for the running cycles and setups, then deletes every +// actor. Boomer stops scheduling tasks before shutdown but does not wait for +// the ones in flight, and a delete that races one of them loses the actor +// lease and fails with Aborted. The running ones get half of ctx's remaining +// time; any still running then are canceled, so a hung RPC cannot spend the +// time the deletes need. DeleteActor with AnyState removes an actor in any +// state, so a canceled cycle or setup still leaves nothing behind. +func (r *runtime) shutdown(ctx context.Context) { + r.stopped.Store(true) + r.cycleMu.Lock() + r.closing = true + r.cycleMu.Unlock() + drained := make(chan struct{}) + go func() { + r.cycles.Wait() + close(drained) + }() + drainCtx := ctx + if deadline, ok := ctx.Deadline(); ok { + var cancel context.CancelFunc + drainCtx, cancel = context.WithTimeout(ctx, time.Until(deadline)/2) + defer cancel() + } + select { + case <-drained: + case <-drainCtx.Done(): + r.cancelCycles() + select { + case <-drained: + case <-ctx.Done(): + } + } + defer r.cancelCycles() + + r.actorsMu.Lock() + names := slices.Collect(maps.Keys(r.actors)) + r.actors = map[string]struct{}{} + r.actorsMu.Unlock() + + const parallelism = 16 + sem := make(chan struct{}, parallelism) + var wg sync.WaitGroup + var leftMu sync.Mutex + left := 0 + for _, name := range names { + wg.Add(1) + sem <- struct{}{} + go func() { + defer wg.Done() + defer func() { <-sem }() + if r.delete(ctx, name) != nil { + leftMu.Lock() + left++ + leftMu.Unlock() + } + }() + } + wg.Wait() + if left > 0 { + slog.Warn("resumepaused: actors left behind after shutdown", + slog.Int("count", left), slog.String("atespace", r.cfg.Atespace)) + } +} + +// delete deletes an actor, retrying while another operation holds its lease. +func (r *runtime) delete(ctx context.Context, name string) error { + const attempts = 5 + var err error + for attempt := range attempts { + if attempt > 0 { + select { + case <-time.After(r.retryBackoff): + case <-ctx.Done(): + return ctx.Err() + } + } + err = r.call(ctx, "DeleteActor", func(ctx context.Context) error { + _, err := r.cfg.APIStub.DeleteActor(ctx, &ateapipb.DeleteActorRequest{ + Actor: r.ref(name), + AnyState: true, + }) + return err + }) + if c := status.Code(err); c != codes.Aborted && c != codes.Unavailable { + return err + } + } + return err +} + +func (r *runtime) ref(name string) *ateapipb.ObjectRef { + return &ateapipb.ObjectRef{Atespace: r.cfg.Atespace, Name: name} +} + +func (r *runtime) ensureAtespace(ctx context.Context) error { + return r.call(ctx, "CreateAtespace", func(ctx context.Context) error { + _, err := r.cfg.APIStub.CreateAtespace(ctx, &ateapipb.CreateAtespaceRequest{ + Atespace: &ateapipb.Atespace{ + Metadata: &ateapipb.ResourceMetadata{Name: r.cfg.Atespace}, + }, + }) + if status.Code(err) == codes.AlreadyExists { + return nil + } + return err + }) +} + +// call times one unary RPC and reports its latency to locust and Prometheus. +func (r *runtime) call(ctx context.Context, name string, do func(context.Context) error) error { + ctx, span := r.cfg.Tracer.Start(ctx, name) + defer span.End() + + start := time.Now() + err := do(ctx) + latency := time.Since(start) + + boomerutil.LogSampledTrace(span, name, latency, boomerutil.SourceClient, err) + r.record(name, latency, err) + return err +} + +// record reports one sample: to locust and Prometheus while boomer runs, to +// Prometheus only once shutdown has begun. +func (r *runtime) record(name string, latency time.Duration, err error) { + stopped := r.stopped.Load() + switch { + case err != nil && stopped: + bmetrics.RecordPromFailure("grpc", name, userClass, latency) + case err != nil: + r.recordFailure("grpc", name, userClass, latency, err.Error()) + case stopped: + bmetrics.RecordPromSuccess("grpc", name, userClass, latency) + default: + r.recordSuccess("grpc", name, userClass, latency, 0) + } +} diff --git a/internal/benchmarking/boomer/resumepaused/resumepaused_test.go b/internal/benchmarking/boomer/resumepaused/resumepaused_test.go new file mode 100644 index 0000000000..7aa1572acf --- /dev/null +++ b/internal/benchmarking/boomer/resumepaused/resumepaused_test.go @@ -0,0 +1,751 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package resumepaused + +import ( + "context" + "maps" + "net/http" + "net/http/httptest" + "slices" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/agent-substrate/substrate/internal/benchmarking/boomer/dynconfig" + "github.com/agent-substrate/substrate/internal/benchmarking/boomer/userclass" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/prometheus/client_golang/prometheus/promhttp" + "go.opentelemetry.io/otel" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// fakeControlClient keeps each actor's state, so a resume of a running actor +// reports Resumed false the way ateapi does. +type fakeControlClient struct { + ateapipb.ControlClient + mu sync.Mutex + calls []string // " " + states map[string]ateapipb.ActorState + // resumeErr, when set, fails every ResumeActor without changing state. + resumeErr error + // resumeFailures is how many ResumeActor calls fail with Unavailable, + // without changing state, before resumes succeed. + resumeFailures int + // pauseErr, when set, fails every PauseActor without changing state. + pauseErr error + // atespaceErr, when set, fails CreateAtespace. + atespaceErr error + // createErr, when set, fails every CreateActor without creating it. + createErr error + // deleteAborts is how many DeleteActor calls fail with Aborted, as when + // another operation holds the actor lease, before deletes succeed. + deleteAborts int + // resumeEntered and resumeGate, when set, hold every ResumeActor: it + // signals resumeEntered and then waits for resumeGate to close or its + // context to end. + resumeEntered chan struct{} + resumeGate chan struct{} +} + +func newFake() *fakeControlClient { + return &fakeControlClient{states: map[string]ateapipb.ActorState{}} +} + +func (f *fakeControlClient) record(rpc, actor string) { + f.calls = append(f.calls, rpc+" "+actor) +} + +func (f *fakeControlClient) setState(name string, s ateapipb.ActorState) { + f.mu.Lock() + defer f.mu.Unlock() + f.states[name] = s +} + +func (f *fakeControlClient) state(name string) ateapipb.ActorState { + f.mu.Lock() + defer f.mu.Unlock() + return f.states[name] +} + +func (f *fakeControlClient) CreateAtespace(ctx context.Context, in *ateapipb.CreateAtespaceRequest, opts ...grpc.CallOption) (*ateapipb.Atespace, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.record("CreateAtespace", in.GetAtespace().GetMetadata().GetName()) + if f.atespaceErr != nil { + return nil, f.atespaceErr + } + return &ateapipb.Atespace{}, nil +} + +func (f *fakeControlClient) CreateActor(ctx context.Context, in *ateapipb.CreateActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) { + f.mu.Lock() + defer f.mu.Unlock() + name := in.GetActor().GetMetadata().GetName() + f.record("CreateActor", name) + if f.createErr != nil { + return nil, f.createErr + } + f.states[name] = ateapipb.ActorState_ACTOR_STATE_SUSPENDED + return &ateapipb.Actor{}, nil +} + +func (f *fakeControlClient) GetActor(ctx context.Context, in *ateapipb.GetActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) { + f.mu.Lock() + defer f.mu.Unlock() + name := in.GetActor().GetName() + f.record("GetActor", name) + s, ok := f.states[name] + if !ok { + return nil, status.Error(codes.NotFound, "no such actor") + } + return &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: s}}, nil +} + +func (f *fakeControlClient) ResumeActor(ctx context.Context, in *ateapipb.ResumeActorRequest, opts ...grpc.CallOption) (*ateapipb.ResumeActorResponse, error) { + if f.resumeGate != nil { + f.resumeEntered <- struct{}{} + select { + case <-f.resumeGate: + case <-ctx.Done(): + return nil, status.FromContextError(ctx.Err()).Err() + } + } + f.mu.Lock() + defer f.mu.Unlock() + name := in.GetActor().GetName() + f.record("ResumeActor", name) + if f.resumeErr != nil { + return nil, f.resumeErr + } + if f.resumeFailures > 0 { + f.resumeFailures-- + return nil, status.Error(codes.Unavailable, "down") + } + switch f.states[name] { + case ateapipb.ActorState_ACTOR_STATE_RUNNING: + return &ateapipb.ResumeActorResponse{Resumed: false}, nil + case ateapipb.ActorState_ACTOR_STATE_SUSPENDED: + f.states[name] = ateapipb.ActorState_ACTOR_STATE_RUNNING + return &ateapipb.ResumeActorResponse{Resumed: true}, nil + } + return nil, status.Error(codes.Aborted, "actor crashed") +} + +func (f *fakeControlClient) PauseActor(ctx context.Context, in *ateapipb.PauseActorRequest, opts ...grpc.CallOption) (*ateapipb.PauseActorResponse, error) { + f.mu.Lock() + defer f.mu.Unlock() + name := in.GetActor().GetName() + f.record("PauseActor", name) + if f.pauseErr != nil { + return nil, f.pauseErr + } + if f.states[name] != ateapipb.ActorState_ACTOR_STATE_RUNNING { + return nil, status.Error(codes.FailedPrecondition, "not running") + } + // A paused actor is SUSPENDED holding a node-local snapshot; the fake + // tracks only the state. + f.states[name] = ateapipb.ActorState_ACTOR_STATE_SUSPENDED + return &ateapipb.PauseActorResponse{}, nil +} + +func (f *fakeControlClient) DeleteActor(ctx context.Context, in *ateapipb.DeleteActorRequest, opts ...grpc.CallOption) (*ateapipb.Actor, error) { + f.mu.Lock() + defer f.mu.Unlock() + name := in.GetActor().GetName() + f.record("DeleteActor", name) + if f.deleteAborts > 0 { + f.deleteAborts-- + return nil, status.Error(codes.Aborted, "lease held") + } + delete(f.states, name) + return &ateapipb.Actor{}, nil +} + +// callsTo returns the actors of every recorded call to rpc, in order. +func (f *fakeControlClient) callsTo(rpc string) []string { + f.mu.Lock() + defer f.mu.Unlock() + var out []string + for _, c := range f.calls { + if r, actor, _ := strings.Cut(c, " "); r == rpc { + out = append(out, actor) + } + } + return out +} + +func (f *fakeControlClient) resetCalls() { + f.mu.Lock() + defer f.mu.Unlock() + f.calls = nil +} + +func newTestRuntime(fake *fakeControlClient) *runtime { + r := newRuntime(&userclass.Config{ + APIStub: fake, + Atespace: "bench", + Dyn: dynconfig.NewHolder(dynconfig.Config{}), + Tracer: otel.Tracer("test"), + }) + r.retryBackoff = time.Millisecond + r.setupRetryWait = 0 + return r +} + +// setUpUsers runs the first iteration of n users, each on its own goroutine +// as boomer runs them, and returns their actors sorted. +func setUpUsers(t *testing.T, r *runtime, n int) []string { + t.Helper() + var wg sync.WaitGroup + for range n { + wg.Go(r.iterate) + } + wg.Wait() + got := userActors(r) + if len(got) != n { + t.Fatalf("set up %d users, got actors %v", n, got) + } + return got +} + +// userActors returns the actor of every user, sorted. +func userActors(r *runtime) []string { + var out []string + r.users.Range(func(_, v any) bool { + out = append(out, v.(string)) + return true + }) + slices.Sort(out) + return out +} + +// trackedActors returns every actor shutdown would delete, sorted. +func trackedActors(r *runtime) []string { + r.actorsMu.Lock() + defer r.actorsMu.Unlock() + return slices.Sorted(maps.Keys(r.actors)) +} + +func TestRegistered(t *testing.T) { + e, ok := userclass.Lookup("resumepaused") + if !ok { + t.Fatal("resumepaused is not registered") + } + if e.LocustFile != "resumepaused.py" || e.UserClass != userClass { + t.Errorf("got entry %+v", e) + } +} + +func TestStartEnsuresTheAtespace(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + if err := r.start(context.Background()); err != nil { + t.Fatalf("start: %v", err) + } + if got := fake.callsTo("CreateAtespace"); !slices.Equal(got, []string{"bench"}) { + t.Errorf("CreateAtespace calls: got %v", got) + } + if got := fake.callsTo("CreateActor"); len(got) != 0 { + t.Errorf("start created actors %v, want none before a user runs", got) + } +} + +func TestStartFailsWhenTheAtespaceCannotBeEnsured(t *testing.T) { + fake := newFake() + fake.atespaceErr = status.Error(codes.PermissionDenied, "denied") + r := newTestRuntime(fake) + if err := r.start(context.Background()); err == nil { + t.Fatal("start succeeded without an atespace") + } +} + +func TestFirstIterationMakesTheUsersActorCold(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + cold := requestCount(t, "ResumeActorCold", "success") + + r.iterate() + actors := userActors(r) + if len(actors) != 1 { + t.Fatalf("got user actors %v, want 1", actors) + } + name := actors[0] + want := []string{"CreateActor " + name, "ResumeActor " + name, "PauseActor " + name} + if !slices.Equal(fake.calls, want) { + t.Errorf("calls: got %v, want %v", fake.calls, want) + } + if got := fake.state(name); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("actor is %v, want paused (SUSPENDED)", got) + } + if got := trackedActors(r); !slices.Equal(got, actors) { + t.Errorf("tracked %v, want %v", got, actors) + } + if got := requestCount(t, "ResumeActorCold", "success") - cold; got != 0 { + t.Errorf("ResumeActorCold successes grew by %v during setup, want 0", got) + } +} + +func TestIterateResumesThenPausesTheUsersActor(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + name := userActors(r)[0] + fake.resetCalls() + before := requestCount(t, "ResumeActorCold", "success") + + for range 3 { + r.iterate() + } + want := []string{"ResumeActor " + name, "PauseActor " + name} + want = slices.Concat(want, want, want) + if !slices.Equal(fake.calls, want) { + t.Errorf("calls: got %v, want %v", fake.calls, want) + } + if got := requestCount(t, "ResumeActorCold", "success") - before; got != 3 { + t.Errorf("ResumeActorCold successes grew by %v, want 3", got) + } +} + +func TestEachUserCyclesItsOwnActor(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + var wg sync.WaitGroup + for range 3 { + wg.Go(func() { + for range 4 { + r.iterate() + } + }) + } + wg.Wait() + + actors := userActors(r) + if len(actors) != 3 { + t.Fatalf("got user actors %v, want 3", actors) + } + if got := fake.callsTo("CreateActor"); len(got) != 3 { + t.Errorf("CreateActor calls = %v, want one per user", got) + } + // Each user warms its actor once, then cycles it three times. + resumes := map[string]int{} + for _, name := range fake.callsTo("ResumeActor") { + resumes[name]++ + } + for _, name := range actors { + if resumes[name] != 4 { + t.Errorf("actor %s resumed %d times, want 4", name, resumes[name]) + } + } +} + +func TestSetupIsNotPaced(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.cfg.Dyn.Store(dynconfig.Config{TargetRPS: 1}) + start := time.Now() + setUpUsers(t, r, 5) + if elapsed := time.Since(start); elapsed > 500*time.Millisecond { + t.Errorf("setting up 5 users at 1 resume per second took %v, want well under 1s", elapsed) + } +} + +func TestSetupDeletesAnActorThatDoesNotWarmAndRetries(t *testing.T) { + fake := newFake() + fake.resumeFailures = 1 + r := newTestRuntime(fake) + + r.iterate() + if got := userActors(r); len(got) != 0 { + t.Fatalf("user has actor %v after a failed warm-up", got) + } + created, deleted := fake.callsTo("CreateActor"), fake.callsTo("DeleteActor") + if len(created) != 1 || !slices.Equal(deleted, created) { + t.Errorf("created %v, deleted %v, want the one actor deleted", created, deleted) + } + if got := trackedActors(r); len(got) != 0 { + t.Errorf("tracked %v, want none", got) + } + + r.iterate() + actors := userActors(r) + if len(actors) != 1 || actors[0] == created[0] { + t.Fatalf("got user actors %v, want a new one", actors) + } + if got := fake.state(actors[0]); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("actor is %v, want paused (SUSPENDED)", got) + } +} + +func TestSetupKeepsTrackOfAnActorItCouldNotDelete(t *testing.T) { + fake := newFake() + fake.resumeErr = status.Error(codes.Unavailable, "down") + fake.deleteAborts = 5 + r := newTestRuntime(fake) + + r.iterate() + created := fake.callsTo("CreateActor") + if got := trackedActors(r); !slices.Equal(got, created) { + t.Errorf("tracked %v, want %v for shutdown to delete", got, created) + } +} + +func TestSetupRetriesAFailedCreate(t *testing.T) { + fake := newFake() + fake.createErr = status.Error(codes.Unavailable, "down") + r := newTestRuntime(fake) + + r.iterate() + if got := fake.callsTo("DeleteActor"); len(got) != 0 { + t.Errorf("deleted %v, want nothing after a failed create", got) + } + if got := trackedActors(r); len(got) != 0 { + t.Errorf("tracked %v, want none", got) + } + + fake.createErr = nil + r.iterate() + if got := userActors(r); len(got) != 1 { + t.Errorf("got user actors %v, want 1", got) + } +} + +func TestIterateFailsAResumeOfARunningActor(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + name := userActors(r)[0] + fake.setState(name, ateapipb.ActorState_ACTOR_STATE_RUNNING) + fake.resetCalls() + before := requestCount(t, "ResumeActorCold", "failure") + recoveries := requestCount(t, "PauseActorRecovery", "success") + + r.iterate() + if got := requestCount(t, "ResumeActorCold", "failure") - before; got != 1 { + t.Errorf("ResumeActorCold failures grew by %v, want 1", got) + } + if got := requestCount(t, "PauseActorRecovery", "success") - recoveries; got != 1 { + t.Errorf("PauseActorRecovery successes grew by %v, want 1", got) + } + // Recovery pauses the running actor. + want := []string{"ResumeActor " + name, "PauseActor " + name} + if !slices.Equal(fake.calls, want) { + t.Errorf("calls: got %v, want %v", fake.calls, want) + } + if got := fake.state(name); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("actor is %v, want paused (SUSPENDED)", got) + } +} + +func TestIterateKeepsTheActorAfterAFailedResume(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + name := userActors(r)[0] + fake.resumeErr = status.Error(codes.ResourceExhausted, "no free workers available") + + r.iterate() + if got := userActors(r); !slices.Equal(got, []string{name}) { + t.Errorf("user actors: got %v, want [%s]", got, name) + } + if got := fake.state(name); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("actor is %v, want paused (SUSPENDED)", got) + } +} + +func TestIteratePacesResumes(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + + r.cfg.Dyn.Store(dynconfig.Config{TargetRPS: 200}) + start := time.Now() + for range 21 { + r.iterate() + } + // The first resume goes at once; the other 20 take 5 ms each. + if elapsed := time.Since(start); elapsed < 95*time.Millisecond { + t.Errorf("21 resumes at 200 per second took %v, want >= 100ms", elapsed) + } +} + +func TestIterateRecordsTheRecoveryPauseOfAPausedActorAsASuccess(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + name := userActors(r)[0] + fake.resetCalls() + fake.resumeErr = status.Error(codes.Aborted, "lease held") + resumeFailures := requestCount(t, "ResumeActorCold", "failure") + pauseFailures := requestCount(t, "PauseActor", "failure") + recoverySuccesses := requestCount(t, "PauseActorRecovery", "success") + recoveryFailures := requestCount(t, "PauseActorRecovery", "failure") + + // The resume fails with the actor still paused, so the recovery pause + // finds it paused: a success, as the actor is cold. + r.iterate() + if got := fake.callsTo("PauseActor"); !slices.Equal(got, []string{name}) { + t.Errorf("PauseActor calls = %v, want the recovery pause of %s", got, name) + } + if got := requestCount(t, "ResumeActorCold", "failure") - resumeFailures; got != 1 { + t.Errorf("ResumeActorCold failures grew by %v, want 1", got) + } + if got := requestCount(t, "PauseActor", "failure") - pauseFailures; got != 0 { + t.Errorf("PauseActor failures grew by %v, want 0", got) + } + if got := requestCount(t, "PauseActorRecovery", "success") - recoverySuccesses; got != 1 { + t.Errorf("PauseActorRecovery successes grew by %v, want 1", got) + } + if got := requestCount(t, "PauseActorRecovery", "failure") - recoveryFailures; got != 0 { + t.Errorf("PauseActorRecovery failures grew by %v, want 0", got) + } +} + +func TestIterateRecordsAFailedRecoveryPause(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + fake.pauseErr = status.Error(codes.Unavailable, "down") + pauseFailures := requestCount(t, "PauseActor", "failure") + recoveryFailures := requestCount(t, "PauseActorRecovery", "failure") + + // The resume succeeds, the measured pause fails, and so does the + // recovery pause of the running actor. + r.iterate() + if got := requestCount(t, "PauseActor", "failure") - pauseFailures; got != 1 { + t.Errorf("PauseActor failures grew by %v, want 1", got) + } + if got := requestCount(t, "PauseActorRecovery", "failure") - recoveryFailures; got != 1 { + t.Errorf("PauseActorRecovery failures grew by %v, want 1", got) + } +} + +func TestShutdownDeletesEveryActor(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + want := setUpUsers(t, r, 3) + r.shutdown(context.Background()) + + got := fake.callsTo("DeleteActor") + slices.Sort(got) + if !slices.Equal(got, want) { + t.Errorf("deleted %v, want %v", got, want) + } +} + +func TestShutdownRetriesAnAbortedDelete(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + name := setUpUsers(t, r, 1)[0] + fake.deleteAborts = 2 + r.shutdown(context.Background()) + + if got := fake.callsTo("DeleteActor"); len(got) != 3 { + t.Errorf("DeleteActor calls = %v, want 3 calls", got) + } + if _, err := fake.GetActor(context.Background(), &ateapipb.GetActorRequest{Actor: r.ref(name)}); status.Code(err) != codes.NotFound { + t.Errorf("actor still exists after shutdown: err = %v", err) + } +} + +func TestShutdownWaitsForARunningCycle(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + name := userActors(r)[0] + fake.resumeEntered = make(chan struct{}) + fake.resumeGate = make(chan struct{}) + cycleDone := make(chan struct{}) + go func() { + r.cycle(name) + close(cycleDone) + }() + <-fake.resumeEntered + + shutdownDone := make(chan struct{}) + go func() { + r.shutdown(context.Background()) + close(shutdownDone) + }() + select { + case <-shutdownDone: + t.Fatal("shutdown returned while a cycle was running") + case <-time.After(20 * time.Millisecond): + } + if got := fake.callsTo("DeleteActor"); len(got) != 0 { + t.Errorf("deleted %v while a cycle was running", got) + } + + close(fake.resumeGate) + <-cycleDone + <-shutdownDone + if got := fake.callsTo("DeleteActor"); !slices.Equal(got, []string{name}) { + t.Errorf("deleted %v, want [%s]", got, name) + } + r.iterate() + if got := fake.callsTo("ResumeActor"); len(got) != 2 { + t.Errorf("ResumeActor calls = %v, want the warm-up and the one cycle only", got) + } +} + +func TestShutdownWaitsForARunningSetupAndDeletesItsActor(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + fake.resumeEntered = make(chan struct{}) + fake.resumeGate = make(chan struct{}) + setupDone := make(chan struct{}) + go func() { + r.iterate() + close(setupDone) + }() + <-fake.resumeEntered + + shutdownDone := make(chan struct{}) + go func() { + r.shutdown(context.Background()) + close(shutdownDone) + }() + select { + case <-shutdownDone: + t.Fatal("shutdown returned while a setup was running") + case <-time.After(20 * time.Millisecond): + } + + close(fake.resumeGate) + <-setupDone + <-shutdownDone + created := fake.callsTo("CreateActor") + if got := fake.callsTo("DeleteActor"); len(created) != 1 || !slices.Equal(got, created) { + t.Errorf("created %v, deleted %v, want the setup's actor deleted", created, got) + } + + // No setup starts once shutdown has begun. + r.iterate() + if got := fake.callsTo("CreateActor"); len(got) != 1 { + t.Errorf("CreateActor calls = %v after shutdown, want the one before", got) + } +} + +func TestShutdownRecordsOnlyToPrometheus(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + names := setUpUsers(t, r, 3) + var toLocust atomic.Int32 + r.recordSuccess = func(string, string, string, time.Duration, int64) { toLocust.Add(1) } + r.recordFailure = func(string, string, string, time.Duration, string) { toLocust.Add(1) } + + r.cycle(names[0]) + if got := toLocust.Load(); got != 2 { + t.Fatalf("a cycle sent %d samples to locust, want the resume and the pause", got) + } + + // A cycle that finishes once shutdown has begun, and the deletes, record + // to Prometheus only: boomer may no longer drain its stats channel. + fake.resumeEntered = make(chan struct{}) + fake.resumeGate = make(chan struct{}) + go r.cycle(names[0]) + <-fake.resumeEntered + resumes := requestCount(t, "ResumeActorCold", "success") + deletes := requestCount(t, "DeleteActor", "success") + toLocust.Store(0) + shutdownDone := make(chan struct{}) + go func() { + r.shutdown(context.Background()) + close(shutdownDone) + }() + for !closing(r) { + time.Sleep(time.Millisecond) + } + close(fake.resumeGate) + <-shutdownDone + + if got := toLocust.Load(); got != 0 { + t.Errorf("shutdown sent %d samples to locust, want 0", got) + } + if got := requestCount(t, "ResumeActorCold", "success") - resumes; got != 1 { + t.Errorf("ResumeActorCold successes grew by %v, want the in-flight resume", got) + } + if got := requestCount(t, "DeleteActor", "success") - deletes; got != 3 { + t.Errorf("DeleteActor successes grew by %v, want 3", got) + } +} + +func TestShutdownCancelsACycleThatOutlastsItsShare(t *testing.T) { + fake := newFake() + r := newTestRuntime(fake) + r.iterate() + name := userActors(r)[0] + fake.resumeEntered = make(chan struct{}) + fake.resumeGate = make(chan struct{}) // never closed: the resume hangs + cycleDone := make(chan struct{}) + go func() { + r.cycle(name) + close(cycleDone) + }() + <-fake.resumeEntered + + ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond) + defer cancel() + r.shutdown(ctx) + if ctx.Err() != nil { + t.Fatal("shutdown spent its whole budget") + } + select { + case <-cycleDone: + case <-time.After(time.Second): + t.Fatal("the hung cycle was not canceled") + } + if got := fake.callsTo("DeleteActor"); !slices.Equal(got, []string{name}) { + t.Errorf("deleted %v, want [%s]", got, name) + } + if _, err := fake.GetActor(context.Background(), &ateapipb.GetActorRequest{Actor: r.ref(name)}); status.Code(err) != codes.NotFound { + t.Errorf("actor still exists after shutdown: err = %v", err) + } +} + +// closing reports whether shutdown has begun. +func closing(r *runtime) bool { + r.cycleMu.Lock() + defer r.cycleMu.Unlock() + return r.closing +} + +// requestCount scrapes the default registry for the locust_requests_total +// of one request name and status, which is global to the test binary. +func requestCount(t *testing.T, name, result string) float64 { + t.Helper() + return scrape(t, `locust_requests_total{method="grpc",name="`+name+`",status="`+result+`",user_class="ResumePausedUser"} `) +} + +func scrape(t *testing.T, prefix string) float64 { + t.Helper() + rec := httptest.NewRecorder() + promhttp.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil)) + for _, line := range strings.Split(rec.Body.String(), "\n") { + if v, ok := strings.CutPrefix(line, prefix); ok { + f, err := strconv.ParseFloat(v, 64) + if err != nil { + t.Fatalf("parse %q: %v", line, err) + } + return f + } + } + return 0 +} diff --git a/internal/contextlogging/contextlogging_test.go b/internal/contextlogging/contextlogging_test.go index f1203b4538..29eb97b8d8 100644 --- a/internal/contextlogging/contextlogging_test.go +++ b/internal/contextlogging/contextlogging_test.go @@ -143,14 +143,14 @@ func TestHandleUngroupedKeepsTraceFieldsTopLevel(t *testing.T) { // tokenValuer stands in for a type that implements slog.LogValuer and // resolves to a proto; the handler must resolve it before deciding. type tokenValuer struct { - m *ateapipb.MintActorJWTResponse + m *ateapipb.MintActorJwtResponse } func (v tokenValuer) LogValue() slog.Value { return slog.AnyValue(v.m) } func TestHandleRedactsProtoAttrsAnywhereInTheRecord(t *testing.T) { const token = "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhY3RvciJ9.c2lnbmF0dXJl" - resp := &ateapipb.MintActorJWTResponse{ActorJwt: token} + resp := &ateapipb.MintActorJwtResponse{ActorJwt: token} ref := &ateapipb.ObjectRef{Atespace: "ate-demo-sandbox", Name: "agent"} tpl := &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{{ Name: "c", Env: []*ateapipb.EnvVar{{Name: "API_KEY", Value: "sk-secret"}}, @@ -202,7 +202,7 @@ func (stringValuer) LogValue() slog.Value { return slog.StringValue("resolved") // groupValuer resolves to a group that holds a proto, the shape a typed // reference with a nested message would take. type groupValuer struct { - m *ateapipb.MintActorJWTResponse + m *ateapipb.MintActorJwtResponse } func (g groupValuer) LogValue() slog.Value { @@ -215,7 +215,7 @@ type countingValuer struct{ n *int } func (c countingValuer) LogValue() slog.Value { *c.n++; return slog.StringValue("once") } func TestRedactValueReportsWhatItChanges(t *testing.T) { - sensitive := &ateapipb.MintActorJWTResponse{ActorJwt: "token"} + sensitive := &ateapipb.MintActorJwtResponse{ActorJwt: "token"} envless := &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{{Name: "c"}}} for _, tc := range []struct { name string @@ -228,7 +228,7 @@ func TestRedactValueReportsWhatItChanges(t *testing.T) { {"nil any", slog.AnyValue(nil), false}, {"clean type", slog.AnyValue(&ateapipb.ObjectRef{Name: "n"}), false}, {"labeled type, nothing set", slog.AnyValue(envless), false}, - {"typed nil labeled type", slog.AnyValue((*ateapipb.MintActorJWTResponse)(nil)), false}, + {"typed nil labeled type", slog.AnyValue((*ateapipb.MintActorJwtResponse)(nil)), false}, {"group of plain values", slog.GroupValue(slog.String("a", "b"), slog.Any("ref", &ateapipb.ObjectRef{})), false}, {"empty group", slog.GroupValue(), false}, {"sensitive", slog.AnyValue(sensitive), true}, @@ -261,23 +261,23 @@ func TestRedactAttrsReturnsTheInputWhenNothingChanges(t *testing.T) { if len(out) != len(in) || &out[0] != &in[0] { t.Fatal("redactAttrs copied a slice with nothing to change") } - sensitive := &ateapipb.MintActorJWTResponse{ActorJwt: "token"} + sensitive := &ateapipb.MintActorJwtResponse{ActorJwt: "token"} in = append(in, slog.Group("g2", slog.Any("r", sensitive))) out = redactAttrs(in) if &out[0] == &in[0] { t.Fatal("redactAttrs returned the input although an attribute changed") } - if in[3].Value.Group()[0].Value.Any().(*ateapipb.MintActorJWTResponse).GetActorJwt() != "token" { + if in[3].Value.Group()[0].Value.Any().(*ateapipb.MintActorJwtResponse).GetActorJwt() != "token" { t.Fatal("redactAttrs modified the input slice") } - if out[3].Value.Group()[0].Value.Any().(*ateapipb.MintActorJWTResponse).GetActorJwt() != protoredact.Placeholder { + if out[3].Value.Group()[0].Value.Any().(*ateapipb.MintActorJwtResponse).GetActorJwt() != protoredact.Placeholder { t.Fatal("redactAttrs did not mask the group member") } } func TestHandleCoversGroupsBoundAttrsAndNilsTogether(t *testing.T) { const token = "eyJhbGciOiJSUzI1NiJ9.token" - sensitive := &ateapipb.MintActorJWTResponse{ActorJwt: token} + sensitive := &ateapipb.MintActorJwtResponse{ActorJwt: token} var buf bytes.Buffer base := slog.New(NewHandler(slog.NewJSONHandler(&buf, &slog.HandlerOptions{ReplaceAttr: dropTime}))) @@ -288,7 +288,7 @@ func TestHandleCoversGroupsBoundAttrsAndNilsTogether(t *testing.T) { With(slog.Group("bound", slog.Any("resp", sensitive))). WithGroup("inner"). Info("x", - slog.Any("nilresp", (*ateapipb.MintActorJWTResponse)(nil)), + slog.Any("nilresp", (*ateapipb.MintActorJwtResponse)(nil)), slog.Any("valuer", groupValuer{sensitive}), slog.String("plain", "keep"), ) @@ -313,7 +313,7 @@ func TestHandleResolvesLogValuersOnce(t *testing.T) { calls := 0 attrs := []slog.Attr{slog.Any("v", countingValuer{&calls}), slog.String("a", "b")} if withProto { - attrs = append(attrs, slog.Any("resp", &ateapipb.MintActorJWTResponse{ActorJwt: "t"})) + attrs = append(attrs, slog.Any("resp", &ateapipb.MintActorJwtResponse{ActorJwt: "t"})) } slog.New(NewHandler(slog.NewJSONHandler(io.Discard, nil))).LogAttrs(context.Background(), slog.LevelInfo, "x", attrs...) if calls != 1 { @@ -419,12 +419,12 @@ func BenchmarkHandle(b *testing.B) { // with ==: a value receiver and a slice field. Comparing two of them as // interfaces panics at run time. type uncomparableMsg struct { - *ateapipb.MintActorJWTResponse + *ateapipb.MintActorJwtResponse _ []int } func (u uncomparableMsg) ProtoReflect() protoreflect.Message { - return u.MintActorJWTResponse.ProtoReflect() + return u.MintActorJwtResponse.ProtoReflect() } func TestHandleToleratesUncomparableMessageTypes(t *testing.T) { @@ -435,8 +435,8 @@ func TestHandleToleratesUncomparableMessageTypes(t *testing.T) { msg uncomparableMsg want string }{ - {"nothing to mask", uncomparableMsg{MintActorJWTResponse: &ateapipb.MintActorJWTResponse{}}, `"v":{}`}, - {"secret set", uncomparableMsg{MintActorJWTResponse: &ateapipb.MintActorJWTResponse{ActorJwt: "token"}}, protoredact.Placeholder}, + {"nothing to mask", uncomparableMsg{MintActorJwtResponse: &ateapipb.MintActorJwtResponse{}}, `"v":{}`}, + {"secret set", uncomparableMsg{MintActorJwtResponse: &ateapipb.MintActorJwtResponse{ActorJwt: "token"}}, protoredact.Placeholder}, } { t.Run(tc.name, func(t *testing.T) { var buf bytes.Buffer diff --git a/internal/e2e/README.md b/internal/e2e/README.md index f66e67b6e6..0045c221a5 100644 --- a/internal/e2e/README.md +++ b/internal/e2e/README.md @@ -56,6 +56,20 @@ installs the fixtures elsewhere: `E2E_SUBSTRATE_TEMPLATE_ATESPACE` / gVisor, 10m on micro-VM, where the golden is a cloud-hypervisor cold boot plus a checkpoint). +## Settings + +The installer reads this setting in addition to those in +[docs/operator-install.md](../../docs/operator-install.md#settings), through the same +flag, environment variable, and configuration document channels. It sets the +worker thread count (`--concurrency`) of the Envoy container in `atenet-egress`; +`0` leaves the manifest unchanged. The +[nighthawk egress benchmark](../../benchmarking/nighthawk-egress/README.md) sets +it to `1` to measure single-CPU Envoy performance. + +| Key | Environment variable | Flag | Default | Applies to | +|------------------------|-----------------------------|---------------------------|---------|------------| +| `e2e.envoyConcurrency` | `E2E_ATE_ENVOY_CONCURRENCY` | `--e2e-envoy-concurrency` | `0` | (global) | + ## After a failure A suite deletes the namespaces it created only when it passed. A failed run diff --git a/internal/e2e/atenet_dataplane.go b/internal/e2e/atenet_dataplane.go index 50ca3f86a2..63438a0d29 100644 --- a/internal/e2e/atenet_dataplane.go +++ b/internal/e2e/atenet_dataplane.go @@ -38,6 +38,16 @@ type AtenetDataplane interface { EgressUpgradeDeniedBody() string PlatformMetricPrefixes([]string) []string RouteDurationSeen(context.Context, string) (bool, error) + ScrapeMetrics(context.Context) (string, error) + EgressConnectStats(before, after string) EgressConnectStats +} + +// EgressConnectStats holds the delta of egress CONNECT connections and requests +// observed between two metric scrapes. A value of -1 indicates that the +// dataplane does not report the counter. +type EgressConnectStats struct { + Connections int + Requests int } // CurrentAtenetDataplane returns the implementation selected for this test @@ -82,6 +92,48 @@ func (envoyAtenetDataplane) RouteDurationSeen(_ context.Context, collectorScrape return len(MissingPlatformMetrics(collectorScrape, []string{"atenet_router_route_duration"})) == 0, nil } +func (envoyAtenetDataplane) ScrapeMetrics(ctx context.Context) (string, error) { + return ScrapeEgressEnvoyMetrics(ctx) +} + +func (envoyAtenetDataplane) EgressConnectStats(before, after string) EgressConnectStats { + beforeStats := parseEnvoyEgressConnectMetrics(before) + afterStats := parseEnvoyEgressConnectMetrics(after) + return EgressConnectStats{ + Connections: afterStats.Connections - beforeStats.Connections, + Requests: afterStats.Requests - beforeStats.Requests, + } +} + +func parseEnvoyEgressConnectMetrics(scrape string) EgressConnectStats { + var stats EgressConnectStats + for line := range strings.SplitSeq(scrape, "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + if !strings.Contains(line, `envoy_http_conn_manager_prefix="egress_connect"`) { + continue + } + fields := strings.Fields(line) + if len(fields) < 2 { + continue + } + val, err := strconv.ParseFloat(fields[len(fields)-1], 64) + if err != nil { + continue + } + switch { + case strings.HasPrefix(line, "envoy_http_downstream_cx_http2_total{"): + stats.Connections += int(val) + case strings.HasPrefix(line, "envoy_http_downstream_rq_xx{") && + strings.Contains(line, `envoy_response_code_class="2"`): + stats.Requests += int(val) + } + } + return stats +} + type agentGatewayAtenetDataplane struct{} func (agentGatewayAtenetDataplane) NewParkingObserver(context.Context) (ParkingObserver, error) { @@ -120,6 +172,14 @@ func (agentGatewayAtenetDataplane) RouteDurationSeen(ctx context.Context, _ stri return len(MissingPlatformMetrics(scrape, []string{"agentgateway_atenet_router_route_duration_seconds"})) == 0, nil } +func (agentGatewayAtenetDataplane) ScrapeMetrics(ctx context.Context) (string, error) { + return ScrapeAgentGatewayRouterMetrics(ctx) +} + +func (agentGatewayAtenetDataplane) EgressConnectStats(_, _ string) EgressConnectStats { + return EgressConnectStats{Connections: -1, Requests: -1} +} + type agentGatewayParkingObserver struct{} func (agentGatewayParkingObserver) WaitForCount(ctx context.Context, cond func(int) bool) (int, error) { diff --git a/internal/e2e/atenet_dataplane_test.go b/internal/e2e/atenet_dataplane_test.go index 149d12016b..9d3b7c6d45 100644 --- a/internal/e2e/atenet_dataplane_test.go +++ b/internal/e2e/atenet_dataplane_test.go @@ -33,3 +33,31 @@ func TestAtenetDataplaneEgressPolicyDenial(t *testing.T) { } }) } + +func TestAtenetDataplaneEgressConnectStats(t *testing.T) { + const before = `# TYPE envoy_http_downstream_cx_http2_total counter +envoy_http_downstream_cx_http2_total{envoy_http_conn_manager_prefix="egress_connect"} 1 +envoy_http_downstream_rq_xx{envoy_response_code_class="2",envoy_http_conn_manager_prefix="egress_connect"} 2 +` + const after = `# TYPE envoy_http_downstream_cx_http2_total counter +envoy_http_downstream_cx_http2_total{envoy_http_conn_manager_prefix="egress_connect"} 4 +envoy_http_downstream_rq_xx{envoy_response_code_class="2",envoy_http_conn_manager_prefix="egress_connect"} 252 +` + + t.Run("envoy", func(t *testing.T) { + t.Setenv(AtenetDataplaneEnv, "") + got := CurrentAtenetDataplane().EgressConnectStats(before, after) + want := EgressConnectStats{Connections: 3, Requests: 250} + if got != want { + t.Errorf("EgressConnectStats() = %+v, want %+v", got, want) + } + }) + t.Run("agentgateway", func(t *testing.T) { + t.Setenv(AtenetDataplaneEnv, "agentgateway") + got := CurrentAtenetDataplane().EgressConnectStats(before, after) + want := EgressConnectStats{Connections: -1, Requests: -1} + if got != want { + t.Errorf("EgressConnectStats() = %+v, want %+v", got, want) + } + }) +} diff --git a/internal/e2e/collector_metrics.go b/internal/e2e/collector_metrics.go index 293a63ddae..218ead3a52 100644 --- a/internal/e2e/collector_metrics.go +++ b/internal/e2e/collector_metrics.go @@ -25,7 +25,9 @@ import ( "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/internal/portforward" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" ) const ( @@ -35,6 +37,8 @@ const ( // AgentGateway exposes native Prometheus metrics; it does not export these // instruments through the OTLP collector. agentGatewayRouterStatsPort = 15020 + // Envoy exposes its Prometheus stats on the admin listener. + egressEnvoyAdminPort = 15000 ) // PlatformMetricPrefixes are the Prometheus metric-name prefixes (OTLP dots @@ -155,6 +159,72 @@ func MissingPlatformMetrics(scrape string, prefixes []string) []string { return missing } +// Port-forward each ready atenet-egress pod's Envoy admin port and read its +// stats/prometheus endpoint, returning the concatenated Prometheus exposition text. +func ScrapeEgressEnvoyMetrics(ctx context.Context) (string, error) { + config, err := ateclient.LoadKubeConfig(KubeConfig, KubeContext) + if err != nil { + return "", fmt.Errorf("loading kubeconfig: %w", err) + } + clientset, err := kubernetes.NewForConfig(config) + if err != nil { + return "", fmt.Errorf("creating k8s client: %w", err) + } + + ns := SystemNamespace() + pods, err := clientset.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{ + LabelSelector: "app=atenet-egress", + }) + if err != nil { + return "", fmt.Errorf("listing atenet-egress pods: %w", err) + } + + var b strings.Builder + ready := 0 + for i := range pods.Items { + pod := &pods.Items[i] + if !portforward.IsPodReady(pod) { + continue + } + ready++ + body, err := scrapeEgressEnvoyPod(ctx, config, clientset, ns, pod.Name) + if err != nil { + return "", err + } + b.WriteString(body) + } + if ready == 0 { + return "", fmt.Errorf("no ready atenet-egress pods in %s", ns) + } + return b.String(), nil +} + +func scrapeEgressEnvoyPod(ctx context.Context, config *rest.Config, clientset kubernetes.Interface, namespace, podName string) (string, error) { + localPort, stop, err := portforward.PodPortForward(ctx, config, clientset, namespace, podName, egressEnvoyAdminPort) + if err != nil { + return "", err + } + defer stop() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, fmt.Sprintf("http://127.0.0.1:%d/stats/prometheus", localPort), nil) + if err != nil { + return "", err + } + resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req) + if err != nil { + return "", fmt.Errorf("scraping egress Envoy metrics on %s: %w", podName, err) + } + defer resp.Body.Close() + body, err := io.ReadAll(resp.Body) + if err != nil { + return "", fmt.Errorf("reading egress Envoy metrics on %s: %w", podName, err) + } + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("egress Envoy metrics on %s returned %d: %s", podName, resp.StatusCode, body) + } + return string(body), nil +} + // LifecycleEventMetric is the count connector's view of the actor lifecycle // events. Reading it rather than a log store keeps this check on the metrics // harness: kind has no place to query log records. diff --git a/internal/e2e/egresspolicy.go b/internal/e2e/egresspolicy.go index d250d83b41..dae98f054b 100644 --- a/internal/e2e/egresspolicy.go +++ b/internal/e2e/egresspolicy.go @@ -34,8 +34,8 @@ import ( // rule on every port would tie with the http rule, which the API rejects. func EgressAllowAll() []*ateapipb.EgressRule { return []*ateapipb.EgressRule{ - {Http: &ateapipb.HTTPRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, - {Https: &ateapipb.HTTPSRule{Hostnames: []string{"*"}}}, + {Http: &ateapipb.HttpRule{Hostnames: []string{"*"}, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}}}, + {Https: &ateapipb.HttpsRule{Hostnames: []string{"*"}}}, } } @@ -43,20 +43,20 @@ func EgressAllowAll() []*ateapipb.EgressRule { // hosts matching patterns (exact names, or "*." plus a name for one leftmost // label) on port 80. func EgressAllowHTTP(patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Http: &ateapipb.HTTPRule{Hostnames: patterns}} + return &ateapipb.EgressRule{Http: &ateapipb.HttpRule{Hostnames: patterns}} } // EgressAllowHTTPS is a rule that lets an actor send HTTPS, intercepted by the // gateway, to the hosts matching patterns on port 443. func EgressAllowHTTPS(patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Https: &ateapipb.HTTPSRule{Hostnames: patterns}} + return &ateapipb.EgressRule{Https: &ateapipb.HttpsRule{Hostnames: patterns}} } // EgressAllowPassthrough is a rule that lets an actor send TLS to the hosts // matching patterns on any port, relayed by the gateway without decryption. func EgressAllowPassthrough(patterns ...string) *ateapipb.EgressRule { return &ateapipb.EgressRule{ - TlsPassthrough: &ateapipb.TLSPassthroughRule{ + TlsPassthrough: &ateapipb.TlsPassthroughRule{ Hostnames: patterns, Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}, }, @@ -67,7 +67,7 @@ func EgressAllowPassthrough(patterns ...string) *ateapipb.EgressRule { // given ports. func EgressAllowPassthroughOnPorts(ports []int32, patterns ...string) *ateapipb.EgressRule { return &ateapipb.EgressRule{ - TlsPassthrough: &ateapipb.TLSPassthroughRule{ + TlsPassthrough: &ateapipb.TlsPassthroughRule{ Hostnames: patterns, Ports: &ateapipb.Ports{Numbers: ports}, }, diff --git a/internal/e2e/fixture.go b/internal/e2e/fixture.go index 65a80d7258..43ecf5ef56 100644 --- a/internal/e2e/fixture.go +++ b/internal/e2e/fixture.go @@ -32,7 +32,8 @@ import ( // fixtureSubstitutions, whose block fragments are CRD-shaped and now serve // only the pool manifests. The inline placeholders match the pool manifests' // values so one (bucket, name) pair renders both halves of a fixture. -func substrateTemplateSubstitutions(bucket, name string, trustBundle bool) (inline, blocks map[string]string) { +// trustBundles names the trust bundles projected into one trust-bundle.pem. +func substrateTemplateSubstitutions(bucket, name string, trustBundles []string) (inline, blocks map[string]string) { inline = map[string]string{ "${BUCKET_NAME}": bucket, "${FIXTURE_SUFFIX}": "-" + name, @@ -46,10 +47,14 @@ func substrateTemplateSubstitutions(bucket, name string, trustBundle bool) (inli // Off unless the caller opts in; see WithTrustBundle. "${TEMPLATE_TRUST_BUNDLE}": "", } - if trustBundle { - // Indented to sit in a template's systemInfo dataSources list. The + if len(trustBundles) > 0 { + // Indented to sit in a template's systemInfo dataSources list. Each // name must be on atelet's supported-bundle allowlist. - blocks["${TEMPLATE_TRUST_BUNDLE}"] = " - trustBundle:\n names:\n - egress-mitm.ate.dev\n path: trust-bundle.pem" + block := " - trustBundle:\n names:\n" + for _, bundle := range trustBundles { + block += " - " + bundle + "\n" + } + blocks["${TEMPLATE_TRUST_BUNDLE}"] = block + " path: trust-bundle.pem" } if !IsMicroVM() { return inline, blocks @@ -118,9 +123,12 @@ type SubstrateFixtureManifests struct { // the k8s namespace GC — and a template leaked by an interrupted earlier run // is cleared before creating its replacement, since templates are immutable. // +// trustBundles names the trust bundles a template with a +// ${TEMPLATE_TRUST_BUNDLE} placeholder projects; nil projects none. +// // Returns the fixture's atespace (the same string that names the k8s // namespace holding the pool) and the created templates. -func DeploySubstrateFixture(t *testing.T, ctx context.Context, clients *Clients, manifests SubstrateFixtureManifests, bucket, name string, trustBundle bool) (string, []*ateapipb.ActorTemplate) { +func DeploySubstrateFixture(t *testing.T, ctx context.Context, clients *Clients, manifests SubstrateFixtureManifests, bucket, name string, trustBundles []string) (string, []*ateapipb.ActorTemplate) { t.Helper() // The pool manifest also carries the namespace. Its cleanup is registered @@ -137,7 +145,7 @@ func DeploySubstrateFixture(t *testing.T, ctx context.Context, clients *Clients, RunCmd(t, "kubectl", delArgs...) }) - tmplInline, tmplBlocks := substrateTemplateSubstitutions(bucket, name, trustBundle) + tmplInline, tmplBlocks := substrateTemplateSubstitutions(bucket, name, trustBundles) rendered := koResolve(t, renderManifest(t, manifests.Template, tmplInline, tmplBlocks)) templates := decodeSubstrateTemplates(t, rendered) diff --git a/internal/e2e/fixtures/probe/main.go b/internal/e2e/fixtures/probe/main.go index 9db51f30db..e1c3a113b0 100644 --- a/internal/e2e/fixtures/probe/main.go +++ b/internal/e2e/fixtures/probe/main.go @@ -318,10 +318,12 @@ func parseFetchHeaders(params []string) (http.Header, error) { // path. Parameters to the fetch are passed as URL query parameters: // // - url=: required. -// - roots=bundle|system: the TLS trust anchors. "bundle" (the default) +// - roots=bundle|system|pem: the TLS trust anchors. "bundle" (the default) // loads the projected trust bundle at trustFile; "system" uses the -// image's system roots. TestActorEgressMITMTrust documents why each mode -// passes or fails. +// image's system roots; "pem" trusts only the certificates in ca. +// TestActorEgressMITMTrust documents why each mode passes or fails. +// - ca=: the certificates roots=pem trusts. Required by roots=pem and +// rejected with any other mode. // - header=:: repeatable; set on the request, so a suite can // pre-seed a header and observe whether the gateway overwrites it. // - dial=: connect to this address instead of the URL's host, @@ -352,12 +354,20 @@ func fetch(w http.ResponseWriter, r *http.Request) { } roots := r.URL.Query().Get("roots") switch roots { - case "", "bundle", "system": + case "", "bundle", "system", "pem": default: // Fail closed on typos: silently treating an unknown value as // "bundle" would flip a suite's negative control into a positive // fetch with a misleading failure message. - resp["error"] = "unknown roots value " + strconv.Quote(roots) + " (want bundle or system)" + resp["error"] = "unknown roots value " + strconv.Quote(roots) + " (want bundle, system or pem)" + writeJSON(w, resp) + return + } + // The same fail-closed reasoning: a ca the fetch would ignore, or a + // roots=pem with nothing to trust, is a suite bug, not a result. + ca := r.URL.Query().Get("ca") + if (roots == "pem") != (ca != "") { + resp["error"] = "the ca parameter goes with roots=pem, and roots=pem needs it" writeJSON(w, resp) return } @@ -376,7 +386,17 @@ func fetch(w http.ResponseWriter, r *http.Request) { } } tlsCfg := &tls.Config{} - if roots != "system" { + switch roots { + case "system": + case "pem": + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM([]byte(ca)) { + resp["error"] = "no certificates parsed from the ca parameter" + writeJSON(w, resp) + return + } + tlsCfg.RootCAs = pool + default: b, err := os.ReadFile(trustFile) if err != nil { resp["error"] = "reading trust bundle: " + err.Error() diff --git a/internal/e2e/fixtures/probe/main_test.go b/internal/e2e/fixtures/probe/main_test.go index f5a44cdd55..cf10505e29 100644 --- a/internal/e2e/fixtures/probe/main_test.go +++ b/internal/e2e/fixtures/probe/main_test.go @@ -16,6 +16,8 @@ package main import ( "encoding/json" + "encoding/pem" + "net" "net/http" "net/http/httptest" "net/url" @@ -24,6 +26,8 @@ import ( "testing" "github.com/google/go-cmp/cmp" + + "github.com/agent-substrate/substrate/internal/testcert" ) // The capabilities e2e assertions are only as trustworthy as this decoder and @@ -263,6 +267,70 @@ func TestFetchRejectsMalformedDial(t *testing.T) { } } +// roots=pem trusts exactly the certificates in ca, which is how a suite +// verifies an origin's own certificate rather than one the gateway minted. +func TestFetchTrustsOnlyTheGivenCA(t *testing.T) { + origin := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte("ok")) + })) + defer origin.Close() + originCA := string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: origin.Certificate().Raw})) + unrelatedCA := string(testcert.NewServerTLS(t, net.ParseIP("127.0.0.1")).RootCA) + + for _, tc := range []struct { + name string + ca string + wantOK bool + }{ + {"origin's CA", originCA, true}, + {"origin's CA among others", unrelatedCA + originCA, true}, + {"unrelated CA", unrelatedCA, false}, + } { + t.Run(tc.name, func(t *testing.T) { + resp := doFetchQuery(t, url.Values{"url": {origin.URL}, "roots": {"pem"}, "ca": {tc.ca}}) + if tc.wantOK { + if resp["error"] != "" || resp["status"] != "200" { + t.Errorf("fetch = status %q, error %q; want 200", resp["status"], resp["error"]) + } + return + } + if !strings.Contains(resp["error"], "certificate") { + t.Errorf("error = %q, want a certificate verification failure", resp["error"]) + } + }) + } +} + +// roots=pem and ca come together, and ca must hold a certificate; anything +// else fails before the request is sent. +func TestFetchRejectsMismatchedCA(t *testing.T) { + origin := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + t.Error("fetch sent the request despite a bad ca parameter") + })) + defer origin.Close() + + for _, tc := range []struct { + name string + query url.Values + wantErr string + }{ + {"roots=pem without ca", url.Values{"roots": {"pem"}}, "ca parameter"}, + {"ca without roots=pem", url.Values{"roots": {"system"}, "ca": {"x"}}, "ca parameter"}, + {"ca without a certificate", url.Values{"roots": {"pem"}, "ca": {"not a certificate"}}, "no certificates parsed"}, + } { + t.Run(tc.name, func(t *testing.T) { + tc.query.Set("url", origin.URL) + resp := doFetchQuery(t, tc.query) + if !strings.Contains(resp["error"], tc.wantErr) { + t.Errorf("error = %q, want one mentioning %q", resp["error"], tc.wantErr) + } + if resp["status"] != "" { + t.Errorf("status = %q, want none", resp["status"]) + } + }) + } +} + func TestFetchTruncatesBody(t *testing.T) { origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(strings.Repeat("x", maxFetchBody+1))) diff --git a/internal/e2e/fixtures/serverpod.yaml.tmpl b/internal/e2e/fixtures/serverpod.yaml.tmpl index 4b5df22ba7..0159e22154 100644 --- a/internal/e2e/fixtures/serverpod.yaml.tmpl +++ b/internal/e2e/fixtures/serverpod.yaml.tmpl @@ -18,10 +18,10 @@ # for what each placeholder means. # # The differences between those servers are the placeholders below: a name, an -# image, a port, a readiness probe, and whatever credentials the server has to -# mount. Everything else was identical in every copy of this manifest that -# existed before it was shared, so a per-server file only offered somewhere for -# the copies to drift apart. +# image, a port and optionally an HTTPS one, a readiness probe, and whatever +# credentials the server has to mount. Everything else was identical in every +# copy of this manifest that existed before it was shared, so a per-server file +# only offered somewhere for the copies to drift apart. # # A plain Pod, not an actor. These servers are scaffolding around the code under # test, and running one inside an actor would make its snapshot or restore @@ -43,6 +43,8 @@ ${ARGS} ports: - name: serve containerPort: ${TARGET_PORT} + # The HTTPS listener, present when the ServerPod sets TLSPort. +${TLS_CONTAINER_PORT} # The suite starts dialing as soon as it has an address, so without a # readiness gate the first connection can land before the listener exists # and come back refused -- which reads as a failure of whatever sits between @@ -83,3 +85,4 @@ spec: - name: serve port: ${PORT} targetPort: ${TARGET_PORT} +${TLS_SERVICE_PORT} diff --git a/internal/e2e/fixtures/testserver/common_test.go b/internal/e2e/fixtures/testserver/common_test.go index d6773c08f7..a64d46153c 100644 --- a/internal/e2e/fixtures/testserver/common_test.go +++ b/internal/e2e/fixtures/testserver/common_test.go @@ -55,26 +55,22 @@ func writeOriginCertificate(t *testing.T) originCertificate { return originCertificate{caPEM: material.RootCA, certFile: certFile, keyFile: keyFile} } +// startOriginServer serves handler the way serveHTTP does: in cleartext on one +// loopback listener and, given a certificate, over TLS on a second. The URLs of +// a scheme the server does not serve are empty. func startOriginServer(t *testing.T, handler http.Handler, certFile, keyFile string) originServer { t.Helper() - listener, err := net.Listen("tcp", "127.0.0.1:0") - if err != nil { - t.Fatalf("listen on loopback: %v", err) - } server := &http.Server{Handler: handler} - if certFile != "" || keyFile != "" { - if err := configureTLSServer(server, certFile, keyFile); err != nil { - t.Fatalf("configure TLS: %v", err) - } + if err := configureTLSServer(server, certFile, keyFile); err != nil { + t.Fatalf("configure TLS: %v", err) + } + cleartext := listenLoopback(t) + var encrypted net.Listener + if server.TLSConfig != nil { + encrypted = listenLoopback(t) } serveErr := make(chan error, 1) - go func() { - if certFile == "" && keyFile == "" { - serveErr <- server.Serve(listener) - return - } - serveErr <- server.ServeTLS(listener, "", "") - }() + go func() { serveErr <- serveListeners(server, cleartext, encrypted) }() t.Cleanup(func() { ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() @@ -86,11 +82,22 @@ func startOriginServer(t *testing.T, handler http.Handler, certFile, keyFile str } }) - address := listener.Addr().String() - return originServer{ - httpURL: "http://" + address, - httpsURL: "https://" + address, - wsURL: "ws://" + address, - wssURL: "wss://" + address, + origin := originServer{ + httpURL: "http://" + cleartext.Addr().String(), + wsURL: "ws://" + cleartext.Addr().String(), + } + if encrypted != nil { + origin.httpsURL = "https://" + encrypted.Addr().String() + origin.wssURL = "wss://" + encrypted.Addr().String() + } + return origin +} + +func listenLoopback(t *testing.T) net.Listener { + t.Helper() + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen on loopback: %v", err) } + return listener } diff --git a/internal/e2e/fixtures/testserver/egressprobe.yaml.tmpl b/internal/e2e/fixtures/testserver/egressprobe.yaml.tmpl index 52134b2489..7dc0b2f676 100644 --- a/internal/e2e/fixtures/testserver/egressprobe.yaml.tmpl +++ b/internal/e2e/fixtures/testserver/egressprobe.yaml.tmpl @@ -25,6 +25,7 @@ metadata: namespace: ${NAMESPACE} labels: app: egressprobe + ate.dev/worker-pool: egressprobe spec: restartPolicy: Never containers: diff --git a/internal/e2e/fixtures/testserver/http.go b/internal/e2e/fixtures/testserver/http.go index 5e2e5a0409..32e57ce5b7 100644 --- a/internal/e2e/fixtures/testserver/http.go +++ b/internal/e2e/fixtures/testserver/http.go @@ -16,6 +16,7 @@ package main import ( "encoding/json" + "fmt" "io" "log" "net/http" @@ -40,6 +41,7 @@ type echoedHeaders struct { // port. The optional body makes the same fixture useful for assertions about a // response inside the egress tunnel. /echoheaders returns the request headers as // they arrived, which is how a suite sees what the gateway put on the wire. +// The /fetch handler for e2e tests that need test requests with bodies. func newHTTPHandler(body, authorizationFile string) http.Handler { mux := http.NewServeMux() mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { @@ -61,6 +63,13 @@ func newHTTPHandler(body, authorizationFile string) http.Handler { log.Printf("testserver http: encoding /echoheaders response: %v", err) } }) + mux.HandleFunc("POST /fetch", func(w http.ResponseWriter, r *http.Request) { + if _, err := io.Copy(io.Discard, r.Body); err != nil { + http.Error(w, fmt.Sprintf("reading request body: %v", err), http.StatusBadRequest) + return + } + w.WriteHeader(http.StatusOK) + }) return mux } @@ -74,7 +83,7 @@ func newHTTPCmd() *cobra.Command { ) cmd := &cobra.Command{ Use: "http", - Short: "Serve an HTTP/1.1 origin answering /healthz and /echoheaders.", + Short: "Serve an HTTP/1.1 origin answering /healthz, /echoheaders and POST /fetch.", Args: cobra.NoArgs, RunE: func(_ *cobra.Command, _ []string) error { server := &http.Server{ @@ -87,9 +96,9 @@ func newHTTPCmd() *cobra.Command { return serveHTTP(server, certFile, keyFile) }, } - cmd.Flags().StringVar(&listenAddress, "listen", ":8080", "Address the HTTP origin listens on.") - cmd.Flags().StringVar(&certFile, "tls-cert", "", "PEM certificate file; enables HTTPS when paired with --tls-key.") - cmd.Flags().StringVar(&keyFile, "tls-key", "", "PEM private key file; enables HTTPS when paired with --tls-cert.") + cmd.Flags().StringVar(&listenAddress, "listen", ":8080", "Address the origin serves cleartext HTTP on.") + cmd.Flags().StringVar(&certFile, "tls-cert", "", "PEM certificate file; with --tls-key, also serves HTTPS on :8443.") + cmd.Flags().StringVar(&keyFile, "tls-key", "", "PEM private key file; with --tls-cert, also serves HTTPS on :8443.") cmd.Flags().StringVar(&authorizationFile, "authorization-file", "", "Enable /credential, requiring a Bearer token matching this file.") cmd.Flags().StringVar(&body, "body", "", "Response body for /healthz.") return cmd diff --git a/internal/e2e/fixtures/testserver/http_test.go b/internal/e2e/fixtures/testserver/http_test.go index ad6fa0e92d..88875a4327 100644 --- a/internal/e2e/fixtures/testserver/http_test.go +++ b/internal/e2e/fixtures/testserver/http_test.go @@ -138,6 +138,100 @@ func TestHTTPSTestResponseUsesHTTP11AndVerifiedIdentity(t *testing.T) { } } +func TestHTTPHandlerFetch(t *testing.T) { + server := startOriginServer(t, newHTTPHandler("", ""), "", "") + client := &http.Client{Timeout: 2 * time.Second} + + t.Run("waits for full POST body before responding 200", func(t *testing.T) { + pr, pw := io.Pipe() + req, err := http.NewRequest(http.MethodPost, server.httpURL+"/fetch", pr) + if err != nil { + t.Fatal(err) + } + + type result struct { + resp *http.Response + err error + } + done := make(chan result, 1) + go func() { + resp, err := client.Do(req) + done <- result{resp: resp, err: err} + }() + + if _, err := pw.Write([]byte("part1")); err != nil { + t.Fatal(err) + } + select { + case res := <-done: + t.Fatalf("POST /fetch responded before body EOF: resp=%v, err=%v", res.resp, res.err) + case <-time.After(50 * time.Millisecond): + } + + if _, err := pw.Write([]byte("part2")); err != nil { + t.Fatal(err) + } + if err := pw.Close(); err != nil { + t.Fatal(err) + } + + res := <-done + if res.err != nil { + t.Fatalf("POST /fetch: %v", res.err) + } + defer res.resp.Body.Close() + if res.resp.StatusCode != http.StatusOK { + t.Errorf("POST /fetch status = %d, want %d", res.resp.StatusCode, http.StatusOK) + } + }) + + t.Run("rejects GET with 405", func(t *testing.T) { + resp, err := client.Get(server.httpURL + "/fetch") + if err != nil { + t.Fatalf("GET /fetch: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusMethodNotAllowed { + t.Errorf("GET /fetch status = %d, want %d", resp.StatusCode, http.StatusMethodNotAllowed) + } + }) +} + +func TestHTTPServesCleartextBesideTLS(t *testing.T) { + cert := writeOriginCertificate(t) + server := startOriginServer(t, newHTTPHandler("both schemes", ""), cert.certFile, cert.keyFile) + + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(cert.caPEM) { + t.Fatal("adding fixture CA to trust pool") + } + client := &http.Client{ + Timeout: 2 * time.Second, + Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: roots, ServerName: "127.0.0.1"}}, + } + for _, url := range []string{server.httpURL + "/healthz", server.httpsURL + "/healthz"} { + response, err := client.Get(url) + if err != nil { + t.Fatalf("GET %s: %v", url, err) + } + body, err := io.ReadAll(response.Body) + response.Body.Close() + if err != nil { + t.Fatalf("reading %s: %v", url, err) + } + if response.StatusCode != http.StatusOK || string(body) != "both schemes" { + t.Errorf("GET %s = %d %q, want 200 %q", url, response.StatusCode, body, "both schemes") + } + } +} + +func TestHTTPWithoutCertificateServesOnlyCleartext(t *testing.T) { + server := startOriginServer(t, newHTTPHandler("", ""), "", "") + if server.httpsURL != "" { + t.Errorf("origin without a certificate serves HTTPS at %s, want cleartext only", server.httpsURL) + } +} + func TestCredentialHandler(t *testing.T) { path := filepath.Join(t.TempDir(), "token") if err := os.WriteFile(path, []byte("expected-token"), 0600); err != nil { diff --git a/internal/e2e/fixtures/testserver/main.go b/internal/e2e/fixtures/testserver/main.go index addd7b57d5..177c89e0c1 100644 --- a/internal/e2e/fixtures/testserver/main.go +++ b/internal/e2e/fixtures/testserver/main.go @@ -22,9 +22,11 @@ // testserver egressprobe --listen=:8080 a client that drives the egress gateway // testserver websocket --listen=:8080 a websocket server that responds to PINGs // -// Each pod runs exactly one subcommand on one listener, so the wire behavior of -// any given pod is unchanged from when these were separate binaries -- the grpc -// pod still speaks nothing but cleartext HTTP/2, for instance. +// Each pod runs exactly one subcommand, so the wire behavior of any given pod is +// unchanged from when these were separate binaries -- the grpc pod still speaks +// nothing but cleartext HTTP/2, for instance. Given --tls-cert and --tls-key, +// http and websocket also serve HTTPS on :8443 beside their cleartext --listen, +// so one pod answers both schemes for the same name. // // The subcommand shape follows Kubernetes' test/images/agnhost: a single // extendable CLI whose modes are cobra subcommands, matching the rest of this diff --git a/internal/e2e/fixtures/testserver/tls.go b/internal/e2e/fixtures/testserver/tls.go index b370552dcc..121b0187c2 100644 --- a/internal/e2e/fixtures/testserver/tls.go +++ b/internal/e2e/fixtures/testserver/tls.go @@ -17,6 +17,8 @@ package main import ( "crypto/tls" "fmt" + "log" + "net" "net/http" ) @@ -42,14 +44,40 @@ func configureTLSServer(server *http.Server, certFile, keyFile string) error { return nil } -func serveHTTP(server *http.Server, certFile, keyFile string) error { - if certFile == "" && keyFile == "" { - return server.ListenAndServe() - } +// tlsListenAddress is where the http and websocket subcommands serve HTTPS +// when given a certificate, beside their cleartext --listen. It is a fixed port. +const tlsListenAddress = ":8443" +// serveHTTP serves server in cleartext on server.Addr and, when certFile and +// keyFile are set, over TLS on tlsListenAddress. +func serveHTTP(server *http.Server, certFile, keyFile string) error { if err := configureTLSServer(server, certFile, keyFile); err != nil { return err } + cleartext, err := net.Listen("tcp", server.Addr) + if err != nil { + return err + } + var encrypted net.Listener + if server.TLSConfig != nil { + if encrypted, err = net.Listen("tcp", tlsListenAddress); err != nil { + _ = cleartext.Close() + return err + } + log.Printf("testserver: serving HTTPS on %s", tlsListenAddress) + } + return serveListeners(server, cleartext, encrypted) +} - return server.ListenAndServeTLS("", "") +// serveListeners serves server in cleartext on cleartext and, unless encrypted +// is nil, over TLS on encrypted, returning when either stops. The caller binds +// both before this starts serving either, so a readiness probe that reaches one +// listener finds the other already listening. +func serveListeners(server *http.Server, cleartext, encrypted net.Listener) error { + errs := make(chan error, 2) + go func() { errs <- server.Serve(cleartext) }() + if encrypted != nil { + go func() { errs <- server.ServeTLS(encrypted, "", "") }() + } + return <-errs } diff --git a/internal/e2e/fixtures/testserver/websocket.go b/internal/e2e/fixtures/testserver/websocket.go index 011043c1ef..d17abcd7c3 100644 --- a/internal/e2e/fixtures/testserver/websocket.go +++ b/internal/e2e/fixtures/testserver/websocket.go @@ -87,9 +87,9 @@ func newWebsocketCmd() *cobra.Command { return serveHTTP(server, certFile, keyFile) }, } - cmd.Flags().StringVar(&listenAddress, "listen", ":80", "Address the websocket server listens on.") - cmd.Flags().StringVar(&certFile, "tls-cert", "", "PEM certificate file; enables HTTPS when paired with --tls-key.") - cmd.Flags().StringVar(&keyFile, "tls-key", "", "PEM private key file; enables HTTPS when paired with --tls-cert.") + cmd.Flags().StringVar(&listenAddress, "listen", ":80", "Address the websocket server serves cleartext on.") + cmd.Flags().StringVar(&certFile, "tls-cert", "", "PEM certificate file; with --tls-key, also serves HTTPS on :8443.") + cmd.Flags().StringVar(&keyFile, "tls-key", "", "PEM private key file; with --tls-cert, also serves HTTPS on :8443.") cmd.Flags().BoolVar(&echo, "echo", false, "Echo every WebSocket message with its original type and payload.") return cmd } diff --git a/internal/e2e/probe.go b/internal/e2e/probe.go index bab9ac5c68..f96707db4b 100644 --- a/internal/e2e/probe.go +++ b/internal/e2e/probe.go @@ -16,6 +16,7 @@ package e2e import ( "context" + "slices" "testing" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -35,7 +36,13 @@ var probeManifests = SubstrateFixtureManifests{ // ProbeOption adjusts what DeployProbe installs. type ProbeOption func(*probeConfig) -type probeConfig struct{ trustBundle bool } +type probeConfig struct{ trustBundles []string } + +// The allowlisted names atelet resolves trust bundles by. +const ( + egressTrustBundleName = "egress-mitm.ate.dev" + systemRootsBundleName = "system-roots.ate.dev" +) // WithTrustBundle projects the egress trust bundle into the probe's // system-info volume, ensuring the cluster-scoped bundle exists first. @@ -43,7 +50,16 @@ type probeConfig struct{ trustBundle bool } // Only suites that test the bundle should ask for it. An actor whose template // includes the bundle won't start if the bundle is missing, so a suite that // just needs a probe shouldn't depend on it. -func WithTrustBundle() ProbeOption { return func(c *probeConfig) { c.trustBundle = true } } +func WithTrustBundle() ProbeOption { + return func(c *probeConfig) { c.trustBundles = append(c.trustBundles, egressTrustBundleName) } +} + +// WithSystemRoots adds the WebPKI roots atelet ships to the probe's projected +// trust file, as the egress demo's template combines them with the egress +// trust bundle. +func WithSystemRoots() ProbeOption { + return func(c *probeConfig) { c.trustBundles = append(c.trustBundles, systemRootsBundleName) } +} // DeployProbe builds the probe fixture image and installs the fixture for the // sandbox class under test, removing it when the test ends. name @@ -59,12 +75,12 @@ func DeployProbe(t *testing.T, bucket, name string, opts ...ProbeOption) (string for _, opt := range opts { opt(&cfg) } - if cfg.trustBundle { + if slices.Contains(cfg.trustBundles, egressTrustBundleName) { // Every actor from this template — the fixture's golden boot included — // fails closed while the bundle is missing, so it has to exist first. EnsureEgressTrustBundle(t, context.Background(), GetClients()) } - atespace, templates := DeploySubstrateFixture(t, context.Background(), GetClients(), probeManifests, bucket, name, cfg.trustBundle) + atespace, templates := DeploySubstrateFixture(t, context.Background(), GetClients(), probeManifests, bucket, name, cfg.trustBundles) return atespace, templates[0] } diff --git a/internal/e2e/probe_test.go b/internal/e2e/probe_test.go index 3b325ff93e..012cec6deb 100644 --- a/internal/e2e/probe_test.go +++ b/internal/e2e/probe_test.go @@ -16,29 +16,36 @@ package e2e import ( "os" + "slices" "strings" "testing" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) -// TestProbeTemplate_TrustBundle checks that the probe template includes the -// trust bundle only when a suite asks for it with WithTrustBundle. +// TestProbeTemplate_TrustBundle checks that the probe template projects +// exactly the trust bundles a suite asks for, into one file. func TestProbeTemplate_TrustBundle(t *testing.T) { t.Setenv(sandboxClassEnv, "") for _, tc := range []struct { name string - cfg probeConfig - want bool + opts []ProbeOption + want []string }{ - {"default", probeConfig{}, false}, - {"WithTrustBundle", probeConfig{trustBundle: true}, true}, + {"default", nil, nil}, + {"WithTrustBundle", []ProbeOption{WithTrustBundle()}, []string{"egress-mitm.ate.dev"}}, + {"WithSystemRoots", []ProbeOption{WithSystemRoots()}, []string{"system-roots.ate.dev"}}, + {"both", []ProbeOption{WithTrustBundle(), WithSystemRoots()}, []string{"egress-mitm.ate.dev", "system-roots.ate.dev"}}, } { t.Run(tc.name, func(t *testing.T) { + var cfg probeConfig + for _, opt := range tc.opts { + opt(&cfg) + } // Strict decoding is what proves the fragment landed at the right // depth: misindented, it would fail the protojson decode or parse // as some other field. - inline, blocks := substrateTemplateSubstitutions("test-bucket", "render", tc.cfg.trustBundle) + inline, blocks := substrateTemplateSubstitutions("test-bucket", "render", cfg.trustBundles) rendered, err := os.ReadFile(renderManifest(t, probeManifests.Template, inline, blocks)) if err != nil { t.Fatalf("reading rendered manifest: %v", err) @@ -48,28 +55,34 @@ func TestProbeTemplate_TrustBundle(t *testing.T) { t.Fatalf("probe template manifest yields %d documents, want 1", len(templates)) } - var source *ateapipb.TrustBundleDataSource + var sources []*ateapipb.TrustBundleDataSource for _, vol := range templates[0].GetVolumes() { for _, ds := range vol.GetSystemInfo().GetDataSources() { if ds.GetTrustBundle() != nil { - source = ds.GetTrustBundle() + sources = append(sources, ds.GetTrustBundle()) } } } - if (source != nil) != tc.want { - t.Fatalf("trustBundle data source present = %v, want %v", source != nil, tc.want) - } - if source == nil { + if tc.want == nil { + if len(sources) != 0 { + t.Fatalf("trustBundle data sources = %v, want none", sources) + } return } - // The projected name must select the bundle atecontroller - // publishes, or actors fail closed on a name atelet rejects. - if names := source.GetNames(); len(names) != 1 || !strings.HasPrefix(EgressTrustBundleObjectName, names[0]+":") { - t.Errorf("trustBundle name = %q, want just the bundle backing %q", names, EgressTrustBundleObjectName) + if len(sources) != 1 { + t.Fatalf("got %d trustBundle data sources, want one projecting every name into one file", len(sources)) } - if source.GetPath() == "" { + if got := sources[0].GetNames(); !slices.Equal(got, tc.want) { + t.Errorf("trustBundle names = %q, want %q", got, tc.want) + } + if sources[0].GetPath() == "" { t.Error("trustBundle projection has no path") } }) } + // The egress name must select the bundle atecontroller publishes, or + // actors fail closed on a name atelet rejects. + if !strings.HasPrefix(EgressTrustBundleObjectName, egressTrustBundleName+":") { + t.Errorf("egress trust bundle name %q does not select %q", egressTrustBundleName, EgressTrustBundleObjectName) + } } diff --git a/internal/e2e/sandbox_test.go b/internal/e2e/sandbox_test.go index 1d63f07355..357c96b893 100644 --- a/internal/e2e/sandbox_test.go +++ b/internal/e2e/sandbox_test.go @@ -102,7 +102,7 @@ func renderPool(t *testing.T, relPath string) *v1alpha1.WorkerPool { // same misplaced-placeholder tripwire renderPool's strict mode does. func renderTemplates(t *testing.T, relPath string) []*ateapipb.ActorTemplate { t.Helper() - inline, blocks := substrateTemplateSubstitutions("test-bucket", "render", false) + inline, blocks := substrateTemplateSubstitutions("test-bucket", "render", nil) rendered, err := os.ReadFile(renderManifest(t, relPath, inline, blocks)) if err != nil { t.Fatalf("reading the rendered %s: %v", relPath, err) diff --git a/internal/e2e/serverpod.go b/internal/e2e/serverpod.go index b94850b67b..33c062ef06 100644 --- a/internal/e2e/serverpod.go +++ b/internal/e2e/serverpod.go @@ -19,6 +19,7 @@ import ( "fmt" "net" "os" + "path" "slices" "strconv" "strings" @@ -63,6 +64,11 @@ type ServerPod struct { // publish a port the container -- uid 65532, every capability dropped -- // cannot bind, such as 80; the Service maps Port down to it. TargetPort int + // TLSPort, when set, publishes the HTTPS listener testserver's http and + // websocket subcommands open on container port 8443 when given a + // certificate, beside the cleartext one on Port. The readiness probe then + // checks HTTPS, the listener that needs the certificate. + TLSPort int // Namespace deploys into an existing namespace instead of a fresh one, for // a suite that has to populate that namespace first: credentials the pod // mounts have to exist before it is scheduled, and DeployServerPod cannot @@ -75,8 +81,6 @@ type ServerPod struct { // HealthPath is the HTTP readiness path, defaulting to /healthz. Ignored // when GRPCProbe is set. HealthPath string - // HealthScheme selects HTTP or HTTPS for the readiness probe. Empty means HTTP. - HealthScheme corev1.URIScheme // Volumes and VolumeMounts carry whatever credentials the server needs. // Typed, rather than more YAML in the template, so the Secret names here // sit beside the code that creates them instead of drifting from it. @@ -84,16 +88,60 @@ type ServerPod struct { VolumeMounts []corev1.VolumeMount } +// serverTLSTargetPort is the container port testserver serves HTTPS on when +// given a certificate (its tlsListenAddress), which ServerPod.TLSPort publishes. +const serverTLSTargetPort = 8443 + +// serviceDNSBundle is where ServiceDNSTLS mounts the server's certificate +// chain and key, in one file. +const serviceDNSBundle = "/run/servicedns.podcert.ate.dev/credential-bundle.pem" + +// ServiceDNSTLS returns spec serving HTTPS as well as HTTP, with a certificate +// from the cluster's service-DNS signer that covers only Server.SvcName. +// Unless spec sets them, HTTP is on port 80 and HTTPS on 443. The server must +// be testserver's http or websocket subcommand. +func ServiceDNSTLS(spec ServerPod) ServerPod { + if spec.Port == 0 { + spec.Port, spec.TargetPort = 80, 8080 + } + if spec.TLSPort == 0 { + spec.TLSPort = 443 + } + spec.Args = append(slices.Clone(spec.Args), "--tls-cert="+serviceDNSBundle, "--tls-key="+serviceDNSBundle) + spec.Volumes = append(slices.Clone(spec.Volumes), corev1.Volume{ + Name: "servicedns", + VolumeSource: corev1.VolumeSource{Projected: &corev1.ProjectedVolumeSource{ + Sources: []corev1.VolumeProjection{{PodCertificate: &corev1.PodCertificateProjection{ + SignerName: "servicedns.podcert.ate.dev/identity", + KeyType: "ECDSAP256", + CredentialBundlePath: path.Base(serviceDNSBundle), + }}}, + }}, + }) + spec.VolumeMounts = append(slices.Clone(spec.VolumeMounts), corev1.VolumeMount{ + Name: "servicedns", + MountPath: path.Dir(serviceDNSBundle), + ReadOnly: true, + }) + return spec +} + // Server is a deployed ServerPod, as the address a caller dials it at. type Server struct { + // Name is the Service's name, which is also the Pod's. + Name string // Namespace is the namespace the server was deployed into, for a suite that // wants to port-forward to it or read its logs on failure. Namespace string - // ClusterIP is the Service's address. Deliberately not its DNS name: an IP - // keeps a caller inside a sandbox off that sandbox's resolver, and makes - // the authority in a gateway's access log exactly what the test deployed. + // ClusterIP is the Service's address. Address dials it rather than a DNS + // name: an IP keeps a caller inside a sandbox off that sandbox's resolver, + // and makes the authority in a gateway's access log exactly what the test + // deployed. A TLS client needs the name its certificate covers instead; see + // SvcName. ClusterIP string Port int + // TLSPort is ServerPod.TLSPort: where the Service publishes HTTPS, or 0. + TLSPort int } // Address is the host:port to dial the server at. @@ -101,6 +149,18 @@ func (s Server) Address() string { return net.JoinHostPort(s.ClusterIP, strconv.Itoa(s.Port)) } +// TLSAddress is the host:port to dial the server's HTTPS listener at. +func (s Server) TLSAddress() string { + return net.JoinHostPort(s.ClusterIP, strconv.Itoa(s.TLSPort)) +} + +// SvcName is the Service's ..svc, the one name a ServiceDNSTLS +// server's certificate covers. It resolves through the search path cluster DNS +// gives every pod, an actor's sandbox and the egress gateway included. +func (s Server) SvcName() string { + return s.Name + "." + s.Namespace + ".svc" +} + // DeployServerPod builds spec's image, applies the shared server manifest, waits // for readiness and returns the address to dial. // @@ -128,8 +188,12 @@ func DeployServerPod(t *testing.T, ctx context.Context, spec ServerPod) Server { t.Fatalf("service %s/%s has no ClusterIP to dial: %q", namespace, spec.Name, service.Spec.ClusterIP) } - server := Server{Namespace: namespace, ClusterIP: service.Spec.ClusterIP, Port: spec.Port} - t.Logf("server %s is serving at %s (namespace %s)", spec.Name, server.Address(), namespace) + server := Server{Name: spec.Name, Namespace: namespace, ClusterIP: service.Spec.ClusterIP, Port: spec.Port, TLSPort: spec.TLSPort} + if server.TLSPort != 0 { + t.Logf("server %s is serving at %s and HTTPS at %s (namespace %s)", spec.Name, server.Address(), server.TLSAddress(), namespace) + } else { + t.Logf("server %s is serving at %s (namespace %s)", spec.Name, server.Address(), namespace) + } return server } @@ -142,6 +206,9 @@ func renderServerPod(t *testing.T, spec ServerPod, namespace string) string { if targetPort == 0 { targetPort = spec.Port } + if spec.TLSPort != 0 && targetPort == serverTLSTargetPort { + t.Fatalf("server %s listens in cleartext on %d, which its HTTPS listener (TLSPort) needs; set TargetPort to another port", spec.Name, serverTLSTargetPort) + } targetPortStr := strconv.Itoa(targetPort) inline := map[string]string{ "${NAME}": spec.Name, @@ -153,6 +220,9 @@ func renderServerPod(t *testing.T, spec ServerPod, namespace string) string { blocks := map[string]string{ "${ARGS}": serverArgs(spec, targetPortStr), "${READINESS_PROBE}": serverReadinessProbe(spec, targetPortStr), + // Indented to the container's and the Service's `ports:` lists. + "${TLS_CONTAINER_PORT}": serverTLSPort(spec, " - name: serve-tls\n containerPort: %d", serverTLSTargetPort), + "${TLS_SERVICE_PORT}": serverTLSPort(spec, " - name: serve-tls\n port: %d\n targetPort: %d", spec.TLSPort, serverTLSTargetPort), // Indented to their parents: volumeMounts is a container field, volumes // a pod one. An empty list takes its whole line, key included. "${VOLUME_MOUNTS}": yamlListBlock(t, "volumeMounts", spec.VolumeMounts, 4), @@ -177,19 +247,30 @@ func serverArgs(spec ServerPod, targetPort string) string { // serverReadinessProbe renders the probe fragment for spec, indented to sit // under the template's `readinessProbe:` key. kubelet dials the container // directly, so the probe names the listen port rather than the published one. +// A server with TLSPort is probed over HTTPS, on the listener that depends on +// its certificate. func serverReadinessProbe(spec ServerPod, targetPort string) string { if spec.GRPCProbe { return " grpc:\n port: " + targetPort } - path := spec.HealthPath - if path == "" { - path = "/healthz" + healthPath := spec.HealthPath + if healthPath == "" { + healthPath = "/healthz" + } + scheme := corev1.URISchemeHTTP + if spec.TLSPort != 0 { + targetPort, scheme = strconv.Itoa(serverTLSTargetPort), corev1.URISchemeHTTPS } - scheme := spec.HealthScheme - if scheme == "" { - scheme = corev1.URISchemeHTTP + return fmt.Sprintf(" httpGet:\n path: %s\n port: %s\n scheme: %s", healthPath, targetPort, scheme) +} + +// serverTLSPort renders format with args for a server with TLSPort, and +// nothing, which drops the placeholder's line, for one without. +func serverTLSPort(spec ServerPod, format string, args ...any) string { + if spec.TLSPort == 0 { + return "" } - return fmt.Sprintf(" httpGet:\n path: %s\n port: %s\n scheme: %s", path, targetPort, scheme) + return fmt.Sprintf(format, args...) } func serverEndpointReady(endpointSlices []discoveryv1.EndpointSlice, pod *corev1.Pod, port int32) bool { @@ -253,7 +334,7 @@ func CreateServerService(t *testing.T, ctx context.Context, spec ServerPod) Serv if net.ParseIP(created.Spec.ClusterIP) == nil { t.Fatalf("origin Service has invalid IP %q", created.Spec.ClusterIP) } - return Server{Namespace: spec.Namespace, ClusterIP: created.Spec.ClusterIP, Port: spec.Port} + return Server{Name: spec.Name, Namespace: spec.Namespace, ClusterIP: created.Spec.ClusterIP, Port: spec.Port, TLSPort: spec.TLSPort} } t.Fatal("origin manifest has no Service") return Server{} diff --git a/internal/e2e/serverpod_test.go b/internal/e2e/serverpod_test.go index f081f48fe3..df6ad7db9b 100644 --- a/internal/e2e/serverpod_test.go +++ b/internal/e2e/serverpod_test.go @@ -193,6 +193,9 @@ func TestRenderServerPod(t *testing.T) { pod, service := renderServerPodDocs(t, tc.spec) container := pod.Spec.Containers[0] + if len(container.Ports) != 1 || len(service.Spec.Ports) != 1 { + t.Fatalf("rendered container ports %+v and Service ports %+v, want one each without TLSPort", container.Ports, service.Spec.Ports) + } if got, want := container.Args, []string{"http", "--listen=:" + strconv.Itoa(tc.wantListen)}; !slices.Equal(got, want) { t.Errorf("container args = %v, want %v", got, want) } @@ -288,11 +291,96 @@ func TestRenderServerPod_Volumes(t *testing.T) { } } -func TestRenderServerPod_HTTPSProbe(t *testing.T) { - pod, _ := renderServerPodDocs(t, ServerPod{Name: "tls", Port: 8443, HealthScheme: corev1.URISchemeHTTPS}) - probe := pod.Spec.Containers[0].ReadinessProbe.HTTPGet +// TestRenderServerPod_TLSPort covers a server answering both schemes: the +// cleartext listener on Port as usual, plus testserver's HTTPS listener on 8443 +// published on TLSPort and probed over HTTPS. +func TestRenderServerPod_TLSPort(t *testing.T) { + pod, service := renderServerPodDocs(t, ServerPod{Name: "tls", Port: 80, TargetPort: 8080, TLSPort: 443}) + container := pod.Spec.Containers[0] + + if len(container.Ports) != 2 || container.Ports[1].Name != "serve-tls" || container.Ports[1].ContainerPort != 8443 { + t.Errorf("container ports = %+v, want serve plus serve-tls on 8443", container.Ports) + } + if len(service.Spec.Ports) != 2 { + t.Fatalf("Service ports = %+v, want serve and serve-tls", service.Spec.Ports) + } + if got := service.Spec.Ports[0]; got.Port != 80 || got.TargetPort.IntValue() != 8080 { + t.Errorf("Service serve port = %+v, want 80 -> 8080", got) + } + if got := service.Spec.Ports[1]; got.Name != "serve-tls" || got.Port != 443 || got.TargetPort.IntValue() != 8443 { + t.Errorf("Service serve-tls port = %+v, want 443 -> 8443", got) + } + probe := container.ReadinessProbe.HTTPGet if probe.Scheme != corev1.URISchemeHTTPS || probe.Port.IntValue() != 8443 { - t.Fatalf("probe = %+v, want HTTPS on 8443", probe) + t.Errorf("probe = %+v, want HTTPS on 8443", probe) + } +} + +func TestServiceDNSTLS(t *testing.T) { + base := ServerPod{ + Name: "wss-origin", + ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", + Args: []string{"websocket", "--echo"}, + HealthPath: "/readyz", + Volumes: []corev1.Volume{{Name: "extra", VolumeSource: corev1.VolumeSource{EmptyDir: &corev1.EmptyDirVolumeSource{}}}}, + VolumeMounts: []corev1.VolumeMount{{Name: "extra", MountPath: "/run/extra"}}, + } + spec := ServiceDNSTLS(base) + + // Pure: the caller's spec, often a package-level var, is left as it was. + if len(base.Args) != 2 || len(base.Volumes) != 1 || len(base.VolumeMounts) != 1 || base.Port != 0 || base.TLSPort != 0 { + t.Errorf("ServiceDNSTLS modified its argument: %+v", base) + } + if spec.Port != 80 || spec.TargetPort != 8080 || spec.TLSPort != 443 || spec.HealthPath != "/readyz" { + t.Errorf("ports or health path = Port %d TargetPort %d TLSPort %d HealthPath %q, want 80, 8080, 443, /readyz", spec.Port, spec.TargetPort, spec.TLSPort, spec.HealthPath) + } + if explicit := ServiceDNSTLS(ServerPod{Port: 8080, TLSPort: 8443}); explicit.Port != 8080 || explicit.TargetPort != 0 || explicit.TLSPort != 8443 { + t.Errorf("ServiceDNSTLS replaced ports the caller set: %+v", explicit) + } + + pod, _ := renderServerPodDocs(t, spec) + container := pod.Spec.Containers[0] + wantArgs := []string{"websocket", "--echo", "--tls-cert=" + serviceDNSBundle, "--tls-key=" + serviceDNSBundle, "--listen=:8080"} + if !slices.Equal(container.Args, wantArgs) { + t.Errorf("args = %q, want %q", container.Args, wantArgs) + } + probe := container.ReadinessProbe.HTTPGet + if probe.Scheme != corev1.URISchemeHTTPS || probe.Path != "/readyz" || probe.Port.IntValue() != 8443 { + t.Errorf("probe = %+v, want HTTPS /readyz on 8443", probe) + } + + if len(pod.Spec.Volumes) != 2 || len(container.VolumeMounts) != 2 { + t.Fatalf("rendered volumes %+v and mounts %+v, want the caller's plus one each", pod.Spec.Volumes, container.VolumeMounts) + } + projected := pod.Spec.Volumes[1].Projected + if projected == nil || len(projected.Sources) != 1 || projected.Sources[0].PodCertificate == nil { + t.Fatalf("added volume = %+v, want a podCertificate projection", pod.Spec.Volumes[1]) + } + cert := projected.Sources[0].PodCertificate + if cert.SignerName != "servicedns.podcert.ate.dev/identity" || cert.KeyType != "ECDSAP256" || cert.CredentialBundlePath != "credential-bundle.pem" { + t.Errorf("podCertificate = %+v, want an ECDSAP256 service-DNS credential bundle", cert) + } + mount := container.VolumeMounts[1] + if mount.Name != pod.Spec.Volumes[1].Name || mount.MountPath+"/"+cert.CredentialBundlePath != serviceDNSBundle || !mount.ReadOnly { + t.Errorf("added mount = %+v, want the podCertificate volume read-only at %s", mount, serviceDNSBundle) + } +} + +func TestServerAddresses(t *testing.T) { + for _, tc := range []struct{ ip, wantAddress, wantTLSAddress string }{ + {"10.96.0.7", "10.96.0.7:80", "10.96.0.7:443"}, + {"fd00:10:96::7", "[fd00:10:96::7]:80", "[fd00:10:96::7]:443"}, + } { + server := Server{Name: "mitm-origin", Namespace: "abcd-123", ClusterIP: tc.ip, Port: 80, TLSPort: 443} + if got, want := server.SvcName(), "mitm-origin.abcd-123.svc"; got != want { + t.Errorf("SvcName() with ClusterIP %s = %q, want %q", tc.ip, got, want) + } + if got := server.Address(); got != tc.wantAddress { + t.Errorf("Address() = %q, want %q", got, tc.wantAddress) + } + if got := server.TLSAddress(); got != tc.wantTLSAddress { + t.Errorf("TLSAddress() = %q, want %q", got, tc.wantTLSAddress) + } } } diff --git a/internal/e2e/statusz.go b/internal/e2e/statusz.go index a837a82874..cae64166fd 100644 --- a/internal/e2e/statusz.go +++ b/internal/e2e/statusz.go @@ -26,8 +26,9 @@ import ( "k8s.io/client-go/kubernetes" ) -// routerStatusPort is the atenet-router Service port for the router's -// --status-port (the "status" container port). +// routerStatusPort is the router's --status-port. The router binds it to +// loopback and the Service does not publish it, so the client forwards to a +// router pod directly. const routerStatusPort = 4040 // StatuszClient reads the atenet router's /statusz page over a @@ -51,7 +52,7 @@ func NewStatuszClient(ctx context.Context) (*StatuszClient, error) { return nil, fmt.Errorf("creating k8s client: %w", err) } - localPort, stop, err := portforward.ServicePortForward(ctx, config, clientset, SystemNamespace(), ResourceName("atenet-router"), routerStatusPort) + localPort, stop, err := portforward.ServicePodPortForward(ctx, config, clientset, SystemNamespace(), ResourceName("atenet-router"), routerStatusPort) if err != nil { return nil, err } diff --git a/internal/e2e/suites/capabilities/capabilities_test.go b/internal/e2e/suites/capabilities/capabilities_test.go index 631462b385..39339350af 100644 --- a/internal/e2e/suites/capabilities/capabilities_test.go +++ b/internal/e2e/suites/capabilities/capabilities_test.go @@ -138,7 +138,7 @@ func deployFixture(t *testing.T, ctx context.Context, clients *e2e.Clients, buck atespace, _ := e2e.DeploySubstrateFixture(t, ctx, clients, e2e.SubstrateFixtureManifests{ Pool: "internal/e2e/fixtures/capabilities/capabilities.yaml.tmpl", Template: "internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl", - }, bucket, "capabilities", false) + }, bucket, "capabilities", nil) return atespace } diff --git a/internal/e2e/suites/demo/demo_test.go b/internal/e2e/suites/demo/demo_test.go index 1b8c52fcd7..b8c1fc6157 100644 --- a/internal/e2e/suites/demo/demo_test.go +++ b/internal/e2e/suites/demo/demo_test.go @@ -259,8 +259,8 @@ func TestDurableDirLifecycle(t *testing.T) { }, }, { - // Suspend from PAUSED uploads the MEMORY local snapshot. - name: "preferredFidelity:MEMORY, suspend from PAUSED", + // Suspend from paused uploads the MEMORY local snapshot. + name: "preferredFidelity:MEMORY, suspend from paused", tc: actorLifecycleTestCase{ fidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, wantMemoryAfterPause: 2, @@ -274,8 +274,8 @@ func TestDurableDirLifecycle(t *testing.T) { }, }, { - // Suspend from PAUSED uploads the VOLUMES local snapshot. - name: "preferredFidelity:VOLUMES, suspend from PAUSED", + // Suspend from paused uploads the VOLUMES local snapshot. + name: "preferredFidelity:VOLUMES, suspend from paused", tc: actorLifecycleTestCase{ fidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, wantMemoryAfterPause: 1, @@ -289,8 +289,8 @@ func TestDurableDirLifecycle(t *testing.T) { }, }, { - // Suspend from PAUSED uploads the ROOTFS local snapshot. - name: "preferredFidelity:ROOTFS, suspend from PAUSED", + // Suspend from paused uploads the ROOTFS local snapshot. + name: "preferredFidelity:ROOTFS, suspend from paused", tc: actorLifecycleTestCase{ fidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, wantMemoryAfterPause: 1, @@ -590,7 +590,7 @@ type actorLifecycleTestCase struct { microVMOnly bool // suspendWhilePaused pauses the actor again before the suspend step, so - // the suspend runs from PAUSED: it must upload the node-local pause + // the suspend runs from paused: it must upload the node-local pause // snapshot instead of checkpointing a running workload. The counter // expectations are unchanged — both origins capture the state after the // second call. @@ -660,7 +660,7 @@ func runActorLifecycleTestCase(t *testing.T, prefix string, createTemplate func( }); err != nil { t.Fatalf("failed to pause Actor: %v", err) } - waitForActorState(ctx, t, clients, actorID, ateapipb.ActorState_ACTOR_STATE_PAUSED) + waitForActorPaused(ctx, t, clients, actorID) // Resuming the actor t.Logf("Resuming Actor %q again...", actorID) @@ -693,7 +693,7 @@ func runActorLifecycleTestCase(t *testing.T, prefix string, createTemplate func( }); err != nil { t.Fatalf("failed to pause Actor before suspend: %v", err) } - waitForActorState(ctx, t, clients, actorID, ateapipb.ActorState_ACTOR_STATE_PAUSED) + waitForActorPaused(ctx, t, clients, actorID) } t.Logf("Suspending Actor %q...", actorID) if _, err := clients.SubstrateAPI.SuspendActor(ctx, &ateapipb.SuspendActorRequest{ @@ -905,7 +905,7 @@ func pauseActor(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj * }); err != nil { t.Fatalf("failed to pause Actor: %v", err) } - waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_PAUSED) + waitForActorPaused(ctx, t, clients, actorName) // Resuming the actor again t.Logf("Resuming Actor %q again...", actorName) @@ -1106,26 +1106,26 @@ func revertActor(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj } validateCounterResponse(t, resp, "after revert from RUNNING", 2, 2) - // Revert from PAUSED. Resume prefers a local snapshot over the external + // Revert from paused. Resume prefers a local snapshot over the external // one, so clearing LocalSnapshot is what prevents this pause's count=2 // checkpoint from restoring on the next resume. if _, err := clients.SubstrateAPI.PauseActor(ctx, &ateapipb.PauseActorRequest{Actor: actorRef}); err != nil { t.Fatalf("failed to pause Actor: %v", err) } - waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_PAUSED) + waitForActorPaused(ctx, t, clients, actorName) reverted, err = clients.SubstrateAPI.RevertActor(ctx, &ateapipb.RevertActorRequest{Actor: actorRef}) if err != nil { - t.Fatalf("RevertActor from PAUSED: %v", err) + t.Fatalf("RevertActor from paused: %v", err) } if got := reverted.GetActor().GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { - t.Fatalf("state after revert from PAUSED = %v, want SUSPENDED", got) + t.Fatalf("state after revert from paused = %v, want SUSPENDED", got) } if hasLocalSnapshot(reverted.GetActor().GetStatus()) { - t.Fatal("local snapshot pointer survived revert from PAUSED") + t.Fatal("local snapshot pointer survived revert from paused") } if got := durableSnapshotURI(reverted.GetActor().GetStatus()); got != snapshotURI { - t.Fatalf("snapshot URI after revert from PAUSED = %q, want %q", got, snapshotURI) + t.Fatalf("snapshot URI after revert from paused = %q, want %q", got, snapshotURI) } // SUSPENDED is rejected with FailedPrecondition rather than no-oping. @@ -1136,15 +1136,15 @@ func revertActor(ctx context.Context, t *testing.T, clients *e2e.Clients, nsObj // Final resume: if the pause checkpoint had been restored, the counter // would advance from 2 to 3. Expecting 2 proves the external snapshot won. if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{Actor: actorRef}); err != nil { - t.Fatalf("failed to resume Actor after PAUSED revert: %v", err) + t.Fatalf("failed to resume Actor after paused revert: %v", err) } waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_RUNNING) resp, err = callActor(t, resources.ActorRef{Atespace: demoAtespace, Name: actorName}) if err != nil { - t.Fatalf("failed to call actor after PAUSED revert: %v", err) + t.Fatalf("failed to call actor after paused revert: %v", err) } - validateCounterResponse(t, resp, "after revert from PAUSED", 2, 2) + validateCounterResponse(t, resp, "after revert from paused", 2, 2) if _, err := clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: actorRef, AnyState: true}); err != nil { t.Fatalf("failed to delete Actor: %v", err) @@ -1271,7 +1271,7 @@ func deletePausedActorAnyState(ctx context.Context, t *testing.T, clients *e2e.C }); err != nil { t.Fatalf("failed to pause Actor: %v", err) } - waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_PAUSED) + waitForActorPaused(ctx, t, clients, actorName) // 4. Delete the paused actor with any-state t.Logf("Deleting paused Actor %q with any-state...", actorName) @@ -1410,6 +1410,26 @@ func waitForActorState(ctx context.Context, t *testing.T, clients *e2e.Clients, waitForActorStateWithTimeout(ctx, t, clients, actorName, expectedState, 60*time.Second) } +// waitForActorPaused waits for the shape a pause leaves behind: SUSPENDED, +// holding a node-local snapshot and pinned to its node. A plain suspend +// leaves neither. +func waitForActorPaused(ctx context.Context, t *testing.T, clients *e2e.Clients, actorName string) { + t.Helper() + waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_SUSPENDED) + resp, err := clients.SubstrateAPI.GetActor(ctx, &ateapipb.GetActorRequest{ + Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName}, + }) + if err != nil { + t.Fatalf("GetActor %q after pause: %v", actorName, err) + } + if !hasLocalSnapshot(resp.GetStatus()) { + t.Fatalf("Actor %q is SUSPENDED after pause but holds no local snapshot: %v", actorName, resp.GetStatus()) + } + if resp.GetStatus().GetAssignedNode() == "" { + t.Fatalf("Actor %q is paused but not pinned to a node: %v", actorName, resp.GetStatus()) + } +} + func waitForActorStateWithTimeout(ctx context.Context, t *testing.T, clients *e2e.Clients, actorName string, expectedState ateapipb.ActorState, timeout time.Duration) { t.Helper() t.Logf("Waiting for Actor %q to be %v...", actorName, expectedState) diff --git a/internal/e2e/suites/demo/updatetemplate_test.go b/internal/e2e/suites/demo/updatetemplate_test.go index bd5978bfd0..e7d8e954bf 100644 --- a/internal/e2e/suites/demo/updatetemplate_test.go +++ b/internal/e2e/suites/demo/updatetemplate_test.go @@ -200,7 +200,7 @@ func runUpdateTemplateTestCase(t *testing.T, fidelity ateapipb.SnapshotFidelity) } // Pause under template B while status.snapshots still holds the last - // committed snapshot from template A. Resuming from PAUSED restores the + // committed snapshot from template A. Resuming from pause restores the // local checkpoint (which was captured under template B, since templates // can only be updated while SUSPENDED) and preserves the in-memory counter // when preferredFidelity is MEMORY. @@ -210,7 +210,7 @@ func runUpdateTemplateTestCase(t *testing.T, fidelity ateapipb.SnapshotFidelity) }); err != nil { t.Fatalf("failed to pause Actor under template B: %v", err) } - waitForActorState(ctx, t, clients, actorID, ateapipb.ActorState_ACTOR_STATE_PAUSED) + waitForActorPaused(ctx, t, clients, actorID) paused, err := clients.SubstrateAPI.GetActor(ctx, &ateapipb.GetActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID}, diff --git a/internal/e2e/suites/egresscredinject/egresscredinject_test.go b/internal/e2e/suites/egresscredinject/egresscredinject_test.go index 919974f40f..16f8ce5bd4 100644 --- a/internal/e2e/suites/egresscredinject/egresscredinject_test.go +++ b/internal/e2e/suites/egresscredinject/egresscredinject_test.go @@ -26,7 +26,6 @@ import ( "io" "net/http" "net/url" - "path" "slices" "strings" "testing" @@ -37,49 +36,21 @@ import ( "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" - corev1 "k8s.io/api/core/v1" ) const probeTemplate = "probe" var probeNamespace string -// servingBundle is the echo-https pod's certificate chain and key, in one file. -const servingBundle = "/run/servicedns.podcert.ate.dev/credential-bundle.pem" - -// The origins the injection fetches reach, one per scheme. -var ( - // echoHTTPSServer serves the certificate kubelet requests for it from the - // cluster's service-DNS signer, which names the Services selecting the pod - // as ..svc. The egress gateway trusts that signer once - // hack/setup-e2e-egress-tls-kind.sh has run. - echoHTTPSServer = e2e.ServerPod{ - Name: "echo-https", - ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", - Args: []string{"http", "--tls-cert=" + servingBundle, "--tls-key=" + servingBundle}, - Port: 443, - TargetPort: 8443, - HealthScheme: corev1.URISchemeHTTPS, - Volumes: []corev1.Volume{{ - Name: "servicedns", - VolumeSource: corev1.VolumeSource{Projected: &corev1.ProjectedVolumeSource{ - Sources: []corev1.VolumeProjection{{PodCertificate: &corev1.PodCertificateProjection{ - SignerName: "servicedns.podcert.ate.dev/identity", - KeyType: "ECDSAP256", - CredentialBundlePath: path.Base(servingBundle), - }}}, - }}, - }}, - VolumeMounts: []corev1.VolumeMount{{Name: "servicedns", MountPath: path.Dir(servingBundle), ReadOnly: true}}, - } - echoHTTPServer = e2e.ServerPod{ - Name: "echo-http", - ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", - Args: []string{"http"}, - Port: 80, - TargetPort: 8080, - } -) +// echoServer is the origin the injection fetches reach, over HTTPS and +// cleartext HTTP alike: testserver's /echoheaders returns the headers it +// received. Its certificate comes from the cluster's service-DNS signer, which +// the egress gateway trusts once hack/setup-e2e-egress-tls-kind.sh has run. +var echoServer = e2e.ServiceDNSTLS(e2e.ServerPod{ + Name: "echo", + ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", + Args: []string{"http"}, +}) // The hosts of the fail-closed cases. const ( @@ -115,7 +86,7 @@ var withPlaceholder = []string{"header=" + url.QueryEscape("Authorization:"+plac // (default-deny), 403 or 500 for a URI naming a provider this gateway // does not serve — over HTTPS and plain HTTP alike. // -// The echo origins run in the cluster, so the HTTPS one needs the gateway to +// The echo origin runs in the cluster, so its HTTPS side needs the gateway to // trust the service-DNS signer its certificate comes from, which // hack/setup-e2e-egress-tls-kind.sh adds after the install. The install must // be made with the bundled provider, which deploys the k8s-credential-provider @@ -132,20 +103,16 @@ func TestActorEgressCredentialInjection(t *testing.T) { ctx := context.Background() clients := e2e.GetClients() - echoHTTPS := e2e.DeployServerPod(t, ctx, echoHTTPSServer) - httpSpec := echoHTTPServer - httpSpec.Namespace = echoHTTPS.Namespace - echoHTTP := e2e.DeployServerPod(t, ctx, httpSpec) - // The names the service-DNS signer certifies. - httpsHost := echoHTTPSServer.Name + "." + echoHTTPS.Namespace + ".svc" - httpHost := echoHTTPServer.Name + "." + echoHTTP.Namespace + ".svc" + echo := e2e.DeployServerPod(t, ctx, echoServer) + // The name the service-DNS signer certifies. + echoHost := echo.SvcName() e2e.ConfigureCredentialProvider(t) probeNamespace, _ = e2e.DeployProbe(t, env["BUCKET_NAME"], "egresscredinject", e2e.WithTrustBundle()) const id = "probe-credinject" - createAndResumeActor(t, ctx, clients, id, httpsHost, httpHost) + createAndResumeActor(t, ctx, clients, id, echoHost) waitForActorState(t, ctx, clients, id, ateapipb.ActorState_ACTOR_STATE_RUNNING) rc, err := e2e.NewRouterClient(ctx) @@ -155,18 +122,18 @@ func TestActorEgressCredentialInjection(t *testing.T) { defer rc.Close() // HTTPS is injected on the gateway's MITM leg and plain HTTP on its - // cleartext leg, each by a rule with the same effect. + // cleartext leg, each by a rule with the same effect. dial is where the + // echo origin serves the scheme. legs := []struct { scheme string - host string - echo e2e.Server + dial string }{ - {"https", httpsHost, echoHTTPS}, - {"http", httpHost, echoHTTP}, + {"https", echo.TLSAddress()}, + {"http", echo.Address()}, } for _, leg := range legs { t.Run(leg.scheme, func(t *testing.T) { - origin := leg.scheme + "://" + leg.host + "/echoheaders" + origin := leg.scheme + "://" + echoHost + "/echoheaders" // The upstream must receive the policy's credential instead of the // actor's placeholder. @@ -210,7 +177,7 @@ func TestActorEgressCredentialInjection(t *testing.T) { for _, leg := range legs { t.Run(tt.name+"/"+leg.scheme, func(t *testing.T) { origin := leg.scheme + "://" + tt.host + "/" - withDial := slices.Concat(withPlaceholder, []string{"dial=" + url.QueryEscape(leg.echo.Address())}) + withDial := slices.Concat(withPlaceholder, []string{"dial=" + url.QueryEscape(leg.dial)}) got := probeFetch(t, ctx, rc, id, origin, withDial) if got.Error != "" { t.Fatalf("fetch of %s failed at the transport (%s), want an HTTP denial %v from the gateway", origin, got.Error, tt.wantStatuses) @@ -327,10 +294,9 @@ func probeFetch(t *testing.T, ctx context.Context, rc *e2e.RouterClient, id, ori } // createAndResumeActor first deletes any actor left by an earlier run, since -// actor records outlive the fixture namespace. echoHTTPSHost and echoHTTPHost -// are the echo origins' names, whose rules inject a credential the provider -// resolves. -func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Clients, id, echoHTTPSHost, echoHTTPHost string) { +// actor records outlive the fixture namespace. echoHost is the echo origin's +// name, whose rules inject a credential the provider resolves. +func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Clients, id, echoHost string) { t.Helper() ref := &ateapipb.ObjectRef{Atespace: probeNamespace, Name: id} // NotFound is the normal case on a fresh run. Other errors don't stop the @@ -356,10 +322,10 @@ func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Client } }) // Exercise each credential outcome on both HTTP and HTTPS: the echo - // origins' credential resolves, and each fail-closed host's cannot. + // origin's credential resolves, and each fail-closed host's cannot. rules := []*ateapipb.EgressRule{ - e2e.EgressInjectHeader("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHTTPSHost), - e2e.EgressInjectHeaderHTTP("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHTTPHost), + e2e.EgressInjectHeader("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost), + e2e.EgressInjectHeaderHTTP("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost), } for _, credential := range []struct { host string diff --git a/internal/e2e/suites/egressmitm/egressmitm_test.go b/internal/e2e/suites/egressmitm/egressmitm_test.go index 55daebaf39..315d93b0db 100644 --- a/internal/e2e/suites/egressmitm/egressmitm_test.go +++ b/internal/e2e/suites/egressmitm/egressmitm_test.go @@ -21,7 +21,9 @@ package egressmitm import ( "context" + "crypto/x509" "encoding/json" + "encoding/pem" "io" "net/http" "net/url" @@ -40,22 +42,26 @@ const probeTemplate = "probe" var probeNamespace string // TestActorEgressMITMTrust proves an actor can do TLS through the MITM -// egress gateway using only the projected trust bundle: +// egress gateway using only the projected trust bundle. The origins run in the +// cluster and serve certificates from the service-DNS signer, which the +// gateway trusts once hack/setup-e2e-egress-tls-kind.sh has run and the +// probe's bundle does not: // // - positive: /fetch with roots=bundle succeeds — the gateway's per-SNI // minted leaf (signed from the egress-mitm-ca-pool) validates against // the anchors atelet projected from the reconciler-published bundle. -// The bundle holds no public CAs, so a pass certifies interception is -// on. -// - negative: /fetch with roots=system fails certificate verification — -// the minted leaf chains to no public CA, proving the traffic really is -// intercepted rather than relayed (were it relayed this fetch would -// succeed, and the positive case would be meaningless). +// The bundle holds no service-DNS CA, so a pass certifies interception +// is on. +// - negative: /fetch trusting only the service-DNS CA fails certificate +// verification — the actor saw the minted leaf, not the origin's own +// certificate (were the traffic relayed, this fetch would succeed, and +// the positive case would be meaningless). // // CI runs this in both lanes (see pr-workflow.yaml) — once per sandbox class, // since trust delivery differs per class (gVisor RO bind vs the micro-VM // unified virtio-fs share). Locally: // +// hack/setup-e2e-egress-tls-kind.sh // hack/run-e2e-kind.sh ./internal/e2e/suites/egressmitm -v -args --no-color // E2E_SANDBOX_CLASS=microvm hack/run-e2e-kind.sh ./internal/e2e/suites/egressmitm -v -args --no-color // @@ -79,10 +85,18 @@ func TestActorEgressMITMTrust(t *testing.T) { // missing.) e2e.EnsureEgressTrustBundle(t, ctx, clients) - probeNamespace, _ = e2e.DeployProbe(t, env["BUCKET_NAME"], "egressmitm", e2e.WithTrustBundle()) + // One pod answers http and https for the intercepted name; passthrough is + // decided by name, so it needs an origin of its own. + mitm := e2e.DeployServerPod(t, ctx, originSpec("mitm-origin")) + passthroughSpec := originSpec("passthrough-origin") + passthroughSpec.Namespace = mitm.Namespace + passthrough := e2e.DeployServerPod(t, ctx, passthroughSpec) + originRoots := "ca=" + url.QueryEscape(e2e.ServiceDNSTrustBundle(t, ctx, clients)) + + probeNamespace, _ = e2e.DeployProbe(t, env["BUCKET_NAME"], "egressmitm", e2e.WithTrustBundle(), e2e.WithSystemRoots()) const id = "probe-mitm" - createAndResumeActor(t, ctx, clients, id) + createAndResumeActor(t, ctx, clients, id, mitm, passthrough) waitForActorState(t, ctx, clients, id, ateapipb.ActorState_ACTOR_STATE_RUNNING) rc, err := e2e.NewRouterClient(ctx) @@ -91,44 +105,29 @@ func TestActorEgressMITMTrust(t *testing.T) { } defer rc.Close() - const origin = "https://" + egressOriginHost + "/" + origin := "https://" + mitm.SvcName() + "/healthz" - // The gateway signs with the pool mounted into its pod, and kubelet - // propagates Secret contents into that mount on its own schedule (up to - // ~1 minute). In CI the pool predates the gateway pod, but a LOCAL rerun - // can recreate the pool moments before this fetch (a prior run's cleanup - // deleted it), leaving the gateway briefly signing with the old CA — so - // certificate failures retry for one propagation window before counting. - deadline := time.Now().Add(2 * time.Minute) - var pos fetchResponse - for { - pos = probeFetch(t, ctx, rc, id, origin, "bundle") - isCertErr := strings.Contains(pos.Error, "certificate") || strings.Contains(pos.Error, "x509") - if pos.Error == "" || !isCertErr || time.Now().After(deadline) { - break - } - time.Sleep(5 * time.Second) - } + pos := fetchUntilReached(t, ctx, rc, id, origin, "bundle") if pos.Error != "" { t.Fatalf("TLS through the MITM egress gateway with the projected trust bundle failed: %s — the projected anchors did not validate the gateway's minted leaf (or interception/minting is broken)", pos.Error) } if pos.Status != "200" { - t.Fatalf("fetch %s via projected bundle: status %s, want 200", origin, pos.Status) + t.Fatalf("fetch %s via projected bundle: status %s, want 200 (a 5xx is the gateway failing to reach or trust the origin; has hack/setup-e2e-egress-tls-kind.sh run?) body %q", origin, pos.Status, pos.Body) } - neg := probeFetch(t, ctx, rc, id, origin, "system") + neg := probeFetch(t, ctx, rc, id, origin, "pem", originRoots) if neg.Error == "" { - t.Errorf("fetch with system roots unexpectedly succeeded (status %s): the minted leaf should chain to no public CA — is the egress gateway actually terminating TLS?", neg.Status) + t.Errorf("fetch trusting only the origin's CA unexpectedly succeeded (status %s): the actor saw the origin's own certificate — is the egress gateway actually terminating TLS?", neg.Status) } else if !strings.Contains(neg.Error, "certificate") && !strings.Contains(neg.Error, "x509") { - t.Errorf("fetch with system roots failed, but not with a certificate-verification error: %s", neg.Error) + t.Errorf("fetch trusting only the origin's CA failed, but not with a certificate-verification error: %s", neg.Error) } // A port in the Host is never dialed: no rule checked it. The gateway // dials the Host's name on the port the actor dialed, and nothing answers // on port 1. for _, tc := range []struct{ origin, dial string }{ - {origin: "https://" + egressOriginHost + ":1/", dial: egressOriginHost + ":443"}, - {origin: "http://" + egressOriginHost + ":1/", dial: egressOriginHost + ":80"}, + {origin: "https://" + mitm.SvcName() + ":1/healthz", dial: mitm.TLSAddress()}, + {origin: "http://" + mitm.SvcName() + ":1/healthz", dial: mitm.Address()}, } { res := probeFetch(t, ctx, rc, id, tc.origin, "bundle", "dial="+url.QueryEscape(tc.dial)) switch { @@ -139,20 +138,20 @@ func TestActorEgressMITMTrust(t *testing.T) { } } - // The gateway relays passthrough TLS unread, so the origin's own certificate - // must validate with the system roots. That is also the proof it was not - // intercepted. - const passthroughOrigin = "https://" + egressOriginPassthroughHost + "/" - pt := probeFetch(t, ctx, rc, id, passthroughOrigin, "system") + // The gateway relays passthrough TLS unread, so the origin's own + // certificate must validate against its CA alone. That is also the proof it + // was not intercepted. + passthroughOrigin := "https://" + passthrough.SvcName() + "/healthz" + pt := fetchUntilReached(t, ctx, rc, id, passthroughOrigin, "pem", originRoots) if pt.Error != "" { - t.Fatalf("TLS passthrough to %s failed: %s — the origin's certificate did not validate against the system roots, so the connection was intercepted, misrouted, or closed", passthroughOrigin, pt.Error) + t.Fatalf("TLS passthrough to %s failed: %s — the origin's certificate did not validate against its CA, so the connection was intercepted, misrouted, or closed", passthroughOrigin, pt.Error) } if pt.Status != "200" { - t.Fatalf("passthrough fetch %s: status %s, want 200", passthroughOrigin, pt.Status) + t.Fatalf("passthrough fetch %s: status %s, want 200; body %q", passthroughOrigin, pt.Status, pt.Body) } // The gateway dials the name the SNI claims. - bySNI := probeFetch(t, ctx, rc, id, passthroughOrigin, "system", "dial="+url.QueryEscape(unreachableAddress)) + bySNI := fetchUntilReached(t, ctx, rc, id, passthroughOrigin, "pem", originRoots, "dial="+url.QueryEscape(unreachableAddress)) switch { case bySNI.Error != "": t.Errorf("TLS passthrough to %s dialed at %s failed: %s — the gateway dialed the actor's address instead of resolving the SNI", passthroughOrigin, unreachableAddress, bySNI.Error) @@ -163,12 +162,17 @@ func TestActorEgressMITMTrust(t *testing.T) { // Envoy closes denied connections at the ClientHello; agentgateway // terminates TLS to return 403. Trust the gateway's CA so a certificate // failure cannot masquerade as either policy-denial response. - for _, tc := range []struct{ name, origin string }{ - {"passthrough rule excludes port 443", "https://" + egressOriginPassthroughWrongPortHost + "/"}, - {"host outside the policy", "https://example.org/"}, + // + // Each name is an origin's fully qualified alias: rules match names + // exactly, so only the wrong-port rule lists the first and none the + // second. A gateway that relayed either would hand the actor a + // certificate the bundle rejects, and the dial keeps the actor off DNS. + for _, tc := range []struct{ name, origin, dial string }{ + {"passthrough rule excludes port 443", "https://" + passthrough.SvcName() + ".cluster.local/healthz", passthrough.TLSAddress()}, + {"host outside the policy", "https://" + mitm.SvcName() + ".cluster.local/healthz", mitm.TLSAddress()}, } { t.Run(tc.name, func(t *testing.T) { - denied := probeFetch(t, ctx, rc, id, tc.origin, "bundle") + denied := probeFetch(t, ctx, rc, id, tc.origin, "bundle", "dial="+url.QueryEscape(tc.dial)) if os.Getenv(e2e.AtenetDataplaneEnv) == "agentgateway" { if denied.Error != "" || denied.Status != "403" { t.Errorf("fetch of %s: error %q, status %s, want HTTP 403", tc.origin, denied.Error, denied.Status) @@ -185,23 +189,75 @@ func TestActorEgressMITMTrust(t *testing.T) { } }) } + + // With every origin in the cluster, no fetch here verifies a public + // certificate, so check that atelet merged its WebPKI roots into the + // projected bundle, as the egress demo's template asks it to. + t.Run("projected bundle holds the system roots", func(t *testing.T) { + if !holdsCertificate(t, probeTrust(t, ctx, rc, id), systemRoot) { + t.Errorf("the probe's projected trust bundle has no certificate for %q; atelet did not merge %s into it", systemRoot, "system-roots.ate.dev") + } + }) +} + +// originSpec is an origin answering /healthz over https and http alike. +func originSpec(name string) e2e.ServerPod { + return e2e.ServiceDNSTLS(e2e.ServerPod{ + Name: name, + ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", + Args: []string{"http"}, + }) } -// egressOriginHost is the one host the probe actor's EgressPolicy allows. const ( - egressOriginHost = "example.com" - egressOriginPassthroughHost = "example.edu" - // egressOriginPassthroughWrongPortHost is allowed by passthrough on 8443 - // only, while the probe dials 443. - egressOriginPassthroughWrongPortHost = "example.net" // unreachableAddress is in the reserved 240/4 block: a gateway that dialed // it instead of the resolved SNI would reach nothing. unreachableAddress = "240.0.0.1:443" + // systemRoot is the common name of a WebPKI root atelet's image ships. + systemRoot = "ISRG Root X1" ) type fetchResponse struct { Status string `json:"status"` Error string `json:"error"` + Body string `json:"body"` +} + +// reachWindow bounds the retries of a fetch that must return 200. +const reachWindow = 2 * time.Minute + +// fetchUntilReached is probeFetch for a fetch that must return 200. It retries +// transient failures for up to reachWindow, then returns the last result for +// the caller to fail on. A denial or any other final status returns at once. +// The failures it rides out: +// +// - a certificate error: the gateway signs with the pool mounted into its +// pod, and kubelet propagates Secret contents into that mount on its own +// schedule (up to ~1 minute). In CI the pool predates the gateway pod, +// but a LOCAL rerun can recreate the pool moments before this fetch (a +// prior run's cleanup deleted it), leaving the gateway briefly signing +// with the old CA; +// - any other transport error, and a 5xx: the actor's route, or the +// gateway's path to the origin, is still propagating. +// +// It never skips. Reaching the origins is part of what the lane proves, and a +// skip would report that as a pass (docs/dev/best-practices/ci-fail-closed.md). +func fetchUntilReached(t *testing.T, ctx context.Context, rc *e2e.RouterClient, id, origin, roots string, extraParams ...string) fetchResponse { + t.Helper() + deadline := time.Now().Add(reachWindow) + for { + resp := probeFetch(t, ctx, rc, id, origin, roots, extraParams...) + if !transientFailure(resp) || time.Now().After(deadline) { + return resp + } + t.Logf("fetch of %s: transient failure (status %q, error %q), retrying", origin, resp.Status, resp.Error) + time.Sleep(5 * time.Second) + } +} + +// transientFailure reports whether fetchUntilReached should retry resp. +func transientFailure(resp fetchResponse) bool { + return resp.Error != "" || strings.HasPrefix(resp.Status, "5") } // probeFetch asks the probe to fetch origin with the given roots mode; @@ -242,11 +298,53 @@ func probeFetch(t *testing.T, ctx context.Context, rc *e2e.RouterClient, id, ori } } +// probeTrust returns the trust bundle the probe's /whoami reads from its +// projected trust file. +func probeTrust(t *testing.T, ctx context.Context, rc *e2e.RouterClient, id string) string { + t.Helper() + ref := resources.ActorRef{Atespace: probeNamespace, Name: id} + resp, err := rc.Get(ctx, ref, "/whoami") + if err != nil { + t.Fatalf("GET /whoami for %q: %v", id, err) + } + defer resp.Body.Close() + var who struct { + Trust string `json:"trust"` + Error string `json:"error"` + } + if err := json.NewDecoder(resp.Body).Decode(&who); err != nil { + t.Fatalf("decoding /whoami for %q (status %d): %v", id, resp.StatusCode, err) + } + if who.Trust == "" { + t.Fatalf("/whoami for %q returned no trust bundle; error: %s", id, who.Error) + } + return who.Trust +} + +// holdsCertificate reports whether bundle has a certificate whose subject +// common name is commonName. +func holdsCertificate(t *testing.T, bundle, commonName string) bool { + t.Helper() + for rest := []byte(bundle); ; { + var block *pem.Block + if block, rest = pem.Decode(rest); block == nil { + return false + } + cert, err := x509.ParseCertificate(block.Bytes) + if err != nil { + t.Fatalf("parsing a certificate in the trust bundle: %v", err) + } + if cert.Subject.CommonName == commonName { + return true + } + } +} + // createAndResumeActor mirrors the identity suite's self-healing actor // lifecycle (actor records outlive the fixture namespace); DeployProbe has // already waited for the template's golden snapshot. -func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Clients, id string) { +func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Clients, id string, mitm, passthrough e2e.Server) { t.Helper() ref := &ateapipb.ObjectRef{Atespace: probeNamespace, Name: id} _, _ = clients.SubstrateAPI.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: ref}) @@ -258,12 +356,14 @@ func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Client t.Fatalf("CreateActor %q: %v", id, err) } // The gateway refuses every tunnel for an actor without a policy. Naming - // only the origin also lets the same actor show a denial. + // only the origins also lets the same actor show a denial. The + // passthrough origin's fully qualified alias is allowed on 8443 only, + // while the probe dials 443. e2e.EnsureEgressPolicy(t, ctx, clients, ref, - e2e.EgressAllowHTTP(egressOriginHost), - e2e.EgressAllowHTTPS(egressOriginHost), - e2e.EgressAllowPassthrough(egressOriginPassthroughHost), - e2e.EgressAllowPassthroughOnPorts([]int32{8443}, egressOriginPassthroughWrongPortHost), + e2e.EgressAllowHTTP(mitm.SvcName()), + e2e.EgressAllowHTTPS(mitm.SvcName()), + e2e.EgressAllowPassthrough(passthrough.SvcName()), + e2e.EgressAllowPassthroughOnPorts([]int32{8443}, passthrough.SvcName()+".cluster.local"), ) t.Cleanup(func() { _, _ = clients.SubstrateAPI.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: ref}) diff --git a/internal/e2e/suites/multiactor/multiactor_test.go b/internal/e2e/suites/multiactor/multiactor_test.go index 58b4b9a1b4..53d8fb1d42 100644 --- a/internal/e2e/suites/multiactor/multiactor_test.go +++ b/internal/e2e/suites/multiactor/multiactor_test.go @@ -52,7 +52,7 @@ func TestTwoActorsShareOneWorker(t *testing.T) { e2e.DeploySubstrateFixture(t, ctx, clients, e2e.SubstrateFixtureManifests{ Pool: "internal/e2e/fixtures/probe/probe-multiactor.yaml.tmpl", Template: "internal/e2e/fixtures/probe/probe-multiactor-template.yaml.tmpl", - }, env["BUCKET_NAME"], "multiactor", false) + }, env["BUCKET_NAME"], "multiactor", nil) names := []string{"ma-first", "ma-second"} pods := map[string]string{} @@ -73,11 +73,14 @@ func TestTwoActorsShareOneWorker(t *testing.T) { } defer rc.Close() - for _, name := range names { - got := whoami(t, ctx, rc, name) - if got.Atespace != multiactorNamespace || got.File != name { - t.Errorf("request for %s was served by atespace=%q actor=%q; each actor must be reached in its own sandbox", - name, got.Atespace, got.File) + // Alternate actors to catch one actor's tunnel serving the other. + for range 3 { + for _, name := range names { + got := whoami(t, ctx, rc, name) + if got.Atespace != multiactorNamespace || got.File != name { + t.Errorf("request for %s was served by atespace=%q actor=%q; each actor must be reached in its own sandbox", + name, got.Atespace, got.File) + } } } diff --git a/internal/e2e/suites/networking/accesslog_test.go b/internal/e2e/suites/networking/accesslog_test.go index f9057bc5c8..f355ba6641 100644 --- a/internal/e2e/suites/networking/accesslog_test.go +++ b/internal/e2e/suites/networking/accesslog_test.go @@ -42,6 +42,7 @@ func TestProtocolGatewayAccessLogMatches(t *testing.T) { actorRef := resources.ActorRef{Atespace: "networking-e2e", Name: "ws-1"} for _, test := range []struct { name, container, line string + authority string want bool }{ { @@ -51,31 +52,38 @@ func TestProtocolGatewayAccessLogMatches(t *testing.T) { want: true, }, { - name: "agentgateway opaque tunnel success", + name: "agentgateway authenticated CONNECT success", container: "agentgateway", line: `CONNECT tunnel terminated target=10.96.0.7:80 actor_name=ws-1 actor_uid=uid-1 atespace=networking-e2e`, want: true, }, { - name: "agentgateway opaque tunnel wrong identity", + name: "agentgateway HTTPS nonstandard port", + container: "agentgateway", + line: `CONNECT tunnel terminated bind=bind/wildcard target=10.96.0.7:8443 actor_name=ws-1 actor_uid="uid-1" atespace=networking-e2e`, + authority: "10.96.0.7:8443", + want: true, + }, + { + name: "agentgateway authenticated CONNECT wrong identity", container: "agentgateway", line: `CONNECT tunnel terminated target=10.96.0.7:80 actor_name=ws-2 actor_uid=uid-1 atespace=networking-e2e substrate.connect.authority="10.96.0.7:80"`, want: false, }, { - name: "agentgateway opaque tunnel wrong target", + name: "agentgateway authenticated CONNECT wrong target", container: "agentgateway", line: `CONNECT tunnel terminated target=10.96.0.8:80 actor_name=ws-1 actor_uid=uid-1 atespace=networking-e2e substrate.connect.authority="10.96.0.7:80"`, want: false, }, { - name: "agentgateway opaque tunnel wrong atespace", + name: "agentgateway authenticated CONNECT wrong atespace", container: "agentgateway", line: `CONNECT tunnel terminated target=10.96.0.7:80 actor_name=ws-1 actor_uid=uid-1 atespace=other`, want: false, }, { - name: "agentgateway opaque tunnel error", + name: "agentgateway authenticated CONNECT error", container: "agentgateway", line: `CONNECT tunnel terminated target=10.96.0.7:80 actor_name=ws-1 actor_uid=uid-1 atespace=networking-e2e error=denied`, want: false, @@ -130,14 +138,13 @@ func TestProtocolGatewayAccessLogMatches(t *testing.T) { }, } { t.Run(test.name, func(t *testing.T) { - got := protocolGatewayAccessLogMatches(gatewayAccessLogLine{container: test.container, text: test.line}, actorRef, "10.96.0.7:80", true) + if test.authority == "" { + test.authority = "10.96.0.7:80" + } + got := protocolGatewayAccessLogMatches(gatewayAccessLogLine{container: test.container, text: test.line}, actorRef, test.authority) if got != test.want { t.Fatalf("protocolGatewayAccessLogMatches() = %t, want %t", got, test.want) } }) } - valid := gatewayAccessLogLine{container: "agentgateway", text: `CONNECT tunnel terminated target=10.96.0.7:80 actor_name=ws-1 actor_uid=uid-1 atespace=networking-e2e`} - if protocolGatewayAccessLogMatches(valid, actorRef, "10.96.0.7:80", false) { - t.Fatal("opaque CONNECT evidence must not satisfy the MITM matcher") - } } diff --git a/internal/e2e/suites/networking/egresspolicy_test.go b/internal/e2e/suites/networking/egresspolicy_test.go index feb7e4268d..410d5a8755 100644 --- a/internal/e2e/suites/networking/egresspolicy_test.go +++ b/internal/e2e/suites/networking/egresspolicy_test.go @@ -16,6 +16,7 @@ package networking import ( "context" + "encoding/json" "fmt" "net/http" "strings" @@ -24,6 +25,7 @@ import ( "github.com/agent-substrate/substrate/internal/e2e" "github.com/agent-substrate/substrate/internal/resources" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) // The EgressPolicy half of egress: the other TestActorEgress* tests give their @@ -38,8 +40,8 @@ func notTransient(status int, _ []byte) bool { return status != http.StatusServi // reached stops the retry loop only on success. A lane expecting the fetch to // work sees more transients than the 503 above (the minted leaf fails -// verification until kubelet has propagated the CA pool, public origins -// hiccup), all of them 502s the actor cannot tell from a denial. +// verification until kubelet has propagated the CA pool), all of them 502s the +// actor cannot tell from a denial. func reached(status int, _ []byte) bool { return status == http.StatusOK } // TestActorEgressPolicyDeniesUnlistedHost: the policy names one hostname, so @@ -95,62 +97,200 @@ func TestActorEgressRequiresPolicy(t *testing.T) { t.Logf("egress was denied as expected: %s", body) } -// hostnamePolicyActor creates an actor whose policy allows HTTPS to -// example.com and nothing else, and waits until it is routable. -func hostnamePolicyActor(t *testing.T, ctx context.Context) (*e2e.RouterClient, resources.ActorRef) { - t.Helper() - _, actorName, _ := createAndResumeActorWithEgress(t, ctx, "egress-sni", e2e.EgressFixture(), e2e.EgressAllowHTTPS("example.com"), e2e.EgressAllowPassthrough("example.edu")) +// TestActorEgressHTTPSByHostname: the policy allows one origin by name for +// interception and another for passthrough, and the gateway handles each the +// way its rule says and refuses a name no rule lists. +// +// Both origins serve service-DNS certificates, which the gateway trusts and +// the actor does not, so the certificate the actor verifies tells the two +// paths apart. +func TestActorEgressHTTPSByHostname(t *testing.T) { + ctx := context.Background() + dataplane := e2e.CurrentAtenetDataplane() + + const mitmBody, passthroughBody = "mitm origin", "passthrough origin" + mitm := e2e.DeployServerPod(t, ctx, egressHTTPSTarget("mitm-origin", mitmBody)) + passthroughSpec := egressHTTPSTarget("passthrough-origin", passthroughBody) + passthroughSpec.Namespace = mitm.Namespace + passthrough := e2e.DeployServerPod(t, ctx, passthroughSpec) + + _, actorName, _ := createAndResumeActorWithEgress(t, ctx, "egress-sni", e2e.EgressFixture(), + e2e.EgressAllowHTTPS(mitm.SvcName()), e2e.EgressAllowPassthrough(passthrough.SvcName())) router := mustRouterClient(t, ctx) - t.Cleanup(func() { router.Close() }) + defer router.Close() actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} waitForActorRoute(t, ctx, router, actorRef) - return router, actorRef -} -func isMitmCert(body string) bool { - // example.* domains use certs signed by Google Trust Services - // If this substring is not present it means dataplane used a minted cert - return !strings.Contains(body, "O=Google Trust Services") + t.Run("mitm", func(t *testing.T) { + // The actor's default roots hold the gateway's CA but not the + // service-DNS signer, so a verified fetch means the gateway minted the + // leaf the actor saw. + url := "https://" + mitm.SvcName() + "/healthz" + status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, url, reached) + if status != http.StatusOK { + t.Fatalf("fetch of %s returned HTTP %d, want 200; body: %s", url, status, body) + } + if got := decodeEgressFetch(t, body); got.Body != mitmBody || !got.TLS { + t.Errorf("fetch of %s = %+v, want verified TLS and body %q", url, got, mitmBody) + } + }) + + t.Run("passthrough", func(t *testing.T) { + // Trusting only the service-DNS signer, the fetch verifies against the + // origin's own certificate alone, which an intercepting gateway would + // have replaced with a minted leaf. + url := "https://" + passthrough.SvcName() + "/healthz" + payload, err := json.Marshal(map[string]string{"url": url, "rootCA": e2e.ServiceDNSTrustBundle(t, ctx, e2e.GetClients())}) + if err != nil { + t.Fatalf("marshaling the fetch request for %s: %v", url, err) + } + status, body := postThroughEgressActorUntil(t, ctx, router, actorRef, "/", payload, reached) + if status != http.StatusOK { + t.Fatalf("fetch of %s returned HTTP %d, want 200; body: %s", url, status, body) + } + if got := decodeEgressFetch(t, body); got.Body != passthroughBody || !got.TLS { + t.Errorf("fetch of %s = %+v, want verified TLS and body %q", url, got, passthroughBody) + } + }) + + t.Run("unlisted host denied", func(t *testing.T) { + // The MITM origin by its fully qualified name, which no rule lists + // (rules match names exactly) and its certificate does not cover. A + // gateway that wrongly intercepted it fails upstream with a 503; one + // that wrongly passed it through hands the actor that certificate. + url := "https://" + mitm.SvcName() + ".cluster.local/healthz" + status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, url, notTransient) + if !dataplane.IsEgressPolicyDenied(status, string(body)) { + t.Fatalf("fetch of %s, outside the policy, returned HTTP %d, want an egress-policy denial; body: %s", url, status, body) + } + // Envoy denies by closing the connection, which the actor reports as + // the same 502 as any failed fetch; a certificate error in it means the + // handshake reached the origin. + if got := decodeEgressFetch(t, body); strings.Contains(got.Error, "x509") || strings.Contains(got.Error, "certificate") { + t.Fatalf("fetch of %s, outside the policy, failed certificate verification instead of being denied, so the gateway passed it through; body: %s", url, body) + } + t.Logf("fetch of %s was denied as expected: %s", url, body) + }) } -// TestActorEgressHTTPSByHostnameMITM: the gateway terminates the TLS and decides -// each request by name: example.com 200, example.org 403. -func TestActorEgressHTTPSByHostnameMITM(t *testing.T) { - ctx := context.Background() +// This test validates that TLS passthrough connection is denied when policy changes +// even if there are outstanding connections that were allowed by the previous policy. +func TestActorEgressPassthroughPolicyRefresh(t *testing.T) { + ctx := t.Context() dataplane := e2e.CurrentAtenetDataplane() - router, actorRef := hostnamePolicyActor(t, ctx) + _, address, rootCA := prepareProtocolOrigin(t, ctx, e2e.ServerPod{ + Name: "passthrough-origin", + ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", + Args: []string{"http"}, + Port: 80, + TargetPort: 8080, + TLSPort: 443, + }, true) - status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.com/", reached) - if status != http.StatusOK { - t.Fatalf("fetch of the allowed host returned HTTP %d, want 200; body: %s", status, body) + actorAtespace, actorName, _ := createAndResumeActorWithEgress(t, ctx, "egress-passthrough-refresh", e2e.EgressFixture(), e2e.EgressAllowPassthrough("*")) + router := mustRouterClient(t, ctx) + defer router.Close() + actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} + waitForActorRoute(t, ctx, router, actorRef) + + // Ask egres actor to POST to /fetch on the test server. Test server /fetch handler + // waits for the entire body before responding. + url := "https://" + address + "/fetch" + postPayload, err := json.Marshal(map[string]string{"url": url, "rootCA": rootCA}) + if err != nil { + t.Fatalf("marshaling /post request: %v", err) + } + + type postResult struct { + status int + body []byte } - if !isMitmCert(string(body)) { - t.Fatalf("request did not use minted cert; body: %s", body) + since := metav1.NewTime(time.Now().Add(-1 * time.Minute)) + firstResult := make(chan postResult, 1) + go func() { + status, body := postThroughEgressActorUntil(t, ctx, router, actorRef, "/post", postPayload, notTransient) + firstResult <- postResult{status: status, body: body} + }() + + // Allow connection to be fully established before changing the policy. + assertEgressGatewayConnect(t, ctx, since, actorAtespace, actorName, "443") + + e2e.EnsureEgressPolicy(t, ctx, e2e.GetClients(), actorRef.ToObjectRef(), e2e.EgressAllowPassthrough("foo.bar.baz")) + // Wait for the policy cached in ext_proc server to expire, so it will fetch the new one. + // ext_proc cache freshness interval is 10 seconds. + time.Sleep(11 * time.Second) + + // AgentGateway intercepts denied TLS connections to return HTTP 403. + // Use the actor's default gateway roots for this request; the first request + // trusts only the origin CA to prove it was passed through. + deniedPayload, err := json.Marshal(map[string]string{"url": url}) + if err != nil { + t.Fatalf("marshaling denied /post request: %v", err) } - status, body = fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.org/", notTransient) + status, body := postThroughEgressActorUntil(t, ctx, router, actorRef, "/post", deniedPayload, notTransient) if !dataplane.IsEgressPolicyDenied(status, string(body)) { - t.Fatalf("fetch of a host outside the policy returned HTTP %d, want an egress-policy denial; body: %s", status, body) + t.Fatalf("second POST to %s after policy update returned HTTP %d, want an egress-policy denial; body: %s", url, status, body) + } + if got := decodeEgressFetch(t, body); strings.Contains(got.Error, "x509") || strings.Contains(got.Error, "certificate") { + t.Fatalf("second POST to %s failed certificate verification instead of being denied; body: %s", url, body) + } + + first := <-firstResult + if first.status != http.StatusOK { + t.Fatalf("first POST to %s returned HTTP %d, want 200; body: %s", url, first.status, first.body) } - t.Logf("denied on the decrypted request: %s", body) } -// TestActorEgressHTTPSByHostnamePassthrough: the gateway acts as TCP proxy fetching -// allowed SNI. -func TestActorEgressHTTPSByHostnamePassthrough(t *testing.T) { - ctx := context.Background() - dataplane := e2e.CurrentAtenetDataplane() - router, actorRef := hostnamePolicyActor(t, ctx) +func TestActorEgressPendingPlaintextHeadersDoNotBlockMITM(t *testing.T) { + ctx := t.Context() + const mitmBody = "mitm origin" + mitm := e2e.DeployServerPod(t, ctx, egressHTTPSTarget("mitm-origin", mitmBody)) + plaintextSpec := egressHTTPTarget() + plaintextSpec.Namespace = mitm.Namespace + plaintext := e2e.DeployServerPod(t, ctx, plaintextSpec) - status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.edu/", reached) + actorAtespace, actorName, _ := createAndResumeActorWithEgress( + t, ctx, "egress-slow-headers", e2e.EgressFixture(), + e2e.EgressAllowHTTPS(mitm.SvcName()), + e2e.EgressAllowHTTP(plaintext.SvcName()), + ) + router := mustRouterClient(t, ctx) + defer router.Close() + actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} + waitForActorRoute(t, ctx, router, actorRef) + + plaintextURL := "http://" + plaintext.SvcName() + "/healthz" + slowPayload, err := json.Marshal(map[string]string{"url": plaintextURL}) + if err != nil { + t.Fatalf("marshaling /slow-headers request: %v", err) + } + + type postResult struct { + status int + body []byte + } + since := metav1.NewTime(time.Now().Add(-1 * time.Minute)) + pendingResult := make(chan postResult, 1) + go func() { + status, body := postThroughEgressActorUntil(t, ctx, router, actorRef, "/slow-headers", slowPayload, notTransient) + pendingResult <- postResult{status: status, body: body} + }() + + // Allow the plaintext request to send its incomplete headers before issuing the MITM request. + assertEgressGatewayConnect(t, ctx, since, actorAtespace, actorName, "80") + + mitmURL := "https://" + mitm.SvcName() + "/healthz" + status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, mitmURL, reached) if status != http.StatusOK { - t.Fatalf("fetch of the allowed host returned HTTP %d, want 200; body: %s", status, body) + t.Fatalf("fetch of %s returned HTTP %d, want 200; body: %s", mitmURL, status, body) } - if isMitmCert(string(body)) { - t.Fatalf("request used minted cert; body: %s", body) + if got := decodeEgressFetch(t, body); got.Body != mitmBody || !got.TLS { + t.Fatalf("fetch of %s = %+v, want verified TLS and body %q", mitmURL, got, mitmBody) } - status, body = fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.org/", notTransient) - if !dataplane.IsEgressPolicyDenied(status, string(body)) { - t.Fatalf("fetch of a host outside the policy returned HTTP %d, want an egress-policy denial; body: %s", status, body) + + pending := <-pendingResult + if pending.status != http.StatusOK { + t.Fatalf("fetch of %s returned HTTP %d, want 200; body: %s", plaintextURL, pending.status, pending.body) } } diff --git a/internal/e2e/suites/networking/grpcingress_test.go b/internal/e2e/suites/networking/grpcingress_test.go index a395bf3f23..e919cb49d2 100644 --- a/internal/e2e/suites/networking/grpcingress_test.go +++ b/internal/e2e/suites/networking/grpcingress_test.go @@ -109,8 +109,9 @@ func TestIngressProtocolDowngrade(t *testing.T) { } body, _ := io.ReadAll(resp.Body) resp.Body.Close() - // The counter cannot speak h2c. Envoy returns HTTP 502; agentgateway - // reports the upstream failure with gRPC's Unavailable status. + // The counter cannot speak h2c. Through atunnel's HTTPS listener Envoy + // returns HTTP 502; through a CONNECT tunnel the failure is gRPC's + // Unavailable status. grpcStatus := resp.Header.Get("Grpc-Status") if grpcStatus == "" { grpcStatus = resp.Trailer.Get("Grpc-Status") @@ -265,7 +266,7 @@ func TestIngressGRPC(t *testing.T) { // shared one would be deleted out from under another. func deployGRPCEchoTemplate(t *testing.T, ctx context.Context, bucket string) e2e.SubstrateFixture { t.Helper() - atespace, _ := e2e.DeploySubstrateFixture(t, ctx, e2e.GetClients(), grpcEchoFixtureManifests, bucket, "networking", false) + atespace, _ := e2e.DeploySubstrateFixture(t, ctx, e2e.GetClients(), grpcEchoFixtureManifests, bucket, "networking", nil) return e2e.SubstrateFixture{ Atespace: atespace, Name: "grpcecho", diff --git a/internal/e2e/suites/networking/microvmclock_test.go b/internal/e2e/suites/networking/microvmclock_test.go index 22b1b25300..43433498f0 100644 --- a/internal/e2e/suites/networking/microvmclock_test.go +++ b/internal/e2e/suites/networking/microvmclock_test.go @@ -75,7 +75,7 @@ func TestMicroVMClockAfterDelayedRestore(t *testing.T) { target, address, rootCA := prepareProtocolOrigin(t, ctx, e2e.ServerPod{ Name: "clock-origin", ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", - Args: []string{"http", "--body=delayed restore clock"}, Port: 8443, + Args: []string{"http", "--body=delayed restore clock"}, Port: 8080, TLSPort: 8443, }, true) logOriginCertificateValidity(t, ctx, target, rootCA, suspendBoundary) @@ -103,7 +103,7 @@ func TestMicroVMClockAfterDelayedRestore(t *testing.T) { if got.Error != "" || got.StatusCode != 200 || got.Body != "delayed restore clock" || got.Protocol != "HTTP/1.1" || !got.TLS { t.Errorf("delayed-restore HTTPS observation = %+v; want one verified TLS HTTP/1.1 8443 request", got) } - assertProtocolGateway(t, ctx, since, actorName, target.Address()) + assertProtocolGateway(t, ctx, since, actorName, target.TLSAddress()) } func logOriginCertificateValidity(t *testing.T, ctx context.Context, target e2e.Server, rootCA string, suspendBoundary time.Time) { diff --git a/internal/e2e/suites/networking/networking_test.go b/internal/e2e/suites/networking/networking_test.go index c8d3585ef3..0291dfe5cd 100644 --- a/internal/e2e/suites/networking/networking_test.go +++ b/internal/e2e/suites/networking/networking_test.go @@ -143,10 +143,79 @@ func TestActorEgress(t *testing.T) { }) } +func TestActorEgressMultiplexing(t *testing.T) { + ctx := context.Background() + dataplane := e2e.CurrentAtenetDataplane() + + origin := egressHTTPTarget() + target := e2e.DeployServerPod(t, ctx, origin) + + fixture := e2e.EgressFixture() + actorAtespace, actorName, _ := createAndResumeActorWithEgress(t, ctx, "egress-mux", fixture, e2e.EgressAllowAll()...) + router := mustRouterClient(t, ctx) + defer router.Close() + + actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} + waitForActorRoute(t, ctx, router, actorRef) + + beforeScrape, err := dataplane.ScrapeMetrics(ctx) + if err != nil { + t.Fatalf("ScrapeMetrics before /mux: %v", err) + } + + url := fmt.Sprintf("http://%s/fetch", target.Address()) + payload, err := json.Marshal(map[string]string{"url": url}) + if err != nil { + t.Fatalf("marshaling the mux request for %s: %v", url, err) + } + + since := metav1.NewTime(time.Now().Add(-1 * time.Minute)) + status, body := postThroughEgressActor(t, ctx, router, actorRef, "/mux", payload) + if status != http.StatusOK { + t.Fatalf("Actor egress mux of %s returned HTTP %d, want 200; body: %s", url, status, body) + } + t.Logf("Actor egress mux of %s succeeded; body: %s", url, body) + + assertEgressGatewayConnect(t, ctx, since, actorAtespace, actorName, strconv.Itoa(origin.Port)) + + afterScrape, err := dataplane.ScrapeMetrics(ctx) + if err != nil { + t.Fatalf("ScrapeMetrics after /mux: %v", err) + } + + const ( + // Different dataplane may have different stream concurrency. + wantConnectionsAtLeast = 1 + wantRequests = 250 + ) + stats := dataplane.EgressConnectStats(beforeScrape, afterScrape) + if stats.Connections != -1 && stats.Connections < wantConnectionsAtLeast { + t.Errorf("egress connect connections = %d, want at least %d", stats.Connections, wantConnectionsAtLeast) + } + if stats.Requests != -1 && stats.Requests != wantRequests { + t.Errorf("egress connect requests = %d, want %d", stats.Requests, wantRequests) + } +} + +// egressHTTPSTarget returns an origin whose /healthz answers with body over +// HTTPS on 443 (and cleartext on 80): testserver's http subcommand serving a +// service-DNS certificate, which the gateway trusts once +// hack/setup-e2e-egress-tls-kind.sh has run. +func egressHTTPSTarget(name, body string) e2e.ServerPod { + return e2e.ServiceDNSTLS(e2e.ServerPod{ + Name: name, + ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", + Args: []string{"http", "--body=" + body}, + }) +} + // TestActorEgressHTTPS covers the same path as TestActorEgress with a TLS // origin, through the gateway's TLS interception. func TestActorEgressHTTPS(t *testing.T) { ctx := context.Background() + const originBody = "https origin" + target := e2e.DeployServerPod(t, ctx, egressHTTPSTarget("httpstarget", originBody)) + fixture := e2e.EgressFixture() actorAtespace, actorName, _ := createAndResumeActorWithEgress(t, ctx, "egress-https", fixture, e2e.EgressAllowAll()...) router := mustRouterClient(t, ctx) @@ -157,11 +226,15 @@ func TestActorEgressHTTPS(t *testing.T) { since := metav1.NewTime(time.Now().Add(-1 * time.Minute)) actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} - status, body := fetchThroughEgressActor(t, ctx, router, actorRef, "https://example.com/") + url := "https://" + target.SvcName() + "/healthz" + status, body := fetchThroughEgressActor(t, ctx, router, actorRef, url) if status != http.StatusOK { - t.Fatalf("Actor HTTPS egress fetch returned HTTP %d, want 200; body: %s", status, body) + t.Fatalf("Actor HTTPS egress fetch of %s returned HTTP %d, want 200; body: %s", url, status, body) + } + if got := decodeEgressFetch(t, body); got.Body != originBody || !got.TLS { + t.Errorf("Actor HTTPS egress fetch of %s = %+v, want verified TLS and body %q", url, got, originBody) } - t.Logf("Actor HTTPS egress fetch succeeded; body: %s", body) + t.Logf("Actor HTTPS egress fetch of %s succeeded; body: %s", url, body) assertEgressGatewayConnect(t, ctx, since, actorAtespace, actorName, "443") } @@ -230,6 +303,24 @@ func fetchThroughEgressActor(t *testing.T, ctx context.Context, router *e2e.Rout return postThroughEgressActor(t, ctx, router, actorRef, "/", payload) } +// egressFetchResponse is the egress demo's answer to a fetch. +type egressFetchResponse struct { + StatusCode int `json:"statusCode"` + Body string `json:"body"` + TLS bool `json:"tls"` + Error string `json:"error"` +} + +// decodeEgressFetch parses the body fetchThroughEgressActor returned. +func decodeEgressFetch(t *testing.T, body []byte) egressFetchResponse { + t.Helper() + var got egressFetchResponse + if err := json.Unmarshal(body, &got); err != nil { + t.Fatalf("decoding the egress demo's fetch response: %v (body %q)", err, body) + } + return got +} + // postThroughEgressActor POSTs payload to path on the egress demo Actor and // returns the status and body it answered with. Retries a non-200 response for // up to 30s: ResumeActor can return before its route reaches atenet-router's @@ -291,9 +382,12 @@ func assertEgressGatewayConnect(t *testing.T, ctx context.Context, since metav1. } return true case "agentgateway": + target, _ := accessLogField(line.text, "target") + tunnelActor, _ := accessLogField(line.text, "actor_name") + tunnelSpace, _ := accessLogField(line.text, "atespace") if strings.Contains(line.text, "CONNECT tunnel terminated") && - strings.Contains(line.text, "target=") && - strings.Contains(line.text, ":"+port) { + strings.HasSuffix(target, ":"+port) && + tunnelActor == actorName && tunnelSpace == atespace { t.Logf("egress gateway tunneled the request: %s", line.text) return true } @@ -507,7 +601,8 @@ func TestActorEgressHTTPSNonStandardPort(t *testing.T) { Name: "https-origin", ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", Args: []string{"http", "--body=" + expectedBody}, - Port: 8443, + Port: 8080, + TLSPort: 8443, }, true) actorName, router, actorRef := prepareProtocolActor(t, ctx, "https-port") since := metav1.NewTime(time.Now().Add(-time.Minute)) @@ -527,7 +622,7 @@ func TestActorEgressHTTPSNonStandardPort(t *testing.T) { if got.Error != "" || got.StatusCode != http.StatusOK || got.Body != expectedBody || got.Protocol != "HTTP/1.1" || !got.TLS { t.Errorf("HTTPS observation = %+v; want verified TLS, HTTP/1.1, status 200 and body %q", got, expectedBody) } - assertProtocolGateway(t, ctx, since, actorName, target.Address()) + assertProtocolGateway(t, ctx, since, actorName, target.TLSAddress()) } func prepareProtocolOrigin(t *testing.T, ctx context.Context, spec e2e.ServerPod, encrypted bool) (e2e.Server, string, string) { @@ -537,6 +632,9 @@ func prepareProtocolOrigin(t *testing.T, ctx context.Context, spec e2e.ServerPod var reserved e2e.Server var rootCA string if encrypted { + if spec.TLSPort == 0 { + t.Fatalf("encrypted origin %s sets no TLSPort to publish its HTTPS listener on", spec.Name) + } reserved = e2e.CreateServerService(t, ctx, spec) // The unified gateway verifies local origins against service-DNS roots. secret, err := e2e.GetClients().K8s.CoreV1().Secrets("podcertificate-controller-system").Get(ctx, "service-dns-ca-pool", metav1.GetOptions{}) @@ -560,7 +658,6 @@ func prepareProtocolOrigin(t *testing.T, ctx context.Context, spec e2e.ServerPod spec.Volumes = []corev1.Volume{{Name: "tls", VolumeSource: corev1.VolumeSource{Secret: &corev1.SecretVolumeSource{SecretName: originSecret.Name}}}} spec.VolumeMounts = []corev1.VolumeMount{{Name: "tls", MountPath: "/run/origin-tls", ReadOnly: true}} spec.Args = append(spec.Args, "--tls-cert=/run/origin-tls/tls.crt", "--tls-key=/run/origin-tls/tls.key") - spec.HealthScheme = corev1.URISchemeHTTPS } target := e2e.DeployServerPod(t, ctx, spec) if encrypted && target.ClusterIP != reserved.ClusterIP { @@ -571,8 +668,12 @@ func prepareProtocolOrigin(t *testing.T, ctx context.Context, spec e2e.ServerPod if net.ParseIP(target.ClusterIP).To4() != nil { family = "IPv4" } - t.Logf("origin %s/%s at %s (%s), TLS=%v", target.Namespace, spec.Name, target.Address(), family, encrypted) - address := protocolOriginRequest(target, spec.Name) + dialed := target.Address() + if encrypted { + dialed = target.TLSAddress() + } + t.Logf("origin %s/%s at %s (%s), TLS=%v", target.Namespace, spec.Name, dialed, family, encrypted) + address := protocolOriginRequest(target) return target, address, rootCA } @@ -775,22 +876,31 @@ func valueString(value *string) string { return *value } -// The unified gateway needs the origin's DNS name for SNI; the outer CONNECT -// log still records target.Address(). -func protocolOriginRequest(target e2e.Server, service string) string { - return net.JoinHostPort(service+"."+target.Namespace+".svc.cluster.local", strconv.Itoa(target.Port)) +// protocolOriginRequest is the host:port an actor fetches target at: its DNS +// name, which the unified gateway needs for SNI, on its TLS port if it has one. +// The CONNECT authority in the gateway's access log is the ClusterIP on the +// same port, target.TLSAddress() or target.Address(). +func protocolOriginRequest(target e2e.Server) string { + port := target.Port + if target.TLSPort != 0 { + port = target.TLSPort + } + return net.JoinHostPort(target.Name+"."+target.Namespace+".svc.cluster.local", strconv.Itoa(port)) } func TestProtocolOriginRequest(t *testing.T) { for _, tc := range []struct { name, ip string + tlsPort int + want string }{ - {"IPv4", "10.0.0.7"}, - {"IPv6", "fd00::7"}, + {"IPv4", "10.0.0.7", 0, "origin.test.svc.cluster.local:8080"}, + {"IPv6", "fd00::7", 0, "origin.test.svc.cluster.local:8080"}, + {"TLS", "10.0.0.7", 8443, "origin.test.svc.cluster.local:8443"}, } { t.Run(tc.name, func(t *testing.T) { - target := e2e.Server{Namespace: "test", ClusterIP: tc.ip, Port: 8443} - if got, want := protocolOriginRequest(target, "origin"), "origin.test.svc.cluster.local:8443"; got != want { + target := e2e.Server{Name: "origin", Namespace: "test", ClusterIP: tc.ip, Port: 8080, TLSPort: tc.tlsPort} + if got, want := protocolOriginRequest(target), tc.want; got != want { t.Fatalf("origin request = %q, want %q", got, want) } }) @@ -856,9 +966,12 @@ func assertProtocolGateway(t *testing.T, ctx context.Context, since metav1.Time, t.Helper() actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} identity := resources.AteomForActorSPIFFEID(actorRef).String() + // The authenticated CONNECT record identifies the actor and destination. + // Callers separately verify the HTTPS response using only the gateway's CA, + // so a successful TLS handshake proves interception rather than passthrough. waitForAccessLog(t, ctx, since, "successful gateway traffic for "+identity+" to "+authority, func(lines []gatewayAccessLogLine) bool { for _, entry := range lines { - if protocolGatewayAccessLogMatches(entry, actorRef, authority, false) { + if protocolGatewayAccessLogMatches(entry, actorRef, authority) { t.Logf("gateway evidence: %s", entry.text) return true } @@ -867,7 +980,7 @@ func assertProtocolGateway(t *testing.T, ctx context.Context, since metav1.Time, }) } -func protocolGatewayAccessLogMatches(entry gatewayAccessLogLine, actorRef resources.ActorRef, authority string, allowOpaque bool) bool { +func protocolGatewayAccessLogMatches(entry gatewayAccessLogLine, actorRef resources.ActorRef, authority string) bool { switch entry.container { case "envoy": gotAuthority, _ := accessLogField(entry.text, "authority") @@ -883,11 +996,9 @@ func protocolGatewayAccessLogMatches(entry gatewayAccessLogLine, actorRef resour } case "agentgateway": gotAuthority, _ := accessLogField(entry.text, "substrate.connect.authority") - if allowOpaque && strings.Contains(entry.text, "CONNECT tunnel terminated") { - // Baseline TCP routes carry opaque CONNECT traffic and therefore do - // not emit an HTTP status. The accepted CONNECT record identifies the - // authenticated actor and destination; the operation assertion above - // supplies end-to-end success. + if strings.Contains(entry.text, "CONNECT tunnel terminated") { + // This record is emitted when the authenticated CONNECT is accepted, + // before the gateway selects interception or passthrough. target, targetOK := accessLogField(entry.text, "target") tunnelActor, actorOK := accessLogField(entry.text, "actor_name") tunnelSpace, spaceOK := accessLogField(entry.text, "atespace") diff --git a/internal/e2e/suites/networking/websocketegress_test.go b/internal/e2e/suites/networking/websocketegress_test.go index 34b318b194..59f5c2cc36 100644 --- a/internal/e2e/suites/networking/websocketegress_test.go +++ b/internal/e2e/suites/networking/websocketegress_test.go @@ -70,7 +70,7 @@ func TestActorEgressSecureWebSocket(t *testing.T) { ctx := t.Context() _, address, _ := prepareProtocolOrigin(t, ctx, e2e.ServerPod{ Name: "wss-origin", ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", - Args: []string{"websocket", "--echo"}, Port: 443, TargetPort: 8443, HealthPath: "/readyz", + Args: []string{"websocket", "--echo"}, Port: 80, TargetPort: 8080, TLSPort: 443, HealthPath: "/readyz", }, true) _, router, actorRef := prepareProtocolActor(t, ctx, "wss") raw := postEgressWithStatus(t, ctx, router, actorRef, "/websocket", map[string]any{ diff --git a/internal/e2e/suites/networking/websocketingress_test.go b/internal/e2e/suites/networking/websocketingress_test.go index 911a434f49..d05153bb72 100644 --- a/internal/e2e/suites/networking/websocketingress_test.go +++ b/internal/e2e/suites/networking/websocketingress_test.go @@ -40,7 +40,7 @@ func deployWebsocketFixture(t *testing.T, ctx context.Context) e2e.SubstrateFixt atespace, _ := e2e.DeploySubstrateFixture(t, ctx, e2e.GetClients(), e2e.SubstrateFixtureManifests{ Pool: "internal/e2e/fixtures/testserver/websocket.yaml.tmpl", Template: "internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl", - }, env["BUCKET_NAME"], "websocket", false) + }, env["BUCKET_NAME"], "websocket", nil) return e2e.SubstrateFixture{ Atespace: atespace, diff --git a/internal/e2e/suites/parking/parking_test.go b/internal/e2e/suites/parking/parking_test.go index fd32884b0b..0df9c7972c 100644 --- a/internal/e2e/suites/parking/parking_test.go +++ b/internal/e2e/suites/parking/parking_test.go @@ -206,7 +206,7 @@ func deployParkingFixture(t *testing.T, ctx context.Context, clients *e2e.Client _, templates := e2e.DeploySubstrateFixture(t, ctx, clients, e2e.SubstrateFixtureManifests{ Pool: "internal/e2e/fixtures/probe/probe-parking.yaml.tmpl", Template: "internal/e2e/fixtures/probe/probe-parking-template.yaml.tmpl", - }, env["BUCKET_NAME"], "parking", false) + }, env["BUCKET_NAME"], "parking", nil) return templates[0] } diff --git a/internal/e2e/suites/security/crossatespace_test.go b/internal/e2e/suites/security/crossatespace_test.go index 104d0b47d8..0637f3cc33 100644 --- a/internal/e2e/suites/security/crossatespace_test.go +++ b/internal/e2e/suites/security/crossatespace_test.go @@ -310,11 +310,11 @@ func deployAtespaces(t *testing.T, ctx context.Context, clients *e2e.Clients, bu a, _ := e2e.DeploySubstrateFixture(t, ctx, clients, e2e.SubstrateFixtureManifests{ Pool: "internal/e2e/fixtures/security/atespace-a-pool.yaml.tmpl", Template: "internal/e2e/fixtures/security/atespace-template.yaml.tmpl", - }, bucket, "sec-a", false) + }, bucket, "sec-a", nil) b, _ := e2e.DeploySubstrateFixture(t, ctx, clients, e2e.SubstrateFixtureManifests{ Pool: "internal/e2e/fixtures/security/atespace-b-namespace.yaml.tmpl", Template: "internal/e2e/fixtures/security/atespace-template.yaml.tmpl", - }, bucket, "sec-b", false) + }, bucket, "sec-b", nil) if a == b { t.Fatalf("both fixtures rendered to atespace %q — the fixture no longer isolates anything", a) } diff --git a/internal/e2e/suites/sizing/sizing_test.go b/internal/e2e/suites/sizing/sizing_test.go index a10f6e5577..6142f41321 100644 --- a/internal/e2e/suites/sizing/sizing_test.go +++ b/internal/e2e/suites/sizing/sizing_test.go @@ -114,7 +114,7 @@ func deploySizedProbe(t *testing.T, ctx context.Context, clients *e2e.Clients, b e2e.DeploySubstrateFixture(t, ctx, clients, e2e.SubstrateFixtureManifests{ Pool: "internal/e2e/fixtures/probe/probe-sized.yaml.tmpl", Template: "internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl", - }, bucket, "sizing", false) + }, bucket, "sizing", nil) } func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Clients, id string) { diff --git a/internal/e2e/trustbundle.go b/internal/e2e/trustbundle.go index df2e260751..11c8318453 100644 --- a/internal/e2e/trustbundle.go +++ b/internal/e2e/trustbundle.go @@ -236,3 +236,24 @@ func waitForEgressTrustBundle(t *testing.T, ctx context.Context, clients *Client } t.Fatalf("timed out waiting for ClusterTrustBundle %q to carry the pool's root certificates (last observed: %.80q...); is atecontroller's EgressMITMTrustReconciler running?", EgressTrustBundleObjectName, last) } + +// ServiceDNSTrustBundle returns the service-DNS signer's roots as PEM, for a +// fetch that has to verify a ServiceDNSTLS server's own certificate rather than +// a leaf the egress gateway minted. +func ServiceDNSTrustBundle(t *testing.T, ctx context.Context, clients *Clients) string { + t.Helper() + // The ClusterTrustBundle podcertcontroller publishes for the signer. + const name = "servicedns.podcert.ate.dev:identity:primary-bundle" + bundles, err := clustertrustbundle.NewClient(clients.K8s, nil) + if err != nil { + t.Fatal(err) + } + ctb, err := bundles.Get(ctx, name, metav1.GetOptions{}) + if err != nil { + t.Fatalf("getting ClusterTrustBundle %q: %v", name, err) + } + if blocks, ok := certificateBlocks(ctb.Spec.TrustBundle); !ok || len(blocks) == 0 { + t.Fatalf("ClusterTrustBundle %q holds no certificates: %.80q", name, ctb.Spec.TrustBundle) + } + return ctb.Spec.TrustBundle +} diff --git a/internal/egresspolicy/egresspolicy_test.go b/internal/egresspolicy/egresspolicy_test.go index 9158bc72ef..c26994db72 100644 --- a/internal/egresspolicy/egresspolicy_test.go +++ b/internal/egresspolicy/egresspolicy_test.go @@ -23,19 +23,19 @@ import ( ) func httpRule(patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Http: &ateapipb.HTTPRule{Hostnames: patterns}} + return &ateapipb.EgressRule{Http: &ateapipb.HttpRule{Hostnames: patterns}} } func httpRuleOnPorts(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Http: &ateapipb.HTTPRule{Hostnames: patterns, Ports: ports}} + return &ateapipb.EgressRule{Http: &ateapipb.HttpRule{Hostnames: patterns, Ports: ports}} } func httpsRule(patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Https: &ateapipb.HTTPSRule{Hostnames: patterns}} + return &ateapipb.EgressRule{Https: &ateapipb.HttpsRule{Hostnames: patterns}} } func passthroughRule(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{TlsPassthrough: &ateapipb.TLSPassthroughRule{Hostnames: patterns, Ports: ports}} + return &ateapipb.EgressRule{TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: patterns, Ports: ports}} } func ports(numbers ...int32) *ateapipb.Ports { return &ateapipb.Ports{Numbers: numbers} } @@ -214,7 +214,7 @@ func TestEvaluateRequest(t *testing.T) { CredentialUri: "ate-secret://k8s/default/token", }}, } - withEffects := &ateapipb.EgressRule{Https: &ateapipb.HTTPSRule{ + withEffects := &ateapipb.EgressRule{Https: &ateapipb.HttpsRule{ Hostnames: []string{"api.example.com"}, Effects: effects, }} @@ -400,7 +400,7 @@ func TestEvaluateRequest(t *testing.T) { } func httpsRuleOnPorts(ports *ateapipb.Ports, patterns ...string) *ateapipb.EgressRule { - return &ateapipb.EgressRule{Https: &ateapipb.HTTPSRule{Hostnames: patterns, Ports: ports}} + return &ateapipb.EgressRule{Https: &ateapipb.HttpsRule{Hostnames: patterns, Ports: ports}} } func TestSNIRules(t *testing.T) { diff --git a/internal/hardware/hardware.go b/internal/hardware/hardware.go index 4fab18123c..99930e506e 100644 --- a/internal/hardware/hardware.go +++ b/internal/hardware/hardware.go @@ -30,9 +30,13 @@ const ( // AttrArchitecture is the CPU architecture ("amd64", "arm64", etc.). AttrArchitecture = "architecture" - // TODO: Add AttrCPUFeatures ("cpu_features"), AttrGVisorAssetHash - // ("gvisor_asset_hash"), and other compatibility attributes as snapshot - // compatibility expands. + // AttrSandboxRuntimeHash identifies the sandbox runtime binaries: a + // sha256 over a SandboxConfig version's asset digests and pause image + // digest, computed by atelet. + AttrSandboxRuntimeHash = "sandbox_runtime_hash" + + // TODO: Add AttrCPUFeatures ("cpu_features") and other compatibility + // attributes as snapshot compatibility expands. ) // ProbeHost inspects the current host and returns its v1 VersionedSandboxCompat diff --git a/internal/imagecache/unpack.go b/internal/imagecache/unpack.go index 3d581bd805..81e64cf6c5 100644 --- a/internal/imagecache/unpack.go +++ b/internal/imagecache/unpack.go @@ -26,6 +26,7 @@ import ( "path/filepath" "sort" "strings" + "time" ) const ( @@ -107,6 +108,10 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo // CAP_DAC_OVERRIDE. Keyed by name so a repeated dir entry's last mode wins. dirModes := map[string]os.FileMode{} + // Records each directory's tar mtime to restore after extraction, since + // creating child entries updates the parent directory's mtime. + dirModTimes := map[string]time.Time{} + // Ancestors an entry needed vs. directories the tar declared: the // difference is recorded as ImplicitDirs (attrs fabricated, see the // whiteoutSet field doc). Whiteout/opaque markers count too — their @@ -199,6 +204,9 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo if closeErr != nil { return nil, fmt.Errorf("while closing file %q: %w", name, closeErr) } + if err := restoreModTime(root, name, hdr.ModTime); err != nil { + return nil, err + } case tar.TypeDir: // Create owner-writable so children can be written even when the image @@ -213,6 +221,7 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo return nil, fmt.Errorf("while creating directory=%q, mode=%v: %w", name, mode, err) } dirModes[name] = mode + dirModTimes[name] = hdr.ModTime declared[name] = true delete(implicit, name) @@ -240,6 +249,8 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo if err := root.Symlink(hdr.Linkname, name); err != nil { return nil, fmt.Errorf("while creating symlink src=%q target=%q: %w", name, hdr.Linkname, err) } + // The symlink's own mtime is not restored: Chtimes follows the link, + // so it would stamp the target instead. case tar.TypeLink: linkname, linkSkip, err := validateTarName(hdr.Linkname) @@ -260,6 +271,8 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo if err := root.Link(linkname, name); err != nil { return nil, fmt.Errorf("while creating hardlink src=%q target=%q: %w", name, linkname, err) } + // No mtime to restore: the link shares the target's inode, which + // already has the target entry's mtime. default: tfStr := string([]byte{hdr.Typeflag}) @@ -268,7 +281,7 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo } } - // Restore the image's intended directory modes now that every child exists. + // Restore the image's intended directory modes and mtime now that every child exists. // Deepest paths first: a child's path is always longer than its parent's, so // length-descending order guarantees a directory is restored before any of its // ancestors — restoring a parent to a non-traversable mode then can't block @@ -282,6 +295,9 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo if err := root.Chmod(name, dirModes[name]); err != nil { return nil, fmt.Errorf("while restoring mode %v on directory %q: %w", dirModes[name], name, err) } + if err := restoreModTime(root, name, dirModTimes[name]); err != nil { + return nil, err + } } // Keep only implicit candidates that survive in the tree as directories @@ -313,3 +329,16 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo return wh, nil } + +// restoreModTime sets the atime and mtime of an unpacked path to the mtime its +// tar entry records, so that every node unpacks a layer to the same tree (see +// the deterministic-unpack note in cmd/ateom-microvm/restore.go). +func restoreModTime(root *os.Root, name string, modTime time.Time) error { + if modTime.IsZero() { + return nil + } + if err := root.Chtimes(name, modTime, modTime); err != nil { + return fmt.Errorf("while restoring mtime on %q: %w", name, err) + } + return nil +} diff --git a/internal/imagecache/unpack_test.go b/internal/imagecache/unpack_test.go index 338b210970..d6f8c375a9 100644 --- a/internal/imagecache/unpack_test.go +++ b/internal/imagecache/unpack_test.go @@ -23,6 +23,7 @@ import ( "slices" "strings" "testing" + "time" ) type tarEntry struct { @@ -31,6 +32,7 @@ type tarEntry struct { mode int64 body string linkname string + modTime time.Time } func defaultMode(typeflag byte) int64 { @@ -59,6 +61,7 @@ func buildTar(t *testing.T, entries []tarEntry) []byte { Mode: mode, Size: int64(len(e.body)), Linkname: e.linkname, + ModTime: e.modTime, } if err := tw.WriteHeader(hdr); err != nil { t.Fatalf("tar.WriteHeader(%+v): %v", hdr, err) @@ -482,6 +485,71 @@ func TestUnpackLayer_ReadOnlyDir(t *testing.T) { } } +// Files and directories keep the mtimes their tar entries record, so Python can +// trust the .pyc files shipped in an image. A directory keeps its mtime although +// its children are written after it, read-only directories included, and a link +// entry never re-stamps its target. +func TestUnpackLayer_RestoresModTimes(t *testing.T) { + var ( + fileTime = time.Date(2025, 9, 10, 9, 37, 24, 0, time.UTC) + dirTime = time.Date(2025, 9, 10, 9, 30, 0, 0, time.UTC) + subTime = time.Date(2024, 2, 29, 12, 0, 0, 0, time.UTC) + staleTime = time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC) + linkTime = time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + ) + entries := []tarEntry{ + {name: "app/", typeflag: tar.TypeDir, modTime: staleTime}, + {name: "app/main.py", typeflag: tar.TypeReg, body: "print()\n", modTime: fileTime}, + {name: "app/sub/", typeflag: tar.TypeDir, modTime: subTime}, + {name: "app/sub/mod.py", typeflag: tar.TypeReg, body: "x = 1\n", modTime: fileTime}, + // Chtimes would follow a symlink, and a hardlink shares the target's + // inode: either way, the target must keep its own mtime. + {name: "app/sym", typeflag: tar.TypeSymlink, linkname: "main.py", modTime: linkTime}, + {name: "app/hard", typeflag: tar.TypeLink, linkname: "app/main.py", modTime: linkTime}, + // A repeated dir entry: the last one's mtime wins, as its mode does. + {name: "app/", typeflag: tar.TypeDir, modTime: dirTime}, + {name: "ro/", typeflag: tar.TypeDir, mode: 0o555, modTime: dirTime}, + {name: "ro/bin", typeflag: tar.TypeReg, mode: 0o755, body: "bin", modTime: fileTime}, + } + dir, _, err := runUnpack(t, entries) + if err != nil { + t.Fatalf("unpackLayer: %v", err) + } + // Plain os.RemoveAll (t.TempDir's cleanup) can't delete under a 0555 dir. + t.Cleanup(func() { + if err := RemoveAllWritable(filepath.Join(dir, "ro")); err != nil { + t.Errorf("RemoveAllWritable(ro): %v", err) + } + }) + + for _, tc := range []struct { + path string + want time.Time + }{ + {path: "app/main.py", want: fileTime}, + {path: "app/sub/mod.py", want: fileTime}, + {path: "app/hard", want: fileTime}, + {path: "app/sub", want: subTime}, + {path: "app", want: dirTime}, + {path: "ro/bin", want: fileTime}, + {path: "ro", want: dirTime}, + } { + fi, err := os.Lstat(filepath.Join(dir, tc.path)) + if err != nil { + t.Errorf("lstat %s: %v", tc.path, err) + continue + } + if got := fi.ModTime(); !got.Equal(tc.want) { + t.Errorf("%s mtime = %v, want %v", tc.path, got.UTC(), tc.want) + } + } + if fi, err := os.Stat(filepath.Join(dir, "ro")); err != nil { + t.Errorf("stat ro: %v", err) + } else if fi.Mode().Perm() != 0o555 { + t.Errorf("ro mode = %v, want the image's 0555 preserved", fi.Mode().Perm()) + } +} + func TestUnpackLayer_PathTraversal(t *testing.T) { tests := []struct { name string diff --git a/internal/installdefaults/installdefaults.go b/internal/installdefaults/installdefaults.go index 31baa189f8..d0e6980c4e 100644 --- a/internal/installdefaults/installdefaults.go +++ b/internal/installdefaults/installdefaults.go @@ -40,14 +40,16 @@ const ( // AteletTrustDomain and the ServiceAccount constants are the trust-domain // and service-account segments of the SPIFFE IDs that atelet, - // atenet-router and atenet-egress Pod certificates carry, as minted by the - // podidentity signer (cmd/podcertcontroller/internal/podidentitysigner). + // atenet-router, atenet-egress and ate-controller Pod certificates carry, + // as minted by the podidentity signer + // (cmd/podcertcontroller/internal/podidentitysigner). // The namespace segment is the namespace they run in, which callers // resolve themselves rather than assume. - AteletTrustDomain = "cluster.local" - AteletServiceAccount = "atelet" - RouterServiceAccount = "atenet-router" - EgressServiceAccount = "atenet-egress" + AteletTrustDomain = "cluster.local" + AteletServiceAccount = "atelet" + RouterServiceAccount = "atenet-router" + EgressServiceAccount = "atenet-egress" + ControllerServiceAccount = "ate-controller" // PodNamespaceEnv is the conventional env var name for the namespace // a pod is running in, exposed via Kubernetes' downward API. @@ -99,3 +101,8 @@ func RouterSPIFFEID(namespace string) string { func EgressSPIFFEID(namespace string) string { return SPIFFEID(namespace, EgressServiceAccount) } + +// ControllerSPIFFEID returns the SPIFFE ID ate-controller presents when it runs in namespace. +func ControllerSPIFFEID(namespace string) string { + return SPIFFEID(namespace, ControllerServiceAccount) +} diff --git a/internal/installdefaults/installdefaults_test.go b/internal/installdefaults/installdefaults_test.go index 0b858f694b..b6c89860a0 100644 --- a/internal/installdefaults/installdefaults_test.go +++ b/internal/installdefaults/installdefaults_test.go @@ -64,6 +64,13 @@ func TestEgressSPIFFEID(t *testing.T) { } } +func TestControllerSPIFFEID(t *testing.T) { + const want = "spiffe://cluster.local/ns/ate-system/sa/ate-controller" + if got := ControllerSPIFFEID(SystemNamespace); got != want { + t.Errorf("ControllerSPIFFEID(%q) = %q, want %q", SystemNamespace, got, want) + } +} + func TestActorJWTIssuer(t *testing.T) { tests := []struct { name string diff --git a/internal/nodepath/nodepath.go b/internal/nodepath/nodepath.go index 5a86208f07..f03b2f7f29 100644 --- a/internal/nodepath/nodepath.go +++ b/internal/nodepath/nodepath.go @@ -56,9 +56,11 @@ func AteletOTLPSocketPath() string { ) } -// AteomsDir is the parent of every per-ateom directory. Each ateom creates -// AteomPath(podUID) under it when it boots, so listing this directory is how a -// scraper with no prior knowledge discovers the node's ateoms. +// AteomsDir is the parent of every per-ateom directory, so listing it is how +// a scraper with no prior knowledge discovers the node's ateoms. atelet +// creates AteomPath(podUID) once it writes the pod's sandbox versions file and +// removes it when the pod is deleted; the ateom also MkdirAlls it at boot. A +// directory may exist before its ateom listens on its socket. func AteomsDir() string { return filepath.Join(BasePath, "ateoms") } @@ -74,6 +76,12 @@ func AteomSocketPath(podUID string) string { ) } +// SandboxVersionsFileName is the base name of the file in AteomPath(podUID) +// where atelet keeps the sandbox versions of a worker's pool, a +// sandboxversions.File. A runtime it lists may not have its assets on the +// node yet. +const SandboxVersionsFileName = "sandbox-versions.json" + // ActorNetNSName names an actor's sandbox network namespace. func ActorNetNSName(actorUID string) string { return "ateom-actor:" + actorUID diff --git a/internal/portforward/portforward.go b/internal/portforward/portforward.go index 7fa750b251..02f65d5a5f 100644 --- a/internal/portforward/portforward.go +++ b/internal/portforward/portforward.go @@ -48,7 +48,18 @@ func ServicePortForward(ctx context.Context, config *rest.Config, clientset kube if err != nil { return 0, nil, err } - return podPortForward(ctx, config, clientset, namespace, pod.Name, targetPort) + return PodPortForward(ctx, config, clientset, namespace, pod.Name, targetPort) +} + +// ServicePodPortForward forwards a random local port straight to +// containerPort on a ready pod behind the named Service, for ports the +// Service does not publish. Otherwise it behaves like ServicePortForward. +func ServicePodPortForward(ctx context.Context, config *rest.Config, clientset kubernetes.Interface, namespace, service string, containerPort int32) (int, func(), error) { + pod, _, err := firstReadyPodForService(ctx, clientset, namespace, service) + if err != nil { + return 0, nil, err + } + return PodPortForward(ctx, config, clientset, namespace, pod.Name, containerPort) } func firstReadyPodForService(ctx context.Context, clientset kubernetes.Interface, namespace, service string) (*corev1.Pod, *corev1.Service, error) { @@ -102,7 +113,7 @@ func resolveTargetPort(svc *corev1.Service, pod *corev1.Pod, servicePort int32) return 0, fmt.Errorf("service %s has no port %d", svc.Name, servicePort) } -func podPortForward(ctx context.Context, config *rest.Config, clientset kubernetes.Interface, namespace, podName string, targetPort int32) (int, func(), error) { +func PodPortForward(ctx context.Context, config *rest.Config, clientset kubernetes.Interface, namespace, podName string, targetPort int32) (int, func(), error) { req := clientset.CoreV1().RESTClient().Post(). Resource("pods"). Namespace(namespace). diff --git a/internal/proto/ateletpb/atelet.pb.go b/internal/proto/ateletpb/atelet.pb.go index fea6efe1c3..16b2fc7103 100644 --- a/internal/proto/ateletpb/atelet.pb.go +++ b/internal/proto/ateletpb/atelet.pb.go @@ -23,6 +23,7 @@ package ateletpb import ( protoreflect "google.golang.org/protobuf/reflect/protoreflect" protoimpl "google.golang.org/protobuf/runtime/protoimpl" + timestamppb "google.golang.org/protobuf/types/known/timestamppb" reflect "reflect" sync "sync" unsafe "unsafe" @@ -42,7 +43,7 @@ const ( ActorMetadataField_ACTOR_METADATA_FIELD_UNSPECIFIED ActorMetadataField = 0 ActorMetadataField_ACTOR_METADATA_FIELD_NAME ActorMetadataField = 1 ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE ActorMetadataField = 2 - ActorMetadataField_ACTOR_METADATA_FIELD_UID ActorMetadataField = 3 + ActorMetadataField_ACTOR_METADATA_FIELD_UID ActorMetadataField = 3 // Keep this in sync with the maximums on fields of this type. ) // Enum value maps for ActorMetadataField. @@ -88,58 +89,6 @@ func (ActorMetadataField) EnumDescriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{0} } -type CheckpointType int32 - -const ( - // Should never happen - CheckpointType_CHECKPOINT_TYPE_UNSPECIFIED CheckpointType = 0 - // Save snapshot only in local filesystem - CheckpointType_CHECKPOINT_TYPE_LOCAL CheckpointType = 1 - // Save snapshot to object storage - CheckpointType_CHECKPOINT_TYPE_EXTERNAL CheckpointType = 2 -) - -// Enum value maps for CheckpointType. -var ( - CheckpointType_name = map[int32]string{ - 0: "CHECKPOINT_TYPE_UNSPECIFIED", - 1: "CHECKPOINT_TYPE_LOCAL", - 2: "CHECKPOINT_TYPE_EXTERNAL", - } - CheckpointType_value = map[string]int32{ - "CHECKPOINT_TYPE_UNSPECIFIED": 0, - "CHECKPOINT_TYPE_LOCAL": 1, - "CHECKPOINT_TYPE_EXTERNAL": 2, - } -) - -func (x CheckpointType) Enum() *CheckpointType { - p := new(CheckpointType) - *p = x - return p -} - -func (x CheckpointType) String() string { - return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) -} - -func (CheckpointType) Descriptor() protoreflect.EnumDescriptor { - return file_atelet_proto_enumTypes[1].Descriptor() -} - -func (CheckpointType) Type() protoreflect.EnumType { - return &file_atelet_proto_enumTypes[1] -} - -func (x CheckpointType) Number() protoreflect.EnumNumber { - return protoreflect.EnumNumber(x) -} - -// Deprecated: Use CheckpointType.Descriptor instead. -func (CheckpointType) EnumDescriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{1} -} - // SnapshotFidelity mirrors the public ateapi enum: the state layers a // snapshot holds, lowest to highest, each including the ones below it. type SnapshotFidelity int32 @@ -156,7 +105,7 @@ const ( SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS SnapshotFidelity = 2 // Capture process memory plus the full filesystem delta on top of the OCI // image (including any attached DurableDir volumes). - SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY SnapshotFidelity = 3 + SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY SnapshotFidelity = 3 // Keep this in sync with the maximums on fields of this type. ) // Enum value maps for SnapshotFidelity. @@ -186,11 +135,11 @@ func (x SnapshotFidelity) String() string { } func (SnapshotFidelity) Descriptor() protoreflect.EnumDescriptor { - return file_atelet_proto_enumTypes[2].Descriptor() + return file_atelet_proto_enumTypes[1].Descriptor() } func (SnapshotFidelity) Type() protoreflect.EnumType { - return &file_atelet_proto_enumTypes[2] + return &file_atelet_proto_enumTypes[1] } func (x SnapshotFidelity) Number() protoreflect.EnumNumber { @@ -199,6 +148,69 @@ func (x SnapshotFidelity) Number() protoreflect.EnumNumber { // Deprecated: Use SnapshotFidelity.Descriptor instead. func (SnapshotFidelity) EnumDescriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{1} +} + +// UploadMode is when a checkpoint's snapshot is uploaded. +type UploadMode int32 + +const ( + UploadMode_UPLOAD_MODE_UNSPECIFIED UploadMode = 0 + // Keep the snapshot locally and upload it on atelet's schedule: at most + // for atelet's upload-delay after the oldest snapshot of the actor + // that is not yet durable was taken. Snapshots superseded before then + // are never uploaded. + // + // Once the actor has stayed suspended on this node for + // atelet's detach-delay, atelet detaches it (deletes all the local snapshots). + // + // TODO: Temporarily the deferred upload and detach are unimplemented. Only a local + // copy is taken. + UploadMode_UPLOAD_MODE_DEFERRED UploadMode = 1 + // Upload the snapshot synchronously before Checkpoint returns, while keeping the local copy. + // + // TODO: Temporarily this deletes the local snapshot after the upload. + UploadMode_UPLOAD_MODE_IMMEDIATE UploadMode = 2 // Keep this in sync with the maximums on fields of this type. +) + +// Enum value maps for UploadMode. +var ( + UploadMode_name = map[int32]string{ + 0: "UPLOAD_MODE_UNSPECIFIED", + 1: "UPLOAD_MODE_DEFERRED", + 2: "UPLOAD_MODE_IMMEDIATE", + } + UploadMode_value = map[string]int32{ + "UPLOAD_MODE_UNSPECIFIED": 0, + "UPLOAD_MODE_DEFERRED": 1, + "UPLOAD_MODE_IMMEDIATE": 2, + } +) + +func (x UploadMode) Enum() *UploadMode { + p := new(UploadMode) + *p = x + return p +} + +func (x UploadMode) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (UploadMode) Descriptor() protoreflect.EnumDescriptor { + return file_atelet_proto_enumTypes[2].Descriptor() +} + +func (UploadMode) Type() protoreflect.EnumType { + return &file_atelet_proto_enumTypes[2] +} + +func (x UploadMode) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use UploadMode.Descriptor instead. +func (UploadMode) EnumDescriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{2} } @@ -927,16 +939,36 @@ func (x *MintActorCertificateResponse) GetActorCertificates() [][]byte { type TerminateRequest struct { state protoimpl.MessageState `protogen:"open.v1"` - // If unset, the atelet will just cleanup node resources for the actor. - WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` - Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` - ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` - ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` - ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` - ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` - Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // The UID of the worker pod whose ateom hosts the actor. If unset, atelet + // only cleans up the actor's node resources; it does not reach an ateom. + // + // +k8s:optional + // +k8s:format=k8s-uuid + WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` + // +k8s:required + // +k8s:format=k8s-uuid + ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` + // A nil spec is tolerated: the delete flow terminates template-less + // actors with a fallback spec. + // + // +k8s:optional + Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *TerminateRequest) Reset() { @@ -1055,24 +1087,51 @@ func (*TerminateResponse) Descriptor() ([]byte, []int) { } type RunRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` - Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` - ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` - ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` - ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` - ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` - Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // The UID of the worker pod whose ateom hosts the actor. + // + // +k8s:required + // +k8s:format=k8s-uuid + WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` + // +k8s:required + // +k8s:format=k8s-uuid + ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` + // +k8s:required + Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` // The sandbox binaries to use for booting this actor from scratch. atelet // fetches the relevant assets and records them with the actor's on-node state // so a later Checkpoint can pin the same version into the snapshot manifest. + // + // +k8s:required SandboxAssets *SandboxAssets `protobuf:"bytes,8,opt,name=sandbox_assets,json=sandboxAssets,proto3" json:"sandbox_assets,omitempty"` // When absent the actor has no egress: its TCP is captured and refused. + // + // +k8s:optional EgressGateway *EgressGateway `protobuf:"bytes,9,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"` // The actor's declared size, from the ActorTemplate's resource limits. atelet // passes these through to the sandbox so it is sized to the actor (not the // whole host or worker pod). Zero means "unset": keep the runtime default. - CpuMilli int64 `protobuf:"varint,10,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores + CpuMilli int64 `protobuf:"varint,10,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). + // +k8s:optional + // +k8s:minimum=0 MemoryBytes int64 `protobuf:"varint,11,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1190,6 +1249,10 @@ type EgressGateway struct { state protoimpl.MessageState `protogen:"open.v1"` // address is required and identifies the remote gateway as an IP address or // DNS name followed by its port. + // + // +k8s:required + // +k8s:maxLength=261 # a bracketed or 253-character host, ':' and a 5-digit port + // +k8s:customValidation # host:port shape Address string `protobuf:"bytes,1,opt,name=address,proto3" json:"address,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1241,8 +1304,15 @@ type AssetFile struct { // the plugin cannot serve it. // Masked in logged requests, since a signed URL carries its credential in // the query. + // + // +k8s:required + // +k8s:maxLength=2048 # matches the SandboxConfig asset url's bound Url string `protobuf:"bytes,1,opt,name=url,proto3" json:"url,omitempty"` // Lower-case hex SHA256; names the cached file and verifies its integrity. + // + // +k8s:required + // +k8s:maxLength=64 + // +k8s:customValidation # 64 hex characters Sha256 string `protobuf:"bytes,2,opt,name=sha256,proto3" json:"sha256,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1295,8 +1365,11 @@ func (x *AssetFile) GetSha256() string { // ArchAssets is the set of assets for a single architecture, keyed by asset // name (a wrapper message because proto map values cannot themselves be maps). type ArchAssets struct { - state protoimpl.MessageState `protogen:"open.v1"` - Files map[string]*AssetFile `protobuf:"bytes,1,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // asset name -> file + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:required + // +k8s:maxProperties=16 + // +k8s:eachKey=+k8s:maxLength=64 + Files map[string]*AssetFile `protobuf:"bytes,1,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // asset name -> file unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1344,14 +1417,27 @@ func (x *ArchAssets) GetFiles() map[string]*AssetFile { // atelet's backend code interprets the asset names (gVisor expects "gvisor", // the release tarball; legacy "runsc", a bare binary, is still accepted). type SandboxAssets struct { - state protoimpl.MessageState `protogen:"open.v1"` - SandboxClass string `protobuf:"bytes,1,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // e.g. "gvisor" - Assets map[string]*ArchAssets `protobuf:"bytes,2,rep,name=assets,proto3" json:"assets,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // arch -> {name -> file} + state protoimpl.MessageState `protogen:"open.v1"` + // TODO: send this as an enum, like ateapi's and ateom's SandboxClass, so + // the tags can bound it and the custom validation goes away. + // + // +k8s:required + // +k8s:maxLength=63 # guardrail; the class names are a few characters + // +k8s:customValidation # gvisor or microvm + SandboxClass string `protobuf:"bytes,1,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // e.g. "gvisor" + // +k8s:required + // +k8s:maxProperties=8 # one entry per architecture + // +k8s:eachKey=+k8s:maxLength=16 + Assets map[string]*ArchAssets `protobuf:"bytes,2,rep,name=assets,proto3" json:"assets,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // arch -> {name -> file} // pause_image is the image for the sandbox's root container. Like the // binaries above it is sandbox configuration, not workload configuration, // and atelet pins it into the snapshot manifest so a restore rebuilds the // sandbox from the same image. Empty for sandboxes without a pause // container (microvm). + // + // +k8s:optional + // +k8s:maxLength=512 + // +k8s:customValidation # must be a well-formed image reference, pinned by digest PauseImage string `protobuf:"bytes,3,opt,name=pause_image,json=pauseImage,proto3" json:"pause_image,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1409,10 +1495,20 @@ func (x *SandboxAssets) GetPauseImage() string { } // WorkloadSpec parallels Pod, but with far fewer configurable fields. +// +// +k8s:customValidation # every volume mount names a declared volume type WorkloadSpec struct { - state protoimpl.MessageState `protogen:"open.v1"` - Containers []*Container `protobuf:"bytes,1,rep,name=containers,proto3" json:"containers,omitempty"` - Volumes []*Volume `protobuf:"bytes,2,rep,name=volumes,proto3" json:"volumes,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:optional + // +k8s:maxItems=10 # matches the template's containers bound + // +k8s:listType=map + // +k8s:listMapKey=name + Containers []*Container `protobuf:"bytes,1,rep,name=containers,proto3" json:"containers,omitempty"` + // +k8s:optional + // +k8s:maxItems=32 # matches the template's volumes bound + // +k8s:listType=map + // +k8s:listMapKey=name + Volumes []*Volume `protobuf:"bytes,2,rep,name=volumes,proto3" json:"volumes,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1498,12 +1594,29 @@ func (*DurableDirVolume) Descriptor() ([]byte, []int) { } type ExternalVolumeSource struct { - state protoimpl.MessageState `protogen:"open.v1"` - StorageVolumeId string `protobuf:"bytes,1,opt,name=storage_volume_id,json=storageVolumeId,proto3" json:"storage_volume_id,omitempty"` - VolumeType string `protobuf:"bytes,2,opt,name=volume_type,json=volumeType,proto3" json:"volume_type,omitempty"` - VolumeContext map[string]string `protobuf:"bytes,3,rep,name=volume_context,json=volumeContext,proto3" json:"volume_context,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` + state protoimpl.MessageState `protogen:"open.v1"` + // Required, unlike the actor status counterpart: the control plane only + // sends volumes that finished provisioning. + // + // +k8s:required + // +k8s:maxLength=256 # matches ExternalVolume.storage_volume_id's bound + // +k8s:customValidation # no control characters + StorageVolumeId string `protobuf:"bytes,1,opt,name=storage_volume_id,json=storageVolumeId,proto3" json:"storage_volume_id,omitempty"` + // +k8s:optional + // +k8s:maxLength=253 # matches ExternalVolume.volume_type's bound + // +k8s:customValidation # optional "substrate.io/" prefix + DNS subdomain + VolumeType string `protobuf:"bytes,2,opt,name=volume_type,json=volumeType,proto3" json:"volume_type,omitempty"` + // volume_context is the driver's provisioning metadata. Its entries are the + // driver's own, so the map is only bounded, not validated. + // + // +k8s:optional + // +k8s:customValidation # at most 40 KiB in total, ten times the CSI limit for map fields + VolumeContext map[string]string `protobuf:"bytes,3,rep,name=volume_context,json=volumeContext,proto3" json:"volume_context,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // publish_context is the driver's attachment metadata for this node, which - // the node plugin needs to complete the mount. + // the node plugin needs to complete the mount. Bounded like volume_context. + // + // +k8s:optional + // +k8s:customValidation # at most 40 KiB in total, ten times the CSI limit for map fields PublishContext map[string]string `protobuf:"bytes,4,rep,name=publish_context,json=publishContext,proto3" json:"publish_context,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1568,8 +1681,13 @@ func (x *ExternalVolumeSource) GetPublishContext() map[string]string { } type ImageVolumeSource struct { - state protoimpl.MessageState `protogen:"open.v1"` - Reference string `protobuf:"bytes,1,opt,name=reference,proto3" json:"reference,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // reference is the OCI image reference, pinned by digest. + // + // +k8s:required + // +k8s:maxLength=512 # matches the template ImageVolumeSource.reference's bound + // +k8s:customValidation # must be a well-formed image reference, pinned by digest + Reference string `protobuf:"bytes,1,opt,name=reference,proto3" json:"reference,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1614,9 +1732,19 @@ func (x *ImageVolumeSource) GetReference() string { // ActorMetadataItem projects one actor identity field to one file at the // given path, relative to the root of the enclosing system-info volume. type ActorMetadataItem struct { - state protoimpl.MessageState `protogen:"open.v1"` - Field ActorMetadataField `protobuf:"varint,1,opt,name=field,proto3,enum=atelet.ActorMetadataField" json:"field,omitempty"` - Path string `protobuf:"bytes,2,opt,name=path,proto3" json:"path,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=3 # keep this in sync with the ActorMetadataField enum + Field ActorMetadataField `protobuf:"varint,1,opt,name=field,proto3,enum=atelet.ActorMetadataField" json:"field,omitempty"` + // path must be a clean relative Unix path; the rule is shared with the + // control plane through internal/resources. + // + // +k8s:required + // +k8s:minLength=1 + // +k8s:maxLength=255 + // +k8s:customValidation # clean relative path + Path string `protobuf:"bytes,2,opt,name=path,proto3" json:"path,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1668,8 +1796,16 @@ func (x *ActorMetadataItem) GetPath() string { // ActorMetadataDataSource projects the actor's identity fields to files, one // per item. Values are written raw with no trailing newline. type ActorMetadataDataSource struct { - state protoimpl.MessageState `protogen:"open.v1"` - Items []*ActorMetadataItem `protobuf:"bytes,1,rep,name=items,proto3" json:"items,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // items must not project the same field twice. + // + // +k8s:required + // +k8s:minItems=1 + // +k8s:maxItems=8 # matches the template's items bound + // +k8s:listType=atomic + // +k8s:unique=map + // +k8s:listMapKey=field + Items []*ActorMetadataItem `protobuf:"bytes,1,rep,name=items,proto3" json:"items,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1713,9 +1849,22 @@ func (x *ActorMetadataDataSource) GetItems() []*ActorMetadataItem { // TrustBundleDataSource corresponds to TrustBundleDataSource in ateapi.proto type TrustBundleDataSource struct { - state protoimpl.MessageState `protogen:"open.v1"` - Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"` - Names []string `protobuf:"bytes,3,rep,name=names,proto3" json:"names,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // path must be a clean relative Unix path; the rule is shared with the + // control plane through internal/resources. + // + // +k8s:required + // +k8s:minLength=1 + // +k8s:maxLength=255 + // +k8s:customValidation # clean relative path + Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"` + // +k8s:required + // +k8s:minItems=1 + // +k8s:maxItems=8 # matches the template's names bound + // +k8s:listType=set + // +k8s:eachVal=+k8s:minLength=1 + // +k8s:eachVal=+k8s:maxLength=253 # matches the template's bundle-name bound + Names []string `protobuf:"bytes,3,rep,name=names,proto3" json:"names,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1764,13 +1913,18 @@ func (x *TrustBundleDataSource) GetNames() []string { return nil } +// SystemInfoDataSource selects exactly one projection to place in the +// volume. type SystemInfoDataSource struct { state protoimpl.MessageState `protogen:"open.v1"` - // Types that are valid to be assigned to DataSource: + // Exactly one of actor_metadata / trust_bundle must be set. // - // *SystemInfoDataSource_ActorMetadata - // *SystemInfoDataSource_TrustBundle - DataSource isSystemInfoDataSource_DataSource `protobuf_oneof:"data_source"` + // +k8s:optional + // +k8s:unionMember + ActorMetadata *ActorMetadataDataSource `protobuf:"bytes,1,opt,name=actor_metadata,json=actorMetadata,proto3" json:"actor_metadata,omitempty"` + // +k8s:optional + // +k8s:unionMember + TrustBundle *TrustBundleDataSource `protobuf:"bytes,2,opt,name=trust_bundle,json=trustBundle,proto3" json:"trust_bundle,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1805,52 +1959,29 @@ func (*SystemInfoDataSource) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{26} } -func (x *SystemInfoDataSource) GetDataSource() isSystemInfoDataSource_DataSource { - if x != nil { - return x.DataSource - } - return nil -} - func (x *SystemInfoDataSource) GetActorMetadata() *ActorMetadataDataSource { if x != nil { - if x, ok := x.DataSource.(*SystemInfoDataSource_ActorMetadata); ok { - return x.ActorMetadata - } + return x.ActorMetadata } return nil } func (x *SystemInfoDataSource) GetTrustBundle() *TrustBundleDataSource { if x != nil { - if x, ok := x.DataSource.(*SystemInfoDataSource_TrustBundle); ok { - return x.TrustBundle - } + return x.TrustBundle } return nil } -type isSystemInfoDataSource_DataSource interface { - isSystemInfoDataSource_DataSource() -} - -type SystemInfoDataSource_ActorMetadata struct { - ActorMetadata *ActorMetadataDataSource `protobuf:"bytes,1,opt,name=actor_metadata,json=actorMetadata,proto3,oneof"` -} - -type SystemInfoDataSource_TrustBundle struct { - TrustBundle *TrustBundleDataSource `protobuf:"bytes,2,opt,name=trust_bundle,json=trustBundle,proto3,oneof"` -} - -func (*SystemInfoDataSource_ActorMetadata) isSystemInfoDataSource_DataSource() {} - -func (*SystemInfoDataSource_TrustBundle) isSystemInfoDataSource_DataSource() {} - // SystemInfoVolume is a read-only volume whose files are generated by atelet // on every Run/Restore, so they carry the values of the actor actually being // started, whatever checkpointed state it boots from. type SystemInfoVolume struct { - state protoimpl.MessageState `protogen:"open.v1"` + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:optional + // +k8s:maxItems=8 # matches the template's data_sources bound + // +k8s:listType=atomic + // +k8s:customValidation # at most one actor_metadata; paths unique across entries DataSources []*SystemInfoDataSource `protobuf:"bytes,1,rep,name=data_sources,json=dataSources,proto3" json:"data_sources,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1893,16 +2024,26 @@ func (x *SystemInfoVolume) GetDataSources() []*SystemInfoDataSource { return nil } +// Volume names one volume and selects exactly one source for it. type Volume struct { state protoimpl.MessageState `protogen:"open.v1"` - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Types that are valid to be assigned to Source: - // - // *Volume_DurableDir - // *Volume_External - // *Volume_SystemInfo - // *Volume_Image - Source isVolume_Source `protobuf_oneof:"source"` + // +k8s:required + // +k8s:format=k8s-short-name + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // Exactly one of durable_dir / external / system_info / image must be set. + // + // +k8s:optional + // +k8s:unionMember + DurableDir *DurableDirVolume `protobuf:"bytes,2,opt,name=durable_dir,json=durableDir,proto3" json:"durable_dir,omitempty"` + // +k8s:optional + // +k8s:unionMember + External *ExternalVolumeSource `protobuf:"bytes,3,opt,name=external,proto3" json:"external,omitempty"` + // +k8s:optional + // +k8s:unionMember + SystemInfo *SystemInfoVolume `protobuf:"bytes,4,opt,name=system_info,json=systemInfo,proto3" json:"system_info,omitempty"` + // +k8s:optional + // +k8s:unionMember + Image *ImageVolumeSource `protobuf:"bytes,5,opt,name=image,proto3" json:"image,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -1944,81 +2085,47 @@ func (x *Volume) GetName() string { return "" } -func (x *Volume) GetSource() isVolume_Source { - if x != nil { - return x.Source - } - return nil -} - func (x *Volume) GetDurableDir() *DurableDirVolume { if x != nil { - if x, ok := x.Source.(*Volume_DurableDir); ok { - return x.DurableDir - } + return x.DurableDir } return nil } func (x *Volume) GetExternal() *ExternalVolumeSource { if x != nil { - if x, ok := x.Source.(*Volume_External); ok { - return x.External - } + return x.External } return nil } func (x *Volume) GetSystemInfo() *SystemInfoVolume { if x != nil { - if x, ok := x.Source.(*Volume_SystemInfo); ok { - return x.SystemInfo - } + return x.SystemInfo } return nil } func (x *Volume) GetImage() *ImageVolumeSource { if x != nil { - if x, ok := x.Source.(*Volume_Image); ok { - return x.Image - } + return x.Image } return nil } -type isVolume_Source interface { - isVolume_Source() -} - -type Volume_DurableDir struct { - DurableDir *DurableDirVolume `protobuf:"bytes,2,opt,name=durable_dir,json=durableDir,proto3,oneof"` -} - -type Volume_External struct { - External *ExternalVolumeSource `protobuf:"bytes,3,opt,name=external,proto3,oneof"` -} - -type Volume_SystemInfo struct { - SystemInfo *SystemInfoVolume `protobuf:"bytes,4,opt,name=system_info,json=systemInfo,proto3,oneof"` -} - -type Volume_Image struct { - Image *ImageVolumeSource `protobuf:"bytes,5,opt,name=image,proto3,oneof"` -} - -func (*Volume_DurableDir) isVolume_Source() {} - -func (*Volume_External) isVolume_Source() {} - -func (*Volume_SystemInfo) isVolume_Source() {} - -func (*Volume_Image) isVolume_Source() {} - type VolumeMount struct { - state protoimpl.MessageState `protogen:"open.v1"` - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - MountPath string `protobuf:"bytes,2,opt,name=mount_path,json=mountPath,proto3" json:"mount_path,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // name must match the name of a Volume. + // + // +k8s:required + // +k8s:format=k8s-short-name + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // mount_path within the container. Must be a clean absolute Unix path. + // + // +k8s:required + // +k8s:maxLength=4096 + // +k8s:customValidation # clean-absolute-path shape; no regex/pattern tag exists + MountPath string `protobuf:"bytes,2,opt,name=mount_path,json=mountPath,proto3" json:"mount_path,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -2068,17 +2175,46 @@ func (x *VolumeMount) GetMountPath() string { } type Container struct { - state protoimpl.MessageState `protogen:"open.v1"` - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - Image string `protobuf:"bytes,2,opt,name=image,proto3" json:"image,omitempty"` - Command []string `protobuf:"bytes,3,rep,name=command,proto3" json:"command,omitempty"` - Args []string `protobuf:"bytes,7,rep,name=args,proto3" json:"args,omitempty"` - Env []*EnvEntry `protobuf:"bytes,4,rep,name=env,proto3" json:"env,omitempty"` - WakeupProbe *WakeupProbe `protobuf:"bytes,5,opt,name=wakeup_probe,json=wakeupProbe,proto3" json:"wakeup_probe,omitempty"` - VolumeMounts []*VolumeMount `protobuf:"bytes,6,rep,name=volume_mounts,json=volumeMounts,proto3" json:"volume_mounts,omitempty"` - SecurityContext *SecurityContext `protobuf:"bytes,8,opt,name=security_context,json=securityContext,proto3" json:"security_context,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:required + // +k8s:format=k8s-short-name + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // +k8s:required + // +k8s:maxLength=512 # matches the template Container.image's bound + // +k8s:customValidation # must be a well-formed image reference, pinned by digest + Image string `protobuf:"bytes,2,opt,name=image,proto3" json:"image,omitempty"` + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=atomic + // +k8s:eachVal=+k8s:maxLength=4096 # argv strings; guardrail, not a contract + Command []string `protobuf:"bytes,3,rep,name=command,proto3" json:"command,omitempty"` + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=atomic + // +k8s:eachVal=+k8s:maxLength=4096 # argv strings; guardrail, not a contract + Args []string `protobuf:"bytes,7,rep,name=args,proto3" json:"args,omitempty"` + // +k8s:optional + // +k8s:maxItems=32 # matches the template's env bound + // +k8s:listType=map # each variable is set at most once + // +k8s:listMapKey=name + Env []*EnvEntry `protobuf:"bytes,4,rep,name=env,proto3" json:"env,omitempty"` + // +k8s:optional + WakeupProbe *WakeupProbe `protobuf:"bytes,5,opt,name=wakeup_probe,json=wakeupProbe,proto3" json:"wakeup_probe,omitempty"` + // Keyed by mount_path: each path hosts exactly one mount, while a volume + // may be mounted at multiple paths. + // + // +k8s:optional + // +k8s:maxItems=32 # matches the template's volume_mounts bound + // +k8s:listType=map + // +k8s:listMapKey=mount_path + // +k8s:customValidation # mounts must not nest + VolumeMounts []*VolumeMount `protobuf:"bytes,6,rep,name=volume_mounts,json=volumeMounts,proto3" json:"volume_mounts,omitempty"` + // +k8s:optional + SecurityContext *SecurityContext `protobuf:"bytes,8,opt,name=security_context,json=securityContext,proto3" json:"security_context,omitempty"` // resources are the cgroup limits for this container, resolved by // ate-api-server from the ActorTemplate. Unset means no limits. + // + // +k8s:optional Resources *ResourceLimits `protobuf:"bytes,9,opt,name=resources,proto3" json:"resources,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -2179,8 +2315,9 @@ func (x *Container) GetResources() *ResourceLimits { // SecurityContext holds security settings for a container's process. type SecurityContext struct { - state protoimpl.MessageState `protogen:"open.v1"` - Capabilities *Capabilities `protobuf:"bytes,1,opt,name=capabilities,proto3" json:"capabilities,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:optional + Capabilities *Capabilities `protobuf:"bytes,1,opt,name=capabilities,proto3" json:"capabilities,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -2225,9 +2362,17 @@ func (x *SecurityContext) GetCapabilities() *Capabilities { // Capabilities adjusts a container's Linux capabilities relative to the default // set. Names carry no "CAP_" prefix; drop applies before add. type Capabilities struct { - state protoimpl.MessageState `protogen:"open.v1"` - Add []string `protobuf:"bytes,1,rep,name=add,proto3" json:"add,omitempty"` - Drop []string `protobuf:"bytes,2,rep,name=drop,proto3" json:"drop,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=set + // +k8s:customValidation # capability grammar; "ALL" not accepted + Add []string `protobuf:"bytes,1,rep,name=add,proto3" json:"add,omitempty"` + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=set + // +k8s:customValidation # capability grammar + Drop []string `protobuf:"bytes,2,rep,name=drop,proto3" json:"drop,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -2281,8 +2426,15 @@ func (x *Capabilities) GetDrop() []string { type ResourceLimits struct { state protoimpl.MessageState `protogen:"open.v1"` // memory_bytes is the memory limit in bytes. 0 means unset. + // + // +k8s:optional + // +k8s:minimum=0 MemoryBytes int64 `protobuf:"varint,1,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // cpu_millis is the CPU limit in milli-cores (1000 = one core). 0 means unset. + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores CpuMillis int64 `protobuf:"varint,2,opt,name=cpu_millis,json=cpuMillis,proto3" json:"cpu_millis,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -2333,9 +2485,16 @@ func (x *ResourceLimits) GetCpuMillis() int64 { } type EnvEntry struct { - state protoimpl.MessageState `protogen:"open.v1"` - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - Value string `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // name may be any printable ASCII character except '='. + // + // +k8s:required + // +k8s:maxLength=256 # guardrail + // +k8s:customValidation # printable ASCII except '=' + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // +k8s:optional + // +k8s:maxLength=32768 # guardrail + Value string `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -2387,9 +2546,14 @@ func (x *EnvEntry) GetValue() string { // WakeupProbe describes how to check that a container is ready to serve. // Only HTTP is supported today. type WakeupProbe struct { - state protoimpl.MessageState `protogen:"open.v1"` - HttpGet *HTTPGetAction `protobuf:"bytes,1,opt,name=http_get,json=httpGet,proto3" json:"http_get,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:required + HttpGet *HttpGetAction `protobuf:"bytes,1,opt,name=http_get,json=httpGet,proto3" json:"http_get,omitempty"` // How long to keep polling before giving up and failing the actor start. + // + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=3600 # matches the template wakeup probe's bound TimeoutSeconds int32 `protobuf:"varint,2,opt,name=timeout_seconds,json=timeoutSeconds,proto3" json:"timeout_seconds,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -2425,7 +2589,7 @@ func (*WakeupProbe) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{35} } -func (x *WakeupProbe) GetHttpGet() *HTTPGetAction { +func (x *WakeupProbe) GetHttpGet() *HttpGetAction { if x != nil { return x.HttpGet } @@ -2439,31 +2603,39 @@ func (x *WakeupProbe) GetTimeoutSeconds() int32 { return 0 } -// HTTPGetAction performs an HTTP GET against the container. -type HTTPGetAction struct { +// HttpGetAction performs an HTTP GET against the container. +type HttpGetAction struct { state protoimpl.MessageState `protogen:"open.v1"` // Path to access on the HTTP server. + // + // +k8s:required + // +k8s:maxLength=1024 # matches the template probe path's bound + // +k8s:customValidation # RFC 3986 path shape; no regex/pattern tag exists Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"` // TCP port to connect to (1..65535). + // + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=65535 Port int32 `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *HTTPGetAction) Reset() { - *x = HTTPGetAction{} +func (x *HttpGetAction) Reset() { + *x = HttpGetAction{} mi := &file_atelet_proto_msgTypes[36] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *HTTPGetAction) String() string { +func (x *HttpGetAction) String() string { return protoimpl.X.MessageStringOf(x) } -func (*HTTPGetAction) ProtoMessage() {} +func (*HttpGetAction) ProtoMessage() {} -func (x *HTTPGetAction) ProtoReflect() protoreflect.Message { +func (x *HttpGetAction) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[36] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2475,19 +2647,19 @@ func (x *HTTPGetAction) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use HTTPGetAction.ProtoReflect.Descriptor instead. -func (*HTTPGetAction) Descriptor() ([]byte, []int) { +// Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead. +func (*HttpGetAction) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{36} } -func (x *HTTPGetAction) GetPath() string { +func (x *HttpGetAction) GetPath() string { if x != nil { return x.Path } return "" } -func (x *HTTPGetAction) GetPort() int32 { +func (x *HttpGetAction) GetPort() int32 { if x != nil { return x.Port } @@ -2530,31 +2702,33 @@ func (*RunResponse) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{37} } -type LocalCheckpointConfiguration struct { +type ObjectStorage struct { state protoimpl.MessageState `protogen:"open.v1"` - // Names a sub-directory on the local filesystem below which the checkpoint - // data will be stored. atelet decides where that directory lives, from the - // actor's UID, so this is a bare name and must not contain a path separator. - // The structure of the checkpoint should generally be treated as opaque. - SnapshotName string `protobuf:"bytes,1,opt,name=snapshot_name,json=snapshotName,proto3" json:"snapshot_name,omitempty"` + // The object storage URI of the snapshot. Object names are appended to it, + // so it addresses the snapshot as a whole rather than any one object. + // + // +k8s:required + // +k8s:maxLength=2048 + // +k8s:customValidation # a parseable snapshot URI + SnapshotUri string `protobuf:"bytes,1,opt,name=snapshot_uri,json=snapshotUri,proto3" json:"snapshot_uri,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *LocalCheckpointConfiguration) Reset() { - *x = LocalCheckpointConfiguration{} +func (x *ObjectStorage) Reset() { + *x = ObjectStorage{} mi := &file_atelet_proto_msgTypes[38] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *LocalCheckpointConfiguration) String() string { +func (x *ObjectStorage) String() string { return protoimpl.X.MessageStringOf(x) } -func (*LocalCheckpointConfiguration) ProtoMessage() {} +func (*ObjectStorage) ProtoMessage() {} -func (x *LocalCheckpointConfiguration) ProtoReflect() protoreflect.Message { +func (x *ObjectStorage) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[38] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2566,41 +2740,81 @@ func (x *LocalCheckpointConfiguration) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use LocalCheckpointConfiguration.ProtoReflect.Descriptor instead. -func (*LocalCheckpointConfiguration) Descriptor() ([]byte, []int) { +// Deprecated: Use ObjectStorage.ProtoReflect.Descriptor instead. +func (*ObjectStorage) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{38} } -func (x *LocalCheckpointConfiguration) GetSnapshotName() string { +func (x *ObjectStorage) GetSnapshotUri() string { if x != nil { - return x.SnapshotName + return x.SnapshotUri } return "" } -type ExternalCheckpointConfiguration struct { +// Snapshot identifies one snapshot of an actor, and where its durable copy +// lives. +type Snapshot struct { state protoimpl.MessageState `protogen:"open.v1"` - // The object storage URI of the snapshot to write. Object names are appended - // to it, so it addresses the snapshot as a whole rather than any one object. - SnapshotUri string `protobuf:"bytes,1,opt,name=snapshot_uri,json=snapshotUri,proto3" json:"snapshot_uri,omitempty"` + // ateapi Snapshot.uuid. Names the snapshot's directory under the actor's + // local-checkpoint directory, so this is a bare name and must not contain + // a path separator. + // + // +k8s:required + // +k8s:format=k8s-short-name + SnapshotUuid string `protobuf:"bytes,1,opt,name=snapshot_uuid,json=snapshotUuid,proto3" json:"snapshot_uuid,omitempty"` + // ateapi Snapshot.generation, starting at 1. Orders the actor's snapshots. + // Checkpoint and PromoteSnapshot fail for a generation below atelet's newest + // for the actor, or for its newest under another snapshot_uuid: the + // caller's view is stale. + // + // TODO: Temporarily, the control plane does not set this and atelet does + // not check it. + // + // +k8s:optional + Generation int32 `protobuf:"varint,2,opt,name=generation,proto3" json:"generation,omitempty"` + // On Checkpoint: the fidelity to capture. + // On Restore: the fidelity to restore the actor with. + // On PromoteSnapshot: the lowest fidelity the snapshot may hold. The call + // fails if its manifest records a lower one. + // In SnapshotReport: the fidelity the snapshot holds. + // + // TODO: Open question: should PromoteSnapshot accept a manifest that records + // a higher fidelity than this? Until that is decided, atelet requires the + // two to be equal. + // + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum + Fidelity SnapshotFidelity `protobuf:"varint,3,opt,name=fidelity,proto3,enum=atelet.SnapshotFidelity" json:"fidelity,omitempty"` + // The snapshot's durable copy in object storage. Required on every + // request: atelet uploads there, or reads from there when it holds no + // usable local copy. + // + // TODO: Temporarily, this is optional while we haven't implemented deferred + // uploads. Restore also uses its presence to choose between the local copy + // and object storage; see RestoreRequest.snapshot. + // + // +k8s:optional + Object *ObjectStorage `protobuf:"bytes,4,opt,name=object,proto3" json:"object,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *ExternalCheckpointConfiguration) Reset() { - *x = ExternalCheckpointConfiguration{} +func (x *Snapshot) Reset() { + *x = Snapshot{} mi := &file_atelet_proto_msgTypes[39] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *ExternalCheckpointConfiguration) String() string { +func (x *Snapshot) String() string { return protoimpl.X.MessageStringOf(x) } -func (*ExternalCheckpointConfiguration) ProtoMessage() {} +func (*Snapshot) ProtoMessage() {} -func (x *ExternalCheckpointConfiguration) ProtoReflect() protoreflect.Message { +func (x *Snapshot) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[39] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2612,45 +2826,67 @@ func (x *ExternalCheckpointConfiguration) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use ExternalCheckpointConfiguration.ProtoReflect.Descriptor instead. -func (*ExternalCheckpointConfiguration) Descriptor() ([]byte, []int) { +// Deprecated: Use Snapshot.ProtoReflect.Descriptor instead. +func (*Snapshot) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{39} } -func (x *ExternalCheckpointConfiguration) GetSnapshotUri() string { +func (x *Snapshot) GetSnapshotUuid() string { if x != nil { - return x.SnapshotUri + return x.SnapshotUuid } return "" } -// ExternalRestoreConfiguration is an external snapshot a restore reads. -// Split from the checkpoint-side ExternalCheckpointConfiguration (a write -// destination) so read-side attributes of a restore source have a home. -type ExternalRestoreConfiguration struct { - state protoimpl.MessageState `protogen:"open.v1"` - // The object storage URI of the snapshot to read. Object names are - // appended to it, so it addresses the snapshot as a whole rather than any - // one object. - SnapshotUri string `protobuf:"bytes,1,opt,name=snapshot_uri,json=snapshotUri,proto3" json:"snapshot_uri,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache +func (x *Snapshot) GetGeneration() int32 { + if x != nil { + return x.Generation + } + return 0 +} + +func (x *Snapshot) GetFidelity() SnapshotFidelity { + if x != nil { + return x.Fidelity + } + return SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED } -func (x *ExternalRestoreConfiguration) Reset() { - *x = ExternalRestoreConfiguration{} +func (x *Snapshot) GetObject() *ObjectStorage { + if x != nil { + return x.Object + } + return nil +} + +// SnapshotReport is a snapshot atelet holds or has uploaded, and when. +type SnapshotReport struct { + state protoimpl.MessageState `protogen:"open.v1"` + Snapshot *Snapshot `protobuf:"bytes,1,opt,name=snapshot,proto3" json:"snapshot,omitempty"` + // When the checkpoint that produced the snapshot committed on this node. + LocalCommitTime *timestamppb.Timestamp `protobuf:"bytes,2,opt,name=local_commit_time,json=localCommitTime,proto3" json:"local_commit_time,omitempty"` + // When the snapshot's manifest, the upload's commit marker, was written to + // object storage. Set only in ActorSnapshotState.latest_durable, and only + // when this atelet wrote it. + DurableCompleteTime *timestamppb.Timestamp `protobuf:"bytes,3,opt,name=durable_complete_time,json=durableCompleteTime,proto3" json:"durable_complete_time,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *SnapshotReport) Reset() { + *x = SnapshotReport{} mi := &file_atelet_proto_msgTypes[40] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *ExternalRestoreConfiguration) String() string { +func (x *SnapshotReport) String() string { return protoimpl.X.MessageStringOf(x) } -func (*ExternalRestoreConfiguration) ProtoMessage() {} +func (*SnapshotReport) ProtoMessage() {} -func (x *ExternalRestoreConfiguration) ProtoReflect() protoreflect.Message { +func (x *SnapshotReport) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[40] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2662,58 +2898,75 @@ func (x *ExternalRestoreConfiguration) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use ExternalRestoreConfiguration.ProtoReflect.Descriptor instead. -func (*ExternalRestoreConfiguration) Descriptor() ([]byte, []int) { +// Deprecated: Use SnapshotReport.ProtoReflect.Descriptor instead. +func (*SnapshotReport) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{40} } -func (x *ExternalRestoreConfiguration) GetSnapshotUri() string { +func (x *SnapshotReport) GetSnapshot() *Snapshot { if x != nil { - return x.SnapshotUri + return x.Snapshot } - return "" + return nil } -type CheckpointRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` - Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` - ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` - ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` - ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` - ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` - // Sandbox binary config is not sent on checkpoint: atelet uses the version the - // actor is currently running (recorded with the actor's on-node state at - // Run/Restore) and records it into the snapshot manifest. - Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` - Type CheckpointType `protobuf:"varint,8,opt,name=type,proto3,enum=atelet.CheckpointType" json:"type,omitempty"` - // The checkpoint configuration, depending on the type. - // - // Types that are valid to be assigned to Config: - // - // *CheckpointRequest_LocalConfig - // *CheckpointRequest_ExternalConfig - Config isCheckpointRequest_Config `protobuf_oneof:"config"` - // Fidelity the checkpoint captures. - Fidelity SnapshotFidelity `protobuf:"varint,11,opt,name=fidelity,proto3,enum=atelet.SnapshotFidelity" json:"fidelity,omitempty"` +func (x *SnapshotReport) GetLocalCommitTime() *timestamppb.Timestamp { + if x != nil { + return x.LocalCommitTime + } + return nil +} + +func (x *SnapshotReport) GetDurableCompleteTime() *timestamppb.Timestamp { + if x != nil { + return x.DurableCompleteTime + } + return nil +} + +// ActorSnapshotState is what atelet holds for one actor on this node. +// Returned by Checkpoint, PromoteSnapshot. +type ActorSnapshotState struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The snapshot atelet restores the actor from locally. Unset when the + // latest checkpoint kept no local copy, and once a detach has made the + // snapshot durable and begun deleting it. Either way, latest_durable is + // then the actor's latest snapshot. + LatestLocal *SnapshotReport `protobuf:"bytes,1,opt,name=latest_local,json=latestLocal,proto3" json:"latest_local,omitempty"` + // The snapshot an in-flight upload is reading, if any. If it is + // latest_local, its local copy stays after the upload. If it is an older + // snapshot, whose upload began before latest_local was taken, atelet + // deletes its local copy once the upload ends. + Uploading *SnapshotReport `protobuf:"bytes,2,opt,name=uploading,proto3" json:"uploading,omitempty"` + // The newest snapshot atelet knows is durable, whether or not the control + // plane has recorded it: one atelet uploaded since it started, or one a + // PromoteSnapshot found already uploaded. Unset if there is none, which says + // nothing about older durable snapshots. + LatestDurable *SnapshotReport `protobuf:"bytes,3,opt,name=latest_durable,json=latestDurable,proto3" json:"latest_durable,omitempty"` + // When atelet next plans to trigger a background upload, which will upload + // whichever snapshot is latest_local at the time of triggering. Unset when + // there is no latest_local, or it is already durable or being uploaded. + // Advisory: a promote or detach can upload it sooner, and an older + // snapshot's upload can delay it. + UploadDueTime *timestamppb.Timestamp `protobuf:"bytes,4,opt,name=upload_due_time,json=uploadDueTime,proto3" json:"upload_due_time,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *CheckpointRequest) Reset() { - *x = CheckpointRequest{} +func (x *ActorSnapshotState) Reset() { + *x = ActorSnapshotState{} mi := &file_atelet_proto_msgTypes[41] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *CheckpointRequest) String() string { +func (x *ActorSnapshotState) String() string { return protoimpl.X.MessageStringOf(x) } -func (*CheckpointRequest) ProtoMessage() {} +func (*ActorSnapshotState) ProtoMessage() {} -func (x *CheckpointRequest) ProtoReflect() protoreflect.Message { +func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[41] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2725,135 +2978,64 @@ func (x *CheckpointRequest) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use CheckpointRequest.ProtoReflect.Descriptor instead. -func (*CheckpointRequest) Descriptor() ([]byte, []int) { +// Deprecated: Use ActorSnapshotState.ProtoReflect.Descriptor instead. +func (*ActorSnapshotState) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{41} } -func (x *CheckpointRequest) GetWorkerPodUid() string { +func (x *ActorSnapshotState) GetLatestLocal() *SnapshotReport { if x != nil { - return x.WorkerPodUid - } - return "" -} - -func (x *CheckpointRequest) GetAtespace() string { - if x != nil { - return x.Atespace - } - return "" -} - -func (x *CheckpointRequest) GetActorName() string { - if x != nil { - return x.ActorName - } - return "" -} - -func (x *CheckpointRequest) GetActorUid() string { - if x != nil { - return x.ActorUid - } - return "" -} - -func (x *CheckpointRequest) GetActorTemplateAtespace() string { - if x != nil { - return x.ActorTemplateAtespace - } - return "" -} - -func (x *CheckpointRequest) GetActorTemplateName() string { - if x != nil { - return x.ActorTemplateName - } - return "" -} - -func (x *CheckpointRequest) GetSpec() *WorkloadSpec { - if x != nil { - return x.Spec + return x.LatestLocal } return nil } -func (x *CheckpointRequest) GetType() CheckpointType { +func (x *ActorSnapshotState) GetUploading() *SnapshotReport { if x != nil { - return x.Type - } - return CheckpointType_CHECKPOINT_TYPE_UNSPECIFIED -} - -func (x *CheckpointRequest) GetConfig() isCheckpointRequest_Config { - if x != nil { - return x.Config + return x.Uploading } return nil } -func (x *CheckpointRequest) GetLocalConfig() *LocalCheckpointConfiguration { +func (x *ActorSnapshotState) GetLatestDurable() *SnapshotReport { if x != nil { - if x, ok := x.Config.(*CheckpointRequest_LocalConfig); ok { - return x.LocalConfig - } + return x.LatestDurable } return nil } -func (x *CheckpointRequest) GetExternalConfig() *ExternalCheckpointConfiguration { +func (x *ActorSnapshotState) GetUploadDueTime() *timestamppb.Timestamp { if x != nil { - if x, ok := x.Config.(*CheckpointRequest_ExternalConfig); ok { - return x.ExternalConfig - } + return x.UploadDueTime } return nil } -func (x *CheckpointRequest) GetFidelity() SnapshotFidelity { - if x != nil { - return x.Fidelity - } - return SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED -} - -type isCheckpointRequest_Config interface { - isCheckpointRequest_Config() -} - -type CheckpointRequest_LocalConfig struct { - LocalConfig *LocalCheckpointConfiguration `protobuf:"bytes,9,opt,name=local_config,json=localConfig,proto3,oneof"` -} - -type CheckpointRequest_ExternalConfig struct { - ExternalConfig *ExternalCheckpointConfiguration `protobuf:"bytes,10,opt,name=external_config,json=externalConfig,proto3,oneof"` -} - -func (*CheckpointRequest_LocalConfig) isCheckpointRequest_Config() {} - -func (*CheckpointRequest_ExternalConfig) isCheckpointRequest_Config() {} - -type CheckpointResponse struct { - state protoimpl.MessageState `protogen:"open.v1"` +// UploadConfig is when a checkpoint's snapshot becomes durable. +type UploadConfig struct { + state protoimpl.MessageState `protogen:"open.v1"` + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=2 # keep this in sync with the UploadMode enum + Mode UploadMode `protobuf:"varint,1,opt,name=mode,proto3,enum=atelet.UploadMode" json:"mode,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *CheckpointResponse) Reset() { - *x = CheckpointResponse{} +func (x *UploadConfig) Reset() { + *x = UploadConfig{} mi := &file_atelet_proto_msgTypes[42] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *CheckpointResponse) String() string { +func (x *UploadConfig) String() string { return protoimpl.X.MessageStringOf(x) } -func (*CheckpointResponse) ProtoMessage() {} +func (*UploadConfig) ProtoMessage() {} -func (x *CheckpointResponse) ProtoReflect() protoreflect.Message { +func (x *UploadConfig) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[42] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2865,47 +3047,83 @@ func (x *CheckpointResponse) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use CheckpointResponse.ProtoReflect.Descriptor instead. -func (*CheckpointResponse) Descriptor() ([]byte, []int) { +// Deprecated: Use UploadConfig.ProtoReflect.Descriptor instead. +func (*UploadConfig) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{42} } -type UploadPausedCheckpointRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - Atespace string `protobuf:"bytes,1,opt,name=atespace,proto3" json:"atespace,omitempty"` - ActorName string `protobuf:"bytes,2,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` - ActorUid string `protobuf:"bytes,3,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` - // For metrics attribution, like on CheckpointRequest. - ActorTemplateAtespace string `protobuf:"bytes,4,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` - ActorTemplateName string `protobuf:"bytes,5,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` - // The local checkpoint to upload: the uuid of the actor's pause snapshot - // (ateapi Snapshot.uuid). - LocalSnapshotName string `protobuf:"bytes,6,opt,name=local_snapshot_name,json=localSnapshotName,proto3" json:"local_snapshot_name,omitempty"` - // Destination object-storage URI (the actor's in-progress snapshot URI). - DestinationSnapshotUri string `protobuf:"bytes,7,opt,name=destination_snapshot_uri,json=destinationSnapshotUri,proto3" json:"destination_snapshot_uri,omitempty"` - // Fidelity the uploaded snapshot must have. The fidelity the pause - // checkpoint captured is not sent: atelet reads it from the local - // snapshot's own manifest, which is authoritative, and rejects the upload - // if the two differ. - DesiredFidelity SnapshotFidelity `protobuf:"varint,8,opt,name=desired_fidelity,json=desiredFidelity,proto3,enum=atelet.SnapshotFidelity" json:"desired_fidelity,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} - -func (x *UploadPausedCheckpointRequest) Reset() { - *x = UploadPausedCheckpointRequest{} +func (x *UploadConfig) GetMode() UploadMode { + if x != nil { + return x.Mode + } + return UploadMode_UPLOAD_MODE_UNSPECIFIED +} + +// +k8s:customValidation # immediate upload requires snapshot.object +type CheckpointRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The UID of the worker pod whose ateom hosts the actor. + // + // +k8s:required + // +k8s:format=k8s-uuid + WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` + // +k8s:required + // +k8s:format=k8s-uuid + ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` + // Sandbox binary config is not sent on checkpoint: atelet uses the version the + // actor is currently running (recorded with the actor's on-node state at + // Run/Restore) and records it into the snapshot manifest. + // + // +k8s:required + Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` + // The snapshot to take, and where it is uploaded. atelet may upload it long + // after Checkpoint returns, without asking the control plane. A request at + // atelet's newest generation for the actor, with the same snapshot_uuid, + // is a retry: atelet returns the actor's current state without + // checkpointing again. + // + // TODO: Temporarily, atelet does not recognize retries. A retry runs the + // checkpoint again, which fails once an earlier attempt has removed the + // workload. + // + // +k8s:required + Snapshot *Snapshot `protobuf:"bytes,8,opt,name=snapshot,proto3" json:"snapshot,omitempty"` + // When the snapshot becomes durable. + // + // +k8s:required + UploadConfig *UploadConfig `protobuf:"bytes,9,opt,name=upload_config,json=uploadConfig,proto3" json:"upload_config,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *CheckpointRequest) Reset() { + *x = CheckpointRequest{} mi := &file_atelet_proto_msgTypes[43] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *UploadPausedCheckpointRequest) String() string { +func (x *CheckpointRequest) String() string { return protoimpl.X.MessageStringOf(x) } -func (*UploadPausedCheckpointRequest) ProtoMessage() {} +func (*CheckpointRequest) ProtoMessage() {} -func (x *UploadPausedCheckpointRequest) ProtoReflect() protoreflect.Message { +func (x *CheckpointRequest) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[43] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -2917,87 +3135,102 @@ func (x *UploadPausedCheckpointRequest) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use UploadPausedCheckpointRequest.ProtoReflect.Descriptor instead. -func (*UploadPausedCheckpointRequest) Descriptor() ([]byte, []int) { +// Deprecated: Use CheckpointRequest.ProtoReflect.Descriptor instead. +func (*CheckpointRequest) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{43} } -func (x *UploadPausedCheckpointRequest) GetAtespace() string { +func (x *CheckpointRequest) GetWorkerPodUid() string { + if x != nil { + return x.WorkerPodUid + } + return "" +} + +func (x *CheckpointRequest) GetAtespace() string { if x != nil { return x.Atespace } return "" } -func (x *UploadPausedCheckpointRequest) GetActorName() string { +func (x *CheckpointRequest) GetActorName() string { if x != nil { return x.ActorName } return "" } -func (x *UploadPausedCheckpointRequest) GetActorUid() string { +func (x *CheckpointRequest) GetActorUid() string { if x != nil { return x.ActorUid } return "" } -func (x *UploadPausedCheckpointRequest) GetActorTemplateAtespace() string { +func (x *CheckpointRequest) GetActorTemplateAtespace() string { if x != nil { return x.ActorTemplateAtespace } return "" } -func (x *UploadPausedCheckpointRequest) GetActorTemplateName() string { +func (x *CheckpointRequest) GetActorTemplateName() string { if x != nil { return x.ActorTemplateName } return "" } -func (x *UploadPausedCheckpointRequest) GetLocalSnapshotName() string { +func (x *CheckpointRequest) GetSpec() *WorkloadSpec { if x != nil { - return x.LocalSnapshotName + return x.Spec } - return "" + return nil } -func (x *UploadPausedCheckpointRequest) GetDestinationSnapshotUri() string { +func (x *CheckpointRequest) GetSnapshot() *Snapshot { if x != nil { - return x.DestinationSnapshotUri + return x.Snapshot } - return "" + return nil } -func (x *UploadPausedCheckpointRequest) GetDesiredFidelity() SnapshotFidelity { +func (x *CheckpointRequest) GetUploadConfig() *UploadConfig { if x != nil { - return x.DesiredFidelity + return x.UploadConfig } - return SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED + return nil } -type UploadPausedCheckpointResponse struct { - state protoimpl.MessageState `protogen:"open.v1"` +type CheckpointResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The actor's state once the checkpoint is taken: latest_local is the new + // snapshot, unless UPLOAD_MODE_IMMEDIATE kept no local copy. The control + // plane can drop its entries for any local-only generation not named here: + // atelet has deleted it, and will never upload it. + // + // TODO: Temporarily, atelet leaves this unset. An unset state says nothing + // about which snapshots atelet holds. + State *ActorSnapshotState `protobuf:"bytes,1,opt,name=state,proto3" json:"state,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } -func (x *UploadPausedCheckpointResponse) Reset() { - *x = UploadPausedCheckpointResponse{} +func (x *CheckpointResponse) Reset() { + *x = CheckpointResponse{} mi := &file_atelet_proto_msgTypes[44] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *UploadPausedCheckpointResponse) String() string { +func (x *CheckpointResponse) String() string { return protoimpl.X.MessageStringOf(x) } -func (*UploadPausedCheckpointResponse) ProtoMessage() {} +func (*CheckpointResponse) ProtoMessage() {} -func (x *UploadPausedCheckpointResponse) ProtoReflect() protoreflect.Message { +func (x *CheckpointResponse) ProtoReflect() protoreflect.Message { mi := &file_atelet_proto_msgTypes[44] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -3009,40 +3242,76 @@ func (x *UploadPausedCheckpointResponse) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use UploadPausedCheckpointResponse.ProtoReflect.Descriptor instead. -func (*UploadPausedCheckpointResponse) Descriptor() ([]byte, []int) { +// Deprecated: Use CheckpointResponse.ProtoReflect.Descriptor instead. +func (*CheckpointResponse) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{44} } +func (x *CheckpointResponse) GetState() *ActorSnapshotState { + if x != nil { + return x.State + } + return nil +} + type RestoreRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` - Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` - ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` - ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` - ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` - ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` - Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` - Type CheckpointType `protobuf:"varint,8,opt,name=type,proto3,enum=atelet.CheckpointType" json:"type,omitempty"` - // The checkpoint configuration, depending on the type. - // - // Types that are valid to be assigned to Config: - // - // *RestoreRequest_LocalConfig - // *RestoreRequest_ExternalConfig - Config isRestoreRequest_Config `protobuf_oneof:"config"` - // Fidelity to restore from the checkpoint. - Fidelity SnapshotFidelity `protobuf:"varint,11,opt,name=fidelity,proto3,enum=atelet.SnapshotFidelity" json:"fidelity,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + // The UID of the worker pod whose ateom hosts the actor. + // + // +k8s:required + // +k8s:format=k8s-uuid + WorkerPodUid string `protobuf:"bytes,1,opt,name=worker_pod_uid,json=workerPodUid,proto3" json:"worker_pod_uid,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + Atespace string `protobuf:"bytes,2,opt,name=atespace,proto3" json:"atespace,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + ActorName string `protobuf:"bytes,3,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` + // +k8s:required + // +k8s:format=k8s-uuid + ActorUid string `protobuf:"bytes,4,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateAtespace string `protobuf:"bytes,5,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateName string `protobuf:"bytes,6,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` + // +k8s:required + Spec *WorkloadSpec `protobuf:"bytes,7,opt,name=spec,proto3" json:"spec,omitempty"` + // Snapshot to restore from. atelet reads its local copy if it holds a + // usable one, and `snapshot.object` otherwise. A local copy is unusable if + // its manifest does not parse or lists a missing file. While atelet is + // detaching the actor, it waits for the detach to finish and then reads + // `snapshot.object`. Fails if neither copy exists: that generation is lost. + // + // TODO: Temporarily, whether `snapshot.object` is set chooses the copy: + // atelet reads only the local copy when it is unset, and only + // `snapshot.object` when it is set, with no fallback either way. + // + // +k8s:required + Snapshot *Snapshot `protobuf:"bytes,8,opt,name=snapshot,proto3" json:"snapshot,omitempty"` // When absent the actor has no egress: its TCP is captured and refused. + // + // +k8s:optional EgressGateway *EgressGateway `protobuf:"bytes,13,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"` // The actor's declared size, from the ActorTemplate's resource limits. For // gVisor and micro-VM VOLUMES restores the sandbox is (re)sized to these; // for a MEMORY micro-VM restore the size baked into the snapshot wins. Zero // means "unset": keep the runtime default. - CpuMilli int64 `protobuf:"varint,14,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores + CpuMilli int64 `protobuf:"varint,14,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). + // +k8s:optional + // +k8s:minimum=0 MemoryBytes int64 `protobuf:"varint,15,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. // The sandbox binaries and pause image to restore with, resolved from the - // ActorTemplate's SandboxConfig. Required. + // ActorTemplate's SandboxConfig. + // + // +k8s:required SandboxAssets *SandboxAssets `protobuf:"bytes,16,opt,name=sandbox_assets,json=sandboxAssets,proto3" json:"sandbox_assets,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -3127,45 +3396,13 @@ func (x *RestoreRequest) GetSpec() *WorkloadSpec { return nil } -func (x *RestoreRequest) GetType() CheckpointType { - if x != nil { - return x.Type - } - return CheckpointType_CHECKPOINT_TYPE_UNSPECIFIED -} - -func (x *RestoreRequest) GetConfig() isRestoreRequest_Config { - if x != nil { - return x.Config - } - return nil -} - -func (x *RestoreRequest) GetLocalConfig() *LocalCheckpointConfiguration { - if x != nil { - if x, ok := x.Config.(*RestoreRequest_LocalConfig); ok { - return x.LocalConfig - } - } - return nil -} - -func (x *RestoreRequest) GetExternalConfig() *ExternalRestoreConfiguration { +func (x *RestoreRequest) GetSnapshot() *Snapshot { if x != nil { - if x, ok := x.Config.(*RestoreRequest_ExternalConfig); ok { - return x.ExternalConfig - } + return x.Snapshot } return nil } -func (x *RestoreRequest) GetFidelity() SnapshotFidelity { - if x != nil { - return x.Fidelity - } - return SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED -} - func (x *RestoreRequest) GetEgressGateway() *EgressGateway { if x != nil { return x.EgressGateway @@ -3194,22 +3431,6 @@ func (x *RestoreRequest) GetSandboxAssets() *SandboxAssets { return nil } -type isRestoreRequest_Config interface { - isRestoreRequest_Config() -} - -type RestoreRequest_LocalConfig struct { - LocalConfig *LocalCheckpointConfiguration `protobuf:"bytes,9,opt,name=local_config,json=localConfig,proto3,oneof"` -} - -type RestoreRequest_ExternalConfig struct { - ExternalConfig *ExternalRestoreConfiguration `protobuf:"bytes,10,opt,name=external_config,json=externalConfig,proto3,oneof"` -} - -func (*RestoreRequest_LocalConfig) isRestoreRequest_Config() {} - -func (*RestoreRequest_ExternalConfig) isRestoreRequest_Config() {} - type RestoreResponse struct { state protoimpl.MessageState `protogen:"open.v1"` unknownFields protoimpl.UnknownFields @@ -3246,11 +3467,164 @@ func (*RestoreResponse) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{46} } +type PromoteSnapshotRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Golden actors are never paused, so the golden atespace is rejected. + // + // +k8s:required + // +k8s:format=k8s-short-name + // +k8s:customValidation # not the golden atespace + Atespace string `protobuf:"bytes,1,opt,name=atespace,proto3" json:"atespace,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + ActorName string `protobuf:"bytes,2,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` + // +k8s:required + // +k8s:format=k8s-uuid + ActorUid string `protobuf:"bytes,3,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` + // For metrics attribution, like on CheckpointRequest. + // + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateAtespace string `protobuf:"bytes,4,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` + // +k8s:optional + // +k8s:format=k8s-short-name + ActorTemplateName string `protobuf:"bytes,5,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` + // The actor's latest snapshot as the control plane records it, and where + // it is, or will be, uploaded. + // + // +k8s:required + // +k8s:subfield(object)=+k8s:required + Snapshot *Snapshot `protobuf:"bytes,6,opt,name=snapshot,proto3" json:"snapshot,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *PromoteSnapshotRequest) Reset() { + *x = PromoteSnapshotRequest{} + mi := &file_atelet_proto_msgTypes[47] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *PromoteSnapshotRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*PromoteSnapshotRequest) ProtoMessage() {} + +func (x *PromoteSnapshotRequest) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[47] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use PromoteSnapshotRequest.ProtoReflect.Descriptor instead. +func (*PromoteSnapshotRequest) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{47} +} + +func (x *PromoteSnapshotRequest) GetAtespace() string { + if x != nil { + return x.Atespace + } + return "" +} + +func (x *PromoteSnapshotRequest) GetActorName() string { + if x != nil { + return x.ActorName + } + return "" +} + +func (x *PromoteSnapshotRequest) GetActorUid() string { + if x != nil { + return x.ActorUid + } + return "" +} + +func (x *PromoteSnapshotRequest) GetActorTemplateAtespace() string { + if x != nil { + return x.ActorTemplateAtespace + } + return "" +} + +func (x *PromoteSnapshotRequest) GetActorTemplateName() string { + if x != nil { + return x.ActorTemplateName + } + return "" +} + +func (x *PromoteSnapshotRequest) GetSnapshot() *Snapshot { + if x != nil { + return x.Snapshot + } + return nil +} + +type PromoteSnapshotResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The actor's state once promoted: latest_durable is its latest snapshot, + // and latest_local remains set if atelet still holds the local copy. + // + // TODO: Temporarily, atelet leaves this unset. An unset state says nothing + // about which snapshots atelet holds. + State *ActorSnapshotState `protobuf:"bytes,1,opt,name=state,proto3" json:"state,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *PromoteSnapshotResponse) Reset() { + *x = PromoteSnapshotResponse{} + mi := &file_atelet_proto_msgTypes[48] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *PromoteSnapshotResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*PromoteSnapshotResponse) ProtoMessage() {} + +func (x *PromoteSnapshotResponse) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[48] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use PromoteSnapshotResponse.ProtoReflect.Descriptor instead. +func (*PromoteSnapshotResponse) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{48} +} + +func (x *PromoteSnapshotResponse) GetState() *ActorSnapshotState { + if x != nil { + return x.State + } + return nil +} + var File_atelet_proto protoreflect.FileDescriptor const file_atelet_proto_rawDesc = "" + "\n" + - "\fatelet.proto\x12\x06atelet\"\xd6\x01\n" + + "\fatelet.proto\x12\x06atelet\x1a\x1fgoogle/protobuf/timestamp.proto\"\xd6\x01\n" + "\x15RegisterWorkerRequest\x123\n" + "\bcapacity\x18\x01 \x01(\v2\x17.atelet.WorkerResourcesR\bcapacity\x12?\n" + "\x0fdefault_runtime\x18\x02 \x01(\v2\x16.atelet.SandboxRuntimeR\x0edefaultRuntime\x12G\n" + @@ -3362,22 +3736,20 @@ const file_atelet_proto_rawDesc = "" + "\x05items\x18\x01 \x03(\v2\x19.atelet.ActorMetadataItemR\x05items\"M\n" + "\x15TrustBundleDataSource\x12\x12\n" + "\x04path\x18\x01 \x01(\tR\x04path\x12\x14\n" + - "\x05names\x18\x03 \x03(\tR\x05namesJ\x04\b\x02\x10\x03R\x04name\"\xb3\x01\n" + - "\x14SystemInfoDataSource\x12H\n" + - "\x0eactor_metadata\x18\x01 \x01(\v2\x1f.atelet.ActorMetadataDataSourceH\x00R\ractorMetadata\x12B\n" + - "\ftrust_bundle\x18\x02 \x01(\v2\x1d.atelet.TrustBundleDataSourceH\x00R\vtrustBundleB\r\n" + - "\vdata_source\"S\n" + + "\x05names\x18\x03 \x03(\tR\x05namesJ\x04\b\x02\x10\x03R\x04name\"\xa0\x01\n" + + "\x14SystemInfoDataSource\x12F\n" + + "\x0eactor_metadata\x18\x01 \x01(\v2\x1f.atelet.ActorMetadataDataSourceR\ractorMetadata\x12@\n" + + "\ftrust_bundle\x18\x02 \x01(\v2\x1d.atelet.TrustBundleDataSourceR\vtrustBundle\"S\n" + "\x10SystemInfoVolume\x12?\n" + - "\fdata_sources\x18\x01 \x03(\v2\x1c.atelet.SystemInfoDataSourceR\vdataSources\"\x8f\x02\n" + + "\fdata_sources\x18\x01 \x03(\v2\x1c.atelet.SystemInfoDataSourceR\vdataSources\"\xfd\x01\n" + "\x06Volume\x12\x12\n" + - "\x04name\x18\x01 \x01(\tR\x04name\x12;\n" + - "\vdurable_dir\x18\x02 \x01(\v2\x18.atelet.DurableDirVolumeH\x00R\n" + - "durableDir\x12:\n" + - "\bexternal\x18\x03 \x01(\v2\x1c.atelet.ExternalVolumeSourceH\x00R\bexternal\x12;\n" + - "\vsystem_info\x18\x04 \x01(\v2\x18.atelet.SystemInfoVolumeH\x00R\n" + - "systemInfo\x121\n" + - "\x05image\x18\x05 \x01(\v2\x19.atelet.ImageVolumeSourceH\x00R\x05imageB\b\n" + - "\x06source\"@\n" + + "\x04name\x18\x01 \x01(\tR\x04name\x129\n" + + "\vdurable_dir\x18\x02 \x01(\v2\x18.atelet.DurableDirVolumeR\n" + + "durableDir\x128\n" + + "\bexternal\x18\x03 \x01(\v2\x1c.atelet.ExternalVolumeSourceR\bexternal\x129\n" + + "\vsystem_info\x18\x04 \x01(\v2\x18.atelet.SystemInfoVolumeR\n" + + "systemInfo\x12/\n" + + "\x05image\x18\x05 \x01(\v2\x19.atelet.ImageVolumeSourceR\x05image\"@\n" + "\vVolumeMount\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x1d\n" + "\n" + @@ -3405,18 +3777,32 @@ const file_atelet_proto_rawDesc = "" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x19\n" + "\x05value\x18\x02 \x01(\tB\x03\x80\x01\x01R\x05value\"h\n" + "\vWakeupProbe\x120\n" + - "\bhttp_get\x18\x01 \x01(\v2\x15.atelet.HTTPGetActionR\ahttpGet\x12'\n" + + "\bhttp_get\x18\x01 \x01(\v2\x15.atelet.HttpGetActionR\ahttpGet\x12'\n" + "\x0ftimeout_seconds\x18\x02 \x01(\x05R\x0etimeoutSeconds\"7\n" + - "\rHTTPGetAction\x12\x12\n" + + "\rHttpGetAction\x12\x12\n" + "\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" + "\x04port\x18\x02 \x01(\x05R\x04port\"\r\n" + - "\vRunResponse\"C\n" + - "\x1cLocalCheckpointConfiguration\x12#\n" + - "\rsnapshot_name\x18\x01 \x01(\tR\fsnapshotName\"D\n" + - "\x1fExternalCheckpointConfiguration\x12!\n" + - "\fsnapshot_uri\x18\x01 \x01(\tR\vsnapshotUri\"A\n" + - "\x1cExternalRestoreConfiguration\x12!\n" + - "\fsnapshot_uri\x18\x01 \x01(\tR\vsnapshotUri\"\xae\x04\n" + + "\vRunResponse\"2\n" + + "\rObjectStorage\x12!\n" + + "\fsnapshot_uri\x18\x01 \x01(\tR\vsnapshotUri\"\xb4\x01\n" + + "\bSnapshot\x12#\n" + + "\rsnapshot_uuid\x18\x01 \x01(\tR\fsnapshotUuid\x12\x1e\n" + + "\n" + + "generation\x18\x02 \x01(\x05R\n" + + "generation\x124\n" + + "\bfidelity\x18\x03 \x01(\x0e2\x18.atelet.SnapshotFidelityR\bfidelity\x12-\n" + + "\x06object\x18\x04 \x01(\v2\x15.atelet.ObjectStorageR\x06object\"\xd6\x01\n" + + "\x0eSnapshotReport\x12,\n" + + "\bsnapshot\x18\x01 \x01(\v2\x10.atelet.SnapshotR\bsnapshot\x12F\n" + + "\x11local_commit_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\x0flocalCommitTime\x12N\n" + + "\x15durable_complete_time\x18\x03 \x01(\v2\x1a.google.protobuf.TimestampR\x13durableCompleteTime\"\x88\x02\n" + + "\x12ActorSnapshotState\x129\n" + + "\flatest_local\x18\x01 \x01(\v2\x16.atelet.SnapshotReportR\vlatestLocal\x124\n" + + "\tuploading\x18\x02 \x01(\v2\x16.atelet.SnapshotReportR\tuploading\x12=\n" + + "\x0elatest_durable\x18\x03 \x01(\v2\x16.atelet.SnapshotReportR\rlatestDurable\x12B\n" + + "\x0fupload_due_time\x18\x04 \x01(\v2\x1a.google.protobuf.TimestampR\ruploadDueTime\"6\n" + + "\fUploadConfig\x12&\n" + + "\x04mode\x18\x01 \x01(\x0e2\x12.atelet.UploadModeR\x04mode\"\x8c\x03\n" + "\x11CheckpointRequest\x12$\n" + "\x0eworker_pod_uid\x18\x01 \x01(\tR\fworkerPodUid\x12\x1a\n" + "\batespace\x18\x02 \x01(\tR\batespace\x12\x1d\n" + @@ -3425,25 +3811,11 @@ const file_atelet_proto_rawDesc = "" + "\tactor_uid\x18\x04 \x01(\tR\bactorUid\x126\n" + "\x17actor_template_atespace\x18\x05 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" + - "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12*\n" + - "\x04type\x18\b \x01(\x0e2\x16.atelet.CheckpointTypeR\x04type\x12I\n" + - "\flocal_config\x18\t \x01(\v2$.atelet.LocalCheckpointConfigurationH\x00R\vlocalConfig\x12R\n" + - "\x0fexternal_config\x18\n" + - " \x01(\v2'.atelet.ExternalCheckpointConfigurationH\x00R\x0eexternalConfig\x124\n" + - "\bfidelity\x18\v \x01(\x0e2\x18.atelet.SnapshotFidelityR\bfidelityB\b\n" + - "\x06config\"\x14\n" + - "\x12CheckpointResponse\"\x8e\x03\n" + - "\x1dUploadPausedCheckpointRequest\x12\x1a\n" + - "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + - "\n" + - "actor_name\x18\x02 \x01(\tR\tactorName\x12\x1b\n" + - "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x126\n" + - "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + - "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12.\n" + - "\x13local_snapshot_name\x18\x06 \x01(\tR\x11localSnapshotName\x128\n" + - "\x18destination_snapshot_uri\x18\a \x01(\tR\x16destinationSnapshotUri\x12C\n" + - "\x10desired_fidelity\x18\b \x01(\x0e2\x18.atelet.SnapshotFidelityR\x0fdesiredFidelity\" \n" + - "\x1eUploadPausedCheckpointResponse\"\xfc\x05\n" + + "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12,\n" + + "\bsnapshot\x18\b \x01(\v2\x10.atelet.SnapshotR\bsnapshot\x129\n" + + "\rupload_config\x18\t \x01(\v2\x14.atelet.UploadConfigR\fuploadConfig\"F\n" + + "\x12CheckpointResponse\x120\n" + + "\x05state\x18\x01 \x01(\v2\x1a.atelet.ActorSnapshotStateR\x05state\"\xa2\x04\n" + "\x0eRestoreRequest\x12$\n" + "\x0eworker_pod_uid\x18\x01 \x01(\tR\fworkerPodUid\x12\x1a\n" + "\batespace\x18\x02 \x01(\tR\batespace\x12\x1d\n" + @@ -3452,43 +3824,49 @@ const file_atelet_proto_rawDesc = "" + "\tactor_uid\x18\x04 \x01(\tR\bactorUid\x126\n" + "\x17actor_template_atespace\x18\x05 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x06 \x01(\tR\x11actorTemplateName\x12(\n" + - "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12*\n" + - "\x04type\x18\b \x01(\x0e2\x16.atelet.CheckpointTypeR\x04type\x12I\n" + - "\flocal_config\x18\t \x01(\v2$.atelet.LocalCheckpointConfigurationH\x00R\vlocalConfig\x12O\n" + - "\x0fexternal_config\x18\n" + - " \x01(\v2$.atelet.ExternalRestoreConfigurationH\x00R\x0eexternalConfig\x124\n" + - "\bfidelity\x18\v \x01(\x0e2\x18.atelet.SnapshotFidelityR\bfidelity\x12A\n" + - "\x0eegress_gateway\x18\r \x01(\v2\x15.atelet.EgressGatewayH\x01R\regressGateway\x88\x01\x01\x12\x1b\n" + + "\x04spec\x18\a \x01(\v2\x14.atelet.WorkloadSpecR\x04spec\x12,\n" + + "\bsnapshot\x18\b \x01(\v2\x10.atelet.SnapshotR\bsnapshot\x12A\n" + + "\x0eegress_gateway\x18\r \x01(\v2\x15.atelet.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + "\tcpu_milli\x18\x0e \x01(\x03R\bcpuMilli\x12!\n" + "\fmemory_bytes\x18\x0f \x01(\x03R\vmemoryBytes\x12<\n" + - "\x0esandbox_assets\x18\x10 \x01(\v2\x15.atelet.SandboxAssetsR\rsandboxAssetsB\b\n" + - "\x06configB\x11\n" + + "\x0esandbox_assets\x18\x10 \x01(\v2\x15.atelet.SandboxAssetsR\rsandboxAssetsB\x11\n" + "\x0f_egress_gateway\"\x11\n" + - "\x0fRestoreResponse*\x9a\x01\n" + + "\x0fRestoreResponse\"\x86\x02\n" + + "\x16PromoteSnapshotRequest\x12\x1a\n" + + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + + "\n" + + "actor_name\x18\x02 \x01(\tR\tactorName\x12\x1b\n" + + "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x126\n" + + "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + + "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12,\n" + + "\bsnapshot\x18\x06 \x01(\v2\x10.atelet.SnapshotR\bsnapshot\"K\n" + + "\x17PromoteSnapshotResponse\x120\n" + + "\x05state\x18\x01 \x01(\v2\x1a.atelet.ActorSnapshotStateR\x05state*\x9a\x01\n" + "\x12ActorMetadataField\x12$\n" + " ACTOR_METADATA_FIELD_UNSPECIFIED\x10\x00\x12\x1d\n" + "\x19ACTOR_METADATA_FIELD_NAME\x10\x01\x12!\n" + "\x1dACTOR_METADATA_FIELD_ATESPACE\x10\x02\x12\x1c\n" + - "\x18ACTOR_METADATA_FIELD_UID\x10\x03*j\n" + - "\x0eCheckpointType\x12\x1f\n" + - "\x1bCHECKPOINT_TYPE_UNSPECIFIED\x10\x00\x12\x19\n" + - "\x15CHECKPOINT_TYPE_LOCAL\x10\x01\x12\x1c\n" + - "\x18CHECKPOINT_TYPE_EXTERNAL\x10\x02*\x90\x01\n" + + "\x18ACTOR_METADATA_FIELD_UID\x10\x03*\x90\x01\n" + "\x10SnapshotFidelity\x12!\n" + "\x1dSNAPSHOT_FIDELITY_UNSPECIFIED\x10\x00\x12\x1d\n" + "\x19SNAPSHOT_FIDELITY_VOLUMES\x10\x01\x12\x1c\n" + "\x18SNAPSHOT_FIDELITY_ROOTFS\x10\x02\x12\x1c\n" + - "\x18SNAPSHOT_FIDELITY_MEMORY\x10\x032\xa8\x02\n" + + "\x18SNAPSHOT_FIDELITY_MEMORY\x10\x03*^\n" + + "\n" + + "UploadMode\x12\x1b\n" + + "\x17UPLOAD_MODE_UNSPECIFIED\x10\x00\x12\x18\n" + + "\x14UPLOAD_MODE_DEFERRED\x10\x01\x12\x19\n" + + "\x15UPLOAD_MODE_IMMEDIATE\x10\x022\xa8\x02\n" + "\fAteomSupport\x12c\n" + "\x14MintActorCertificate\x12#.atelet.MintActorCertificateRequest\x1a$.atelet.MintActorCertificateResponse\"\x00\x12Q\n" + "\x0eRegisterWorker\x12\x1d.atelet.RegisterWorkerRequest\x1a\x1e.atelet.RegisterWorkerResponse\"\x00\x12`\n" + - "\x13RequestActorSuspend\x12\".atelet.RequestActorSuspendRequest\x1a#.atelet.RequestActorSuspendResponse\"\x002\xf3\x02\n" + + "\x13RequestActorSuspend\x12\".atelet.RequestActorSuspendRequest\x1a#.atelet.RequestActorSuspendResponse\"\x002\xde\x02\n" + "\vAteomHerder\x120\n" + "\x03Run\x12\x12.atelet.RunRequest\x1a\x13.atelet.RunResponse\"\x00\x12E\n" + "\n" + "Checkpoint\x12\x19.atelet.CheckpointRequest\x1a\x1a.atelet.CheckpointResponse\"\x00\x12<\n" + - "\aRestore\x12\x16.atelet.RestoreRequest\x1a\x17.atelet.RestoreResponse\"\x00\x12i\n" + - "\x16UploadPausedCheckpoint\x12%.atelet.UploadPausedCheckpointRequest\x1a&.atelet.UploadPausedCheckpointResponse\"\x00\x12B\n" + + "\aRestore\x12\x16.atelet.RestoreRequest\x1a\x17.atelet.RestoreResponse\"\x00\x12T\n" + + "\x0fPromoteSnapshot\x12\x1e.atelet.PromoteSnapshotRequest\x1a\x1f.atelet.PromoteSnapshotResponse\"\x00\x12B\n" + "\tTerminate\x12\x18.atelet.TerminateRequest\x1a\x19.atelet.TerminateResponse\"\x00B>Z atelet.WorkerResources @@ -3573,12 +3954,12 @@ var file_atelet_proto_depIdxs = []int32{ 22, // 8: atelet.RunRequest.spec:type_name -> atelet.WorkloadSpec 21, // 9: atelet.RunRequest.sandbox_assets:type_name -> atelet.SandboxAssets 18, // 10: atelet.RunRequest.egress_gateway:type_name -> atelet.EgressGateway - 50, // 11: atelet.ArchAssets.files:type_name -> atelet.ArchAssets.FilesEntry - 51, // 12: atelet.SandboxAssets.assets:type_name -> atelet.SandboxAssets.AssetsEntry + 52, // 11: atelet.ArchAssets.files:type_name -> atelet.ArchAssets.FilesEntry + 53, // 12: atelet.SandboxAssets.assets:type_name -> atelet.SandboxAssets.AssetsEntry 33, // 13: atelet.WorkloadSpec.containers:type_name -> atelet.Container 31, // 14: atelet.WorkloadSpec.volumes:type_name -> atelet.Volume - 52, // 15: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry - 53, // 16: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry + 54, // 15: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry + 55, // 16: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry 0, // 17: atelet.ActorMetadataItem.field:type_name -> atelet.ActorMetadataField 26, // 18: atelet.ActorMetadataDataSource.items:type_name -> atelet.ActorMetadataItem 27, // 19: atelet.SystemInfoDataSource.actor_metadata:type_name -> atelet.ActorMetadataDataSource @@ -3594,43 +3975,50 @@ var file_atelet_proto_depIdxs = []int32{ 34, // 29: atelet.Container.security_context:type_name -> atelet.SecurityContext 36, // 30: atelet.Container.resources:type_name -> atelet.ResourceLimits 35, // 31: atelet.SecurityContext.capabilities:type_name -> atelet.Capabilities - 39, // 32: atelet.WakeupProbe.http_get:type_name -> atelet.HTTPGetAction - 22, // 33: atelet.CheckpointRequest.spec:type_name -> atelet.WorkloadSpec - 1, // 34: atelet.CheckpointRequest.type:type_name -> atelet.CheckpointType - 41, // 35: atelet.CheckpointRequest.local_config:type_name -> atelet.LocalCheckpointConfiguration - 42, // 36: atelet.CheckpointRequest.external_config:type_name -> atelet.ExternalCheckpointConfiguration - 2, // 37: atelet.CheckpointRequest.fidelity:type_name -> atelet.SnapshotFidelity - 2, // 38: atelet.UploadPausedCheckpointRequest.desired_fidelity:type_name -> atelet.SnapshotFidelity - 22, // 39: atelet.RestoreRequest.spec:type_name -> atelet.WorkloadSpec - 1, // 40: atelet.RestoreRequest.type:type_name -> atelet.CheckpointType - 41, // 41: atelet.RestoreRequest.local_config:type_name -> atelet.LocalCheckpointConfiguration - 43, // 42: atelet.RestoreRequest.external_config:type_name -> atelet.ExternalRestoreConfiguration - 2, // 43: atelet.RestoreRequest.fidelity:type_name -> atelet.SnapshotFidelity - 18, // 44: atelet.RestoreRequest.egress_gateway:type_name -> atelet.EgressGateway - 21, // 45: atelet.RestoreRequest.sandbox_assets:type_name -> atelet.SandboxAssets - 19, // 46: atelet.ArchAssets.FilesEntry.value:type_name -> atelet.AssetFile - 20, // 47: atelet.SandboxAssets.AssetsEntry.value:type_name -> atelet.ArchAssets - 13, // 48: atelet.AteomSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest - 3, // 49: atelet.AteomSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest - 11, // 50: atelet.AteomSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest - 17, // 51: atelet.AteomHerder.Run:input_type -> atelet.RunRequest - 44, // 52: atelet.AteomHerder.Checkpoint:input_type -> atelet.CheckpointRequest - 48, // 53: atelet.AteomHerder.Restore:input_type -> atelet.RestoreRequest - 46, // 54: atelet.AteomHerder.UploadPausedCheckpoint:input_type -> atelet.UploadPausedCheckpointRequest - 15, // 55: atelet.AteomHerder.Terminate:input_type -> atelet.TerminateRequest - 14, // 56: atelet.AteomSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse - 10, // 57: atelet.AteomSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse - 12, // 58: atelet.AteomSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse - 40, // 59: atelet.AteomHerder.Run:output_type -> atelet.RunResponse - 45, // 60: atelet.AteomHerder.Checkpoint:output_type -> atelet.CheckpointResponse - 49, // 61: atelet.AteomHerder.Restore:output_type -> atelet.RestoreResponse - 47, // 62: atelet.AteomHerder.UploadPausedCheckpoint:output_type -> atelet.UploadPausedCheckpointResponse - 16, // 63: atelet.AteomHerder.Terminate:output_type -> atelet.TerminateResponse - 56, // [56:64] is the sub-list for method output_type - 48, // [48:56] is the sub-list for method input_type - 48, // [48:48] is the sub-list for extension type_name - 48, // [48:48] is the sub-list for extension extendee - 0, // [0:48] is the sub-list for field type_name + 39, // 32: atelet.WakeupProbe.http_get:type_name -> atelet.HttpGetAction + 1, // 33: atelet.Snapshot.fidelity:type_name -> atelet.SnapshotFidelity + 41, // 34: atelet.Snapshot.object:type_name -> atelet.ObjectStorage + 42, // 35: atelet.SnapshotReport.snapshot:type_name -> atelet.Snapshot + 56, // 36: atelet.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp + 56, // 37: atelet.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp + 43, // 38: atelet.ActorSnapshotState.latest_local:type_name -> atelet.SnapshotReport + 43, // 39: atelet.ActorSnapshotState.uploading:type_name -> atelet.SnapshotReport + 43, // 40: atelet.ActorSnapshotState.latest_durable:type_name -> atelet.SnapshotReport + 56, // 41: atelet.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp + 2, // 42: atelet.UploadConfig.mode:type_name -> atelet.UploadMode + 22, // 43: atelet.CheckpointRequest.spec:type_name -> atelet.WorkloadSpec + 42, // 44: atelet.CheckpointRequest.snapshot:type_name -> atelet.Snapshot + 45, // 45: atelet.CheckpointRequest.upload_config:type_name -> atelet.UploadConfig + 44, // 46: atelet.CheckpointResponse.state:type_name -> atelet.ActorSnapshotState + 22, // 47: atelet.RestoreRequest.spec:type_name -> atelet.WorkloadSpec + 42, // 48: atelet.RestoreRequest.snapshot:type_name -> atelet.Snapshot + 18, // 49: atelet.RestoreRequest.egress_gateway:type_name -> atelet.EgressGateway + 21, // 50: atelet.RestoreRequest.sandbox_assets:type_name -> atelet.SandboxAssets + 42, // 51: atelet.PromoteSnapshotRequest.snapshot:type_name -> atelet.Snapshot + 44, // 52: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState + 19, // 53: atelet.ArchAssets.FilesEntry.value:type_name -> atelet.AssetFile + 20, // 54: atelet.SandboxAssets.AssetsEntry.value:type_name -> atelet.ArchAssets + 13, // 55: atelet.AteomSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest + 3, // 56: atelet.AteomSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest + 11, // 57: atelet.AteomSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest + 17, // 58: atelet.AteomHerder.Run:input_type -> atelet.RunRequest + 46, // 59: atelet.AteomHerder.Checkpoint:input_type -> atelet.CheckpointRequest + 48, // 60: atelet.AteomHerder.Restore:input_type -> atelet.RestoreRequest + 50, // 61: atelet.AteomHerder.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest + 15, // 62: atelet.AteomHerder.Terminate:input_type -> atelet.TerminateRequest + 14, // 63: atelet.AteomSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse + 10, // 64: atelet.AteomSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse + 12, // 65: atelet.AteomSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse + 40, // 66: atelet.AteomHerder.Run:output_type -> atelet.RunResponse + 47, // 67: atelet.AteomHerder.Checkpoint:output_type -> atelet.CheckpointResponse + 49, // 68: atelet.AteomHerder.Restore:output_type -> atelet.RestoreResponse + 51, // 69: atelet.AteomHerder.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse + 16, // 70: atelet.AteomHerder.Terminate:output_type -> atelet.TerminateResponse + 63, // [63:71] is the sub-list for method output_type + 55, // [55:63] is the sub-list for method input_type + 55, // [55:55] is the sub-list for extension type_name + 55, // [55:55] is the sub-list for extension extendee + 0, // [0:55] is the sub-list for field type_name } func init() { file_atelet_proto_init() } @@ -3639,31 +4027,14 @@ func file_atelet_proto_init() { return } file_atelet_proto_msgTypes[14].OneofWrappers = []any{} - file_atelet_proto_msgTypes[26].OneofWrappers = []any{ - (*SystemInfoDataSource_ActorMetadata)(nil), - (*SystemInfoDataSource_TrustBundle)(nil), - } - file_atelet_proto_msgTypes[28].OneofWrappers = []any{ - (*Volume_DurableDir)(nil), - (*Volume_External)(nil), - (*Volume_SystemInfo)(nil), - (*Volume_Image)(nil), - } - file_atelet_proto_msgTypes[41].OneofWrappers = []any{ - (*CheckpointRequest_LocalConfig)(nil), - (*CheckpointRequest_ExternalConfig)(nil), - } - file_atelet_proto_msgTypes[45].OneofWrappers = []any{ - (*RestoreRequest_LocalConfig)(nil), - (*RestoreRequest_ExternalConfig)(nil), - } + file_atelet_proto_msgTypes[45].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_atelet_proto_rawDesc), len(file_atelet_proto_rawDesc)), NumEnums: 3, - NumMessages: 51, + NumMessages: 53, NumExtensions: 0, NumServices: 2, }, diff --git a/internal/proto/ateletpb/atelet.proto b/internal/proto/ateletpb/atelet.proto index 48c5a0a4a1..aff782894d 100644 --- a/internal/proto/ateletpb/atelet.proto +++ b/internal/proto/ateletpb/atelet.proto @@ -16,6 +16,8 @@ syntax = "proto3"; package atelet; +import "google/protobuf/timestamp.proto"; + option go_package = "github.com/agent-substrate/substrate/internal/proto/ateletpb"; // AteomSupport provides callouts for ateom. @@ -242,19 +244,31 @@ service AteomHerder { // ateom. rpc Run(RunRequest) returns (RunResponse) {} - // Checkpoint tells atelet to save the current state of the workload on an - // ateom to object storage, and then completely the ateom to a blank state - // (back to "available" state.) + // Checkpoint saves the workload's state as a new snapshot of the actor and + // returns the ateom to a blank state. upload_config says when the snapshot + // becomes durable. Once ateom has captured the workload, and before the new + // snapshot is uploaded or stored locally, atelet deletes the actor's older + // local snapshots, except one still being uploaded, which it deletes when + // that upload ends. A Checkpoint that then fails to upload or store the new + // snapshot leaves the actor with no local snapshot. rpc Checkpoint(CheckpointRequest) returns (CheckpointResponse) {} - // Restore restores a workload from checkpoint onto an ateom. + // Restore restores a workload from a snapshot onto an ateom. rpc Restore(RestoreRequest) returns (RestoreResponse) {} - // UploadPausedCheckpoint copies a local (pause) checkpoint from this node's - // disk to object storage. Unlike Checkpoint it drives no ateom: the actor is - // paused, its sandbox is gone; the checkpoint files plus their manifest - // already sit under the actor's local-checkpoints directory. - rpc UploadPausedCheckpoint(UploadPausedCheckpointRequest) returns (UploadPausedCheckpointResponse) {} + // PromoteSnapshot makes `snapshot` durable in object storage while keeping + // its local copy on this node. Used when a snapshot needs to be in object + // storage (e.g. for tagging) without evicting the paused actor from the + // node. Waits for an in-flight detach, and fails for an actor running on + // this node. + // + // Idempotent: once the snapshot is durable, atelet answers from the + // manifest at `snapshot.object`, and fails if there is none. + // + // TODO: Temporarily, this deletes all of the actor's local snapshots after + // the upload and does not check whether the actor is running. Its only + // caller, the suspend of a paused actor, relies on the deletion. + rpc PromoteSnapshot(PromoteSnapshotRequest) returns (PromoteSnapshotResponse) {} // Terminate tells atelet to terminate/kill any running workload for an actor, // unmount its volumes, and clean up actor state on the node. @@ -262,16 +276,39 @@ service AteomHerder { } message TerminateRequest { - // If unset, the atelet will just cleanup node resources for the actor. + // The UID of the worker pod whose ateom hosts the actor. If unset, atelet + // only cleans up the actor's node resources; it does not reach an ateom. + // + // +k8s:optional + // +k8s:format=k8s-uuid string worker_pod_uid = 1; + // +k8s:required + // +k8s:format=k8s-short-name string atespace = 2; + + // +k8s:required + // +k8s:format=k8s-short-name string actor_name = 3; + + // +k8s:required + // +k8s:format=k8s-uuid string actor_uid = 4; + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_atespace = 5; + + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_name = 6; + // A nil spec is tolerated: the delete flow terminates template-less + // actors with a fallback spec. + // + // +k8s:optional WorkloadSpec spec = 7; } @@ -279,29 +316,60 @@ message TerminateResponse { } message RunRequest { + // The UID of the worker pod whose ateom hosts the actor. + // + // +k8s:required + // +k8s:format=k8s-uuid string worker_pod_uid = 1; + // +k8s:required + // +k8s:format=k8s-short-name string atespace = 2; + + // +k8s:required + // +k8s:format=k8s-short-name string actor_name = 3; + + // +k8s:required + // +k8s:format=k8s-uuid string actor_uid = 4; + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_atespace = 5; + + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_name = 6; + // +k8s:required WorkloadSpec spec = 7; // The sandbox binaries to use for booting this actor from scratch. atelet // fetches the relevant assets and records them with the actor's on-node state // so a later Checkpoint can pin the same version into the snapshot manifest. + // + // +k8s:required SandboxAssets sandbox_assets = 8; // When absent the actor has no egress: its TCP is captured and refused. + // + // +k8s:optional optional EgressGateway egress_gateway = 9; // The actor's declared size, from the ActorTemplate's resource limits. atelet // passes these through to the sandbox so it is sized to the actor (not the // whole host or worker pod). Zero means "unset": keep the runtime default. - int64 cpu_milli = 10; // CPU limit in millicores (1000 = one core). + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores + int64 cpu_milli = 10; // CPU limit in millicores (1000 = one core). + + // +k8s:optional + // +k8s:minimum=0 int64 memory_bytes = 11; // Memory limit in bytes. } @@ -309,6 +377,10 @@ message RunRequest { message EgressGateway { // address is required and identifies the remote gateway as an IP address or // DNS name followed by its port. + // + // +k8s:required + // +k8s:maxLength=261 # a bracketed or 253-character host, ':' and a 5-digit port + // +k8s:customValidation # host:port shape string address = 1; } @@ -320,14 +392,25 @@ message AssetFile { // the plugin cannot serve it. // Masked in logged requests, since a signed URL carries its credential in // the query. + // + // +k8s:required + // +k8s:maxLength=2048 # matches the SandboxConfig asset url's bound string url = 1 [ debug_redact = true ]; + // Lower-case hex SHA256; names the cached file and verifies its integrity. + // + // +k8s:required + // +k8s:maxLength=64 + // +k8s:customValidation # 64 hex characters string sha256 = 2; } // ArchAssets is the set of assets for a single architecture, keyed by asset // name (a wrapper message because proto map values cannot themselves be maps). message ArchAssets { + // +k8s:required + // +k8s:maxProperties=16 + // +k8s:eachKey=+k8s:maxLength=64 map files = 1; // asset name -> file } @@ -337,7 +420,17 @@ message ArchAssets { // atelet's backend code interprets the asset names (gVisor expects "gvisor", // the release tarball; legacy "runsc", a bare binary, is still accepted). message SandboxAssets { - string sandbox_class = 1; // e.g. "gvisor" + // TODO: send this as an enum, like ateapi's and ateom's SandboxClass, so + // the tags can bound it and the custom validation goes away. + // + // +k8s:required + // +k8s:maxLength=63 # guardrail; the class names are a few characters + // +k8s:customValidation # gvisor or microvm + string sandbox_class = 1; // e.g. "gvisor" + + // +k8s:required + // +k8s:maxProperties=8 # one entry per architecture + // +k8s:eachKey=+k8s:maxLength=16 map assets = 2; // arch -> {name -> file} // pause_image is the image for the sandbox's root container. Like the @@ -345,12 +438,27 @@ message SandboxAssets { // and atelet pins it into the snapshot manifest so a restore rebuilds the // sandbox from the same image. Empty for sandboxes without a pause // container (microvm). + // + // +k8s:optional + // +k8s:maxLength=512 + // +k8s:customValidation # must be a well-formed image reference, pinned by digest string pause_image = 3; } // WorkloadSpec parallels Pod, but with far fewer configurable fields. +// +// +k8s:customValidation # every volume mount names a declared volume message WorkloadSpec { + // +k8s:optional + // +k8s:maxItems=10 # matches the template's containers bound + // +k8s:listType=map + // +k8s:listMapKey=name repeated Container containers = 1; + + // +k8s:optional + // +k8s:maxItems=32 # matches the template's volumes bound + // +k8s:listType=map + // +k8s:listMapKey=name repeated Volume volumes = 2; } @@ -358,16 +466,40 @@ message DurableDirVolume { } message ExternalVolumeSource { + // Required, unlike the actor status counterpart: the control plane only + // sends volumes that finished provisioning. + // + // +k8s:required + // +k8s:maxLength=256 # matches ExternalVolume.storage_volume_id's bound + // +k8s:customValidation # no control characters string storage_volume_id = 1; + + // +k8s:optional + // +k8s:maxLength=253 # matches ExternalVolume.volume_type's bound + // +k8s:customValidation # optional "substrate.io/" prefix + DNS subdomain string volume_type = 2; + + // volume_context is the driver's provisioning metadata. Its entries are the + // driver's own, so the map is only bounded, not validated. + // + // +k8s:optional + // +k8s:customValidation # at most 40 KiB in total, ten times the CSI limit for map fields map volume_context = 3; // publish_context is the driver's attachment metadata for this node, which - // the node plugin needs to complete the mount. + // the node plugin needs to complete the mount. Bounded like volume_context. + // + // +k8s:optional + // +k8s:customValidation # at most 40 KiB in total, ten times the CSI limit for map fields map publish_context = 4; } message ImageVolumeSource { + // reference is the OCI image reference, pinned by digest. + // + // +k8s:required + // +k8s:maxLength=512 # matches the template ImageVolumeSource.reference's bound + // +k8s:customValidation # must be a well-formed image reference, pinned by digest string reference = 1; } @@ -377,83 +509,197 @@ enum ActorMetadataField { ACTOR_METADATA_FIELD_NAME = 1; ACTOR_METADATA_FIELD_ATESPACE = 2; ACTOR_METADATA_FIELD_UID = 3; + // Keep this in sync with the maximums on fields of this type. } // ActorMetadataItem projects one actor identity field to one file at the // given path, relative to the root of the enclosing system-info volume. message ActorMetadataItem { + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=3 # keep this in sync with the ActorMetadataField enum ActorMetadataField field = 1; + + // path must be a clean relative Unix path; the rule is shared with the + // control plane through internal/resources. + // + // +k8s:required + // +k8s:minLength=1 + // +k8s:maxLength=255 + // +k8s:customValidation # clean relative path string path = 2; } // ActorMetadataDataSource projects the actor's identity fields to files, one // per item. Values are written raw with no trailing newline. message ActorMetadataDataSource { + // items must not project the same field twice. + // + // +k8s:required + // +k8s:minItems=1 + // +k8s:maxItems=8 # matches the template's items bound + // +k8s:listType=atomic + // +k8s:unique=map + // +k8s:listMapKey=field repeated ActorMetadataItem items = 1; } // TrustBundleDataSource corresponds to TrustBundleDataSource in ateapi.proto message TrustBundleDataSource { + // path must be a clean relative Unix path; the rule is shared with the + // control plane through internal/resources. + // + // +k8s:required + // +k8s:minLength=1 + // +k8s:maxLength=255 + // +k8s:customValidation # clean relative path string path = 1; + // +k8s:required + // +k8s:minItems=1 + // +k8s:maxItems=8 # matches the template's names bound + // +k8s:listType=set + // +k8s:eachVal=+k8s:minLength=1 + // +k8s:eachVal=+k8s:maxLength=253 # matches the template's bundle-name bound repeated string names = 3; reserved 2; reserved "name"; } +// SystemInfoDataSource selects exactly one projection to place in the +// volume. message SystemInfoDataSource { - oneof data_source { - ActorMetadataDataSource actor_metadata = 1; - TrustBundleDataSource trust_bundle = 2; - } + // Exactly one of actor_metadata / trust_bundle must be set. + // + // +k8s:optional + // +k8s:unionMember + ActorMetadataDataSource actor_metadata = 1; + + // +k8s:optional + // +k8s:unionMember + TrustBundleDataSource trust_bundle = 2; } // SystemInfoVolume is a read-only volume whose files are generated by atelet // on every Run/Restore, so they carry the values of the actor actually being // started, whatever checkpointed state it boots from. message SystemInfoVolume { + // +k8s:optional + // +k8s:maxItems=8 # matches the template's data_sources bound + // +k8s:listType=atomic + // +k8s:customValidation # at most one actor_metadata; paths unique across entries repeated SystemInfoDataSource data_sources = 1; } +// Volume names one volume and selects exactly one source for it. message Volume { + // +k8s:required + // +k8s:format=k8s-short-name string name = 1; - oneof source { - DurableDirVolume durable_dir = 2; - ExternalVolumeSource external = 3; - SystemInfoVolume system_info = 4; - ImageVolumeSource image = 5; - } + // Exactly one of durable_dir / external / system_info / image must be set. + // + // +k8s:optional + // +k8s:unionMember + DurableDirVolume durable_dir = 2; + + // +k8s:optional + // +k8s:unionMember + ExternalVolumeSource external = 3; + + // +k8s:optional + // +k8s:unionMember + SystemInfoVolume system_info = 4; + + // +k8s:optional + // +k8s:unionMember + ImageVolumeSource image = 5; } message VolumeMount { + // name must match the name of a Volume. + // + // +k8s:required + // +k8s:format=k8s-short-name string name = 1; + + // mount_path within the container. Must be a clean absolute Unix path. + // + // +k8s:required + // +k8s:maxLength=4096 + // +k8s:customValidation # clean-absolute-path shape; no regex/pattern tag exists string mount_path = 2; } message Container { + // +k8s:required + // +k8s:format=k8s-short-name string name = 1; + + // +k8s:required + // +k8s:maxLength=512 # matches the template Container.image's bound + // +k8s:customValidation # must be a well-formed image reference, pinned by digest string image = 2; + + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=atomic + // +k8s:eachVal=+k8s:maxLength=4096 # argv strings; guardrail, not a contract repeated string command = 3; + + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=atomic + // +k8s:eachVal=+k8s:maxLength=4096 # argv strings; guardrail, not a contract repeated string args = 7; + + // +k8s:optional + // +k8s:maxItems=32 # matches the template's env bound + // +k8s:listType=map # each variable is set at most once + // +k8s:listMapKey=name repeated EnvEntry env = 4; + + // +k8s:optional WakeupProbe wakeup_probe = 5; + + // Keyed by mount_path: each path hosts exactly one mount, while a volume + // may be mounted at multiple paths. + // + // +k8s:optional + // +k8s:maxItems=32 # matches the template's volume_mounts bound + // +k8s:listType=map + // +k8s:listMapKey=mount_path + // +k8s:customValidation # mounts must not nest repeated VolumeMount volume_mounts = 6; + + // +k8s:optional SecurityContext security_context = 8; // resources are the cgroup limits for this container, resolved by // ate-api-server from the ActorTemplate. Unset means no limits. + // + // +k8s:optional ResourceLimits resources = 9; } // SecurityContext holds security settings for a container's process. message SecurityContext { + // +k8s:optional Capabilities capabilities = 1; } // Capabilities adjusts a container's Linux capabilities relative to the default // set. Names carry no "CAP_" prefix; drop applies before add. message Capabilities { + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=set + // +k8s:customValidation # capability grammar; "ALL" not accepted repeated string add = 1; + + // +k8s:optional + // +k8s:maxItems=64 + // +k8s:listType=set + // +k8s:customValidation # capability grammar repeated string drop = 2; } @@ -461,58 +707,66 @@ message Capabilities { // map so each resource.Quantity is parsed exactly once, in ate-api-server. message ResourceLimits { // memory_bytes is the memory limit in bytes. 0 means unset. + // + // +k8s:optional + // +k8s:minimum=0 int64 memory_bytes = 1; + // cpu_millis is the CPU limit in milli-cores (1000 = one core). 0 means unset. + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores int64 cpu_millis = 2; } message EnvEntry { + // name may be any printable ASCII character except '='. + // + // +k8s:required + // +k8s:maxLength=256 # guardrail + // +k8s:customValidation # printable ASCII except '=' string name = 1; + + // +k8s:optional + // +k8s:maxLength=32768 # guardrail string value = 2 [ debug_redact = true ]; } // WakeupProbe describes how to check that a container is ready to serve. // Only HTTP is supported today. message WakeupProbe { - HTTPGetAction http_get = 1; + // +k8s:required + HttpGetAction http_get = 1; + // How long to keep polling before giving up and failing the actor start. + // + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=3600 # matches the template wakeup probe's bound int32 timeout_seconds = 2; } -// HTTPGetAction performs an HTTP GET against the container. -message HTTPGetAction { +// HttpGetAction performs an HTTP GET against the container. +message HttpGetAction { // Path to access on the HTTP server. + // + // +k8s:required + // +k8s:maxLength=1024 # matches the template probe path's bound + // +k8s:customValidation # RFC 3986 path shape; no regex/pattern tag exists string path = 1; + // TCP port to connect to (1..65535). + // + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=65535 int32 port = 2; } message RunResponse { } -message LocalCheckpointConfiguration { - // Names a sub-directory on the local filesystem below which the checkpoint - // data will be stored. atelet decides where that directory lives, from the - // actor's UID, so this is a bare name and must not contain a path separator. - // The structure of the checkpoint should generally be treated as opaque. - string snapshot_name = 1; -} - -message ExternalCheckpointConfiguration { - // The object storage URI of the snapshot to write. Object names are appended - // to it, so it addresses the snapshot as a whole rather than any one object. - string snapshot_uri = 1; -} - -enum CheckpointType { - // Should never happen - CHECKPOINT_TYPE_UNSPECIFIED = 0; - // Save snapshot only in local filesystem - CHECKPOINT_TYPE_LOCAL = 1; - // Save snapshot to object storage - CHECKPOINT_TYPE_EXTERNAL = 2; -} - // SnapshotFidelity mirrors the public ateapi enum: the state layers a // snapshot holds, lowest to highest, each including the ones below it. enum SnapshotFidelity { @@ -528,111 +782,321 @@ enum SnapshotFidelity { // Capture process memory plus the full filesystem delta on top of the OCI // image (including any attached DurableDir volumes). SNAPSHOT_FIDELITY_MEMORY = 3; + // Keep this in sync with the maximums on fields of this type. } -// ExternalRestoreConfiguration is an external snapshot a restore reads. -// Split from the checkpoint-side ExternalCheckpointConfiguration (a write -// destination) so read-side attributes of a restore source have a home. -message ExternalRestoreConfiguration { - // The object storage URI of the snapshot to read. Object names are - // appended to it, so it addresses the snapshot as a whole rather than any - // one object. +message ObjectStorage { + // The object storage URI of the snapshot. Object names are appended to it, + // so it addresses the snapshot as a whole rather than any one object. + // + // +k8s:required + // +k8s:maxLength=2048 + // +k8s:customValidation # a parseable snapshot URI string snapshot_uri = 1; } +// Snapshot identifies one snapshot of an actor, and where its durable copy +// lives. +message Snapshot { + // ateapi Snapshot.uuid. Names the snapshot's directory under the actor's + // local-checkpoint directory, so this is a bare name and must not contain + // a path separator. + // + // +k8s:required + // +k8s:format=k8s-short-name + string snapshot_uuid = 1; + + // ateapi Snapshot.generation, starting at 1. Orders the actor's snapshots. + // Checkpoint and PromoteSnapshot fail for a generation below atelet's newest + // for the actor, or for its newest under another snapshot_uuid: the + // caller's view is stale. + // + // TODO: Temporarily, the control plane does not set this and atelet does + // not check it. + // + // +k8s:optional + int32 generation = 2; + + // On Checkpoint: the fidelity to capture. + // On Restore: the fidelity to restore the actor with. + // On PromoteSnapshot: the lowest fidelity the snapshot may hold. The call + // fails if its manifest records a lower one. + // In SnapshotReport: the fidelity the snapshot holds. + // + // TODO: Open question: should PromoteSnapshot accept a manifest that records + // a higher fidelity than this? Until that is decided, atelet requires the + // two to be equal. + // + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=3 # keep this in sync with the SnapshotFidelity enum + SnapshotFidelity fidelity = 3; + + // The snapshot's durable copy in object storage. Required on every + // request: atelet uploads there, or reads from there when it holds no + // usable local copy. + // + // TODO: Temporarily, this is optional while we haven't implemented deferred + // uploads. Restore also uses its presence to choose between the local copy + // and object storage; see RestoreRequest.snapshot. + // + // +k8s:optional + ObjectStorage object = 4; +} + +// SnapshotReport is a snapshot atelet holds or has uploaded, and when. +message SnapshotReport { + Snapshot snapshot = 1; + + // When the checkpoint that produced the snapshot committed on this node. + google.protobuf.Timestamp local_commit_time = 2; + + // When the snapshot's manifest, the upload's commit marker, was written to + // object storage. Set only in ActorSnapshotState.latest_durable, and only + // when this atelet wrote it. + google.protobuf.Timestamp durable_complete_time = 3; +} + +// ActorSnapshotState is what atelet holds for one actor on this node. +// Returned by Checkpoint, PromoteSnapshot. +message ActorSnapshotState { + // The snapshot atelet restores the actor from locally. Unset when the + // latest checkpoint kept no local copy, and once a detach has made the + // snapshot durable and begun deleting it. Either way, latest_durable is + // then the actor's latest snapshot. + SnapshotReport latest_local = 1; + + // The snapshot an in-flight upload is reading, if any. If it is + // latest_local, its local copy stays after the upload. If it is an older + // snapshot, whose upload began before latest_local was taken, atelet + // deletes its local copy once the upload ends. + SnapshotReport uploading = 2; + + // The newest snapshot atelet knows is durable, whether or not the control + // plane has recorded it: one atelet uploaded since it started, or one a + // PromoteSnapshot found already uploaded. Unset if there is none, which says + // nothing about older durable snapshots. + SnapshotReport latest_durable = 3; + + // When atelet next plans to trigger a background upload, which will upload + // whichever snapshot is latest_local at the time of triggering. Unset when + // there is no latest_local, or it is already durable or being uploaded. + // Advisory: a promote or detach can upload it sooner, and an older + // snapshot's upload can delay it. + google.protobuf.Timestamp upload_due_time = 4; +} + +// UploadConfig is when a checkpoint's snapshot becomes durable. +message UploadConfig { + // +k8s:required + // +k8s:minimum=1 + // +k8s:maximum=2 # keep this in sync with the UploadMode enum + UploadMode mode = 1; +} + +// UploadMode is when a checkpoint's snapshot is uploaded. +enum UploadMode { + UPLOAD_MODE_UNSPECIFIED = 0; + + // Keep the snapshot locally and upload it on atelet's schedule: at most + // for atelet's upload-delay after the oldest snapshot of the actor + // that is not yet durable was taken. Snapshots superseded before then + // are never uploaded. + // + // Once the actor has stayed suspended on this node for + // atelet's detach-delay, atelet detaches it (deletes all the local snapshots). + // + // TODO: Temporarily the deferred upload and detach are unimplemented. Only a local + // copy is taken. + UPLOAD_MODE_DEFERRED = 1; + + // Upload the snapshot synchronously before Checkpoint returns, while keeping the local copy. + // + // TODO: Temporarily this deletes the local snapshot after the upload. + UPLOAD_MODE_IMMEDIATE = 2; + // Keep this in sync with the maximums on fields of this type. +} + +// +k8s:customValidation # immediate upload requires snapshot.object message CheckpointRequest { + // The UID of the worker pod whose ateom hosts the actor. + // + // +k8s:required + // +k8s:format=k8s-uuid string worker_pod_uid = 1; + // +k8s:required + // +k8s:format=k8s-short-name string atespace = 2; + + // +k8s:required + // +k8s:format=k8s-short-name string actor_name = 3; + + // +k8s:required + // +k8s:format=k8s-uuid string actor_uid = 4; + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_atespace = 5; + + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_name = 6; // Sandbox binary config is not sent on checkpoint: atelet uses the version the // actor is currently running (recorded with the actor's on-node state at // Run/Restore) and records it into the snapshot manifest. + // + // +k8s:required WorkloadSpec spec = 7; - CheckpointType type = 8; - - // The checkpoint configuration, depending on the type. - oneof config { - LocalCheckpointConfiguration local_config = 9; - ExternalCheckpointConfiguration external_config = 10; - } + // The snapshot to take, and where it is uploaded. atelet may upload it long + // after Checkpoint returns, without asking the control plane. A request at + // atelet's newest generation for the actor, with the same snapshot_uuid, + // is a retry: atelet returns the actor's current state without + // checkpointing again. + // + // TODO: Temporarily, atelet does not recognize retries. A retry runs the + // checkpoint again, which fails once an earlier attempt has removed the + // workload. + // + // +k8s:required + Snapshot snapshot = 8; - // Fidelity the checkpoint captures. - SnapshotFidelity fidelity = 11; + // When the snapshot becomes durable. + // + // +k8s:required + UploadConfig upload_config = 9; } message CheckpointResponse { -} - -message UploadPausedCheckpointRequest { - string atespace = 1; - string actor_name = 2; - string actor_uid = 3; - - // For metrics attribution, like on CheckpointRequest. - string actor_template_atespace = 4; - string actor_template_name = 5; - - // The local checkpoint to upload: the uuid of the actor's pause snapshot - // (ateapi Snapshot.uuid). - string local_snapshot_name = 6; - - // Destination object-storage URI (the actor's in-progress snapshot URI). - string destination_snapshot_uri = 7; - - // Fidelity the uploaded snapshot must have. The fidelity the pause - // checkpoint captured is not sent: atelet reads it from the local - // snapshot's own manifest, which is authoritative, and rejects the upload - // if the two differ. - SnapshotFidelity desired_fidelity = 8; -} - -message UploadPausedCheckpointResponse { + // The actor's state once the checkpoint is taken: latest_local is the new + // snapshot, unless UPLOAD_MODE_IMMEDIATE kept no local copy. The control + // plane can drop its entries for any local-only generation not named here: + // atelet has deleted it, and will never upload it. + // + // TODO: Temporarily, atelet leaves this unset. An unset state says nothing + // about which snapshots atelet holds. + ActorSnapshotState state = 1; } message RestoreRequest { + // The UID of the worker pod whose ateom hosts the actor. + // + // +k8s:required + // +k8s:format=k8s-uuid string worker_pod_uid = 1; + // +k8s:required + // +k8s:format=k8s-short-name string atespace = 2; + + // +k8s:required + // +k8s:format=k8s-short-name string actor_name = 3; + + // +k8s:required + // +k8s:format=k8s-uuid string actor_uid = 4; + // The template identity is carried for metrics attribution. + // + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_atespace = 5; + + // +k8s:optional + // +k8s:format=k8s-short-name string actor_template_name = 6; + // +k8s:required WorkloadSpec spec = 7; - CheckpointType type = 8; - - // The checkpoint configuration, depending on the type. - oneof config { - LocalCheckpointConfiguration local_config = 9; - ExternalRestoreConfiguration external_config = 10; - } - - // Fidelity to restore from the checkpoint. - SnapshotFidelity fidelity = 11; + // Snapshot to restore from. atelet reads its local copy if it holds a + // usable one, and `snapshot.object` otherwise. A local copy is unusable if + // its manifest does not parse or lists a missing file. While atelet is + // detaching the actor, it waits for the detach to finish and then reads + // `snapshot.object`. Fails if neither copy exists: that generation is lost. + // + // TODO: Temporarily, whether `snapshot.object` is set chooses the copy: + // atelet reads only the local copy when it is unset, and only + // `snapshot.object` when it is set, with no fallback either way. + // + // +k8s:required + Snapshot snapshot = 8; // When absent the actor has no egress: its TCP is captured and refused. + // + // +k8s:optional optional EgressGateway egress_gateway = 13; // The actor's declared size, from the ActorTemplate's resource limits. For // gVisor and micro-VM VOLUMES restores the sandbox is (re)sized to these; // for a MEMORY micro-VM restore the size baked into the snapshot wins. Zero // means "unset": keep the runtime default. - int64 cpu_milli = 14; // CPU limit in millicores (1000 = one core). + // + // +k8s:optional + // +k8s:minimum=0 + // +k8s:maximum=999999 # the control plane caps cpu limits strictly below 1000 cores + int64 cpu_milli = 14; // CPU limit in millicores (1000 = one core). + + // +k8s:optional + // +k8s:minimum=0 int64 memory_bytes = 15; // Memory limit in bytes. // The sandbox binaries and pause image to restore with, resolved from the - // ActorTemplate's SandboxConfig. Required. + // ActorTemplate's SandboxConfig. + // + // +k8s:required SandboxAssets sandbox_assets = 16; } message RestoreResponse { } + +message PromoteSnapshotRequest { + // Golden actors are never paused, so the golden atespace is rejected. + // + // +k8s:required + // +k8s:format=k8s-short-name + // +k8s:customValidation # not the golden atespace + string atespace = 1; + + // +k8s:required + // +k8s:format=k8s-short-name + string actor_name = 2; + + // +k8s:required + // +k8s:format=k8s-uuid + string actor_uid = 3; + + // For metrics attribution, like on CheckpointRequest. + // + // +k8s:optional + // +k8s:format=k8s-short-name + string actor_template_atespace = 4; + + // +k8s:optional + // +k8s:format=k8s-short-name + string actor_template_name = 5; + + // The actor's latest snapshot as the control plane records it, and where + // it is, or will be, uploaded. + // + // +k8s:required + // +k8s:subfield(object)=+k8s:required + Snapshot snapshot = 6; +} + +message PromoteSnapshotResponse { + // The actor's state once promoted: latest_durable is its latest snapshot, + // and latest_local remains set if atelet still holds the local copy. + // + // TODO: Temporarily, atelet leaves this unset. An unset state says nothing + // about which snapshots atelet holds. + ActorSnapshotState state = 1; +} diff --git a/internal/proto/ateletpb/atelet_grpc.pb.go b/internal/proto/ateletpb/atelet_grpc.pb.go index bec22fb739..ce18960786 100644 --- a/internal/proto/ateletpb/atelet_grpc.pb.go +++ b/internal/proto/ateletpb/atelet_grpc.pb.go @@ -251,11 +251,11 @@ var AteomSupport_ServiceDesc = grpc.ServiceDesc{ } const ( - AteomHerder_Run_FullMethodName = "/atelet.AteomHerder/Run" - AteomHerder_Checkpoint_FullMethodName = "/atelet.AteomHerder/Checkpoint" - AteomHerder_Restore_FullMethodName = "/atelet.AteomHerder/Restore" - AteomHerder_UploadPausedCheckpoint_FullMethodName = "/atelet.AteomHerder/UploadPausedCheckpoint" - AteomHerder_Terminate_FullMethodName = "/atelet.AteomHerder/Terminate" + AteomHerder_Run_FullMethodName = "/atelet.AteomHerder/Run" + AteomHerder_Checkpoint_FullMethodName = "/atelet.AteomHerder/Checkpoint" + AteomHerder_Restore_FullMethodName = "/atelet.AteomHerder/Restore" + AteomHerder_PromoteSnapshot_FullMethodName = "/atelet.AteomHerder/PromoteSnapshot" + AteomHerder_Terminate_FullMethodName = "/atelet.AteomHerder/Terminate" ) // AteomHerderClient is the client API for AteomHerder service. @@ -270,17 +270,29 @@ type AteomHerderClient interface { // Run tells atelet to create a new containerized workload from scratch on an // ateom. Run(ctx context.Context, in *RunRequest, opts ...grpc.CallOption) (*RunResponse, error) - // Checkpoint tells atelet to save the current state of the workload on an - // ateom to object storage, and then completely the ateom to a blank state - // (back to "available" state.) + // Checkpoint saves the workload's state as a new snapshot of the actor and + // returns the ateom to a blank state. upload_config says when the snapshot + // becomes durable. Once ateom has captured the workload, and before the new + // snapshot is uploaded or stored locally, atelet deletes the actor's older + // local snapshots, except one still being uploaded, which it deletes when + // that upload ends. A Checkpoint that then fails to upload or store the new + // snapshot leaves the actor with no local snapshot. Checkpoint(ctx context.Context, in *CheckpointRequest, opts ...grpc.CallOption) (*CheckpointResponse, error) - // Restore restores a workload from checkpoint onto an ateom. + // Restore restores a workload from a snapshot onto an ateom. Restore(ctx context.Context, in *RestoreRequest, opts ...grpc.CallOption) (*RestoreResponse, error) - // UploadPausedCheckpoint copies a local (pause) checkpoint from this node's - // disk to object storage. Unlike Checkpoint it drives no ateom: the actor is - // paused, its sandbox is gone; the checkpoint files plus their manifest - // already sit under the actor's local-checkpoints directory. - UploadPausedCheckpoint(ctx context.Context, in *UploadPausedCheckpointRequest, opts ...grpc.CallOption) (*UploadPausedCheckpointResponse, error) + // PromoteSnapshot makes `snapshot` durable in object storage while keeping + // its local copy on this node. Used when a snapshot needs to be in object + // storage (e.g. for tagging) without evicting the paused actor from the + // node. Waits for an in-flight detach, and fails for an actor running on + // this node. + // + // Idempotent: once the snapshot is durable, atelet answers from the + // manifest at `snapshot.object`, and fails if there is none. + // + // TODO: Temporarily, this deletes all of the actor's local snapshots after + // the upload and does not check whether the actor is running. Its only + // caller, the suspend of a paused actor, relies on the deletion. + PromoteSnapshot(ctx context.Context, in *PromoteSnapshotRequest, opts ...grpc.CallOption) (*PromoteSnapshotResponse, error) // Terminate tells atelet to terminate/kill any running workload for an actor, // unmount its volumes, and clean up actor state on the node. Terminate(ctx context.Context, in *TerminateRequest, opts ...grpc.CallOption) (*TerminateResponse, error) @@ -324,10 +336,10 @@ func (c *ateomHerderClient) Restore(ctx context.Context, in *RestoreRequest, opt return out, nil } -func (c *ateomHerderClient) UploadPausedCheckpoint(ctx context.Context, in *UploadPausedCheckpointRequest, opts ...grpc.CallOption) (*UploadPausedCheckpointResponse, error) { +func (c *ateomHerderClient) PromoteSnapshot(ctx context.Context, in *PromoteSnapshotRequest, opts ...grpc.CallOption) (*PromoteSnapshotResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - out := new(UploadPausedCheckpointResponse) - err := c.cc.Invoke(ctx, AteomHerder_UploadPausedCheckpoint_FullMethodName, in, out, cOpts...) + out := new(PromoteSnapshotResponse) + err := c.cc.Invoke(ctx, AteomHerder_PromoteSnapshot_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } @@ -356,17 +368,29 @@ type AteomHerderServer interface { // Run tells atelet to create a new containerized workload from scratch on an // ateom. Run(context.Context, *RunRequest) (*RunResponse, error) - // Checkpoint tells atelet to save the current state of the workload on an - // ateom to object storage, and then completely the ateom to a blank state - // (back to "available" state.) + // Checkpoint saves the workload's state as a new snapshot of the actor and + // returns the ateom to a blank state. upload_config says when the snapshot + // becomes durable. Once ateom has captured the workload, and before the new + // snapshot is uploaded or stored locally, atelet deletes the actor's older + // local snapshots, except one still being uploaded, which it deletes when + // that upload ends. A Checkpoint that then fails to upload or store the new + // snapshot leaves the actor with no local snapshot. Checkpoint(context.Context, *CheckpointRequest) (*CheckpointResponse, error) - // Restore restores a workload from checkpoint onto an ateom. + // Restore restores a workload from a snapshot onto an ateom. Restore(context.Context, *RestoreRequest) (*RestoreResponse, error) - // UploadPausedCheckpoint copies a local (pause) checkpoint from this node's - // disk to object storage. Unlike Checkpoint it drives no ateom: the actor is - // paused, its sandbox is gone; the checkpoint files plus their manifest - // already sit under the actor's local-checkpoints directory. - UploadPausedCheckpoint(context.Context, *UploadPausedCheckpointRequest) (*UploadPausedCheckpointResponse, error) + // PromoteSnapshot makes `snapshot` durable in object storage while keeping + // its local copy on this node. Used when a snapshot needs to be in object + // storage (e.g. for tagging) without evicting the paused actor from the + // node. Waits for an in-flight detach, and fails for an actor running on + // this node. + // + // Idempotent: once the snapshot is durable, atelet answers from the + // manifest at `snapshot.object`, and fails if there is none. + // + // TODO: Temporarily, this deletes all of the actor's local snapshots after + // the upload and does not check whether the actor is running. Its only + // caller, the suspend of a paused actor, relies on the deletion. + PromoteSnapshot(context.Context, *PromoteSnapshotRequest) (*PromoteSnapshotResponse, error) // Terminate tells atelet to terminate/kill any running workload for an actor, // unmount its volumes, and clean up actor state on the node. Terminate(context.Context, *TerminateRequest) (*TerminateResponse, error) @@ -389,8 +413,8 @@ func (UnimplementedAteomHerderServer) Checkpoint(context.Context, *CheckpointReq func (UnimplementedAteomHerderServer) Restore(context.Context, *RestoreRequest) (*RestoreResponse, error) { return nil, status.Error(codes.Unimplemented, "method Restore not implemented") } -func (UnimplementedAteomHerderServer) UploadPausedCheckpoint(context.Context, *UploadPausedCheckpointRequest) (*UploadPausedCheckpointResponse, error) { - return nil, status.Error(codes.Unimplemented, "method UploadPausedCheckpoint not implemented") +func (UnimplementedAteomHerderServer) PromoteSnapshot(context.Context, *PromoteSnapshotRequest) (*PromoteSnapshotResponse, error) { + return nil, status.Error(codes.Unimplemented, "method PromoteSnapshot not implemented") } func (UnimplementedAteomHerderServer) Terminate(context.Context, *TerminateRequest) (*TerminateResponse, error) { return nil, status.Error(codes.Unimplemented, "method Terminate not implemented") @@ -470,20 +494,20 @@ func _AteomHerder_Restore_Handler(srv interface{}, ctx context.Context, dec func return interceptor(ctx, in, info, handler) } -func _AteomHerder_UploadPausedCheckpoint_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(UploadPausedCheckpointRequest) +func _AteomHerder_PromoteSnapshot_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(PromoteSnapshotRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomHerderServer).UploadPausedCheckpoint(ctx, in) + return srv.(AteomHerderServer).PromoteSnapshot(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomHerder_UploadPausedCheckpoint_FullMethodName, + FullMethod: AteomHerder_PromoteSnapshot_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomHerderServer).UploadPausedCheckpoint(ctx, req.(*UploadPausedCheckpointRequest)) + return srv.(AteomHerderServer).PromoteSnapshot(ctx, req.(*PromoteSnapshotRequest)) } return interceptor(ctx, in, info, handler) } @@ -526,8 +550,8 @@ var AteomHerder_ServiceDesc = grpc.ServiceDesc{ Handler: _AteomHerder_Restore_Handler, }, { - MethodName: "UploadPausedCheckpoint", - Handler: _AteomHerder_UploadPausedCheckpoint_Handler, + MethodName: "PromoteSnapshot", + Handler: _AteomHerder_PromoteSnapshot_Handler, }, { MethodName: "Terminate", diff --git a/internal/proto/ateompb/ateom.pb.go b/internal/proto/ateompb/ateom.pb.go index 87fde50518..bfadf9f340 100644 --- a/internal/proto/ateompb/ateom.pb.go +++ b/internal/proto/ateompb/ateom.pb.go @@ -1142,7 +1142,7 @@ func (x *ImageVolumeMount) GetMountPath() string { // Only HTTP is supported today. type WakeupProbe struct { state protoimpl.MessageState `protogen:"open.v1"` - HttpGet *HTTPGetAction `protobuf:"bytes,1,opt,name=http_get,json=httpGet,proto3" json:"http_get,omitempty"` + HttpGet *HttpGetAction `protobuf:"bytes,1,opt,name=http_get,json=httpGet,proto3" json:"http_get,omitempty"` // How long to keep polling before giving up and failing the actor start. TimeoutSeconds int32 `protobuf:"varint,2,opt,name=timeout_seconds,json=timeoutSeconds,proto3" json:"timeout_seconds,omitempty"` unknownFields protoimpl.UnknownFields @@ -1179,7 +1179,7 @@ func (*WakeupProbe) Descriptor() ([]byte, []int) { return file_ateom_proto_rawDescGZIP(), []int{13} } -func (x *WakeupProbe) GetHttpGet() *HTTPGetAction { +func (x *WakeupProbe) GetHttpGet() *HttpGetAction { if x != nil { return x.HttpGet } @@ -1193,8 +1193,8 @@ func (x *WakeupProbe) GetTimeoutSeconds() int32 { return 0 } -// HTTPGetAction performs an HTTP GET against the container. -type HTTPGetAction struct { +// HttpGetAction performs an HTTP GET against the container. +type HttpGetAction struct { state protoimpl.MessageState `protogen:"open.v1"` // Path to access on the HTTP server. Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"` @@ -1204,20 +1204,20 @@ type HTTPGetAction struct { sizeCache protoimpl.SizeCache } -func (x *HTTPGetAction) Reset() { - *x = HTTPGetAction{} +func (x *HttpGetAction) Reset() { + *x = HttpGetAction{} mi := &file_ateom_proto_msgTypes[14] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *HTTPGetAction) String() string { +func (x *HttpGetAction) String() string { return protoimpl.X.MessageStringOf(x) } -func (*HTTPGetAction) ProtoMessage() {} +func (*HttpGetAction) ProtoMessage() {} -func (x *HTTPGetAction) ProtoReflect() protoreflect.Message { +func (x *HttpGetAction) ProtoReflect() protoreflect.Message { mi := &file_ateom_proto_msgTypes[14] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -1229,19 +1229,19 @@ func (x *HTTPGetAction) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use HTTPGetAction.ProtoReflect.Descriptor instead. -func (*HTTPGetAction) Descriptor() ([]byte, []int) { +// Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead. +func (*HttpGetAction) Descriptor() ([]byte, []int) { return file_ateom_proto_rawDescGZIP(), []int{14} } -func (x *HTTPGetAction) GetPath() string { +func (x *HttpGetAction) GetPath() string { if x != nil { return x.Path } return "" } -func (x *HTTPGetAction) GetPort() int32 { +func (x *HttpGetAction) GetPort() int32 { if x != nil { return x.Port } @@ -2127,9 +2127,9 @@ const file_ateom_proto_rawDesc = "" + "\n" + "mount_path\x18\x02 \x01(\tR\tmountPath\"g\n" + "\vWakeupProbe\x12/\n" + - "\bhttp_get\x18\x01 \x01(\v2\x14.ateom.HTTPGetActionR\ahttpGet\x12'\n" + + "\bhttp_get\x18\x01 \x01(\v2\x14.ateom.HttpGetActionR\ahttpGet\x12'\n" + "\x0ftimeout_seconds\x18\x02 \x01(\x05R\x0etimeoutSeconds\"7\n" + - "\rHTTPGetAction\x12\x12\n" + + "\rHttpGetAction\x12\x12\n" + "\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" + "\x04port\x18\x02 \x01(\x05R\x04port\"\x15\n" + "\x13RunWorkloadResponse\"\xdb\x04\n" + @@ -2256,7 +2256,7 @@ var file_ateom_proto_goTypes = []any{ (*SystemInfoVolumeMount)(nil), // 14: ateom.SystemInfoVolumeMount (*ImageVolumeMount)(nil), // 15: ateom.ImageVolumeMount (*WakeupProbe)(nil), // 16: ateom.WakeupProbe - (*HTTPGetAction)(nil), // 17: ateom.HTTPGetAction + (*HttpGetAction)(nil), // 17: ateom.HttpGetAction (*RunWorkloadResponse)(nil), // 18: ateom.RunWorkloadResponse (*CheckpointWorkloadRequest)(nil), // 19: ateom.CheckpointWorkloadRequest (*CheckpointWorkloadResponse)(nil), // 20: ateom.CheckpointWorkloadResponse @@ -2286,7 +2286,7 @@ var file_ateom_proto_depIdxs = []int32{ 15, // 11: ateom.Container.image_volume_mounts:type_name -> ateom.ImageVolumeMount 10, // 12: ateom.Container.container_spec:type_name -> ateom.ContainerSpec 11, // 13: ateom.ContainerSpec.resources:type_name -> ateom.ResourceLimits - 17, // 14: ateom.WakeupProbe.http_get:type_name -> ateom.HTTPGetAction + 17, // 14: ateom.WakeupProbe.http_get:type_name -> ateom.HttpGetAction 8, // 15: ateom.CheckpointWorkloadRequest.spec:type_name -> ateom.WorkloadSpec 29, // 16: ateom.CheckpointWorkloadRequest.runtime_asset_paths:type_name -> ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntry 0, // 17: ateom.CheckpointWorkloadRequest.fidelity:type_name -> ateom.SnapshotFidelity diff --git a/internal/proto/ateompb/ateom.proto b/internal/proto/ateompb/ateom.proto index 557cfa803b..22002d1df0 100644 --- a/internal/proto/ateompb/ateom.proto +++ b/internal/proto/ateompb/ateom.proto @@ -269,13 +269,13 @@ message ImageVolumeMount { // WakeupProbe describes how to check that a container is ready to serve. // Only HTTP is supported today. message WakeupProbe { - HTTPGetAction http_get = 1; + HttpGetAction http_get = 1; // How long to keep polling before giving up and failing the actor start. int32 timeout_seconds = 2; } -// HTTPGetAction performs an HTTP GET against the container. -message HTTPGetAction { +// HttpGetAction performs an HTTP GET against the container. +message HttpGetAction { // Path to access on the HTTP server. string path = 1; // TCP port to connect to (1..65535). diff --git a/internal/protoredact/protoredact_test.go b/internal/protoredact/protoredact_test.go index bd6bc5126b..e8a4f59816 100644 --- a/internal/protoredact/protoredact_test.go +++ b/internal/protoredact/protoredact_test.go @@ -278,10 +278,10 @@ func TestRedactedHandlesTypedNilWithoutPanicking(t *testing.T) { // A typed nil proto pointer, as a handler returns alongside an error and // the interceptor then logs, must come back as a nil message of the same // type, for a redactable type and for a clean one alike. - var jwt *ateapipb.MintActorJWTResponse + var jwt *ateapipb.MintActorJwtResponse if r, changed := protoredact.Redacted(jwt); changed { t.Errorf("typed nil redactable: reported changed, got %#v", r) - } else if m, ok := r.(*ateapipb.MintActorJWTResponse); !ok || m != nil { + } else if m, ok := r.(*ateapipb.MintActorJwtResponse); !ok || m != nil { t.Errorf("typed nil redactable: got %#v", r) } var ref *ateapipb.ObjectRef @@ -302,7 +302,7 @@ func TestRedactedReturnsMessagesWithNothingToMaskWithoutCopying(t *testing.T) { &ateapipb.Worker{}, &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{{Name: "c", Image: "img"}}}, &ateapipb.ListActorTemplatesResponse{ActorTemplates: []*ateapipb.ActorTemplate{{Containers: []*ateapipb.Container{{Name: "c"}}}}}, - &ateapipb.MintActorJWTResponse{}, + &ateapipb.MintActorJwtResponse{}, &ateletpb.RunRequest{Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Env: []*ateletpb.EnvEntry{{Name: "K"}}}}}}, } { if protoredact.NeedsRedaction(m) { @@ -314,7 +314,7 @@ func TestRedactedReturnsMessagesWithNothingToMaskWithoutCopying(t *testing.T) { } // A populated labeled field, however deep, forces a masked copy. for _, m := range []proto.Message{ - &ateapipb.MintActorJWTResponse{ActorJwt: "t"}, + &ateapipb.MintActorJwtResponse{ActorJwt: "t"}, &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{{Env: []*ateapipb.EnvVar{{Name: "K", Value: "v"}}}}}, &ateapipb.ListActorTemplatesResponse{ActorTemplates: []*ateapipb.ActorTemplate{{}, {Containers: []*ateapipb.Container{{}, {Env: []*ateapipb.EnvVar{{Name: "K", Value: "v"}}}}}}}, &ateletpb.RunRequest{Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Env: []*ateletpb.EnvEntry{{Name: "K", Value: "v"}}}}}}, @@ -335,7 +335,7 @@ func TestNeedsRedactionToleratesTypedNilOfAnyImplementation(t *testing.T) { // A typed nil generated message answers ProtoReflect safely; a typed nil // dynamicpb.Message returns the nil receiver, which faults on use. Both // must be reported clean and returned unchanged. - var gen *ateapipb.MintActorJWTResponse + var gen *ateapipb.MintActorJwtResponse var dyn *dynamicpb.Message for _, m := range []proto.Message{gen, dyn} { if protoredact.NeedsRedaction(m) { @@ -351,7 +351,7 @@ func TestNeedsRedactionIsSafeUnderConcurrentFirstUse(t *testing.T) { // The per-type answer is memoized on first sight; many goroutines // meeting several types at once must agree and not race. msgs := []proto.Message{ - &ateapipb.MintActorJWTResponse{ActorJwt: "t"}, + &ateapipb.MintActorJwtResponse{ActorJwt: "t"}, &ateapipb.ListActorsResponse{}, &ateapipb.ActorTemplate{Containers: []*ateapipb.Container{{Env: []*ateapipb.EnvVar{{Name: "K", Value: "v"}}}}}, &ateapipb.Worker{}, @@ -416,7 +416,7 @@ func isDebugRedact(fd protoreflect.FieldDescriptor) bool { func TestDebugRedactFieldsArePinned(t *testing.T) { want := map[string]bool{ "ateapi.EnvVar.value": true, - "ateapi.MintActorJWTResponse.actor_jwt": true, + "ateapi.MintActorJwtResponse.actor_jwt": true, "atelet.AssetFile.url": true, "atelet.EnvEntry.value": true, "ateom.ContainerSpec.env": true, @@ -455,7 +455,7 @@ func TestSecretLikeFieldsAreLabeled(t *testing.T) { "page_token": "opaque list cursor", "next_page_token": "opaque list cursor", "ateapi.CredentialHeader.credential_uri": "ate-secret:// reference to a credential, not the value", - "ateapi.CredentialHeader.actor_jwt": "ActorJWTSource: audiences and lifetime for minting, not a token", + "ateapi.CredentialHeader.actor_jwt": "ActorJwtSource: audiences and lifetime for minting, not a token", } forEachField(func(fd protoreflect.FieldDescriptor) { if !secretLikeName.MatchString(string(fd.Name())) || isDebugRedact(fd) { diff --git a/internal/resources/validate.go b/internal/resources/validate.go index ff3d411a65..050a3303a7 100644 --- a/internal/resources/validate.go +++ b/internal/resources/validate.go @@ -17,20 +17,26 @@ package resources import ( "encoding/hex" "fmt" + "net" "net/netip" "net/url" "os" "path/filepath" "reflect" + "regexp" + "strconv" "strings" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/volumepath" + "github.com/distribution/reference" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" "google.golang.org/protobuf/proto" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/api/validate/content" + "k8s.io/apimachinery/pkg/util/validation" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -173,42 +179,19 @@ func ValidateRuntimeAssetPath(root, p string, fldPath *field.Path) field.ErrorLi return nil } -// ValidateContainerNames ensures every application container name is safe to -// use as an OCI bundle path component. Each must be a DNS-1123 label (no -// separator or ".."), must not be the reserved "pause" name (which would -// collide with the sandbox-infra bundle and race its concurrent writer), and -// must be unique (duplicates map to the same bundle path and corrupt each -// other). -func ValidateContainerNames(names []string) error { - seen := make(map[string]struct{}) - for _, name := range names { - if errs := content.IsDNS1123Label(name); len(errs) > 0 { - return fmt.Errorf("invalid container name %q: %s", name, strings.Join(errs, "; ")) - } - if name == "pause" { - return fmt.Errorf("invalid container name %q: reserved for sandbox infrastructure", name) - } - if _, dup := seen[name]; dup { - return fmt.Errorf("duplicate container name %q", name) - } - seen[name] = struct{}{} - } - return nil -} - -// ValidateRunscHash ensures the runsc SHA-256 hash is exactly 64 hex -// characters before it is used to build the on-disk binary path -// (static-files/runsc-) and, on a cache hit, returned for ateom to +// ValidateAssetHash ensures a sandbox asset's SHA-256 hash is exactly 64 hex +// characters before it is used to build the on-disk asset path +// (static-files/asset-) and, on a cache hit, returned for ateom to // execute. Without this, a hash containing path separators or ".." could // point the cache-hit early return (and the download target) at an arbitrary // binary outside the static-files dir. -func ValidateRunscHash(sha256Hash string) error { +func ValidateAssetHash(sha256Hash string) error { if len(sha256Hash) != 64 { - return fmt.Errorf("invalid runsc sha256 hash: want 64 hex chars, got %d", len(sha256Hash)) + return fmt.Errorf("invalid asset sha256 hash: want 64 hex chars, got %d", len(sha256Hash)) } // Same decoder atelet's digest comparison uses. if _, err := hex.DecodeString(sha256Hash); err != nil { - return fmt.Errorf("invalid runsc sha256 hash %q: must be hex", sha256Hash) + return fmt.Errorf("invalid asset sha256 hash %q: must be hex", sha256Hash) } return nil } @@ -302,13 +285,21 @@ func ValidateUUID(uuid string, fldPath *field.Path) field.ErrorList { return nil } -// cpuLimitMax bounds cpu limits: they must be less than 1000 cores. -var cpuLimitMax = resource.MustParse("1k") +// cpuLimitMax bounds cpu limits. The wire carries ceil(cores × 1000) +// millicores and caps it at 999999 (strictly below 1000 cores), so a limit +// above 999.999 cores rounds up out of range. +var cpuLimitMax = resource.MustParse("999.999") + +// memoryLimitMax bounds memory limits at what the wire's int64 byte count +// holds; resource.Quantity.Value wraps silently past it. +var memoryLimitMax = resource.MustParse("9223372036854775807") // ValidateLimit validates one resource limit entry at fldPath: only cpu and -// memory are supported, the quantity must be greater than zero, and the cpu -// limit must be less than 1000 cores. An empty quantity is left to the -// required tag. +// memory are supported, the quantity must be greater than zero, and it must +// fit the scalars the control plane converts it to (cpuLimitMax and +// memoryLimitMax). The bounds compare Quantities rather than the converted +// integers, which wrap for very large values. An empty quantity is left to +// the required tag. func ValidateLimit(fldPath *field.Path, name, quantity string) field.ErrorList { if name != ResourceCPU && name != ResourceMemory { return field.ErrorList{field.NotSupported(fldPath.Child("name"), name, []string{ResourceCPU, ResourceMemory})} @@ -324,8 +315,221 @@ func ValidateLimit(fldPath *field.Path, name, quantity string) field.ErrorList { if q.Sign() <= 0 { errs = append(errs, field.Invalid(fldPath.Child("quantity"), quantity, "must be greater than zero")) } - if name == ResourceCPU && q.Cmp(cpuLimitMax) >= 0 { - errs = append(errs, field.Invalid(fldPath.Child("quantity"), quantity, "cpu limit must be less than 1000 cores")) + switch { + case name == ResourceCPU && q.Cmp(cpuLimitMax) > 0: + errs = append(errs, field.Invalid(fldPath.Child("quantity"), quantity, "cpu limit must be at most 999.999 cores")) + case name == ResourceMemory && q.Cmp(memoryLimitMax) > 0: + errs = append(errs, field.Invalid(fldPath.Child("quantity"), quantity, "memory limit must be at most 9223372036854775807 bytes")) + } + return errs +} + +// ValidatePinnedImage requires a well-formed OCI image reference pinned by +// digest (e.g. "name@sha256:..."): changing the image content under a fixed +// reference invalidates snapshots. It parses with the same grammar the +// container runtimes use, so a malformed digest is rejected rather than +// treated as pinned. +func ValidatePinnedImage(fldPath *field.Path, value string) field.ErrorList { + if value == "" { + return nil + } + ref, err := reference.ParseNormalizedNamed(value) + if err != nil { + return field.ErrorList{field.Invalid(fldPath, value, fmt.Sprintf("must be a well-formed image reference: %v", err))} + } + if _, ok := ref.(reference.Digested); !ok { + return field.ErrorList{field.Invalid(fldPath, value, "must be pinned by digest (changing the image invalidates snapshots)")} + } + return nil +} + +// capabilityRE constrains Linux capability names: uppercase, without the +// "CAP_" prefix (which is added when the OCI spec is written; the prefixed +// spelling would silently grant nothing). +var capabilityRE = regexp.MustCompile(`^[A-Z][A-Z0-9_]*$`) + +// ValidateCapabilities checks each capability name. "ALL" is accepted only +// when allowAll is set, which is the case for drop but not add. +func ValidateCapabilities(fldPath *field.Path, caps []string, allowAll bool) field.ErrorList { + var errs field.ErrorList + for i, c := range caps { + p := fldPath.Index(i) + switch { + case c == "ALL" && !allowAll: + errs = append(errs, field.Invalid(p, c, "add does not accept 'ALL'; name the individual capabilities the container needs")) + case c == "ALL": + case len(c) > 63: + errs = append(errs, field.TooLong(p, nil, 63)) + case strings.HasPrefix(c, "CAP_"): + errs = append(errs, field.Invalid(p, c, "must be named without the 'CAP_' prefix (e.g. 'NET_BIND_SERVICE')")) + case !capabilityRE.MatchString(c): + errs = append(errs, field.Invalid(p, c, "must be an uppercase capability name like 'NET_BIND_SERVICE'")) + } + } + return errs +} + +// mountPathBadSegmentRE matches '.' or '..' path segments. +var mountPathBadSegmentRE = regexp.MustCompile(`(^|/)[.][.]?(/|$)`) + +// ValidateMountPath requires a clean absolute Unix path that starts with '/', +// is not '/', and contains no ':', '.' or '..' segments, '//', trailing '/', +// or control characters. +func ValidateMountPath(fldPath *field.Path, p string) field.ErrorList { + if p == "" { + return nil + } + bad := !strings.HasPrefix(p, "/") || len(p) == 1 || + strings.HasSuffix(p, "/") || strings.Contains(p, "//") || + strings.Contains(p, ":") || mountPathBadSegmentRE.MatchString(p) + if !bad { + for _, r := range p { + if r < 0x20 || r == 0x7f { + bad = true + break + } + } + } + if bad { + return field.ErrorList{field.Invalid(fldPath, p, "must be a clean absolute Unix path: must start with '/', not be '/', and contain no ':', '..', '.', '//', trailing '/', or control characters")} + } + return nil +} + +// ValidateNestedMountPaths rejects mount paths that nest under or over one +// another: volumes cannot mount onto other volumes. paths[i] is reported at +// fldPath[i].mount_path. Identical paths are left to the list-key uniqueness +// check. +func ValidateNestedMountPaths(fldPath *field.Path, paths []string) field.ErrorList { + var errs field.ErrorList + for i, path := range paths { + if path == "" { + continue + } + for _, prior := range paths[:i] { + if prior == "" || prior == path { + continue + } + if strings.HasPrefix(path, prior+"/") || strings.HasPrefix(prior, path+"/") { + errs = append(errs, field.Invalid(fldPath.Index(i).Child("mount_path"), path, + fmt.Sprintf("must not nest under or over another mount (%q)", prior))) + } + } + } + return errs +} + +// httpGetPathRE constrains wakeup probe paths to RFC 3986 path-segment +// characters only, with well-formed percent-escapes, and no query string +// or fragment. +var httpGetPathRE = regexp.MustCompile(`^/([A-Za-z0-9\-._~!$&'()*+,;=:@/]|%[0-9A-Fa-f]{2})*$`) + +// ValidateHTTPGetPath checks a wakeup probe's HTTP path. +func ValidateHTTPGetPath(fldPath *field.Path, p string) field.ErrorList { + if p == "" { + return nil + } + if !httpGetPathRE.MatchString(p) { + return field.ErrorList{field.Invalid(fldPath, p, "must be a URL path starting with '/', using only RFC 3986 path-segment characters, without query or fragment")} + } + return nil +} + +// envVarNameRE constrains env var names to any printable ASCII character +// except '='. +var envVarNameRE = regexp.MustCompile(`^[ -<>-~]+$`) + +// ValidateEnvVarName checks an environment variable name. +func ValidateEnvVarName(fldPath *field.Path, name string) field.ErrorList { + if name == "" { + return nil + } + if !envVarNameRE.MatchString(name) { + return field.ErrorList{field.Invalid(fldPath, name, "may contain any printable ASCII character except '='")} + } + return nil +} + +// ValidateProjectedPath applies volumepath.ValidateProjected to a file path +// projected into a system-info volume. atelet re-checks it before writing to +// the host. +func ValidateProjectedPath(fldPath *field.Path, p string) field.ErrorList { + if p == "" { + return nil + } + if err := volumepath.ValidateProjected(p); err != nil { + return field.ErrorList{field.Invalid(fldPath, p, err.Error())} + } + return nil +} + +// ValidateStorageVolumeID rejects control characters (U+0000-U+0008, U+000B, +// U+000C, U+000E-U+001F, U+007F-U+009F) in an external volume's storage ID. +func ValidateStorageVolumeID(fldPath *field.Path, id string) field.ErrorList { + for _, r := range id { + if (r >= 0x0000 && r <= 0x0008) || + r == 0x000B || + r == 0x000C || + (r >= 0x000E && r <= 0x001F) || + (r >= 0x007F && r <= 0x009F) { + return field.ErrorList{field.Invalid(fldPath, id, "must not contain control characters (U+0000-U+0008, U+000B, U+000C, U+000E-U+001F, U+007F-U+009F)")} + } + } + return nil +} + +// ValidateVolumeType allows an optional "substrate.io/" prefix, followed by a +// valid DNS-1123 subdomain. +func ValidateVolumeType(fldPath *field.Path, volumeType string) field.ErrorList { + if volumeType == "" { + return nil + } + var errs field.ErrorList + for _, msg := range validation.IsDNS1123Subdomain(strings.TrimPrefix(volumeType, "substrate.io/")) { + errs = append(errs, field.Invalid(fldPath, volumeType, msg)) + } + return errs +} + +// csiMapMaxBytes bounds a map a CSI driver produced. The CSI spec caps +// map fields at 4 KiB; this allows ten times that so a future +// revision of the limit does not turn compliant drivers away. It is a +// guardrail against unbounded input, not a contract drivers are held to. +const csiMapMaxBytes = 40 * 1024 + +// ValidateCSIMap bounds a map a CSI driver produced (a volume or publish +// context) at csiMapMaxBytes, counting keys and values. The entries are the +// driver's own and are not validated further. +func ValidateCSIMap(fldPath *field.Path, m map[string]string) field.ErrorList { + size := 0 + for k, v := range m { + size += len(k) + len(v) + } + if size > csiMapMaxBytes { + return field.ErrorList{field.TooLong(fldPath, nil, csiMapMaxBytes)} + } + return nil +} + +// ValidateHostPort requires "host:port", where host is an IP address or a DNS +// subdomain name (a trailing dot, marking it fully qualified, is allowed) and +// port is a number in 1..65535. +func ValidateHostPort(fldPath *field.Path, value string) field.ErrorList { + if value == "" { + return nil + } + host, port, err := net.SplitHostPort(value) + if err != nil { + return field.ErrorList{field.Invalid(fldPath, value, "must be host:port")} + } + var errs field.ErrorList + if _, err := netip.ParseAddr(host); err != nil { + for _, msg := range content.IsDNS1123Subdomain(strings.TrimSuffix(host, ".")) { + errs = append(errs, field.Invalid(fldPath, value, "host: "+msg)) + } + } + if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 { + errs = append(errs, field.Invalid(fldPath, value, "port must be a number between 1 and 65535")) } return errs } diff --git a/internal/resources/validate_test.go b/internal/resources/validate_test.go index 7db4c4997a..aacdadda4b 100644 --- a/internal/resources/validate_test.go +++ b/internal/resources/validate_test.go @@ -15,6 +15,7 @@ package resources import ( + "fmt" "os" "path/filepath" "regexp" @@ -123,33 +124,7 @@ func TestValidateWorkerPodUID(t *testing.T) { } } -func TestValidateContainerNames(t *testing.T) { - tests := []struct { - name string - names []string - wantErr bool - }{ - {"no containers", nil, false}, - {"single valid", []string{"worker"}, false}, - {"multiple valid", []string{"worker", "sidecar"}, false}, - {"separator", []string{"a/b"}, true}, - {"traversal", []string{".."}, true}, - {"empty name", []string{""}, true}, - {"uppercase", []string{"Worker"}, true}, - {"reserved pause", []string{"pause"}, true}, - {"reserved pause among valid", []string{"worker", "pause"}, true}, - {"duplicate", []string{"worker", "worker"}, true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - if err := ValidateContainerNames(tt.names); (err != nil) != tt.wantErr { - t.Errorf("ValidateContainerNames(%v) err = %v, wantErr %v", tt.names, err, tt.wantErr) - } - }) - } -} - -func TestValidateRunscHash(t *testing.T) { +func TestValidateAssetHash(t *testing.T) { const valid = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" tests := []struct { @@ -167,8 +142,8 @@ func TestValidateRunscHash(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - if err := ValidateRunscHash(tt.hash); (err != nil) != tt.wantErr { - t.Errorf("ValidateRunscHash(%q) err = %v, wantErr %v", tt.hash, err, tt.wantErr) + if err := ValidateAssetHash(tt.hash); (err != nil) != tt.wantErr { + t.Errorf("ValidateAssetHash(%q) err = %v, wantErr %v", tt.hash, err, tt.wantErr) } }) } @@ -296,14 +271,20 @@ func TestValidateLimit(t *testing.T) { want field.ErrorList }{ {name: "cpu below the bound", limit: "cpu", quantity: "999"}, + {name: "cpu at the millicore bound", limit: "cpu", quantity: "999.999"}, {name: "memory", limit: "memory", quantity: "1Gi"}, - {name: "memory has no upper bound", limit: "memory", quantity: "1000"}, + {name: "memory at the int64 bound", limit: "memory", quantity: "9223372036854775807"}, {name: "missing quantity left to tags", limit: "cpu"}, {name: "unsupported name", limit: "gpu", quantity: "1", want: field.ErrorList{field.NotSupported[string](path.Child("name"), nil, nil)}}, {name: "malformed quantity", limit: "cpu", quantity: "x", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, {name: "zero quantity", limit: "memory", quantity: "0", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, {name: "negative quantity", limit: "memory", quantity: "-1", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, - {name: "cpu at the bound", limit: "cpu", quantity: "1000", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, + {name: "cpu at 1000 cores", limit: "cpu", quantity: "1000", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, + {name: "cpu rounds up to 1000 cores in millicores", limit: "cpu", quantity: "999.9995", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, + {name: "cpu far beyond int64 millicores", limit: "cpu", quantity: "1e30", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, + {name: "memory past the int64 bound", limit: "memory", quantity: "9223372036854775808", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, + {name: "memory rounds up past the int64 bound", limit: "memory", quantity: "9223372036854775807.5", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, + {name: "memory far beyond int64 bytes", limit: "memory", quantity: "1e30", want: field.ErrorList{field.Invalid(quantityPath, nil, "")}}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -395,6 +376,27 @@ func TestValidateActorDirs(t *testing.T) { } } +const testPinnedImage = "example.com/app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + +// stringRuleCase is one input to a single-string rule and whether it fails. +type stringRuleCase struct { + name string + value string + wantErr bool +} + +func runStringRule(t *testing.T, rule func(*field.Path, string) field.ErrorList, tests []stringRuleCase) { + t.Helper() + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + errs := rule(field.NewPath("f"), tt.value) + if (len(errs) > 0) != tt.wantErr { + t.Errorf("value %q: errs = %v, wantErr %v", tt.value, errs, tt.wantErr) + } + }) + } +} + func TestValidateRuntimeAssetPath(t *testing.T) { // on macOS the temp dir is behind the /var symlink root, err := filepath.EvalSymlinks(t.TempDir()) @@ -452,3 +454,178 @@ func TestValidateRuntimeAssetPath(t *testing.T) { }) } } + +func TestValidatePinnedImage(t *testing.T) { + runStringRule(t, ValidatePinnedImage, []stringRuleCase{ + {"empty is left to tags", "", false}, + {"pinned", testPinnedImage, false}, + {"tag only", "example.com/app:v1", true}, + {"malformed digest", "example.com/app@sha256:abc", true}, + {"malformed reference", "Example.com/App", true}, + }) +} + +func TestValidateMountPath(t *testing.T) { + runStringRule(t, ValidateMountPath, []stringRuleCase{ + {"empty is left to tags", "", false}, + {"clean absolute", "/data/cache", false}, + {"relative", "data", true}, + {"root", "/", true}, + {"trailing slash", "/data/", true}, + {"double slash", "/data//x", true}, + {"colon", "/data:x", true}, + {"dot segment", "/data/./x", true}, + {"dot-dot segment", "/data/../x", true}, + {"control character", "/data\x01", true}, + }) +} + +func TestValidateHTTPGetPath(t *testing.T) { + runStringRule(t, ValidateHTTPGetPath, []stringRuleCase{ + {"empty is left to tags", "", false}, + {"simple", "/healthz", false}, + {"percent escape", "/a%20b", false}, + {"no leading slash", "healthz", true}, + {"query", "/healthz?x=1", true}, + {"fragment", "/healthz#x", true}, + {"bad percent escape", "/a%2", true}, + }) +} + +func TestValidateEnvVarName(t *testing.T) { + runStringRule(t, ValidateEnvVarName, []stringRuleCase{ + {"empty is left to tags", "", false}, + {"conventional", "HOME_DIR", false}, + {"printable punctuation", "a.b-c", false}, + {"equals sign", "A=B", true}, + {"control character", "A\tB", true}, + {"non-ASCII", "é", true}, + }) +} + +func TestValidateProjectedPath(t *testing.T) { + runStringRule(t, ValidateProjectedPath, []stringRuleCase{ + {"empty is left to tags", "", false}, + {"relative file", "identity/name", false}, + {"absolute", "/etc/name", true}, + {"escape", "../name", true}, + }) +} + +func TestValidateStorageVolumeID(t *testing.T) { + runStringRule(t, ValidateStorageVolumeID, []stringRuleCase{ + {"empty", "", false}, + {"plain", "projects/p/disks/d-1", false}, + {"newline is allowed", "a\nb", false}, + {"NUL", "a\x00b", true}, + {"DEL", "a\x7fb", true}, + {"C1 control", "a\u0085b", true}, + }) +} + +func TestValidateVolumeType(t *testing.T) { + runStringRule(t, ValidateVolumeType, []stringRuleCase{ + {"empty", "", false}, + {"dns subdomain", "pd.csi.storage.gke.io", false}, + {"substrate prefix", "substrate.io/gcs", false}, + {"uppercase", "PD", true}, + {"other prefix", "example.io/gcs", true}, + }) +} + +func TestValidateCapabilities(t *testing.T) { + tests := []struct { + name string + caps []string + allowAll bool + wantErrs int + }{ + {"valid names", []string{"NET_BIND_SERVICE", "SYS_PTRACE"}, false, 0}, + {"ALL rejected for add", []string{"ALL"}, false, 1}, + {"ALL accepted for drop", []string{"ALL"}, true, 0}, + {"CAP_ prefix", []string{"CAP_NET_ADMIN"}, false, 1}, + {"lowercase", []string{"net_admin"}, false, 1}, + {"too long", []string{strings.Repeat("A", 64)}, false, 1}, + {"each bad entry reported", []string{"CAP_X", "ok", "NET_ADMIN"}, false, 2}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if errs := ValidateCapabilities(field.NewPath("add"), tt.caps, tt.allowAll); len(errs) != tt.wantErrs { + t.Errorf("errs = %v, want %d", errs, tt.wantErrs) + } + }) + } +} + +func TestValidateNestedMountPaths(t *testing.T) { + path := field.NewPath("volume_mounts") + tests := []struct { + name string + paths []string + want field.ErrorList + }{ + {name: "siblings", paths: []string{"/a", "/b", "/ab"}}, + {name: "identical paths are left to the list key", paths: []string{"/a", "/a"}}, + {name: "empty paths are skipped", paths: []string{"", "/a"}}, + { + name: "nested under an earlier mount", + paths: []string{"/a", "/a/b"}, + want: field.ErrorList{field.Invalid(path.Index(1).Child("mount_path"), nil, "")}, + }, + { + name: "nested over an earlier mount", + paths: []string{"/a/b", "/a"}, + want: field.ErrorList{field.Invalid(path.Index(1).Child("mount_path"), nil, "")}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + field.ErrorMatcher{}.ByType().ByField().Test(t, tt.want, ValidateNestedMountPaths(path, tt.paths)) + }) + } +} + +func TestValidateCSIMap(t *testing.T) { + path := field.NewPath("publish_context") + many := make(map[string]string, 100) + for i := range 100 { + many[fmt.Sprintf("key-%03d", i)] = "v" + } + tests := []struct { + name string + m map[string]string + want field.ErrorList + }{ + {name: "nil"}, + {name: "one long value", m: map[string]string{"token": strings.Repeat("x", 4000)}}, + {name: "many small entries", m: many}, + {name: "over the CSI spec limit, within the headroom", m: map[string]string{"k": strings.Repeat("x", 8*1024)}}, + {name: "exactly the bound", m: map[string]string{"k": strings.Repeat("x", 40*1024-1)}}, + {name: "one byte over", m: map[string]string{"k": strings.Repeat("x", 40*1024)}, want: field.ErrorList{field.TooLong(path, nil, 40*1024)}}, + {name: "over across entries", m: map[string]string{"a": strings.Repeat("x", 20*1024), "b": strings.Repeat("y", 20*1024)}, want: field.ErrorList{field.TooLong(path, nil, 40*1024)}}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + field.ErrorMatcher{}.ByType().ByField().Test(t, tt.want, ValidateCSIMap(path, tt.m)) + }) + } +} + +func TestValidateHostPort(t *testing.T) { + runStringRule(t, ValidateHostPort, []stringRuleCase{ + {name: "empty is left to presence tags", value: ""}, + {name: "dns name", value: "atenet-egress.ate-system.svc:443"}, + {name: "fully qualified dns name", value: "atenet-egress.ate-system.svc.cluster.local.:443"}, + {name: "ipv4", value: "10.0.0.1:8080"}, + {name: "ipv6", value: "[fd00::1]:443"}, + {name: "missing port", value: "atenet-egress.ate-system.svc", wantErr: true}, + {name: "empty host", value: ":443", wantErr: true}, + {name: "bare dot host", value: ".:443", wantErr: true}, + {name: "uppercase host", value: "Egress.Example:443", wantErr: true}, + {name: "host with a path", value: "egress/../x:443", wantErr: true}, + {name: "port zero", value: "egress.example:0", wantErr: true}, + {name: "port too large", value: "egress.example:65536", wantErr: true}, + {name: "named port", value: "egress.example:https", wantErr: true}, + {name: "unbracketed ipv6", value: "fd00::1:443", wantErr: true}, + }) +} diff --git a/internal/sandboxversions/sandboxversions.go b/internal/sandboxversions/sandboxversions.go new file mode 100644 index 0000000000..d9dcf6496f --- /dev/null +++ b/internal/sandboxversions/sandboxversions.go @@ -0,0 +1,96 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package sandboxversions reads and writes a worker's sandbox versions file, +// which atelet keeps in the worker's directory (nodepath.AteomPath) as +// nodepath.SandboxVersionsFileName. +package sandboxversions + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" +) + +// File is the content of a worker's sandbox versions file: the sandbox +// runtimes the worker can run, as the compatibility attributes atelet computed +// for them. The ateom adds each runtime's attributes to the ones it probes +// itself and registers the result as RegisterWorkerRequest's default_runtime +// and restorable_runtimes. +type File struct { + // DefaultRuntime is the runtime new and cold-booted actors start under, + // from the default version of the pool's SandboxConfig. + DefaultRuntime Attributes `json:"defaultRuntime"` + // RestorableRuntimes are the other runtimes the worker can restore + // snapshots from, from the config's other Enabled versions in spec order. + // None repeats another runtime or the default. + RestorableRuntimes []Attributes `json:"restorableRuntimes,omitempty"` +} + +// Attributes maps a compatibility attribute key, e.g. "sandbox_runtime_hash", +// to its value. +type Attributes map[string]string + +// tempPrefix starts the name of every temporary file Write renames into +// place. It is dot-prefixed so a directory listing does not show one. +const tempPrefix = ".sandbox-versions-" + +// Write atomically replaces the file at path with v, unless it already holds +// the same content. It reports whether it wrote. +func Write(path string, v *File) (bool, error) { + // encoding/json sorts map keys, so equal content encodes to equal bytes. + data, err := json.MarshalIndent(v, "", " ") + if err != nil { + return false, fmt.Errorf("while encoding %s: %w", path, err) + } + if cur, err := os.ReadFile(path); err == nil && bytes.Equal(cur, data) { + return false, nil + } + dir := filepath.Dir(path) + f, err := os.CreateTemp(dir, tempPrefix+"*") + if err != nil { + return false, err + } + tmp := f.Name() + _, err = f.Write(data) + if cerr := f.Close(); err == nil { + err = cerr + } + if err == nil { + err = os.Rename(tmp, path) + } + if err != nil { + os.Remove(tmp) + return false, fmt.Errorf("while writing %s: %w", path, err) + } + return true, nil +} + +// Read returns the content of the file at path. A missing file is an error +// wrapping fs.ErrNotExist. +func Read(path string) (*File, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + dec := json.NewDecoder(bytes.NewReader(data)) + dec.DisallowUnknownFields() + v := &File{} + if err := dec.Decode(v); err != nil { + return nil, fmt.Errorf("while decoding %s: %w", path, err) + } + return v, nil +} diff --git a/internal/sandboxversions/sandboxversions_test.go b/internal/sandboxversions/sandboxversions_test.go new file mode 100644 index 0000000000..5665d6e6c2 --- /dev/null +++ b/internal/sandboxversions/sandboxversions_test.go @@ -0,0 +1,177 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package sandboxversions + +import ( + "errors" + "io/fs" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/google/go-cmp/cmp" +) + +func testVersions(defaultHash string) *File { + return &File{ + DefaultRuntime: Attributes{"sandbox_runtime_hash": defaultHash}, + RestorableRuntimes: []Attributes{{"sandbox_runtime_hash": "sha256:41ab"}}, + } +} + +func TestWrite(t *testing.T) { + tests := []struct { + name string + // existing is written first, when set; dir is left out when noDir. + existing *File + noDir bool + v *File + wantWrote bool + wantErr bool + }{{ + name: "new file", + v: testVersions("sha256:7c2e"), + wantWrote: true, + }, { + name: "equal content is not rewritten", + existing: testVersions("sha256:7c2e"), + v: testVersions("sha256:7c2e"), + wantWrote: false, + }, { + name: "different content replaces the file", + existing: testVersions("sha256:7c2e"), + v: testVersions("sha256:9d01"), + wantWrote: true, + }, { + name: "missing directory", + noDir: true, + v: testVersions("sha256:7c2e"), + wantErr: true, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + dir := t.TempDir() + if tt.noDir { + dir = filepath.Join(dir, "missing") + } + path := filepath.Join(dir, "sandbox-versions.json") + var before time.Time + if tt.existing != nil { + if _, err := Write(path, tt.existing); err != nil { + t.Fatal(err) + } + st, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + before = st.ModTime() + // A rewrite must land on a later mtime to be told apart. + time.Sleep(10 * time.Millisecond) + } + + wrote, err := Write(path, tt.v) + if (err != nil) != tt.wantErr { + t.Fatalf("Write() error = %v, want error %v", err, tt.wantErr) + } + if wrote != tt.wantWrote { + t.Errorf("Write() wrote = %v, want %v", wrote, tt.wantWrote) + } + if tt.wantErr { + if entries, err := os.ReadDir(filepath.Dir(dir)); err == nil { + for _, e := range entries { + if strings.HasPrefix(e.Name(), tempPrefix) { + t.Errorf("temporary file %q left behind", e.Name()) + } + } + } + return + } + got, err := Read(path) + if err != nil { + t.Fatalf("Read() after Write(): %v", err) + } + if diff := cmp.Diff(tt.v, got); diff != "" { + t.Errorf("Read() mismatch (-want +got):\n%s", diff) + } + st, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if changed := st.ModTime().After(before); tt.existing != nil && changed != tt.wantWrote { + t.Errorf("file mtime changed = %v, want %v", changed, tt.wantWrote) + } + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + if len(entries) != 1 { + t.Errorf("directory holds %d entries, want only the file", len(entries)) + } + }) + } +} + +func TestRead(t *testing.T) { + tests := []struct { + name string + // contents is written to the file when set; the file is left out + // otherwise. + contents string + want *File + wantNotExist bool + wantErr bool + }{{ + name: "missing file", + wantNotExist: true, + wantErr: true, + }, { + name: "corrupt file", + contents: "{not json", + wantErr: true, + }, { + name: "unknown field", + contents: `{"defaultRuntime": {"sandbox_runtime_hash": "sha256:7c2e"}, "nope": 1}`, + wantErr: true, + }, { + name: "valid file", + contents: `{"defaultRuntime": {"sandbox_runtime_hash": "sha256:7c2e"}, "restorableRuntimes": [{"sandbox_runtime_hash": "sha256:41ab"}]}`, + want: &File{ + DefaultRuntime: Attributes{"sandbox_runtime_hash": "sha256:7c2e"}, + RestorableRuntimes: []Attributes{{"sandbox_runtime_hash": "sha256:41ab"}}, + }, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + path := filepath.Join(t.TempDir(), "sandbox-versions.json") + if tt.contents != "" { + if err := os.WriteFile(path, []byte(tt.contents), 0o600); err != nil { + t.Fatal(err) + } + } + got, err := Read(path) + if (err != nil) != tt.wantErr { + t.Fatalf("Read() error = %v, want error %v", err, tt.wantErr) + } + if errors.Is(err, fs.ErrNotExist) != tt.wantNotExist { + t.Errorf("Read() error = %v, want fs.ErrNotExist %v", err, tt.wantNotExist) + } + if diff := cmp.Diff(tt.want, got); diff != "" { + t.Errorf("Read() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/internal/serverboot/logging_test.go b/internal/serverboot/logging_test.go index 1f25dfcd74..ed17344eab 100644 --- a/internal/serverboot/logging_test.go +++ b/internal/serverboot/logging_test.go @@ -284,7 +284,7 @@ func TestInitLoggerInstallsRedactingHandler(t *testing.T) { } const token = "eyJhbGciOiJSUzI1NiJ9.secret-token" - slog.Info("Handle RPC", slog.Any("resp", &ateapipb.MintActorJWTResponse{ActorJwt: token})) + slog.Info("Handle RPC", slog.Any("resp", &ateapipb.MintActorJwtResponse{ActorJwt: token})) got := buf.String() if strings.Contains(got, token) { t.Fatalf("default logger wrote a debug_redact field: %s", got) diff --git a/internal/serverboot/serverboot.go b/internal/serverboot/serverboot.go index f673b6ca7c..652a44ce62 100644 --- a/internal/serverboot/serverboot.go +++ b/internal/serverboot/serverboot.go @@ -14,8 +14,9 @@ // Package serverboot collects the startup boilerplate shared by the // long-running substrate server binaries (ateapi, atelet, ateom-gvisor, -// ateom-microvm): slog wiring, OTel tracer + meter providers, a Prometheus + -// /readyz HTTP surface, and a couple of small helpers for startup fail-fast. +// ateom-microvm): slog wiring, OTel tracer + meter providers, a Prometheus +// /metrics server, a /readyz and /healthz server for the kubelet probes, and a +// couple of small helpers for startup fail-fast. package serverboot import ( @@ -27,6 +28,7 @@ import ( "net/http" "os" "sync/atomic" + "time" "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/contextlogging" @@ -208,17 +210,27 @@ const ( tracesExporterEnv = "OTEL_TRACES_EXPORTER" ) -// metricsPushEnabled applies OTEL_METRICS_EXPORTER with the shared exporter -// rules: the OTLP reader is installed while otlp is selected, as it is when the -// variable is unset. prometheus, the specification's pull exporter, is known -// only when pullServed, i.e. the component serves its own /metrics; elsewhere -// it is skipped as unknown, so a push-only component keeps exporting. -func metricsPushEnabled(ctx context.Context, pullServed bool) bool { +// metricsExporters applies OTEL_METRICS_EXPORTER with the shared exporter +// rules. prometheus, the specification's pull exporter, is known only when +// pullServed, i.e. the component serves its own /metrics; elsewhere it is +// skipped as unknown, so a push-only component keeps exporting. Unset selects +// every exporter the component has. +func metricsExporters(ctx context.Context, pullServed bool) Exporters { known := []string{ExporterOTLP} + def := Exporters{ExporterOTLP: true} if pullServed { known = append(known, exporterPrometheus) + def[exporterPrometheus] = true } - return resolveExportersEnv(ctx, metricsExporterEnv, known, Exporters{ExporterOTLP: true}).Has(ExporterOTLP) + exporters := resolveExportersEnv(ctx, metricsExporterEnv, known, def) + slog.InfoContext(ctx, "Metrics exporters resolved", slog.String("exporters", exporters.String())) + return exporters +} + +// metricsPushEnabled reports whether the OTLP reader is installed: while otlp +// is selected, as it is when the variable is unset. +func metricsPushEnabled(ctx context.Context, pullServed bool) bool { + return metricsExporters(ctx, pullServed).Has(ExporterOTLP) } // tracesPushEnabled applies OTEL_TRACES_EXPORTER with the shared exporter @@ -247,31 +259,45 @@ func InitMetrics(ctx context.Context, serviceName string) (*sdkmetric.MeterProvi // it does not own (atecontroller: controller-runtime's), so every metric // reaches the backend exactly once on whichever path is active. With the OTLP // export, reg is bridged onto the push and the OTel instruments stay off it, -// since the bridge would otherwise push them a second time. When -// OTEL_METRICS_EXPORTER does not select otlp, nothing is pushed and the OTel -// instruments register on reg, so its endpoint serves both. +// since the bridge would otherwise push them a second time. With prometheus +// alone, nothing is pushed and the OTel instruments register on reg, so its +// endpoint serves both. +// +// servePull reports whether OTEL_METRICS_EXPORTER leaves the scraped endpoint +// on, which the caller must then serve from reg: unset, prometheus, or a list +// with it. otlp and none turn it off, and with none no reader is installed. // // wrapProducer, when non-nil, wraps the bridge producer on the OTLP path, for // a rewrite the push backend needs but the scraped endpoint does not. func InitMetricsBridged(ctx context.Context, serviceName string, reg interface { promclient.Registerer promclient.Gatherer -}, wrapProducer func(sdkmetric.Producer) sdkmetric.Producer) (*sdkmetric.MeterProvider, error) { +}, wrapProducer func(sdkmetric.Producer) sdkmetric.Producer) (mp *sdkmetric.MeterProvider, servePull bool, err error) { if serviceName == "" { - return nil, fmt.Errorf("serviceName is required") + return nil, false, fmt.Errorf("serviceName is required") } - if metricsPushEnabled(ctx, true) { + exporters := metricsExporters(ctx, true) + servePull = exporters.Has(exporterPrometheus) + switch { + case exporters.Has(ExporterOTLP): producer := prombridge.NewMetricProducer(prombridge.WithGatherer(reg)) if wrapProducer != nil { producer = wrapProducer(producer) } - return newMeterProvider(ctx, serviceName, true, false, nil, []sdkmetric.Producer{producer}) + mp, err = newMeterProvider(ctx, serviceName, true, false, nil, []sdkmetric.Producer{producer}) + case servePull: + var promReader *prometheus.Exporter + if promReader, err = prometheus.New(prometheus.WithRegisterer(reg)); err != nil { + return nil, false, fmt.Errorf("create Prometheus metric exporter: %w", err) + } + mp, err = newMeterProvider(ctx, serviceName, false, false, nil, nil, promReader) + default: + mp, err = newMeterProvider(ctx, serviceName, false, false, nil, nil) } - promReader, err := prometheus.New(prometheus.WithRegisterer(reg)) if err != nil { - return nil, fmt.Errorf("create Prometheus metric exporter: %w", err) + return nil, false, err } - return newMeterProvider(ctx, serviceName, false, false, nil, nil, promReader) + return mp, servePull, nil } // InitMetricsPushOnlyVia is InitMetrics without the Prometheus reader, for a @@ -362,64 +388,69 @@ func (r *Readiness) MarkNotReady() { r.notReady.Store(true) } // Ready reports whether /readyz returns 200. func (r *Readiness) Ready() bool { return !r.notReady.Load() } -// MetricsServerOptions configures StartMetricsServer. -type MetricsServerOptions struct { - // Addr is the TCP listen address (e.g. ":9090"). - Addr string - // Readiness, if non-nil, enables a /readyz handler: 200 while - // Ready, 503 after MarkNotReady. A zero-value Readiness never - // flips, giving a static 200 for binaries with no drain sequence. - // Nil serves no /readyz at all. - Readiness *Readiness - // EnableHealthz adds an always-200 /healthz for liveness probes, - // which must keep succeeding while a draining server fails /readyz. - EnableHealthz bool +// httpServerReadHeaderTimeout bounds how long a client can take to send the +// request headers, so a slow client cannot hold a connection open forever. +const httpServerReadHeaderTimeout = 10 * time.Second + +// StartMetricsServer runs an HTTP server exposing only /metrics +// (Prometheus). Blocks until the server stops; designed to be `go`-launched. +// The kubelet probes use StartHealthServer, so a change to the metrics +// endpoint does not change the health checks. A serve failure exits the +// process: the probes no longer report a metrics server that cannot bind, so +// without the exit the only sign of the failure is one log line. +func StartMetricsServer(ctx context.Context, addr string) { + slog.InfoContext(ctx, "Starting metrics HTTP server", slog.String("addr", addr)) + if err := listenAndServe(addr, metricsMux()); err != nil { + slog.Error("Metrics HTTP server failed", slog.Any("err", err)) + os.Exit(1) + } } -// StartMetricsServer runs an HTTP server exposing /metrics (Prometheus) -// and optionally /readyz and /healthz. Blocks until http.ListenAndServe -// returns; designed to be `go`-launched. -func StartMetricsServer(ctx context.Context, opts MetricsServerOptions) { - slog.InfoContext(ctx, "Starting metrics HTTP server", slog.String("addr", opts.Addr)) - if err := http.ListenAndServe(opts.Addr, metricsMux(opts)); err != nil { - slog.Error("Failed to start prometheus metrics server", slog.Any("err", err)) +// StartHealthServer runs an HTTP server exposing /readyz and /healthz for the +// kubelet probes. /readyz returns 200 while readiness is Ready and 503 after +// MarkNotReady. /healthz always returns 200, so liveness keeps succeeding +// while a draining server fails readiness. Blocks until the server stops; +// designed to be `go`-launched. A serve failure exits the process: a pod whose +// probes can never succeed never turns Ready, so dying loudly lets the kubelet +// restart it instead of leaving a pod that looks alive but never gets work. +func StartHealthServer(ctx context.Context, addr string, readiness *Readiness) { + slog.InfoContext(ctx, "Starting health HTTP server", slog.String("addr", addr)) + if err := listenAndServe(addr, healthMux(readiness)); err != nil { + slog.Error("Health HTTP server failed", slog.Any("err", err)) + os.Exit(1) } } -// StartReadinessServer runs an HTTP server exposing only /readyz. Blocks until -// http.ListenAndServe returns; designed to be `go`-launched. A serve failure -// exits the process: a worker whose readiness endpoint cannot come up never -// turns Ready and never registers, so dying loudly lets the kubelet restart it -// instead of leaving a pod that looks alive but can never receive work. -func StartReadinessServer(ctx context.Context, addr string, readiness *Readiness) { - slog.InfoContext(ctx, "Starting readiness HTTP server", slog.String("addr", addr)) - if err := http.ListenAndServe(addr, readinessMux(readiness)); err != nil { - slog.Error("Readiness HTTP server failed", slog.Any("err", err)) - os.Exit(1) +func listenAndServe(addr string, handler http.Handler) error { + return newHTTPServer(addr, handler).ListenAndServe() +} + +// newHTTPServer builds the server for StartMetricsServer and +// StartHealthServer; split out so tests can check its timeouts. +func newHTTPServer(addr string, handler http.Handler) *http.Server { + return &http.Server{ + Addr: addr, + Handler: handler, + ReadHeaderTimeout: httpServerReadHeaderTimeout, } } // metricsMux builds the handler for StartMetricsServer; split out so // tests can exercise the endpoints without binding a port. -func metricsMux(opts MetricsServerOptions) *http.ServeMux { +func metricsMux() *http.ServeMux { mux := http.NewServeMux() mux.Handle("/metrics", promhttp.Handler()) - if opts.Readiness != nil { - mux.Handle("/readyz", readinessHandler(opts.Readiness)) - } - if opts.EnableHealthz { - mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte("ok")) - }) - } return mux } -// readinessMux builds the handler for StartReadinessServer. -func readinessMux(readiness *Readiness) *http.ServeMux { +// healthMux builds the handler for StartHealthServer. +func healthMux(readiness *Readiness) *http.ServeMux { mux := http.NewServeMux() mux.Handle("/readyz", readinessHandler(readiness)) + mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("ok")) + }) return mux } diff --git a/internal/serverboot/serverboot_test.go b/internal/serverboot/serverboot_test.go index b685a434b0..1ee6ab44c5 100644 --- a/internal/serverboot/serverboot_test.go +++ b/internal/serverboot/serverboot_test.go @@ -179,10 +179,7 @@ func TestMeterProviderRelayAttribute(t *testing.T) { func TestReadyzDrainsWhileHealthzStaysUp(t *testing.T) { readiness := &Readiness{} - mux := metricsMux(MetricsServerOptions{ - Readiness: readiness, - EnableHealthz: true, - }) + mux := healthMux(readiness) if got := getCode(t, mux, "/readyz"); got != http.StatusOK { t.Errorf("/readyz before drain = %d, want %d", got, http.StatusOK) @@ -202,40 +199,45 @@ func TestReadyzDrainsWhileHealthzStaysUp(t *testing.T) { } func TestReadyzStaticWithZeroValueReadiness(t *testing.T) { - mux := metricsMux(MetricsServerOptions{Readiness: &Readiness{}}) + mux := healthMux(&Readiness{}) if got := getCode(t, mux, "/readyz"); got != http.StatusOK { t.Errorf("/readyz with zero-value Readiness = %d, want %d", got, http.StatusOK) } } -func TestReadinessMux(t *testing.T) { - readiness := &Readiness{} - mux := readinessMux(readiness) - - if got := getCode(t, mux, "/readyz"); got != http.StatusOK { - t.Errorf("/readyz before drain = %d, want %d", got, http.StatusOK) - } - if got := getCode(t, mux, "/metrics"); got != http.StatusNotFound { - t.Errorf("/metrics = %d, want %d", got, http.StatusNotFound) +// The probes and /metrics live on separate servers, so turning one off can +// never take the other with it. +func TestMetricsAndHealthServeSeparateEndpoints(t *testing.T) { + servers := map[string]*http.ServeMux{ + "metrics": metricsMux(), + "health": healthMux(&Readiness{}), } - - readiness.MarkNotReady() - if got := getCode(t, mux, "/readyz"); got != http.StatusServiceUnavailable { - t.Errorf("/readyz during drain = %d, want %d", got, http.StatusServiceUnavailable) - } -} - -func TestReadyzAbsentWithoutReadiness(t *testing.T) { - mux := metricsMux(MetricsServerOptions{}) - if got := getCode(t, mux, "/readyz"); got != http.StatusNotFound { - t.Errorf("/readyz with nil Readiness = %d, want %d", got, http.StatusNotFound) + for _, tc := range []struct { + server string + path string + want int + }{ + {server: "metrics", path: "/metrics", want: http.StatusOK}, + {server: "metrics", path: "/readyz", want: http.StatusNotFound}, + {server: "metrics", path: "/healthz", want: http.StatusNotFound}, + {server: "health", path: "/metrics", want: http.StatusNotFound}, + {server: "health", path: "/readyz", want: http.StatusOK}, + {server: "health", path: "/healthz", want: http.StatusOK}, + } { + t.Run(tc.server+tc.path, func(t *testing.T) { + if got := getCode(t, servers[tc.server], tc.path); got != tc.want { + t.Errorf("%s server GET %s = %d, want %d", tc.server, tc.path, got, tc.want) + } + }) } } -func TestHealthzAbsentUnlessEnabled(t *testing.T) { - mux := metricsMux(MetricsServerOptions{Readiness: &Readiness{}}) - if got := getCode(t, mux, "/healthz"); got != http.StatusNotFound { - t.Errorf("/healthz without EnableHealthz = %d, want %d", got, http.StatusNotFound) +// A server with no ReadHeaderTimeout lets a slow client hold a connection +// open forever. +func TestHTTPServerBoundsHeaderRead(t *testing.T) { + srv := newHTTPServer(":0", http.NewServeMux()) + if srv.ReadHeaderTimeout <= 0 { + t.Errorf("ReadHeaderTimeout = %v, want > 0", srv.ReadHeaderTimeout) } } @@ -439,23 +441,32 @@ func bridgedRegistry(t *testing.T) *prometheus.Registry { return reg } -// With the OTLP export, the bridged family and the OTel instrument are each -// pushed once. The instrument must stay off the bridged registry: there the -// bridge would push it a second time, under its Prometheus name. +// With the OTLP export, whether selected or the default, the bridged family and +// the OTel instrument are each pushed once. The instrument must stay off the +// bridged registry: there the bridge would push it a second time, under its +// Prometheus name. func TestInitMetricsBridgedPushesEachMetricOnce(t *testing.T) { - collector := startMetricsCollector(t, "otlp") - reg := bridgedRegistry(t) - mp, err := InitMetricsBridged(t.Context(), "test-bridged-otlp", reg, nil) - if err != nil { - t.Fatalf("InitMetricsBridged: %v", err) - } - addOne(t, mp, "ate.test.bridged.count") - if gatheredNames(t, reg)["ate_test_bridged_count_total"] { - t.Error("the OTel instrument is registered on the bridged registry") - } - pushed := collector.flush(t, mp) - if pushed["ate.test.bridged.count"] != 1 || pushed["test_bridged_family"] != 1 || pushed["ate_test_bridged_count_total"] != 0 { - t.Errorf("want ate.test.bridged.count and test_bridged_family pushed once each and nothing else for them, got %v", pushed) + for _, tt := range []struct{ name, value string }{ + {"unset", ""}, + {"otlp", "otlp"}, + {"list", "otlp,prometheus"}, + } { + t.Run(tt.name, func(t *testing.T) { + collector := startMetricsCollector(t, tt.value) + reg := bridgedRegistry(t) + mp, _, err := InitMetricsBridged(t.Context(), "test-bridged-otlp", reg, nil) + if err != nil { + t.Fatalf("InitMetricsBridged: %v", err) + } + addOne(t, mp, "ate.test.bridged.count") + if gatheredNames(t, reg)["ate_test_bridged_count_total"] { + t.Error("the OTel instrument is registered on the bridged registry") + } + pushed := collector.flush(t, mp) + if pushed["ate.test.bridged.count"] != 1 || pushed["test_bridged_family"] != 1 || pushed["ate_test_bridged_count_total"] != 0 { + t.Errorf("want ate.test.bridged.count and test_bridged_family pushed once each and nothing else for them, got %v", pushed) + } + }) } } @@ -469,7 +480,7 @@ func TestInitMetricsBridgedWrapsPushedProducer(t *testing.T) { return inner.Produce(ctx) }) } - mp, err := InitMetricsBridged(t.Context(), "test-bridged-wrap", bridgedRegistry(t), wrap) + mp, _, err := InitMetricsBridged(t.Context(), "test-bridged-wrap", bridgedRegistry(t), wrap) if err != nil { t.Fatalf("InitMetricsBridged: %v", err) } @@ -486,27 +497,82 @@ type producerFunc func(context.Context) ([]metricdata.ScopeMetrics, error) func (f producerFunc) Produce(ctx context.Context) ([]metricdata.ScopeMetrics, error) { return f(ctx) } -// With OTEL_METRICS_EXPORTER=none, the OTel instrument is served from the +// With OTEL_METRICS_EXPORTER=prometheus, the OTel instrument is served from the // bridged registry next to its own families, and nothing is pushed. -func TestInitMetricsBridgedExporterNoneServesFromRegistry(t *testing.T) { - collector := startMetricsCollector(t, "none") +func TestInitMetricsBridgedPrometheusServesFromRegistry(t *testing.T) { + collector := startMetricsCollector(t, "prometheus") reg := bridgedRegistry(t) - mp, err := InitMetricsBridged(t.Context(), "test-bridged-none", reg, nil) + mp, servePull, err := InitMetricsBridged(t.Context(), "test-bridged-prometheus", reg, nil) if err != nil { t.Fatalf("InitMetricsBridged: %v", err) } - addOne(t, mp, "ate.test.bridgednone.count") + if !servePull { + t.Error("OTEL_METRICS_EXPORTER=prometheus turned the scrape endpoint off") + } + addOne(t, mp, "ate.test.bridgedprom.count") names := gatheredNames(t, reg) - if !names["ate_test_bridgednone_count_total"] || !names["test_bridged_family"] { + if !names["ate_test_bridgedprom_count_total"] || !names["test_bridged_family"] { t.Errorf("the bridged registry does not serve both the instrument and its own family: %v", names) } + if pushed := collector.flush(t, mp); pushed != nil { + t.Errorf("OTEL_METRICS_EXPORTER=prometheus still pushed %v", pushed) + } +} + +// With OTEL_METRICS_EXPORTER=none there is no push and no endpoint to serve, +// so the instrument is not registered anywhere. +func TestInitMetricsBridgedExporterNoneExportsNothing(t *testing.T) { + collector := startMetricsCollector(t, "none") + reg := bridgedRegistry(t) + mp, servePull, err := InitMetricsBridged(t.Context(), "test-bridged-none", reg, nil) + if err != nil { + t.Fatalf("InitMetricsBridged: %v", err) + } + if servePull { + t.Error("OTEL_METRICS_EXPORTER=none left the scrape endpoint on") + } + addOne(t, mp, "ate.test.bridgednone.count") + if gatheredNames(t, reg)["ate_test_bridgednone_count_total"] { + t.Error("the instrument is registered on the bridged registry") + } if pushed := collector.flush(t, mp); pushed != nil { t.Errorf("OTEL_METRICS_EXPORTER=none still pushed %v", pushed) } } +// servePull follows the policy table: unset and the list keep both paths. +func TestInitMetricsBridgedServePull(t *testing.T) { + tests := []struct { + name string + value string + wantPush bool + wantPull bool + }{ + {"unset", "", true, true}, + {"otlp", "otlp", true, false}, + {"prometheus", "prometheus", false, true}, + {"list", "otlp,prometheus", true, true}, + {"none", "none", false, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + collector := startMetricsCollector(t, tt.value) + mp, servePull, err := InitMetricsBridged(t.Context(), "test-bridged-table", bridgedRegistry(t), nil) + if err != nil { + t.Fatalf("InitMetricsBridged: %v", err) + } + if servePull != tt.wantPull { + t.Errorf("servePull = %t, want %t", servePull, tt.wantPull) + } + if pushed := collector.flush(t, mp) != nil; pushed != tt.wantPush { + t.Errorf("pushed = %t, want %t", pushed, tt.wantPush) + } + }) + } +} + func TestInitMetricsBridgedRequiresServiceName(t *testing.T) { - if _, err := InitMetricsBridged(t.Context(), "", prometheus.NewRegistry(), nil); err == nil { + if _, _, err := InitMetricsBridged(t.Context(), "", prometheus.NewRegistry(), nil); err == nil { t.Error("InitMetricsBridged(\"\") must return an error") } } diff --git a/internal/snapshotcache/snapshotcache.go b/internal/snapshotcache/snapshotcache.go new file mode 100644 index 0000000000..a18773a649 --- /dev/null +++ b/internal/snapshotcache/snapshotcache.go @@ -0,0 +1,157 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package snapshotcache disposes of staged restore files and evicts preserved +// snapshot files from the host page cache after an actor restore completes. +package snapshotcache + +import ( + "errors" + "io/fs" + "log/slog" + "os" + "path/filepath" + "sync" + "time" + + "golang.org/x/sys/unix" +) + +// File identifies a snapshot data file pinned by its FileInfo at discovery time +// so delayed post-writeback passes cannot evict or unlink a newer file written +// at the same path. +type File struct { + Path string + Info fs.FileInfo +} + +var ( + cachestatUnsupportedOnce sync.Once + + // cachestatFn and fadviseFn are seams for unit tests so eviction logic can + // be verified deterministically regardless of the backing filesystem (ext4, + // tmpfs, overlayfs) or kernel version. + cachestatFn = unix.Cachestat + fadviseFn = unix.Fadvise + + // RetryDelays are the backoff intervals between post-restore eviction + // passes when dirty pages are waiting for background kernel writeback. + RetryDelays = []time.Duration{ + 500 * time.Millisecond, + 1500 * time.Millisecond, + 3000 * time.Millisecond, + } +) + +// FindFiles walks dir and returns regular files whose base name satisfies +// match, pinned by their FileInfo at discovery time. +func FindFiles(dir string, match func(name string) bool) []File { + if dir == "" { + return nil + } + var targets []File + _ = filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error { + if err != nil || !d.Type().IsRegular() || !match(d.Name()) { + return nil + } + info, err := d.Info() + if err != nil { + return nil + } + targets = append(targets, File{Path: path, Info: info}) + return nil + }) + return targets +} + +// DropFile unlinks t.Path if it still refers to the same inode and size. +func DropFile(t File) { + st, err := os.Stat(t.Path) + if err != nil { + if !errors.Is(err, os.ErrNotExist) { + slog.Debug("could not stat staged snapshot file", slog.String("path", t.Path), slog.Any("err", err)) + } + return + } + if !os.SameFile(st, t.Info) || st.Size() != t.Info.Size() { + return + } + if err := os.Remove(t.Path); err != nil && !errors.Is(err, os.ErrNotExist) { + slog.Warn("could not drop staged snapshot file", slog.String("path", t.Path), slog.Any("err", err)) + } +} + +// DropFiles calls DropFile for each file in files. +func DropFiles(files []File) { + for _, t := range files { + DropFile(t) + } +} + +// EvictFiles evicts clean cached pages of targets via POSIX_FADV_DONTNEED and +// returns true if any target still exists and has pages (clean or dirty) +// remaining in cache. Files with dirty or writeback pages are skipped so +// FADV_DONTNEED never forces synchronous disk writeback I/O. +func EvictFiles(targets []File) bool { + anyRemaining := false + for _, t := range targets { + f, err := os.Open(t.Path) + if err != nil { + continue + } + info, err := f.Stat() + if err == nil && os.SameFile(info, t.Info) && info.Size() == t.Info.Size() { + var cstat unix.Cachestat_t + if err := cachestatFn(uint(f.Fd()), &unix.CachestatRange{}, &cstat, 0); err != nil { + if errors.Is(err, unix.ENOSYS) { + cachestatUnsupportedOnce.Do(func() { + slog.Warn("cachestat(2) syscall not supported by kernel; skipping preserved snapshot page-cache eviction", slog.Any("err", err)) + }) + } + _ = f.Close() + continue + } + if cstat.Cache == 0 { + _ = f.Close() + continue + } + anyRemaining = true + if cstat.Dirty > 0 || cstat.Writeback > 0 { + _ = f.Close() + continue + } + // length zero extends the Fadvise to the whole file. + _ = fadviseFn(int(f.Fd()), 0, 0, unix.FADV_DONTNEED) + } + _ = f.Close() + } + return anyRemaining +} + +// EvictWithRetries runs EvictFiles immediately and, while pages remain in +// cache, retries after each interval in RetryDelays to evict pages once +// background kernel writeback cleans them. +func EvictWithRetries(targets []File) { + if !EvictFiles(targets) { + return + } + for _, delay := range RetryDelays { + time.Sleep(delay) + if !EvictFiles(targets) { + return + } + } +} diff --git a/internal/snapshotcache/snapshotcache_test.go b/internal/snapshotcache/snapshotcache_test.go new file mode 100644 index 0000000000..d4843aaa2d --- /dev/null +++ b/internal/snapshotcache/snapshotcache_test.go @@ -0,0 +1,195 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package snapshotcache + +import ( + "os" + "path/filepath" + "testing" + + "golang.org/x/sys/unix" +) + +func TestFindAndDropFiles(t *testing.T) { + dir := filepath.Join(t.TempDir(), "restore-state") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + for _, name := range []string{"checkpoint.img", "pages.img", "durable-dir.tar", "manifest.json"} { + if err := os.WriteFile(filepath.Join(dir, name), []byte("staged data"), 0o600); err != nil { + t.Fatal(err) + } + } + + files := FindFiles(dir, func(name string) bool { + ext := filepath.Ext(name) + return ext == ".img" || ext == ".tar" + }) + if len(files) != 3 { + t.Fatalf("FindFiles() = %d files, want 3", len(files)) + } + + DropFiles(files) + + for _, name := range []string{"checkpoint.img", "pages.img", "durable-dir.tar"} { + if _, err := os.Stat(filepath.Join(dir, name)); !os.IsNotExist(err) { + t.Fatalf("staged %s still exists after DropFiles: err=%v", name, err) + } + } + if _, err := os.Stat(filepath.Join(dir, "manifest.json")); err != nil { + t.Fatalf("manifest.json should be preserved: %v", err) + } +} + +func TestDropFileSkipsReplacedInode(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "pages.img") + if err := os.WriteFile(path, []byte("old"), 0o600); err != nil { + t.Fatal(err) + } + files := FindFiles(dir, func(name string) bool { return name == "pages.img" }) + if len(files) != 1 { + t.Fatalf("FindFiles() = %d, want 1", len(files)) + } + + replacement := filepath.Join(dir, "pages.img.new") + if err := os.WriteFile(replacement, []byte("new"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.Rename(replacement, path); err != nil { + t.Fatal(err) + } + + DropFiles(files) + got, err := os.ReadFile(path) + if err != nil { + t.Fatalf("replaced file was deleted: %v", err) + } + if string(got) != "new" { + t.Fatalf("replaced file content = %q, want %q", string(got), "new") + } +} + +func TestEvictFiles(t *testing.T) { + origCachestat := cachestatFn + origFadvise := fadviseFn + t.Cleanup(func() { + cachestatFn = origCachestat + fadviseFn = origFadvise + }) + + dir := t.TempDir() + img := filepath.Join(dir, "pages.img") + if err := os.WriteFile(img, []byte("checkpoint pages"), 0o600); err != nil { + t.Fatal(err) + } + targets := FindFiles(dir, func(name string) bool { return name == "pages.img" }) + if len(targets) != 1 { + t.Fatalf("FindFiles() = %d, want 1", len(targets)) + } + + t.Run("evicts clean cached pages", func(t *testing.T) { + var fadviseCalls int + cachestatFn = func(_ uint, _ *unix.CachestatRange, cstat *unix.Cachestat_t, _ uint) error { + *cstat = unix.Cachestat_t{Cache: 16, Dirty: 0, Writeback: 0} + return nil + } + fadviseFn = func(_ int, _ int64, _ int64, advice int) error { + if advice != unix.FADV_DONTNEED { + t.Errorf("fadvise advice = %d, want FADV_DONTNEED", advice) + } + fadviseCalls++ + return nil + } + if !EvictFiles(targets) { + t.Errorf("EvictFiles() = false, want true when pages were cached") + } + if fadviseCalls != 1 { + t.Errorf("fadviseCalls = %d, want 1", fadviseCalls) + } + }) + + t.Run("skips dirty or writeback pages without calling fadvise", func(t *testing.T) { + var fadviseCalls int + cachestatFn = func(_ uint, _ *unix.CachestatRange, cstat *unix.Cachestat_t, _ uint) error { + *cstat = unix.Cachestat_t{Cache: 16, Dirty: 4, Writeback: 0} + return nil + } + fadviseFn = func(int, int64, int64, int) error { + fadviseCalls++ + return nil + } + if !EvictFiles(targets) { + t.Errorf("EvictFiles() = false, want true so retry passes run after writeback") + } + if fadviseCalls != 0 { + t.Errorf("fadviseCalls = %d, want 0 for dirty pages", fadviseCalls) + } + }) + + t.Run("returns false when no pages are cached", func(t *testing.T) { + var fadviseCalls int + cachestatFn = func(_ uint, _ *unix.CachestatRange, cstat *unix.Cachestat_t, _ uint) error { + *cstat = unix.Cachestat_t{Cache: 0} + return nil + } + fadviseFn = func(int, int64, int64, int) error { + fadviseCalls++ + return nil + } + if EvictFiles(targets) { + t.Errorf("EvictFiles() = true, want false when Cache == 0") + } + if fadviseCalls != 0 { + t.Errorf("fadviseCalls = %d, want 0", fadviseCalls) + } + }) + + t.Run("returns false and skips fadvise on ENOSYS", func(t *testing.T) { + var fadviseCalls int + cachestatFn = func(uint, *unix.CachestatRange, *unix.Cachestat_t, uint) error { + return unix.ENOSYS + } + fadviseFn = func(int, int64, int64, int) error { + fadviseCalls++ + return nil + } + if EvictFiles(targets) { + t.Errorf("EvictFiles() = true on ENOSYS, want false") + } + if fadviseCalls != 0 { + t.Errorf("fadviseCalls = %d, want 0", fadviseCalls) + } + }) + + t.Run("stops eviction when inode is replaced", func(t *testing.T) { + cachestatFn = func(_ uint, _ *unix.CachestatRange, cstat *unix.Cachestat_t, _ uint) error { + *cstat = unix.Cachestat_t{Cache: 16} + return nil + } + replacement := filepath.Join(dir, "pages.img.new") + if err := os.WriteFile(replacement, []byte("new checkpoint pages"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.Rename(replacement, img); err != nil { + t.Fatal(err) + } + if EvictFiles(targets) { + t.Errorf("EvictFiles() = true after inode replacement, want false") + } + }) +} diff --git a/internal/versionlabel/cmd/main.go b/internal/versionlabel/cmd/main.go deleted file mode 100644 index afe127f070..0000000000 --- a/internal/versionlabel/cmd/main.go +++ /dev/null @@ -1,40 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// Command cmd prints "