diff --git a/README.md b/README.md index 3f6d4b021b..6432c37fb3 100644 --- a/README.md +++ b/README.md @@ -250,7 +250,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit * [Egress Traffic](docs/egress-traffic.md): Which protocols an Actor may reach the outside world with and which are blocked. * [Enabling man-in-the-middle (MITM) interception for Actor Egress policy](docs/egress-trust-bundle.md): Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception. * [Request Parking](docs/request-parking.md): How the router parks requests through transient worker-pool saturation. -* [Rolling Upgrade Runbook](docs/upgrade.md): Upgrade a running substrate node by node without losing actor state. +* [Upgrade Runbook](docs/upgrade.md): Upgrade a running install to a newer supported release, and roll back if needed. * [Security Guide](docs/security.md): Security overview, security principles, and threat modeling. * [Roadmap](docs/roadmap.md): Current limitations and what is planned next. * [Benchmarking Guide](benchmarking/README.md): Locust-based load tests, monitoring stack, and the orchestrated benchmark harness. diff --git a/benchmarking/README.md b/benchmarking/README.md index c057b4e6f3..8c45b2c7ea 100644 --- a/benchmarking/README.md +++ b/benchmarking/README.md @@ -111,7 +111,7 @@ Four flags control the optional post-run measurements described in Defaults to the in-cluster service installed by [Optional: Prometheus + Grafana](#optional-prometheus--grafana). * `--atelet-lag-s`: how long to wait after the run before reading the - atelet's snapshot metrics. Defaults to 70. + atelet's snapshot and active actor metrics. Defaults to 70. * `--phase-breakdown` / `--no-phase-breakdown`: read the atelet and worker pod logs once the run ends and add the suspend/resume phase percentiles to `stats.jsonl`. On by default; needs `pods/log` access in `ate-system` and @@ -522,7 +522,9 @@ them are checked into the repository. those two keys, because it is what CI orchestration reads to decide whether a trial ran at all. * `stats.csv`, `stats_history.csv`, `failures.csv`, `exceptions.csv`: Locust's - own CSV output. + own CSV output. `stats_history.csv` has a row per operation every second as + well as the `Aggregated` row, so each operation's latency can be charted + over the run; filter to `Aggregated` for run-wide numbers. * `logs.txt`, `traces.txt`: the runner log, and the trace IDs seen during the run. * `stats.jsonl`: one JSON object per line, one per metric. Every row carries the same five keys: `timestamp`, `tag`, `test_name`, `metric`, and a flat @@ -565,7 +567,9 @@ comparing numbers across runs: * **Actors are derived, not counted.** Locust only sees virtual users, so the numerator is the peak user count times `--actors-per-user`. No server-side gauge counts resident actors: `ate.actor.stats.sampled_actors` drops any - actor without a live resource measurement, so suspended ones fall out. + actor without a live resource measurement, so suspended ones fall out. It + is recorded as `active_actors` in `server_summary.json`, a running-actor + count rather than a resident one. * **The denominators are read once, after the run.** A cluster that autoscaled mid-run is measured at its final size, so the ratio pairs a peak from one moment with a capacity from another. @@ -591,11 +595,29 @@ actually did, independent of what the load generator reported. * `snapshots.size_p50_mb` through `size_p99_mb`: actor memory image sizes. * `snapshots.size_avg_mb`: mean memory image size. * `snapshots.restore_p50_s` through `restore_p99_s`, `restore_mean_s`, and the - same for `checkpoint_*`: atelet restore and checkpoint latency. + same for `checkpoint_*`: how long the atelet took to resume or suspend an + actor (`ate_actor_{restore,checkpoint}_duration`, `total` phase). Every + snapshot kind counts, as in the sizes: the actor's own snapshot, a + template's first start from its golden snapshot, and a pause. The buckets + reach 60s, so a slower operation reads as 60s. * `snapshots.checkpoints_in_window`, `checkpoints_cumulative`: checkpoint volume over the steady-state window, and since the atelet started. * `snapshots.checkpoint_mb_s`: bytes written per second spent checkpointing, not per second of wall clock. +* `active_actors`: running actors (`ate_actor_stats_sampled_actors`), as a + percentile `summary` of the cluster-wide count, `atelets` seen, and a + `timeseries` every 10s over the whole run (ramp-up included). Each point has + the cluster-wide count and `active_atelets` hosting an actor. As the + assumptions above note, the gauge drops actors without a live measurement, + so suspended ones fall out and the count dips while actors are suspended. + The atelet stops exporting when a node has no running actor, so at a step + where at least one atelet reported, a missing one counts as 0. A step where + none reported is skipped, not 0, since a failed scrape looks the same; this + mostly hits 1-user runs. If the metric never appeared during the run, every + field is `null`. `atelets` counts atelet processes seen in the run, not + nodes: the collector keeps an exited atelet's last value for about 5 + minutes, so a restart can briefly count twice, and an atelet scraped both + directly and through a collector counts twice. Every distribution reports p50, p90, p95 and p99 over the steady-state window. The steady-state window runs from the first to the last Locust sample at 90% or @@ -608,13 +630,13 @@ every 10s. The atelet exports before Prometheus scrapes it, so the harvest waits `--atelet-lag-s` seconds (default 70, enough for the OTel SDK's 60s default -export and a 10s scrape) -and reads the snapshot window half that late. The window ends at the last -full-load sample, so teardown suspends are left out. +export and a 10s scrape) and reads the snapshot and `active_actors` windows +half that late, stamping `active_actors` points back by the same amount. The +window ends at the last full-load sample, so teardown suspends are left out. -`metadata.start_ts` and `end_ts` bound the whole run, which the packing -`timeseries` covers. `steady_start_ts` and `steady_end_ts` bound the -steady-state window. A flat subset of the numbers also goes into a +`metadata.start_ts` and `end_ts` bound the whole run, which the packing and +`active_actors` timeseries cover. `steady_start_ts` and `steady_end_ts` bound +the steady-state window. A flat subset of the numbers also goes into a `server_summary` row in `stats.jsonl`. Neither the Kubernetes API nor Prometheus is required. If either is unreachable, diff --git a/benchmarking/automation/manifests/nighthawk-ingress-runner-job.yaml.tmpl b/benchmarking/automation/manifests/nighthawk-ingress-runner-job.yaml.tmpl index c63c8b9937..6f6be20688 100644 --- a/benchmarking/automation/manifests/nighthawk-ingress-runner-job.yaml.tmpl +++ b/benchmarking/automation/manifests/nighthawk-ingress-runner-job.yaml.tmpl @@ -20,6 +20,9 @@ apiVersion: v1 kind: Namespace metadata: name: benchmarking + labels: + # ate-system admits API clients only from namespaces with this label. + ate.dev/api-access: "true" --- apiVersion: v1 kind: ServiceAccount diff --git a/benchmarking/automation/manifests/runner-job.yaml.tmpl b/benchmarking/automation/manifests/runner-job.yaml.tmpl index 9267de1219..eb5e4acc5d 100644 --- a/benchmarking/automation/manifests/runner-job.yaml.tmpl +++ b/benchmarking/automation/manifests/runner-job.yaml.tmpl @@ -20,6 +20,9 @@ apiVersion: v1 kind: Namespace metadata: name: benchmarking + labels: + # ate-system admits API clients only from namespaces with this label. + ate.dev/api-access: "true" --- apiVersion: v1 kind: ServiceAccount diff --git a/benchmarking/automation/test_orchestrator.py b/benchmarking/automation/test_orchestrator.py index d5fea10c9f..b85c368107 100644 --- a/benchmarking/automation/test_orchestrator.py +++ b/benchmarking/automation/test_orchestrator.py @@ -135,6 +135,39 @@ def test_no_placeholder_survives(self): self.assertNotIn("${", orchestrator.render_template(self.TMPL, subs)) +class RunnerNamespaceTest(unittest.TestCase): + """Every runner Job's namespace carries the label ate-system's + NetworkPolicy admits API clients by; without it the runner cannot reach + ate-api-server.""" + + HERE = os.path.dirname(__file__) + POLICIES = os.path.join(HERE, "..", "..", "manifests", "ate-install", "ate-system-networkpolicies.yaml") + + def api_client_labels(self): + for doc in yaml.safe_load_all(open(self.POLICIES)): + if not doc or doc["metadata"]["name"] != "ate-api-server": + continue + for rule in doc["spec"]["ingress"]: + for peer in rule.get("from", []): + if "namespaceSelector" in peer: + return peer["namespaceSelector"]["matchLabels"] + self.fail("ate-api-server NetworkPolicy admits no namespace by label") + + def test_namespace_has_api_access_label(self): + want = self.api_client_labels() + for name in ("runner-job.yaml.tmpl", "nighthawk-ingress-runner-job.yaml.tmpl"): + with self.subTest(template=name): + text = open(os.path.join(self.HERE, "manifests", name)).read() + ns = next( + yaml.safe_load(d) + for d in text.split("\n---\n") + if "kind: Namespace" in d + ) + self.assertEqual(ns["metadata"]["name"], "benchmarking") + for k, v in want.items(): + self.assertEqual(ns["metadata"].get("labels", {}).get(k), v) + + class JobNameTest(unittest.TestCase): COMMIT = "ac41c06deadbeef" diff --git a/benchmarking/locust/runner.py b/benchmarking/locust/runner.py index 9c722b5b1c..42cdcc8ddb 100644 --- a/benchmarking/locust/runner.py +++ b/benchmarking/locust/runner.py @@ -161,10 +161,11 @@ def parse_args() -> argparse.Namespace: type=int, default=70, help=( - "Seconds to wait after the run, and to shift the snapshot window " - "by, so the atelet's last export is scraped. Must cover the " - "atelet's OTEL_METRIC_EXPORT_INTERVAL plus one Prometheus scrape; " - "the default fits the OTel SDK's 60s default export and a 10s scrape" + "Seconds to wait after the run, so the atelet's last export is " + "scraped; its snapshot and active actor windows are read half this " + "late. Must cover the atelet's OTEL_METRIC_EXPORT_INTERVAL plus one " + "Prometheus scrape; the default fits the OTel SDK's 60s default export " + "and a 10s scrape" ), ) p.add_argument( @@ -344,6 +345,10 @@ def run_test(args: argparse.Namespace, csv_prefix: Path, logs: TextIO, traces: T "-t", args.duration, "-u", str(args.users), "--csv", str(csv_prefix), + # Every operation's row in stats_history.csv, not just Aggregated, so + # each one's latency can be charted over the run. Readers of the file + # here keep to the Aggregated rows. + "--csv-full-history", ] if with_boomer: # Master mode so boomer can connect as a worker on localhost:5557. diff --git a/benchmarking/locust/server_telemetry.py b/benchmarking/locust/server_telemetry.py index dff9c2dd91..3db4837dcc 100644 --- a/benchmarking/locust/server_telemetry.py +++ b/benchmarking/locust/server_telemetry.py @@ -14,9 +14,9 @@ """Harvests server-side Prometheus ground-truth timeseries during benchmark trials. -Queries Prometheus over [T_start, T_end] and the steady-state window [T_steady, T_end] -to capture cluster packing, node and pod PSI stalls, and snapshot sizes, latencies -and throughput. +Queries Prometheus over [T_start, T_end] and the steady-state window inside it +to capture cluster packing, node and pod PSI stalls, snapshot sizes, latencies +and throughput, and running actors. """ import csv @@ -274,12 +274,35 @@ def _steady_values( return vals -def _rpc_buckets(method: str) -> str: - """The duration bucket selector for one AteomHerder RPC.""" - return ( - 'rpc_server_call_duration_seconds_bucket' - f'{{rpc_method="atelet.AteomHerder/{method}"}}' - ) +def _snapshot_op(op: str, suffix: str) -> str: + """atelet's histogram for a whole suspend (checkpoint) or resume (restore). + + Buckets reach 60s. Every snapshot kind counts, like the size metric, which + has no kind label. + """ + return f'ate_actor_{op}_duration_seconds_{suffix}{{ate_snapshot_phase="total"}}' + + +def _series_samples(series: dict[str, Any]) -> dict[int, float]: + """A range series' finite values by timestamp; malformed points are skipped. + + Inf is dropped as well as NaN: these reach server_summary.json, and + json.dumps would emit a bare Infinity that strict parsers reject. + """ + samples: dict[int, float] = {} + for pt in series.get("values", []): + try: + t, val = int(pt[0]), float(pt[1]) + except (ValueError, IndexError, TypeError): + continue + if not math.isnan(val) and not math.isinf(val): + samples[t] = val + return samples + + +def _process_key(metric: dict[str, str]) -> str: + """`exported_instance` when a collector re-exports, else the scraped one.""" + return metric.get("exported_instance") or metric.get("instance", "") def _harvest_cluster_packing( @@ -302,18 +325,9 @@ def _harvest_cluster_packing( for series in packing_series: metric = series.get("metric", {}) state = metric.get("ate_worker_state", "unknown") - instance = metric.get("exported_instance") or metric.get("instance", "") - samples = by_instance.setdefault(instance, {}) - for pt in series.get("values", []): - try: - t = int(pt[0]) - val = float(pt[1]) - # Inf as well as NaN: these reach server_summary.json, and - # json.dumps would emit a bare Infinity that strict parsers reject. - if not math.isnan(val) and not math.isinf(val): - samples.setdefault(t, {})[state] = val - except (ValueError, IndexError): - continue + samples = by_instance.setdefault(_process_key(metric), {}) + for t, val in _series_samples(series).items(): + samples.setdefault(t, {})[state] = val # The collector re-exports exited ateapis until they expire, so read # the process whose series runs latest, which is a live one. @@ -436,15 +450,13 @@ def quantiles(buckets: str, scale: float = 1.0) -> list[float | None]: c_end = round(count_end) if count_end is not None else None snap_avg = _ratio(delta(snap_selector % "_sum"), count, scale=mb) - def rpc_mean(method: str) -> float | None: - sel = f'{{rpc_method="atelet.AteomHerder/{method}"}}' + def op_mean(op: str) -> float | None: return _ratio( - delta(f"rpc_server_call_duration_seconds_sum{sel}"), - delta(f"rpc_server_call_duration_seconds_count{sel}"), + delta(_snapshot_op(op, "sum")), delta(_snapshot_op(op, "count")) ) - restore_q = quantiles(_rpc_buckets("Restore")) - ckpt_q = quantiles(_rpc_buckets("Checkpoint")) + restore_q = quantiles(_snapshot_op("restore", "bucket")) + ckpt_q = quantiles(_snapshot_op("checkpoint", "bucket")) # All files' bytes over `total`-phase seconds. written = delta("atelet_snapshot_size_bytes_sum") @@ -465,16 +477,91 @@ def rpc_mean(method: str) -> float | None: "restore_p90_s": restore_q[1], "restore_p95_s": restore_q[2], "restore_p99_s": restore_q[3], - "restore_mean_s": rpc_mean("Restore"), + "restore_mean_s": op_mean("restore"), "checkpoint_p50_s": ckpt_q[0], "checkpoint_p90_s": ckpt_q[1], "checkpoint_p95_s": ckpt_q[2], "checkpoint_p99_s": ckpt_q[3], - "checkpoint_mean_s": rpc_mean("Checkpoint"), + "checkpoint_mean_s": op_mean("checkpoint"), "checkpoint_mb_s": checkpoint_mb_s, } +def _count_active_actors( + prom_url: str, + start_ts: int, + end_ts: int, + steady_start_ts: int, + steady_end_ts: int, + lag_s: int = 0, +) -> dict[str, Any]: + """Running actors over the run, cluster-wide. + + `summary` is percentiles of the cluster-wide total over the steady window. + `timeseries` covers the whole run, ramp-up included: every 10s, the total + and `active_atelets` hosting an actor. `atelets` counts the atelet + processes seen in the run, not nodes. + + The atelet drops a template's series when its last actor on the node + leaves, so at a step where at least one atelet reported, a missing series + is zero actors. A step where none reported is skipped, since a failed + scrape looks the same as no actors anywhere. If the metric was never seen, + every field stays null. + + The collector re-exports an exited atelet's last value for about 5 minutes, + so a restarted atelet can briefly count twice. An atelet scraped directly + and through a collector shows up under 2 keys and is not merged. + + The atelet exports it like the snapshot block, so it is read `lag_s // 2` + late and each point is stamped back by the same amount. + """ + shift = lag_s // 2 + first, last = start_ts + shift, max(end_ts, start_ts + 1) + shift + steady_first, steady_last = steady_start_ts + shift, steady_end_ts + shift + res = query_prometheus_range( + prom_url, + "sum by (instance, exported_instance) (ate_actor_stats_sampled_actors)", + first, last, step="10s", + ) + # Same key as packing; 2 series under one key (e.g. 2 collector replicas) + # describe the same atelet, so they merge by max, not sum. + by_atelet: dict[str, dict[int, float]] = {} + for series in res: + new = _series_samples(series) + if not new: + continue + samples = by_atelet.setdefault(_process_key(series.get("metric", {})), {}) + for t, v in new.items(): + samples[t] = max(samples.get(t, v), v) + + if not by_atelet: + return { + "summary": compute_percentiles([]), + "atelets": None, + "timeseries": [], + } + + # Range query samples land on first + k * step. + points, steady_totals = [], [] + for t in range(first, last + 1, 10): + if not any(t in s for s in by_atelet.values()): + continue + counts = [s.get(t, 0.0) for s in by_atelet.values()] + total = sum(counts) + points.append({ + "timestamp": t - shift, + "active_actors": total, + "active_atelets": sum(1 for v in counts if v > 0), + }) + if steady_first <= t <= steady_last: + steady_totals.append(total) + return { + "summary": compute_percentiles(steady_totals), + "atelets": len(by_atelet), + "timeseries": points, + } + + def harvest_server_telemetry( prom_url: str, start_ts: int, @@ -485,8 +572,9 @@ def harvest_server_telemetry( ) -> dict[str, Any]: """Harvests the ground truth metric streams from Prometheus. - Only the atelet-exported snapshot block is read late, by up to `lag_s`; - packing and PSI are scraped directly and read over the run window as is. + Only the atelet-exported snapshot and active actor blocks are read late, by + `lag_s // 2`; packing and PSI aren't shifted and are read over the run + window as is. """ steady_end = end_ts if steady_end_ts is None else steady_end_ts return { @@ -502,6 +590,9 @@ def harvest_server_telemetry( "snapshots": _harvest_snapshots( prom_url, steady_start_ts, steady_end, lag_s ), + "active_actors": _count_active_actors( + prom_url, start_ts, end_ts, steady_start_ts, steady_end, lag_s + ), } @@ -539,8 +630,8 @@ def log(msg: str) -> None: ) log( f"Detected steady-state window: [{steady_start}, {steady_end}] " - f"({steady_end - steady_start}s), snapshot reads at " - f"[{steady_start + lag_s // 2}, {steady_end + lag_s // 2}]" + f"({steady_end - steady_start}s), atelet-exported reads (snapshots, " + f"active actors) at [{steady_start + lag_s // 2}, {steady_end + lag_s // 2}]" ) telemetry = harvest_server_telemetry( @@ -595,6 +686,9 @@ def log(msg: str) -> None: measurements[f"{rpc}_{p}_s"] = snaps.get(f"{rpc}_{p}_s") measurements[f"{rpc}_mean_s"] = snaps.get(f"{rpc}_mean_s") measurements["checkpoint_mb_s"] = snaps.get("checkpoint_mb_s") + active = telemetry.get("active_actors", {}).get("summary", {}) + for p in ps: + measurements[f"active_actors_{p}"] = active.get(p) jsonl_row = { "timestamp": data_ts, diff --git a/benchmarking/locust/unit_tests/test_server_telemetry.py b/benchmarking/locust/unit_tests/test_server_telemetry.py index 3dfeebdd78..7d52929d37 100644 --- a/benchmarking/locust/unit_tests/test_server_telemetry.py +++ b/benchmarking/locust/unit_tests/test_server_telemetry.py @@ -40,8 +40,8 @@ "end_ts": 105, "steady_start_ts": 100} -def ranges(packing, node=(), pod=()): - """A query_prometheus_range fake answering packing, node PSI and pod PSI. +def ranges(packing, node=(), pod=(), active=()): + """A query_prometheus_range fake answering packing, PSI and active actors. Pod PSI only answers the pod-slice selector, so a query that loses the id regex (and would also sum the pause container) gets nothing back. Node PSI @@ -54,6 +54,8 @@ def ranges(packing, node=(), pod=()): def fake(_url, query, *_args, **_kwargs): if "ate_workerpool_workers" in query: return packing + if "ate_actor_stats_sampled_actors" in query: + return list(active) if 'container="node"' in query: return list(node) if worker_nodes in query and pod_slice in query else [] return list(pod) if pod_slice in query else [] @@ -84,17 +86,20 @@ def parse(*extra): # The memory image selector: pages.img (gVisor) or memory-ranges (microVM). MEMORY_IMAGE = '{file_name=~"pages.img|memory-ranges"}' +# A whole suspend or resume, every snapshot kind. +SNAPSHOT_OP = '{ate_snapshot_phase="total"}' + # Each cumulative sum _harvest_snapshots reads, by a substring of its query. SUMS = { "count": "atelet_snapshot_size_bytes_count" + MEMORY_IMAGE, "size": "atelet_snapshot_size_bytes_sum" + MEMORY_IMAGE, "bytes": "sum(atelet_snapshot_size_bytes_sum)", - "seconds": "ate_actor_checkpoint_duration_seconds_sum", - "restore_sum": 'seconds_sum{rpc_method="atelet.AteomHerder/Restore"}', - "restore_count": 'seconds_count{rpc_method="atelet.AteomHerder/Restore"}', - "checkpoint_sum": 'seconds_sum{rpc_method="atelet.AteomHerder/Checkpoint"}', + # Checkpoint seconds: both checkpoint_mb_s and checkpoint_mean_s read it. + "seconds": "ate_actor_checkpoint_duration_seconds_sum" + SNAPSHOT_OP, + "restore_sum": "ate_actor_restore_duration_seconds_sum" + SNAPSHOT_OP, + "restore_count": "ate_actor_restore_duration_seconds_count" + SNAPSHOT_OP, "checkpoint_count": - 'seconds_count{rpc_method="atelet.AteomHerder/Checkpoint"}', + "ate_actor_checkpoint_duration_seconds_count" + SNAPSHOT_OP, } @@ -123,14 +128,15 @@ def fake(_url, query, time_ts=None): def history_csv(rows): - """A stats_history.csv built from (timestamp, user count) pairs.""" + """A stats_history.csv from (timestamp, users) or (timestamp, users, name).""" f = tempfile.NamedTemporaryFile( "w", delete=False, suffix=".csv", encoding="utf-8" ) writer = csv.DictWriter(f, fieldnames=["Timestamp", "Name", "User Count"]) writer.writeheader() - for ts, users in rows: - writer.writerow({"Timestamp": ts, "Name": "Aggregated", "User Count": users}) + for ts, users, *name in rows: + writer.writerow({"Timestamp": ts, "Name": name[0] if name else "Aggregated", + "User Count": users}) f.close() return Path(f.name) @@ -234,16 +240,18 @@ def test_packing_and_checkpoint_math(self, mock_instant, mock_range): "values": [[100, "0.0"], [105, "0.0"]]} for n in "ab"] pods = [{"metric": {"pod": f"benchmark-ateom-{p}"}, "values": [[100, "0.5"], [105, "0.5"]]} for p in "ab"] - mock_range.side_effect = ranges([partial, full, idle], node=quiet, pod=pods) + mock_range.side_effect = ranges( + [partial, full, idle], node=quiet, pod=pods, + active=[{"metric": {"instance": "atelet-a"}, "values": [[135, "3"]]}]) # The default 70s lag reads the snapshot window at 135 and 140. mock_instant.side_effect = snapshot_prom( "11.5", start={"count": "100", "size": "0", "restore_sum": "0", - "restore_count": "0", "checkpoint_sum": "0", + "restore_count": "0", "seconds": "0", "checkpoint_count": "0"}, end={"count": "150", "size": str(100 * MB), "bytes": str(100 * MB), "seconds": "50", "restore_sum": "4", "restore_count": "8", - "checkpoint_sum": "100", "checkpoint_count": "50"}, + "checkpoint_count": "25"}, at=(135, 140)) with tempfile.TemporaryDirectory() as td: @@ -268,12 +276,14 @@ def test_packing_and_checkpoint_math(self, mock_instant, mock_range): sleep.assert_called_once_with(65) # until end 105 + 70, from 110 self.assertEqual(summary["metadata"]["atelet_lag_s"], 70) - # Packing and PSI are not shifted. - self.assertEqual({c.args[2:4] for c in mock_range.call_args_list}, - {(100, 105)}) + # Packing and PSI are not shifted; active actors are, like snapshots. + windows = {("sampled_actors" in c.args[1], c.args[2:4]) + for c in mock_range.call_args_list} + self.assertEqual(windows, {(False, (100, 105)), (True, (135, 140))}) self.assertEqual( set(summary), - {"metadata", "cluster_packing", "node_psi", "pod_psi", "snapshots"}, + {"metadata", "cluster_packing", "node_psi", "pod_psi", "snapshots", + "active_actors"}, ) packing = summary["cluster_packing"] self.assertEqual(packing["summary"]["p50"], 0.8) # 3 + 1 busy / 5 workers @@ -293,7 +303,7 @@ def test_packing_and_checkpoint_math(self, mock_instant, mock_range): self.assertEqual(row["metric"], "server_summary") m = row["measurements"] # Every source answered, so a key wired to a wrong name would read None. - self.assertEqual(len(m), 45) + self.assertEqual(len(m), 49) self.assertEqual([k for k, v in m.items() if v is None], []) # Every value a string, so one row's types match every other row's. self.assertTrue(all(isinstance(v, str) for v in m.values())) @@ -304,6 +314,7 @@ def test_packing_and_checkpoint_math(self, mock_instant, mock_range): self.assertEqual(m["snapshot_size_p95_mb"], "11.5") self.assertEqual(m["restore_mean_s"], "0.5") self.assertEqual(m["checkpoint_mean_s"], "2.0") + self.assertEqual(m["active_actors_p99"], "3.0") @mock.patch("server_telemetry.query_prometheus_range") @mock.patch("server_telemetry.query_prometheus_instant") @@ -500,6 +511,134 @@ def test_snapshot_selector_covers_both_runtimes(self, mock_instant): self.assertFalse(any('file_name="pages.img"' in q for q in sizes)) self.assertIn("sum(atelet_snapshot_size_bytes_sum)", queries) + @mock.patch("server_telemetry.query_prometheus_instant") + def test_restore_and_checkpoint_read_atelet_histograms(self, mock_instant): + # 42s, above 10s: the buckets reach 60s. + mock_instant.side_effect = snapshot_prom( + "42.0", + start={"restore_sum": "0", "restore_count": "0", + "seconds": "0", "checkpoint_count": "0"}, + end={"restore_sum": "300", "restore_count": "10", + "seconds": "40", "checkpoint_count": "20"}) + snaps = snapshots() + self.assertEqual(snaps["restore_p99_s"], 42.0) + self.assertEqual(snaps["checkpoint_p50_s"], 42.0) + self.assertEqual(snaps["restore_mean_s"], 30.0) # 300s / 10 + self.assertEqual(snaps["checkpoint_mean_s"], 2.0) # 40s / 20 + + queries = [c.args[1] for c in mock_instant.call_args_list] + self.assertFalse(any("rpc_server_call_duration" in q for q in queries)) + for op in ("restore", "checkpoint"): + self.assertTrue(any( + q.startswith("histogram_quantile") + and f"ate_actor_{op}_duration_seconds_bucket{SNAPSHOT_OP}" in q + for q in queries)) + # Every kind, like the sizes, which have no kind label. + self.assertFalse(any("ate_snapshot_kind" in q for q in queries)) + + @mock.patch("server_telemetry.query_prometheus_range") + def test_active_actors(self, mock_range): + def active(lag_s=0, steady=(100, 130)): + return server_telemetry._count_active_actors( + "http://localhost:9090", 100, 130, *steady, lag_s=lag_s) + + # At 120 no atelet reported, so the step is skipped; at 130 only a + # did, so b's missing series counts as 0. + mock_range.return_value = [ + {"metric": {"instance": "a"}, + "values": [[100, "2"], [110, "3"], [130, "4"]]}, + {"metric": {"instance": "b"}, "values": [[100, "1"], [110, "1"]]}, + ] + out = active() + self.assertEqual([p["active_actors"] for p in out["timeseries"]], + [3.0, 4.0, 4.0]) + self.assertEqual([p["timestamp"] for p in out["timeseries"]], + [100, 110, 130]) + self.assertEqual((out["summary"]["min"], out["summary"]["max"]), + (3.0, 4.0)) + self.assertEqual(out["atelets"], 2) + query = mock_range.call_args.args[1] + self.assertIn("sum by (instance, exported_instance)", query) + self.assertIn("ate_actor_stats_sampled_actors", query) + + # Atelets hosting an actor at each point; b drops out at 130. + self.assertEqual([p["active_atelets"] for p in out["timeseries"]], + [2, 2, 1]) + self.assertEqual(set(out["timeseries"][0]), + {"timestamp", "active_actors", "active_atelets"}) + + # The steady window bounds the summary but not the timeseries. + out = active(steady=(100, 110)) + self.assertEqual(out["summary"]["min"], 3.0) + self.assertEqual(len(out["timeseries"]), 3) + + # Read half the lag late, like the snapshot block, but stamped back + # onto the run's clock; the steady window still picks the late reads. + mock_range.return_value = [{"metric": {"instance": "a"}, + "values": [[135, "3"], [145, "5"]]}] + out = active(lag_s=70, steady=(100, 100)) + self.assertEqual(mock_range.call_args.args[2:4], (135, 165)) + self.assertEqual([p["timestamp"] for p in out["timeseries"]], [100, 110]) + self.assertEqual(out["summary"]["max"], 3.0) + + # One atelet re-exported by 2 collector replicas merges by max, not sum. + mock_range.return_value = [ + {"metric": {"instance": f"meter-{i}", "exported_instance": "a"}, + "values": [[100, "3"], [110, v]]} for i, v in ((0, "2"), (1, "3")) + ] + out = active() + self.assertEqual([p["active_actors"] for p in out["timeseries"]], + [3.0, 3.0]) + self.assertEqual(out["atelets"], 1) + + # A reported 0 is not hosting. + mock_range.return_value = [ + {"metric": {"instance": "a"}, "values": [[100, "0"], [110, "2"]]}, + {"metric": {"instance": "b"}, "values": [[100, "3"], [110, "0"]]}, + ] + out = active() + self.assertEqual([p["active_atelets"] for p in out["timeseries"]][:2], + [1, 1]) + + # Never seen (not scraped) is unmeasured, not zero. + mock_range.return_value = [] + out = active() + self.assertEqual(out["summary"], NO_PERCENTILES) + self.assertIsNone(out["atelets"]) + self.assertEqual(out["timeseries"], []) + + def test_series_samples_skips_bad_points(self): + series = {"values": [[100, "1"], [110, "NaN"], [120, "+Inf"], + [130, "-Inf"], [140], [150, None], ["x", "2"], + [160, "2.5"]]} + self.assertEqual(server_telemetry._series_samples(series), + {100: 1.0, 160: 2.5}) + self.assertEqual(server_telemetry._series_samples({}), {}) + + def test_steady_window_ignores_per_request_rows(self): + # --csv-full-history adds a row per operation; only Aggregated counts. + path = history_csv([("100", "5"), ("100", "500", "ResumeActor"), + ("110", "10"), ("110", "500", "ResumeActor")]) + try: + self.assertEqual( + server_telemetry.get_steady_state_window(path, 100, 150), + (110, 110)) + finally: + path.unlink() + + def test_locust_writes_full_history(self): + class Launched(Exception): + pass + + with mock.patch.object(runner, "needs_boomer", return_value=False), \ + mock.patch.object(runner.subprocess, "Popen", + side_effect=Launched) as popen, \ + contextlib.redirect_stdout(io.StringIO()), \ + self.assertRaises(Launched): + runner.run_test(parse(), Path("/tmp/unit"), io.StringIO(), + io.StringIO()) + self.assertIn("--csv-full-history", popen.call_args.args[0]) + def test_prometheus_url_flag(self): self.assertEqual(parse().prometheus_url, runner.DEFAULT_PROMETHEUS_URL) self.assertEqual(parse("--prometheus-url", "http://x:9090").prometheus_url, diff --git a/benchmarking/nighthawk-egress/nighthawk-server.yaml b/benchmarking/nighthawk-egress/nighthawk-server.yaml index b30073b9b3..1f1cc7f3ec 100644 --- a/benchmarking/nighthawk-egress/nighthawk-server.yaml +++ b/benchmarking/nighthawk-egress/nighthawk-server.yaml @@ -20,7 +20,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: nighthawk-server - namespace: ate-system + namespace: nighthawk data: nighthawk.yaml: | # SDS requires a node id and cluster. @@ -218,7 +218,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: nighthawk-server - namespace: ate-system + namespace: nighthawk labels: app: nighthawk-server spec: @@ -303,7 +303,7 @@ apiVersion: v1 kind: Service metadata: name: nighthawk-server - namespace: ate-system + namespace: nighthawk spec: type: ClusterIP ipFamilyPolicy: PreferDualStack diff --git a/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml b/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml index 4e0f223d6d..dd224690a5 100644 --- a/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml +++ b/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml @@ -23,7 +23,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: traffic-driver-invoker - namespace: ate-system + namespace: nighthawk data: envoy.yaml: | static_resources: @@ -90,7 +90,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: traffic-driver-invoker - namespace: ate-system + namespace: nighthawk labels: app: traffic-driver-invoker spec: @@ -145,7 +145,7 @@ apiVersion: v1 kind: Service metadata: name: traffic-driver-invoker - namespace: ate-system + namespace: nighthawk spec: type: ClusterIP ipFamilyPolicy: PreferDualStack diff --git a/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl b/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl index 6434b5c6ee..0c607fad69 100644 --- a/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl +++ b/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl @@ -59,7 +59,7 @@ volumes: - system-roots.ate.dev path: egress-trust-bundle/trust-bundle.pem sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY diff --git a/benchmarking/workloads/deploy.sh b/benchmarking/workloads/deploy.sh index e524156d7a..113094967f 100755 --- a/benchmarking/workloads/deploy.sh +++ b/benchmarking/workloads/deploy.sh @@ -139,12 +139,11 @@ substitute() { # than defaulted) so a stale config from a dirty teardown fails loudly # instead of silently binding these workloads. gvisor-default is applied by # hack/install-ate.sh; microvm is applied by hack/install-microvm-deps.sh. - # The protojson templates take the sandbox class as its proto enum spelling. local manifest="$1" - local sandbox_config_name sandbox_class_enum worker_resources="" + local sandbox_config_name worker_resources="" case "${SANDBOX_CLASS}" in - gvisor) sandbox_config_name="gvisor-default" sandbox_class_enum="SANDBOX_CLASS_GVISOR" ;; - microvm) sandbox_config_name="microvm" sandbox_class_enum="SANDBOX_CLASS_MICROVM" ;; + gvisor) sandbox_config_name="gvisor-default" ;; + microvm) sandbox_config_name="microvm" ;; esac # One flow-style line, so an unset WORKER_MEMORY leaves only a blank line. if [[ -n "${WORKER_MEMORY}" ]]; then @@ -153,7 +152,6 @@ substitute() { sed -e "s|\${BUCKET_NAME}|${BUCKET_NAME}|g" \ -e "s|\${WORKER_COUNT}|${WORKER_COUNT}|g" \ -e "s|\${SANDBOX_CLASS}|${SANDBOX_CLASS}|g" \ - -e "s|\${SANDBOX_CLASS_ENUM}|${sandbox_class_enum}|g" \ -e "s|\${SANDBOX_CONFIG_NAME}|${sandbox_config_name}|g" \ -e "s|\${OTLP_ENDPOINT}|${OTLP_ENDPOINT}|g" \ -e "s|\${ACTOR_MEMORY}|${ACTOR_MEMORY}|g" \ diff --git a/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl index 5d984e4daf..8ba59e96b2 100644 --- a/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl @@ -47,5 +47,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-durdir-data/ sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl index 10c7272666..febfad850d 100644 --- a/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl @@ -44,5 +44,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-durdir-full/ sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl index 67812376f6..92e776603b 100644 --- a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl @@ -58,5 +58,5 @@ snapshotConfig: onCommit: SNAPSHOT_CONTENT_SCOPE_FULL storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-storage/ sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/benchmarking/workloads/manifests/glutton-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-template.yaml.tmpl index bfa9469172..8babdd0516 100644 --- a/benchmarking/workloads/manifests/glutton-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-template.yaml.tmpl @@ -64,5 +64,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton/ sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl b/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl index 7ffa43a637..013d5667af 100644 --- a/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl @@ -34,5 +34,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/kernelmem/ sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/benchmarking/workloads/manifests/sleep-template.yaml.tmpl b/benchmarking/workloads/manifests/sleep-template.yaml.tmpl index 9549ca8672..99e5c7f352 100644 --- a/benchmarking/workloads/manifests/sleep-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sleep-template.yaml.tmpl @@ -38,5 +38,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/sleep/ sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl b/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl index 8d8fd737c9..ab3c93561b 100644 --- a/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl @@ -47,7 +47,7 @@ containers: - name: SWEPERF_DISABLE_INTERNAL_SLEEP value: "1" sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY diff --git a/benchmarking/workloads/manifests/usermem-template.yaml.tmpl b/benchmarking/workloads/manifests/usermem-template.yaml.tmpl index 21eab89cb3..d8d5ef5403 100644 --- a/benchmarking/workloads/manifests/usermem-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/usermem-template.yaml.tmpl @@ -33,5 +33,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/usermem/ sandboxConfig: - sandboxClass: ${SANDBOX_CLASS_ENUM} + sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/charts/substrate-crds/templates/ate.dev_csidriverconfigs.yaml b/charts/substrate-crds/templates/ate.dev_csidriverconfigs.yaml index ebc1473eae..43d5c971b4 100644 --- a/charts/substrate-crds/templates/ate.dev_csidriverconfigs.yaml +++ b/charts/substrate-crds/templates/ate.dev_csidriverconfigs.yaml @@ -88,7 +88,9 @@ spec: description: Enabled controls whether TLS is used. type: boolean serverName: - description: ServerName override for TLS verification. + description: |- + ServerName specifies the expected server name (SAN) for TLS certificate verification. + Required when TLS is enabled. type: string usePodIdentity: description: UsePodIdentity indicates whether to reuse Substrate's @@ -101,6 +103,9 @@ spec: - message: tls.usePodIdentity must be true when tls.enabled is true; manual certificates are not yet supported rule: '!self.enabled || (has(self.usePodIdentity) && self.usePodIdentity)' + - message: tls.serverName is required when tls.enabled is true + rule: '!self.enabled || (has(self.serverName) && size(self.serverName) + > 0)' required: - controllerEndpoint - driverName diff --git a/charts/substrate/templates/ate-controller.yaml b/charts/substrate/templates/ate-controller.yaml index 3e2ce5596a..1db603bb1e 100644 --- a/charts/substrate/templates/ate-controller.yaml +++ b/charts/substrate/templates/ate-controller.yaml @@ -84,6 +84,7 @@ spec: - '--router-service-account={{ include "substrate.fullname" (list "atenet-router" .) }}' - "--ateapi-ca-file=/run/servicedns-ca/trust-bundle.pem" - "--ateapi-client-cert=/run/podidentity.podcert.ate.dev/credential-bundle.pem" + - "--dns-egress-policy=enforce-in-atunnel" env: - name: POD_NAMESPACE valueFrom: diff --git a/cmd/ate-setup/internal/steps/substrate_test.go b/cmd/ate-setup/internal/steps/substrate_test.go index 6b2419461d..c61ba66fe8 100644 --- a/cmd/ate-setup/internal/steps/substrate_test.go +++ b/cmd/ate-setup/internal/steps/substrate_test.go @@ -45,7 +45,7 @@ func TestActorTemplateFromManifestDemos(t *testing.T) { benchmarkValues := map[string]string{ "${ACTOR_MEMORY}": "256Mi", "${OTLP_ENDPOINT}": "http://otel-collector.otel-system:4317", - "${SANDBOX_CLASS_ENUM}": "SANDBOX_CLASS_GVISOR", + "${SANDBOX_CLASS}": "gvisor", "${SANDBOX_CONFIG_NAME}": "gvisor-default", } diff --git a/cmd/ateapi/internal/apivalidation/actor.go b/cmd/ateapi/internal/apivalidation/actor.go index fc1b6895a7..fa6a623786 100644 --- a/cmd/ateapi/internal/apivalidation/actor.go +++ b/cmd/ateapi/internal/apivalidation/actor.go @@ -99,6 +99,11 @@ func ValidateMintAteomActorCertificateRequest(ctx context.Context, req *ateapipb return Validate_MintAteomActorCertificateRequest(ctx, op, nil, req, nil) } +func ValidateMintAteomActorJwtRequest(ctx context.Context, req *ateapipb.MintAteomActorJwtRequest) field.ErrorList { + op := operation.Operation{Type: operation.Create} + return Validate_MintAteomActorJwtRequest(ctx, op, nil, req, nil) +} + func ValidateActorUpdate(ctx context.Context, fldPath *field.Path, newVal, oldVal *ateapipb.Actor, requireStatus bool) field.ErrorList { // Call the generated validation. op := operation.Operation{Type: operation.Update} diff --git a/cmd/ateapi/internal/apivalidation/actor_template.go b/cmd/ateapi/internal/apivalidation/actor_template.go index 483861fe91..15599571b8 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template.go +++ b/cmd/ateapi/internal/apivalidation/actor_template.go @@ -19,6 +19,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/internal/resources" + atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/api/resource" @@ -64,11 +65,11 @@ func ValidateActorTemplateUpdate(ctx context.Context, fldPath *field.Path, newVa func ValidateCustom_CreateActorTemplateRequest_ActorTemplate(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateapipb.ActorTemplate) field.ErrorList { var errs field.ErrorList if value.GetSnapshotConfig().GetPreferredFidelity() == ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS && - value.GetSandboxConfig().GetSandboxClass() != ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM { + value.GetSandboxConfig().GetSandboxClass() != string(atev1alpha1.SandboxClassMicroVM) { errs = append(errs, field.Invalid( fldPath.Child("snapshot_config", "preferred_fidelity"), value.GetSnapshotConfig().GetPreferredFidelity().String(), - "ROOTFS fidelity requires sandbox_config.sandbox_class SANDBOX_CLASS_MICROVM")) + "ROOTFS fidelity requires sandbox_config.sandbox_class microvm")) } declared := make(map[string]bool, len(value.GetVolumes())) for _, vol := range value.GetVolumes() { @@ -166,6 +167,10 @@ func ValidateCustom_ExternalVolumeTemplate_Capacity(_ context.Context, _ operati return nil } +func ValidateCustom_SandboxConfig_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateSandboxClass(fldPath, *value) +} + // ValidateCustom_SnapshotConfig_StorageLocation ensures an // ActorTemplate's snapshotConfig.location is a well-formed // URI with a bucket, so a bad location fails fast. diff --git a/cmd/ateapi/internal/apivalidation/actor_template_test.go b/cmd/ateapi/internal/apivalidation/actor_template_test.go index 35d10bcf6f..4cf66a6db7 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template_test.go +++ b/cmd/ateapi/internal/apivalidation/actor_template_test.go @@ -80,7 +80,7 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) { "rootfs fidelity on microvm accepted", &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { tmpl.SnapshotConfig.PreferredFidelity = ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS - tmpl.SandboxConfig = &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM, ConfigName: "microvm-default"} + tmpl.SandboxConfig = &ateapipb.SandboxConfig{SandboxClass: "microvm", ConfigName: "microvm-default"} })}, nil, }, { @@ -167,9 +167,9 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) { })}, field.ErrorList{field.Required(field.NewPath("actor_template", "sandbox_config"), "")}, }, { - "unspecified sandbox_config.sandbox_class", + "missing sandbox_config.sandbox_class", &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { - tmpl.SandboxConfig.SandboxClass = ateapipb.SandboxClass_SANDBOX_CLASS_UNSPECIFIED + tmpl.SandboxConfig.SandboxClass = "" })}, field.ErrorList{field.Required(field.NewPath("actor_template", "sandbox_config", "sandbox_class"), "")}, }, { @@ -342,19 +342,27 @@ func TestValidateActorTemplate(t *testing.T) { mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig = nil }, want: field.ErrorList{field.Required(field.NewPath("sandbox_config"), "")}, }, { - name: "unspecified sandbox_class", - mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.SandboxConfig.SandboxClass = ateapipb.SandboxClass_SANDBOX_CLASS_UNSPECIFIED - }, - want: field.ErrorList{field.Required(field.NewPath("sandbox_config", "sandbox_class"), "")}, + name: "missing sandbox_class", + mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.SandboxClass = "" }, + want: field.ErrorList{field.Required(field.NewPath("sandbox_config", "sandbox_class"), "")}, + }, { + name: "microvm sandbox_class", + mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.SandboxClass = "microvm" }, }, { - name: "sandbox_class outside the enum", - mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.SandboxClass = ateapipb.SandboxClass(99) }, - want: field.ErrorList{field.Invalid(field.NewPath("sandbox_config", "sandbox_class"), nil, "").WithOrigin("maximum")}, + name: "unsupported sandbox_class", + mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.SandboxClass = "kata" }, + want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_config", "sandbox_class"), nil, nil)}, }, { - name: "negative sandbox_class", - mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.SandboxClass = ateapipb.SandboxClass(-1) }, - want: field.ErrorList{field.Invalid(field.NewPath("sandbox_config", "sandbox_class"), nil, "").WithOrigin("minimum")}, + name: "sandbox_class is case-sensitive", + mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.SandboxClass = "GVISOR" }, + want: field.ErrorList{field.NotSupported[string](field.NewPath("sandbox_config", "sandbox_class"), nil, nil)}, + }, { + name: "sandbox_class too long", + mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.SandboxClass = strings.Repeat("g", 64) }, + want: field.ErrorList{ + field.TooLong(field.NewPath("sandbox_config", "sandbox_class"), nil, 63).WithOrigin("maxLength"), + field.NotSupported[string](field.NewPath("sandbox_config", "sandbox_class"), nil, nil), + }, }, { name: "missing config_name", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.SandboxConfig.ConfigName = "" }, @@ -895,7 +903,7 @@ func validActorTemplate(mutations ...func(*ateapipb.ActorTemplate)) *ateapipb.Ac StorageLocation: "gs://my-bucket/snapshots", PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, } for _, m := range mutations { m(template) diff --git a/cmd/ateapi/internal/apivalidation/actor_test.go b/cmd/ateapi/internal/apivalidation/actor_test.go index b031dabb0e..ac3f0c6163 100644 --- a/cmd/ateapi/internal/apivalidation/actor_test.go +++ b/cmd/ateapi/internal/apivalidation/actor_test.go @@ -639,14 +639,13 @@ func TestValidateActorUpdate(t *testing.T) { }, { "provisioning transition on an existing volume is valid", validInput(withStatus(func(s *ateapipb.ActorStatus) { - s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock", Status: ateapipb.ExternalVolume_STATUS_PENDING}} + s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock"}} })), validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.ExternalVolumes = []*ateapipb.ExternalVolume{{ Name: "vol-a", VolumeType: "substrate.io/mock", StorageVolumeId: "csi-426d29b7", - Status: ateapipb.ExternalVolume_STATUS_CREATED, VolumeContext: map[string]string{"attachment": "iqn.2026-08.io.ate:vol-a"}, }} })), diff --git a/cmd/ateapi/internal/apivalidation/validation_test.go b/cmd/ateapi/internal/apivalidation/validation_test.go index 8f6b950c94..1803f9e1f2 100644 --- a/cmd/ateapi/internal/apivalidation/validation_test.go +++ b/cmd/ateapi/internal/apivalidation/validation_test.go @@ -876,7 +876,6 @@ func validExternalVolume(mutate ...func(*ateapipb.ExternalVolume)) *ateapipb.Ext Name: "my-vol", StorageVolumeId: "valid-storage-id", VolumeType: "mock", - Status: ateapipb.ExternalVolume_STATUS_CREATED, } for _, m := range mutate { m(v) @@ -934,14 +933,6 @@ func TestValidateExternalVolume(t *testing.T) { }, { name: "valid volume with substrate.io prefixed volume type", obj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "substrate.io/mock" }), - }, { - name: "negative status", - obj: valid(func(v *ateapipb.ExternalVolume) { v.Status = ateapipb.ExternalVolume_Status(-1) }), - want: field.ErrorList{field.Invalid(field.NewPath("status"), nil, "").WithOrigin("minimum")}, - }, { - name: "status outside the enum", - obj: valid(func(v *ateapipb.ExternalVolume) { v.Status = ateapipb.ExternalVolume_Status(4) }), - want: field.ErrorList{field.Invalid(field.NewPath("status"), nil, "").WithOrigin("maximum")}, }, { name: "storage volume id at the bound", obj: valid(func(v *ateapipb.ExternalVolume) { v.StorageVolumeId = strings.Repeat("x", 256) }), @@ -1011,15 +1002,9 @@ func TestValidateExternalVolume_Update(t *testing.T) { newObj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeType = "pd.csi.storage.gke.io" }), want: field.ErrorList{field.Invalid(field.NewPath("volume_type"), nil, "").WithOrigin("update")}, }, { - name: "status and volume_context changed is valid", - oldObj: valid(func(v *ateapipb.ExternalVolume) { - v.Status = ateapipb.ExternalVolume_STATUS_PENDING - v.VolumeContext = nil - }), - newObj: valid(func(v *ateapipb.ExternalVolume) { - v.Status = ateapipb.ExternalVolume_STATUS_CREATED - v.VolumeContext = map[string]string{"foo": "bar"} - }), + name: "volume_context changed is valid", + oldObj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeContext = nil }), + newObj: valid(func(v *ateapipb.ExternalVolume) { v.VolumeContext = map[string]string{"foo": "bar"} }), }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { diff --git a/cmd/ateapi/internal/apivalidation/worker.go b/cmd/ateapi/internal/apivalidation/worker.go index 13a1ae00b0..cfb0ea9562 100644 --- a/cmd/ateapi/internal/apivalidation/worker.go +++ b/cmd/ateapi/internal/apivalidation/worker.go @@ -18,6 +18,7 @@ import ( "context" "fmt" + "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/api/validate" @@ -81,6 +82,18 @@ func ValidateRequestActorSuspendRequest(ctx context.Context, req *ateapipb.Reque return Validate_RequestActorSuspendRequest(ctx, op, nil, req, nil) } +func ValidateMarkWorkerAvailableRequest(ctx context.Context, req *ateapipb.MarkWorkerAvailableRequest) field.ErrorList { + // Call the generated validation. + op := operation.Operation{Type: operation.Create} + return Validate_MarkWorkerAvailableRequest(ctx, op, nil, req, nil) +} + +func ValidateMarkWorkerUnavailableRequest(ctx context.Context, req *ateapipb.MarkWorkerUnavailableRequest) field.ErrorList { + // Call the generated validation. + op := operation.Operation{Type: operation.Create} + return Validate_MarkWorkerUnavailableRequest(ctx, op, nil, req, nil) +} + func ValidateReportActorSnapshotStateRequest(ctx context.Context, req *ateapipb.ReportActorSnapshotStateRequest) field.ErrorList { op := operation.Operation{Type: operation.Create} return Validate_ReportActorSnapshotStateRequest(ctx, op, nil, req, nil) @@ -102,6 +115,14 @@ func ValidateWorkerUpdate(ctx context.Context, fldPath *field.Path, newVal, oldV return errs } +func ValidateCustom_Worker_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateSandboxClass(fldPath, *value) +} + +func ValidateCustom_SandboxRuntime_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateSandboxClass(fldPath, *value) +} + // This is needed because DV doesn't have a standard format for IP addresses yet. func ValidateCustom_Worker_Ips(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList { return validateDualStackIPs(fldPath, value) diff --git a/cmd/ateapi/internal/apivalidation/worker_test.go b/cmd/ateapi/internal/apivalidation/worker_test.go index 02b7b58a09..30c246ebdc 100644 --- a/cmd/ateapi/internal/apivalidation/worker_test.go +++ b/cmd/ateapi/internal/apivalidation/worker_test.go @@ -145,7 +145,14 @@ func TestValidateCreateWorkerRequest(t *testing.T) { }, { name: "sandbox_class too long", req: validReq(validWorker(apiWorkerName, func(w *ateapipb.Worker) { w.SandboxClass = strings.Repeat("x", 64) })), - want: field.ErrorList{field.TooLong(field.NewPath("worker", "sandbox_class"), nil, 63).WithOrigin("maxLength")}, + want: field.ErrorList{ + field.TooLong(field.NewPath("worker", "sandbox_class"), nil, 63).WithOrigin("maxLength"), + field.NotSupported[string](field.NewPath("worker", "sandbox_class"), nil, nil), + }, + }, { + name: "unsupported sandbox_class", + req: validReq(validWorker(apiWorkerName, func(w *ateapipb.Worker) { w.SandboxClass = "kata" })), + want: field.ErrorList{field.NotSupported[string](field.NewPath("worker", "sandbox_class"), nil, nil)}, }, { name: "valid labels", req: validReq(validWorker(apiWorkerName, func(w *ateapipb.Worker) { @@ -408,7 +415,14 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { }, { name: "default_runtime.sandbox_class too long", req: valid(func(r *ateapipb.RegisterWorkerRequest) { r.DefaultRuntime.SandboxClass = strings.Repeat("c", 64) }), - want: field.ErrorList{field.TooLong(defaultRuntimePath.Child("sandbox_class"), "", 63).WithOrigin("maxLength")}, + want: field.ErrorList{ + field.TooLong(defaultRuntimePath.Child("sandbox_class"), "", 63).WithOrigin("maxLength"), + field.NotSupported[string](defaultRuntimePath.Child("sandbox_class"), nil, nil), + }, + }, { + name: "unsupported default_runtime.sandbox_class", + req: valid(func(r *ateapipb.RegisterWorkerRequest) { r.DefaultRuntime.SandboxClass = "kata" }), + want: field.ErrorList{field.NotSupported[string](defaultRuntimePath.Child("sandbox_class"), nil, nil)}, }, { name: "missing default_runtime.version", req: valid(func(r *ateapipb.RegisterWorkerRequest) { r.DefaultRuntime.Version = nil }), @@ -481,6 +495,13 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { r.RestorableRuntimes = []*ateapipb.SandboxRuntime{{SandboxClass: "gvisor"}} }), want: field.ErrorList{field.Required(restorablePath.Index(0).Child("version"), "")}, + }, { + name: "unsupported restorable_runtimes sandbox_class", + req: valid(func(r *ateapipb.RegisterWorkerRequest) { + r.RestorableRuntimes = []*ateapipb.SandboxRuntime{validSandboxRuntime()} + r.RestorableRuntimes[0].SandboxClass = "GVISOR" + }), + want: field.ErrorList{field.NotSupported[string](restorablePath.Index(0).Child("sandbox_class"), nil, nil)}, }, { name: "restorable_runtimes at the bound", req: valid(func(r *ateapipb.RegisterWorkerRequest) { r.RestorableRuntimes = sandboxRuntimes(32) }), @@ -519,6 +540,13 @@ func TestValidateWorkerUpdate_SandboxRuntimes(t *testing.T) { s.DefaultRuntime.SandboxClass = "" }, want: field.ErrorList{field.Required(statusPath.Child("default_runtime", "sandbox_class"), "")}, + }, { + name: "unsupported default_runtime.sandbox_class", + mutate: func(s *ateapipb.WorkerStatus) { + s.DefaultRuntime = validSandboxRuntime() + s.DefaultRuntime.SandboxClass = "kata" + }, + want: field.ErrorList{field.NotSupported[string](statusPath.Child("default_runtime", "sandbox_class"), nil, nil)}, }, { name: "missing default_runtime.version", mutate: func(s *ateapipb.WorkerStatus) { diff --git a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go index cee99621a5..c05764ba1c 100644 --- a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go +++ b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go @@ -3871,40 +3871,6 @@ func Validate_ExternalVolume( errs = append(errs, fn(fldPath.Child("volume_type"), &obj.VolumeType, oldVal, oldObj != nil)...) } - { // field ateapipb.ExternalVolume.Status - fn := func( - fldPath *field.Path, - obj, oldObj *ateapipb.ExternalVolume_Status, - oldValueCorrelated bool) (errs field.ErrorList) { - // don't revalidate unchanged data - if oldValueCorrelated && op.Type == operation.Update { - if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { - return nil - } - } - // call field-attached validations - earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { - earlyReturn = true - } - if earlyReturn { - return // do not proceed - } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3); len(e) != 0 { - errs = append(errs, e...) - } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { - errs = append(errs, e...) - } - return - } - oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.ExternalVolume) *ateapipb.ExternalVolume_Status { - return &oldObj.Status - }) - errs = append(errs, fn(fldPath.Child("status"), &obj.Status, oldVal, oldObj != nil)...) - } - { // field ateapipb.ExternalVolume.VolumeContext fn := func( fldPath *field.Path, @@ -5619,6 +5585,124 @@ func Validate_ListWorkersRequest( return errs } +// Validate_MarkWorkerAvailableRequest validates an instance of MarkWorkerAvailableRequest according +// to declarative validation rules in the API schema. +func Validate_MarkWorkerAvailableRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.MarkWorkerAvailableRequest) (errs field.ErrorList) { + + { // field ateapipb.MarkWorkerAvailableRequest.Worker + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.ObjectRef, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + func() { // cohort = "atespace" + earlyReturn := false + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.ForbiddenValue).MarkBeta().MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkBeta().MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkBeta().MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + }() + // call the type's validation function + errs = append(errs, Validate_ObjectRef(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.MarkWorkerAvailableRequest) *ateapipb.ObjectRef { + return oldObj.Worker + }) + errs = append(errs, fn(fldPath.Child("worker"), obj.Worker, oldVal, oldObj != nil)...) + } + + return errs +} + +// Validate_MarkWorkerUnavailableRequest validates an instance of MarkWorkerUnavailableRequest according +// to declarative validation rules in the API schema. +func Validate_MarkWorkerUnavailableRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.MarkWorkerUnavailableRequest) (errs field.ErrorList) { + + { // field ateapipb.MarkWorkerUnavailableRequest.Worker + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.ObjectRef, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + func() { // cohort = "atespace" + earlyReturn := false + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.ForbiddenValue).MarkBeta().MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkBeta().MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkBeta().MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + }() + // call the type's validation function + errs = append(errs, Validate_ObjectRef(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.MarkWorkerUnavailableRequest) *ateapipb.ObjectRef { + return oldObj.Worker + }) + errs = append(errs, fn(fldPath.Child("worker"), obj.Worker, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_MintActorCertificateRequest validates an instance of MintActorCertificateRequest according // to declarative validation rules in the API schema. func Validate_MintActorCertificateRequest( @@ -5960,6 +6044,142 @@ func Validate_MintAteomActorCertificateRequest( return errs } +// Validate_MintAteomActorJwtRequest validates an instance of MintAteomActorJwtRequest according +// to declarative validation rules in the API schema. +func Validate_MintAteomActorJwtRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateapipb.MintAteomActorJwtRequest) (errs field.ErrorList) { + + { // field ateapipb.MintAteomActorJwtRequest.Actor + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.ObjectRef, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + func() { // cohort = "atespace" + earlyReturn := false + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.RequiredValue).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.Subfield(ctx, op, fldPath, obj, oldObj, "atespace", + func(o *ateapipb.ObjectRef) *string { return &o.Atespace }, validate.DirectEqual, validate.OptionalValue).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + }() + // call the type's validation function + errs = append(errs, Validate_ObjectRef(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.MintAteomActorJwtRequest) *ateapipb.ObjectRef { + return oldObj.Actor + }) + errs = append(errs, fn(fldPath.Child("actor"), obj.Actor, oldVal, oldObj != nil)...) + } + + { // field ateapipb.MintAteomActorJwtRequest.ActorUid + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.UUID(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.MintAteomActorJwtRequest) *string { + return &oldObj.ActorUid + }) + errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) + } + + { // field ateapipb.MintAteomActorJwtRequest.Audiences + fn := func( + fldPath *field.Path, + obj, oldObj []string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.MaxItems(ctx, op, fldPath, obj, oldObj, 16).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.RequiredSlice(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.EachValSliceVal(ctx, op, fldPath, obj, oldObj, validate.DirectEqual, nil, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MinLength(ctx, op, fldPath, obj, oldObj, 1) + }); len(e) != 0 { + errs = append(errs, e...) + } + if e := validate.EachValSliceVal(ctx, op, fldPath, obj, oldObj, validate.DirectEqual, nil, + func(ctx context.Context, op operation.Operation, fldPath *field.Path, obj, oldObj *string) field.ErrorList { + return validate.MaxLength(ctx, op, fldPath, obj, oldObj, 512) + }); len(e) != 0 { + errs = append(errs, e...) + } + // lists with set semantics require unique values + if e := validate.ValSliceUnique(ctx, op, fldPath, obj, oldObj, validate.DirectEqual); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.MintAteomActorJwtRequest) []string { + return oldObj.Audiences + }) + errs = append(errs, fn(fldPath.Child("audiences"), obj.Audiences, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_ObjectRef validates an instance of ObjectRef according // to declarative validation rules in the API schema. func Validate_ObjectRef( @@ -7044,7 +7264,7 @@ func Validate_SandboxConfig( { // field ateapipb.SandboxConfig.SandboxClass fn := func( fldPath *field.Path, - obj, oldObj *ateapipb.SandboxClass, + obj, oldObj *string, oldValueCorrelated bool) (errs field.ErrorList) { // don't revalidate unchanged data if oldValueCorrelated && op.Type == operation.Update { @@ -7061,16 +7281,17 @@ func Validate_SandboxConfig( if earlyReturn { return // do not proceed } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 2); len(e) != 0 { + // custom validation + if e := ValidateCustom_SandboxConfig_SandboxClass(ctx, op, fldPath, obj, oldObj); len(e) != 0 { errs = append(errs, e...) } - if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 { errs = append(errs, e...) } return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.SandboxConfig) *ateapipb.SandboxClass { + func(oldObj *ateapipb.SandboxConfig) *string { return &oldObj.SandboxClass }) errs = append(errs, fn(fldPath.Child("sandbox_class"), &obj.SandboxClass, oldVal, oldObj != nil)...) @@ -7137,6 +7358,10 @@ func Validate_SandboxRuntime( if earlyReturn { return // do not proceed } + // custom validation + if e := ValidateCustom_SandboxRuntime_SandboxClass(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 { errs = append(errs, e...) } @@ -7516,6 +7741,36 @@ func Validate_Snapshot( errs = append(errs, fn(fldPath.Child("durable_snapshot"), obj.DurableSnapshot, oldVal, oldObj != nil)...) } + { // field ateapipb.Snapshot.SandboxRuntime + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SandboxRuntime, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SandboxRuntime(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.Snapshot) *ateapipb.SandboxRuntime { + return oldObj.SandboxRuntime + }) + errs = append(errs, fn(fldPath.Child("sandbox_runtime"), obj.SandboxRuntime, oldVal, oldObj != nil)...) + } + return errs } @@ -9648,6 +9903,10 @@ func Validate_Worker( if earlyReturn { return // do not proceed } + // custom validation + if e := ValidateCustom_Worker_SandboxClass(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 { errs = append(errs, e...) } @@ -10055,6 +10314,36 @@ func Validate_WorkerAssignment( errs = append(errs, fn(fldPath.Child("worker_epoch"), &obj.WorkerEpoch, oldVal, oldObj != nil)...) } + { // field ateapipb.WorkerAssignment.SandboxRuntime + fn := func( + fldPath *field.Path, + obj, oldObj *ateapipb.SandboxRuntime, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if ateDeepEqual(obj, oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + // call the type's validation function + errs = append(errs, Validate_SandboxRuntime(ctx, op, fldPath, obj, oldObj)...) + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateapipb.WorkerAssignment) *ateapipb.SandboxRuntime { + return oldObj.SandboxRuntime + }) + errs = append(errs, fn(fldPath.Child("sandbox_runtime"), obj.SandboxRuntime, oldVal, oldObj != nil)...) + } + return errs } @@ -10154,7 +10443,7 @@ func Validate_WorkerStatus( if earlyReturn { return // do not proceed } - if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 2); len(e) != 0 { + if e := validate.Maximum(ctx, op, fldPath, obj, oldObj, 3); len(e) != 0 { errs = append(errs, e...) } if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { diff --git a/cmd/ateapi/internal/authz/authorizer.go b/cmd/ateapi/internal/authz/authorizer.go index 59fcaca465..57d8a78056 100644 --- a/cmd/ateapi/internal/authz/authorizer.go +++ b/cmd/ateapi/internal/authz/authorizer.go @@ -53,7 +53,8 @@ type Authorizer struct { // Check verifies that the principal in ctx has relation on object. // Structural hierarchy links (global:root as parent_global of every atespace -// and worker, and each atespace as parent_atespace of its actors and actor templates) and +// and worker, and each atespace as parent_atespace of its actors, actor +// templates and tags) and // the caller's bootstrap owner and system role grants, if any, are injected as OpenFGA // ContextualTuples at evaluation time rather than persisted in the tuple table. func (a *Authorizer) Check(ctx context.Context, relation, object string) error { @@ -130,19 +131,20 @@ func (a *Authorizer) checkRaw(ctx context.Context, user, relation, object string // Why contextual tuples are used instead of storing parent links in the database: // 1. Deterministic structure: Every `atespace:` and `worker:` // unconditionally has `global:root` as its `parent_global`, and every -// `actor:/` and `actor_template:/` has -// `atespace:` as its `parent_atespace`. Because these +// `actor:/`, `actor_template:/` and +// `tag:/` has `atespace:` as its +// `parent_atespace`. Because these // relationships are derived purely from the object type/ID, storing a row // per resource in the OpenFGA `tuple` table would be redundant. // 2. No write amplification on create: `CreateAtespace`, `CreateWorker`, -// `CreateActorTemplate`, and `CreateActor` can insert their rows without +// `CreateActorTemplate`, `CreateActor` and `CreateTag` can insert their rows without // opening an OpenFGA write transaction just to link the parent. func contextualTuples(object string) []*openfgav1.TupleKey { objectType, id, _ := strings.Cut(object, ":") switch objectType { case "atespace", "worker": return []*openfgav1.TupleKey{parentGlobalTuple(object)} - case "actor", "actor_template": + case "actor", "actor_template", "tag": atespace, _, ok := strings.Cut(id, "/") if !ok { return nil diff --git a/cmd/ateapi/internal/authz/authz_util.go b/cmd/ateapi/internal/authz/authz_util.go index 3882a11f5d..608d660469 100644 --- a/cmd/ateapi/internal/authz/authz_util.go +++ b/cmd/ateapi/internal/authz/authz_util.go @@ -74,6 +74,11 @@ const ( RelationCanGetEgressPolicy = "can_get_egress_policy" RelationCanUpdateEgressPolicy = "can_update_egress_policy" RelationCanDeleteEgressPolicy = "can_delete_egress_policy" + RelationCanCreateTag = "can_create_tag" + RelationCanListTags = "can_list_tags" + RelationCanUseTag = "can_use_tag" + RelationCanMintActorJWT = "can_mint_actor_jwt" + RelationCanMintActorCertificate = "can_mint_actor_certificate" // maxTuplesPerWrite is OpenFGA's default maximum number of tuples allowed in a single Write request. maxTuplesPerWrite = 100 @@ -133,6 +138,11 @@ func ActorTemplateObject(atespace, name string) string { return "actor_template:" + atespacedID(atespace, name) } +// TagObject formats a tag as an OpenFGA object string. +func TagObject(atespace, name string) string { + return "tag:" + atespacedID(atespace, name) +} + // atespacedID formats the object ID of an atespaced resource as // "/". contextualTuples parses the atespace back out of it. func atespacedID(atespace, name string) string { diff --git a/cmd/ateapi/internal/authz/interceptor.go b/cmd/ateapi/internal/authz/interceptor.go index 29ae0165d0..9fd77198c8 100644 --- a/cmd/ateapi/internal/authz/interceptor.go +++ b/cmd/ateapi/internal/authz/interceptor.go @@ -24,16 +24,23 @@ import ( // UnaryServerInterceptor returns a gRPC unary interceptor that enforces per-RPC // permissions registered in defaultRPCPermissions using authorizer. When -// enforce is false, only rules marked alwaysEnforce are checked. +// enforce is false, only rules marked alwaysEnforce are checked. When enforce +// is true, an RPC with no rule is denied. func UnaryServerInterceptor(authorizer *Authorizer, enforce bool) grpc.UnaryServerInterceptor { return func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) { if IsBypassed(ctx) { return handler(ctx, req) } rule, registered := defaultRPCPermissions[info.FullMethod] - if !registered || (!enforce && !rule.alwaysEnforce) { + if registered && rule.authorizedInHandler != "" { return handler(ctx, req) } + if !enforce && !rule.alwaysEnforce { + return handler(ctx, req) + } + if !registered { + return nil, apierror.PermissionDenied("permission denied: no authorization rule for %s", info.FullMethod) + } // Authenticate first, so an unauthenticated caller gets Unauthenticated // rather than a validation error for its request. diff --git a/cmd/ateapi/internal/authz/interceptor_test.go b/cmd/ateapi/internal/authz/interceptor_test.go index ae71b3ab23..e8c5da651a 100644 --- a/cmd/ateapi/internal/authz/interceptor_test.go +++ b/cmd/ateapi/internal/authz/interceptor_test.go @@ -261,6 +261,50 @@ func TestUnaryServerInterceptor_EnforcementDisabled(t *testing.T) { } } +// An RPC with no rule is denied when enforcement is on and passes through when +// it is off; an RPC its handler authorizes passes through either way. Neither +// reaches OpenFGA, so the authorizer is nil. +func TestUnaryServerInterceptor_UnregisteredAndHandlerAuthorizedRPCs(t *testing.T) { + const unregistered = "/ateapi.Control/NotARealMethod" + aliceCtx := principal.InjectContext(context.Background(), principal.PrincipalInfo{ + ID: "alice@example.com", + Kind: principal.KindJWT, + }) + + tests := []struct { + name string + enforce bool + ctx context.Context + fullMethod string + wantCode codes.Code + }{ + {"unregistered RPC is denied with enforcement on", true, aliceCtx, unregistered, codes.PermissionDenied}, + {"unregistered RPC is denied without a principal", true, context.Background(), unregistered, codes.PermissionDenied}, + {"unregistered RPC passes with enforcement off", false, aliceCtx, unregistered, codes.OK}, + {"RegisterWorker passes with enforcement on", true, context.Background(), ateapipb.WorkerService_RegisterWorker_FullMethodName, codes.OK}, + {"RequestActorSuspend passes with enforcement on", true, context.Background(), ateapipb.WorkerService_RequestActorSuspend_FullMethodName, codes.OK}, + {"MintAteomActorCertificate passes with enforcement on", true, context.Background(), ateapipb.WorkerService_MintAteomActorCertificate_FullMethodName, codes.OK}, + {"MintAteomActorJwt passes with enforcement on", true, context.Background(), ateapipb.WorkerService_MintAteomActorJwt_FullMethodName, codes.OK}, + {"RegisterWorker passes with enforcement off", false, context.Background(), ateapipb.WorkerService_RegisterWorker_FullMethodName, codes.OK}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + interceptor := UnaryServerInterceptor(nil, tc.enforce) + handlerCalled := false + _, err := interceptor(tc.ctx, struct{}{}, &grpc.UnaryServerInfo{FullMethod: tc.fullMethod}, func(context.Context, any) (any, error) { + handlerCalled = true + return "ok", nil + }) + if apierror.Code(err) != tc.wantCode { + t.Fatalf("apierror.Code(err) = %v, want %v (err: %v)", apierror.Code(err), tc.wantCode, err) + } + if wantHandler := tc.wantCode == codes.OK; handlerCalled != wantHandler { + t.Fatalf("handlerCalled = %v, want %v", handlerCalled, wantHandler) + } + }) + } +} + // TestAccessPolicyRPCsAlwaysEnforced guards against an AccessPolicy RPC being // added without a governance rule, which would leave it unchecked while // enforcement is disabled. @@ -435,7 +479,7 @@ func TestUnaryServerInterceptor_ActorAndTemplateChecks(t *testing.T) { } } -func TestUnaryServerInterceptor_LifecycleEgressAndWorkerChecks(t *testing.T) { +func TestUnaryServerInterceptor_LifecycleEgressWorkerAndTagChecks(t *testing.T) { ctx := context.Background() pool := startPostgres(t) fgaServer, err := NewOpenFGAServer(pool) @@ -455,6 +499,8 @@ func TestUnaryServerInterceptor_LifecycleEgressAndWorkerChecks(t *testing.T) { {"editor", "editor", AtespaceObject("team-a")}, {"viewer", "viewer", AtespaceObject("team-a")}, {"outsider", "editor", AtespaceObject("team-b")}, + {"outsider-with-viewer", "editor", AtespaceObject("team-b")}, + {"outsider-with-viewer", "viewer", AtespaceObject("team-a")}, } { writeTestTuple(t, ctx, pool, policyManager, g.user, g.role, g.object) } @@ -481,7 +527,41 @@ func TestUnaryServerInterceptor_LifecycleEgressAndWorkerChecks(t *testing.T) { }}} deleteWorker := rpcCall{ateapipb.Control_DeleteWorker_FullMethodName, &ateapipb.DeleteWorkerRequest{Worker: worker}} drainWorker := rpcCall{ateapipb.Control_DrainWorker_FullMethodName, &ateapipb.DrainWorkerRequest{Worker: worker}} + markAvailable := rpcCall{ateapipb.Control_MarkWorkerAvailable_FullMethodName, &ateapipb.MarkWorkerAvailableRequest{Worker: worker}} + markUnavailable := rpcCall{ateapipb.Control_MarkWorkerUnavailable_FullMethodName, &ateapipb.MarkWorkerUnavailableRequest{Worker: worker}} listAssignments := rpcCall{ateapipb.Control_ListWorkerActorAssignments_FullMethodName, &ateapipb.ListWorkerActorAssignmentsRequest{Worker: worker}} + tag := &ateapipb.ObjectRef{Atespace: "team-a", Name: "v1"} + createTag := rpcCall{ateapipb.Control_CreateTag_FullMethodName, &ateapipb.CreateTagRequest{Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "v1"}, + SourceActor: actor, + }}} + // A tag in team-b of an actor in team-a. + createCrossTag := rpcCall{ateapipb.Control_CreateTag_FullMethodName, &ateapipb.CreateTagRequest{Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-b", Name: "v1"}, + SourceActor: actor, + }}} + getTag := rpcCall{ateapipb.Control_GetTag_FullMethodName, &ateapipb.GetTagRequest{Tag: tag}} + listTags := rpcCall{ateapipb.Control_ListTags_FullMethodName, &ateapipb.ListTagsRequest{Atespace: "team-a"}} + listAllTags := rpcCall{ateapipb.Control_ListTags_FullMethodName, &ateapipb.ListTagsRequest{}} + updateTag := rpcCall{ateapipb.Control_UpdateTag_FullMethodName, &ateapipb.UpdateTagRequest{Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "v1"}, + }}} + deleteTag := rpcCall{ateapipb.Control_DeleteTag_FullMethodName, &ateapipb.DeleteTagRequest{Tag: tag}} + mintJWT := rpcCall{ateapipb.Control_MintActorJwt_FullMethodName, &ateapipb.MintActorJwtRequest{Actor: actor}} + mintCert := rpcCall{ateapipb.Control_MintActorCertificate_FullMethodName, &ateapipb.MintActorCertificateRequest{Actor: actor}} + // cloneTag creates an actor in atespace from team-a's tag v1, with a + // template in that same atespace so only the tag check can differ. + cloneTag := func(atespace string, sourceTag *ateapipb.ObjectRef) rpcCall { + return rpcCall{ateapipb.Control_CreateActor_FullMethodName, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: atespace, Name: "clone"}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: atespace, Name: "tmpl"}, + SourceTag: sourceTag, + }}} + } + cloneInTeamA := cloneTag("team-a", tag) + cloneInTeamB := cloneTag("team-b", tag) + cloneGolden := cloneTag("team-a", &ateapipb.ObjectRef{Atespace: "ate-golden", Name: "golden-v1"}) + createWithoutTag := cloneTag("team-a", nil) tests := []struct { name string @@ -534,6 +614,47 @@ func TestUnaryServerInterceptor_LifecycleEgressAndWorkerChecks(t *testing.T) { {"atespace editor cannot list workers", "editor", listWorkers, codes.PermissionDenied}, {"atespace editor cannot get worker", "editor", getWorker, codes.PermissionDenied}, {"atespace editor cannot drain worker", "editor", drainWorker, codes.PermissionDenied}, + {"global owner marks worker available", "global-owner", markAvailable, codes.OK}, + {"global owner marks worker unavailable", "global-owner", markUnavailable, codes.OK}, + {"global viewer cannot mark worker available", "global-viewer", markAvailable, codes.PermissionDenied}, + {"global viewer cannot mark worker unavailable", "global-viewer", markUnavailable, codes.PermissionDenied}, + {"atespace editor cannot mark worker available", "editor", markAvailable, codes.PermissionDenied}, + + // Tags: viewers read, editors write, all within the owning atespace. + {"editor creates tag", "editor", createTag, codes.OK}, + {"viewer cannot create tag", "viewer", createTag, codes.PermissionDenied}, + {"outsider cannot create tag in another atespace", "outsider", createTag, codes.PermissionDenied}, + {"outsider cannot tag another atespace's actor", "outsider", createCrossTag, codes.PermissionDenied}, + {"global owner creates tag", "global-owner", createTag, codes.OK}, + {"viewer gets tag", "viewer", getTag, codes.OK}, + {"global viewer gets tag", "global-viewer", getTag, codes.OK}, + {"outsider cannot get tag", "outsider", getTag, codes.PermissionDenied}, + {"viewer lists tags", "viewer", listTags, codes.OK}, + {"outsider cannot list tags", "outsider", listTags, codes.PermissionDenied}, + {"global viewer lists all tags", "global-viewer", listAllTags, codes.OK}, + {"atespace editor cannot list all tags", "editor", listAllTags, codes.PermissionDenied}, + {"editor updates tag", "editor", updateTag, codes.OK}, + {"editor deletes tag", "editor", deleteTag, codes.OK}, + {"viewer cannot update tag", "viewer", updateTag, codes.PermissionDenied}, + {"viewer cannot delete tag", "viewer", deleteTag, codes.PermissionDenied}, + {"outsider cannot delete tag", "outsider", deleteTag, codes.PermissionDenied}, + {"bootstrap owner deletes tag", "bootstrap-owner", deleteTag, codes.OK}, + + // Cloning from a tag needs editor on the tag's atespace, whatever its + // scope; golden tags and actors created without a tag are unaffected. + {"editor clones tag", "editor", cloneInTeamA, codes.OK}, + {"viewer cannot clone tag", "viewer", cloneInTeamA, codes.PermissionDenied}, + {"editor of another atespace cannot clone tag", "outsider", cloneInTeamB, codes.PermissionDenied}, + {"viewer of tag's atespace cannot clone tag into another atespace", "outsider-with-viewer", cloneInTeamB, codes.PermissionDenied}, + {"global owner clones tag into another atespace", "global-owner", cloneInTeamB, codes.OK}, + {"bootstrap owner clones tag", "bootstrap-owner", cloneInTeamA, codes.OK}, + {"editor clones golden tag", "editor", cloneGolden, codes.OK}, + {"editor creates actor without a tag", "editor", createWithoutTag, codes.OK}, + + // Only the egress gateway mints actor credentials; not even owners. + {"editor cannot mint actor JWT", "editor", mintJWT, codes.PermissionDenied}, + {"global owner cannot mint actor JWT", "global-owner", mintJWT, codes.PermissionDenied}, + {"bootstrap owner cannot mint actor certificate", "bootstrap-owner", mintCert, codes.PermissionDenied}, } interceptor := UnaryServerInterceptor(authorizer, true) @@ -601,7 +722,11 @@ func TestUnaryServerInterceptor_SystemPrincipals(t *testing.T) { }}} deleteWorker := rpcCall{ateapipb.Control_DeleteWorker_FullMethodName, &ateapipb.DeleteWorkerRequest{Worker: worker}} drainWorker := rpcCall{ateapipb.Control_DrainWorker_FullMethodName, &ateapipb.DrainWorkerRequest{Worker: worker}} + markAvailable := rpcCall{ateapipb.Control_MarkWorkerAvailable_FullMethodName, &ateapipb.MarkWorkerAvailableRequest{Worker: worker}} + markUnavailable := rpcCall{ateapipb.Control_MarkWorkerUnavailable_FullMethodName, &ateapipb.MarkWorkerUnavailableRequest{Worker: worker}} listAssignments := rpcCall{ateapipb.Control_ListWorkerActorAssignments_FullMethodName, &ateapipb.ListWorkerActorAssignmentsRequest{Worker: worker}} + mintJWT := rpcCall{ateapipb.Control_MintActorJwt_FullMethodName, &ateapipb.MintActorJwtRequest{Actor: actor}} + mintCert := rpcCall{ateapipb.Control_MintActorCertificate_FullMethodName, &ateapipb.MintActorCertificateRequest{Actor: actor}} mtls := func(id string) principal.PrincipalInfo { return principal.PrincipalInfo{ID: id, Kind: principal.KindMTLS} @@ -623,6 +748,8 @@ func TestUnaryServerInterceptor_SystemPrincipals(t *testing.T) { {"controller updates worker", mtls(controllerID), updateWorker, codes.OK}, {"controller deletes worker", mtls(controllerID), deleteWorker, codes.OK}, {"controller drains worker", mtls(controllerID), drainWorker, codes.OK}, + {"controller marks worker available", mtls(controllerID), markAvailable, codes.OK}, + {"controller marks worker unavailable", mtls(controllerID), markUnavailable, codes.OK}, {"controller cannot list worker assignments", mtls(controllerID), listAssignments, codes.PermissionDenied}, {"controller cannot get actor", mtls(controllerID), getActor, codes.PermissionDenied}, {"controller cannot resume actor", mtls(controllerID), resume, codes.PermissionDenied}, @@ -636,6 +763,8 @@ func TestUnaryServerInterceptor_SystemPrincipals(t *testing.T) { {"egress gateway cannot resume actor", mtls(egressID), resume, codes.PermissionDenied}, {"egress gateway lists actors", mtls(egressID), listAllActors, codes.OK}, {"egress gateway cannot get worker", mtls(egressID), getWorker, codes.PermissionDenied}, + {"egress gateway mints actor JWT", mtls(egressID), mintJWT, codes.OK}, + {"egress gateway mints actor certificate", mtls(egressID), mintCert, codes.OK}, // atenet-router resumes actors. Both it and atenet-egress list actors // as their ateapi health check. @@ -647,10 +776,13 @@ func TestUnaryServerInterceptor_SystemPrincipals(t *testing.T) { {"ingress router cannot delete actor", mtls(routerID), deleteActor, codes.PermissionDenied}, {"ingress router cannot get egress policy", mtls(routerID), getEgress, codes.PermissionDenied}, {"ingress router cannot create worker", mtls(routerID), createWorker, codes.PermissionDenied}, + {"ingress router cannot mint actor JWT", mtls(routerID), mintJWT, codes.PermissionDenied}, + {"controller cannot mint actor certificate", mtls(controllerID), mintCert, codes.PermissionDenied}, // A bearer token whose subject is a granted SPIFFE ID gets nothing. {"jwt with controller ID cannot create worker", jwt(controllerID), createWorker, codes.PermissionDenied}, {"jwt with egress ID cannot get actor", jwt(egressID), getActor, codes.PermissionDenied}, + {"jwt with egress ID cannot mint actor JWT", jwt(egressID), mintJWT, codes.PermissionDenied}, {"jwt with router ID cannot resume actor", jwt(routerID), resume, codes.PermissionDenied}, // Other SPIFFE IDs get nothing. diff --git a/cmd/ateapi/internal/authz/model.fga b/cmd/ateapi/internal/authz/model.fga index 03e9fa70d3..b65bec8163 100644 --- a/cmd/ateapi/internal/authz/model.fga +++ b/cmd/ateapi/internal/authz/model.fga @@ -55,13 +55,13 @@ type global define can_list_atespaces: viewer define can_create_worker: owner or controller define can_list_workers: viewer or controller - define can_get: viewer # Listing across all atespaces (List RPCs with an empty atespace). # atenet-router and atenet-egress, one binary, list actors as their ateapi # health check. define can_list_actors: viewer or ingress_router or egress_gateway define can_list_actor_templates: viewer + define can_list_tags: viewer # --- Policy Governance --- define can_create_access_policy: owner @@ -93,6 +93,8 @@ type atespace define can_create_actor_template: editor define can_list_actors: viewer define can_list_actor_templates: viewer + define can_create_tag: editor + define can_list_tags: viewer define can_get: viewer define can_delete: owner @@ -106,8 +108,25 @@ type actor_template # can_get for now; a separate relation so running a template can later # require more than reading it. define can_use_template: viewer from parent_atespace + define can_delete: editor from parent_atespace + +# A named snapshot owned by an atespace. Its scope controls which atespaces +# CreateActor accepts it in, which the handler enforces; can_use_tag controls +# who may clone it. Managing the tag stays with its owning atespace. +# +# TODO: A tag is a durable copy of an actor's memory and disk, including any +# secrets it held, and it outlives the actor. Any atespace editor can tag any +# actor in the atespace, publish it, and clone it. Restrict who can create, +# publish, and clone tags once the tag scoping design is settled. +type tag + relations + define parent_atespace: [atespace] + + define can_get: viewer from parent_atespace define can_update: editor from parent_atespace define can_delete: editor from parent_atespace + # Create actors from this tag (CreateActor source_tag). + define can_use_tag: editor from parent_atespace type actor relations @@ -129,11 +148,14 @@ type actor define can_update_egress_policy: editor from parent_atespace define can_delete_egress_policy: editor from parent_atespace - # --- Hardened System & Infrastructure Operations --- - # Node Restriction: Only the node hosting the scheduled worker can mint ateom actor credentials define can_mint_ateom_actor_credential: host_node + # Credentials asserting the actor's own identity. Only the egress gateway + # mints them; an editor holding one could impersonate the actor. + define can_mint_actor_jwt: egress_gateway from parent_atespace + define can_mint_actor_certificate: egress_gateway from parent_atespace + # Data Plane Ingress: Routers (or editors) can resume define can_resume: [user] or editor from parent_atespace or ingress_router from parent_atespace diff --git a/cmd/ateapi/internal/authz/model_test.fga.yaml b/cmd/ateapi/internal/authz/model_test.fga.yaml index 1a9b70adcd..0fb8c2f48e 100644 --- a/cmd/ateapi/internal/authz/model_test.fga.yaml +++ b/cmd/ateapi/internal/authz/model_test.fga.yaml @@ -38,10 +38,13 @@ tuples: relation: viewer object: atespace:space-1 - # Templates + # Templates and tags - user: atespace:space-1 relation: parent_atespace object: actor_template:private-template + - user: atespace:space-1 + relation: parent_atespace + object: tag:v1 # Physical Topology & Scheduling (Node -> Worker -> Actor) - user: node:node-1 @@ -102,7 +105,7 @@ tests: can_create_access_policy: true can_get_access_policy: true can_update_access_policy: true - can_get: true + can_list_tags: true - user: user:carol object: atespace:space-1 assertions: @@ -134,7 +137,7 @@ tests: can_create_access_policy: false can_get_access_policy: true can_update_access_policy: false - can_get: true + can_list_tags: true - user: user:frank object: atespace:space-1 assertions: @@ -221,40 +224,35 @@ tests: - name: 3. Actor Template Operations (Inherited from Atespace) check: - - user: user:alice # Owner can get, use, update, delete private template + - user: user:alice # Owner can get, use, delete private template object: actor_template:private-template assertions: can_get: true can_use_template: true - can_update: true can_delete: true - - user: user:bob # Editor can get, use, update, delete private template + - user: user:bob # Editor can get, use, delete private template object: actor_template:private-template assertions: can_get: true can_use_template: true - can_update: true can_delete: true - - user: user:dave # Viewer can get and use, but NOT update or delete + - user: user:dave # Viewer can get and use, but NOT delete object: actor_template:private-template assertions: can_get: true can_use_template: true - can_update: false can_delete: false - user: user:frank # Global viewer can get and use template via parent global object: actor_template:private-template assertions: can_get: true can_use_template: true - can_update: false can_delete: false - user: user:eve # External cannot access private template object: actor_template:private-template assertions: can_get: false can_use_template: false - can_update: false can_delete: false - name: 4. Actor Instance Operations (Inherited from Atespace) @@ -382,7 +380,6 @@ tests: can_list_actors: false can_list_atespaces: false can_create_atespace: false - can_get: false viewer: false - user: user:sys-controller object: worker:pod-1 @@ -403,6 +400,8 @@ tests: assertions: can_get: true can_get_egress_policy: true + can_mint_actor_jwt: true + can_mint_actor_certificate: true can_update: false can_delete: false can_create_egress_policy: false @@ -452,3 +451,77 @@ tests: controller: false egress_gateway: false ingress_router: false + + - name: 8. Tag Operations (Inherited from Atespace) + check: + - user: user:bob # Editor creates, reads, publishes, deletes and clones tags + object: atespace:space-1 + assertions: + can_create_tag: true + can_list_tags: true + - user: user:bob + object: tag:v1 + assertions: + can_get: true + can_update: true + can_delete: true + can_use_tag: true + - user: user:alice # Atespace owner may clone + object: tag:v1 + assertions: + can_use_tag: true + - user: user:carol # Global owner may clone + object: tag:v1 + assertions: + can_use_tag: true + - user: user:dave # Viewer reads tags only + object: atespace:space-1 + assertions: + can_create_tag: false + can_list_tags: true + - user: user:dave + object: tag:v1 + assertions: + can_get: true + can_update: false + can_delete: false + can_use_tag: false + - user: user:frank # Global viewer reads tags in every atespace + object: tag:v1 + assertions: + can_get: true + can_update: false + can_use_tag: false + - user: user:alice # Atespace owner cannot list across all atespaces + object: global:root + assertions: + can_list_tags: false + - user: user:eve # External + object: tag:v1 + assertions: + can_get: false + can_delete: false + can_use_tag: false + - user: user:sys-egress # System roles hold no tag permissions + object: atespace:space-1 + assertions: + can_create_tag: false + can_list_tags: false + + - name: 9. Actor Credentials (Egress Gateway Only) + check: + - user: user:carol # Global owner + object: actor:runner-1 + assertions: + can_mint_actor_jwt: false + can_mint_actor_certificate: false + - user: user:bob # Atespace editor + object: actor:runner-1 + assertions: + can_mint_actor_jwt: false + can_mint_actor_certificate: false + - user: user:sys-router + object: actor:runner-1 + assertions: + can_mint_actor_jwt: false + can_mint_actor_certificate: false diff --git a/cmd/ateapi/internal/authz/registry.go b/cmd/ateapi/internal/authz/registry.go index a6526b95b1..db8de54a6b 100644 --- a/cmd/ateapi/internal/authz/registry.go +++ b/cmd/ateapi/internal/authz/registry.go @@ -116,6 +116,23 @@ func onActorTemplate(relation string, ref resourceRef, fldPath *field.Path) ([]c return []check{{relation: relation, object: ActorTemplateObject(ref.GetAtespace(), ref.GetName())}}, nil } +func onTag(relation string, ref resourceRef, fldPath *field.Path) ([]check, field.ErrorList) { + if errs := validateRef(ref, fldPath); len(errs) > 0 { + return nil, errs + } + return []check{{relation: relation, object: TagObject(ref.GetAtespace(), ref.GetName())}}, nil +} + +// onSourceTag checks can_use_tag on the tag an actor is created from, if the +// request names one. Golden tags are exempt: the handler only accepts one +// built from the actor's own template, which can_use_template already covers. +func onSourceTag(ref *ateapipb.ObjectRef, fldPath *field.Path) ([]check, field.ErrorList) { + if ref == nil || ref.GetAtespace() == resources.GoldenActorAtespace { + return nil, nil + } + return onTag(RelationCanUseTag, ref, fldPath) +} + // both requires the checks from a and b, reporting the validation errors of // both. func both(aChecks []check, aErrs field.ErrorList, bChecks []check, bErrs field.ErrorList) ([]check, field.ErrorList) { @@ -132,6 +149,9 @@ type rpcRule struct { // disabled. AccessPolicy RPCs set it so that nobody can grant themselves // access while enforcement is off and keep that grant once it is turned on. alwaysEnforce bool + // authorizedInHandler, when set, says how the RPC's handler authorizes + // the caller; the interceptor passes such RPCs through unchecked. + authorizedInHandler string } func rule(extract checkExtractor) rpcRule { @@ -142,8 +162,15 @@ func governance(extract checkExtractor) rpcRule { return rpcRule{extract: extract, alwaysEnforce: true} } -// defaultRPCPermissions is the declarative registry mapping gRPC full method names -// to their permission rules. +// inHandler registers an RPC whose handler authorizes the caller, as how +// describes. +func inHandler(how string) rpcRule { + return rpcRule{authorizedInHandler: how} +} + +// defaultRPCPermissions is the declarative registry mapping gRPC full method +// names to their permission rules. With enforcement on, the interceptor denies +// any RPC missing from it. var defaultRPCPermissions = map[string]rpcRule{ // Atespaces. ateapipb.Control_CreateAtespace_FullMethodName: rule(checksOf(func(*ateapipb.CreateAtespaceRequest) ([]check, field.ErrorList) { @@ -197,11 +224,15 @@ var defaultRPCPermissions = map[string]rpcRule{ })), // Actors. Creating or updating an actor also requires can_use_template on - // the actor template it runs, which may live in another atespace. + // the actor template it runs, which may live in another atespace. Creating + // one from an explicit source tag also requires can_use_tag on that tag; + // the handler checks the tag's scope, completion and template. ateapipb.Control_CreateActor_FullMethodName: rule(checksOf(func(r *ateapipb.CreateActorRequest) ([]check, field.ErrorList) { actorChecks, actorErrs := onAtespace(RelationCanCreateActor, r.GetActor().GetMetadata().GetAtespace(), field.NewPath("actor", "metadata", "atespace")) templateChecks, templateErrs := onActorTemplate(RelationCanUseTemplate, r.GetActor().GetActorTemplate(), field.NewPath("actor", "actor_template")) - return both(actorChecks, actorErrs, templateChecks, templateErrs) + checks, errs := both(actorChecks, actorErrs, templateChecks, templateErrs) + tagChecks, tagErrs := onSourceTag(r.GetActor().GetSourceTag(), field.NewPath("actor", "source_tag")) + return both(checks, errs, tagChecks, tagErrs) })), ateapipb.Control_GetActor_FullMethodName: rule(checksOf(func(r *ateapipb.GetActorRequest) ([]check, field.ErrorList) { return onActor(RelationCanGet, r.GetActor(), field.NewPath("actor")) @@ -265,7 +296,54 @@ var defaultRPCPermissions = map[string]rpcRule{ ateapipb.Control_DrainWorker_FullMethodName: rule(checksOf(func(r *ateapipb.DrainWorkerRequest) ([]check, field.ErrorList) { return onWorker(RelationCanDrain, r.GetWorker().GetName(), field.NewPath("worker", "name")) })), + ateapipb.Control_MarkWorkerAvailable_FullMethodName: rule(checksOf(func(r *ateapipb.MarkWorkerAvailableRequest) ([]check, field.ErrorList) { + return onWorker(RelationCanUpdate, r.GetWorker().GetName(), field.NewPath("worker", "name")) + })), + ateapipb.Control_MarkWorkerUnavailable_FullMethodName: rule(checksOf(func(r *ateapipb.MarkWorkerUnavailableRequest) ([]check, field.ErrorList) { + return onWorker(RelationCanUpdate, r.GetWorker().GetName(), field.NewPath("worker", "name")) + })), ateapipb.Control_ListWorkerActorAssignments_FullMethodName: rule(checksOf(func(r *ateapipb.ListWorkerActorAssignmentsRequest) ([]check, field.ErrorList) { return onWorker(RelationCanListActorAssignments, r.GetWorker().GetName(), field.NewPath("worker", "name")) })), + + // Actor identity credentials. + ateapipb.Control_MintActorJwt_FullMethodName: rule(checksOf(func(r *ateapipb.MintActorJwtRequest) ([]check, field.ErrorList) { + return onActor(RelationCanMintActorJWT, r.GetActor(), field.NewPath("actor")) + })), + ateapipb.Control_MintActorCertificate_FullMethodName: rule(checksOf(func(r *ateapipb.MintActorCertificateRequest) ([]check, field.ErrorList) { + return onActor(RelationCanMintActorCertificate, r.GetActor(), field.NewPath("actor")) + })), + + // Tags. CreateTag stores the tag in its source actor's atespace, so that is + // the atespace checked. Validation also requires metadata.atespace to + // match, but it runs after this interceptor. + // + // TODO: Creating and publishing tags expose an actor's data; see the TODO + // on type tag in model.fga. + ateapipb.Control_CreateTag_FullMethodName: rule(checksOf(func(r *ateapipb.CreateTagRequest) ([]check, field.ErrorList) { + return onAtespace(RelationCanCreateTag, r.GetTag().GetSourceActor().GetAtespace(), field.NewPath("tag", "source_actor", "atespace")) + })), + ateapipb.Control_GetTag_FullMethodName: rule(checksOf(func(r *ateapipb.GetTagRequest) ([]check, field.ErrorList) { + return onTag(RelationCanGet, r.GetTag(), field.NewPath("tag")) + })), + ateapipb.Control_ListTags_FullMethodName: rule(checksOf(func(r *ateapipb.ListTagsRequest) ([]check, field.ErrorList) { + return onAtespaceOrGlobal(RelationCanListTags, r.GetAtespace(), field.NewPath("atespace")) + })), + ateapipb.Control_UpdateTag_FullMethodName: rule(checksOf(func(r *ateapipb.UpdateTagRequest) ([]check, field.ErrorList) { + return onTag(RelationCanUpdate, r.GetTag().GetMetadata(), field.NewPath("tag", "metadata")) + })), + ateapipb.Control_DeleteTag_FullMethodName: rule(checksOf(func(r *ateapipb.DeleteTagRequest) ([]check, field.ErrorList) { + return onTag(RelationCanDelete, r.GetTag(), field.NewPath("tag")) + })), + + // WorkerService is served only to atelet, which each handler + // authenticates by its mTLS SPIFFE ID. + ateapipb.WorkerService_RegisterWorker_FullMethodName: inHandler("ateletauth: caller is atelet on the Worker's node"), + ateapipb.WorkerService_RequestActorSuspend_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"), + // TODO: also require that the Actor runs on the caller's node. + ateapipb.WorkerService_MintAteomActorCertificate_FullMethodName: inHandler("ateletauth: caller is atelet"), + ateapipb.WorkerService_MintAteomActorJwt_FullMethodName: inHandler("ateletauth: caller is atelet on the node hosting the Actor"), + // TODO(#2570): not served yet; the handler must authenticate atelet on the + // node of the Worker hosting the Actor when it lands. + ateapipb.WorkerService_ReportActorSnapshotState_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"), } diff --git a/cmd/ateapi/internal/authz/registry_test.go b/cmd/ateapi/internal/authz/registry_test.go index 225d916c3a..ce3f8ceaf6 100644 --- a/cmd/ateapi/internal/authz/registry_test.go +++ b/cmd/ateapi/internal/authz/registry_test.go @@ -30,6 +30,7 @@ func TestDefaultRPCPermissions(t *testing.T) { localTemplate := &ateapipb.ObjectRef{Atespace: "team-a", Name: "tmpl"} sharedTemplate := &ateapipb.ObjectRef{Atespace: "shared", Name: "tmpl"} workerRef := &ateapipb.ObjectRef{Name: "w-1"} + tagRef := &ateapipb.ObjectRef{Atespace: "team-a", Name: "v1"} tests := []struct { name string @@ -107,6 +108,33 @@ func TestDefaultRPCPermissions(t *testing.T) { {RelationCanUseTemplate, "actor_template:shared/tmpl"}, }, }, + { + name: "CreateActor from a tag also checks the tag", + fullMethod: ateapipb.Control_CreateActor_FullMethodName, + req: &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: actorMeta, + ActorTemplate: sharedTemplate, + SourceTag: &ateapipb.ObjectRef{Atespace: "team-b", Name: "v1"}, + }}, + want: []check{ + {RelationCanCreateActor, "atespace:team-a"}, + {RelationCanUseTemplate, "actor_template:shared/tmpl"}, + {RelationCanUseTag, "tag:team-b/v1"}, + }, + }, + { + name: "CreateActor from a golden tag does not check the tag", + fullMethod: ateapipb.Control_CreateActor_FullMethodName, + req: &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: actorMeta, + ActorTemplate: sharedTemplate, + SourceTag: &ateapipb.ObjectRef{Atespace: "ate-golden", Name: "0b5c7d4e-golden"}, + }}, + want: []check{ + {RelationCanCreateActor, "atespace:team-a"}, + {RelationCanUseTemplate, "actor_template:shared/tmpl"}, + }, + }, { name: "GetActor", fullMethod: ateapipb.Control_GetActor_FullMethodName, @@ -232,12 +260,81 @@ func TestDefaultRPCPermissions(t *testing.T) { req: &ateapipb.DrainWorkerRequest{Worker: workerRef}, want: []check{{RelationCanDrain, "worker:w-1"}}, }, + { + name: "MarkWorkerAvailable", + fullMethod: ateapipb.Control_MarkWorkerAvailable_FullMethodName, + req: &ateapipb.MarkWorkerAvailableRequest{Worker: workerRef}, + want: []check{{RelationCanUpdate, "worker:w-1"}}, + }, + { + name: "MarkWorkerUnavailable", + fullMethod: ateapipb.Control_MarkWorkerUnavailable_FullMethodName, + req: &ateapipb.MarkWorkerUnavailableRequest{Worker: workerRef}, + want: []check{{RelationCanUpdate, "worker:w-1"}}, + }, { name: "ListWorkerActorAssignments", fullMethod: ateapipb.Control_ListWorkerActorAssignments_FullMethodName, req: &ateapipb.ListWorkerActorAssignmentsRequest{Worker: workerRef}, want: []check{{RelationCanListActorAssignments, "worker:w-1"}}, }, + + // Actor identity credentials. + { + name: "MintActorJWT", + fullMethod: ateapipb.Control_MintActorJwt_FullMethodName, + req: &ateapipb.MintActorJwtRequest{Actor: actorRef}, + want: []check{{RelationCanMintActorJWT, "actor:team-a/runner"}}, + }, + { + name: "MintActorCertificate", + fullMethod: ateapipb.Control_MintActorCertificate_FullMethodName, + req: &ateapipb.MintActorCertificateRequest{Actor: actorRef}, + want: []check{{RelationCanMintActorCertificate, "actor:team-a/runner"}}, + }, + + // Tags. + { + name: "CreateTag checks its source actor's atespace", + fullMethod: ateapipb.Control_CreateTag_FullMethodName, + req: &ateapipb.CreateTagRequest{Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "other", Name: "v1"}, + SourceActor: actorRef, + }}, + want: []check{{RelationCanCreateTag, "atespace:team-a"}}, + }, + { + name: "GetTag", + fullMethod: ateapipb.Control_GetTag_FullMethodName, + req: &ateapipb.GetTagRequest{Tag: tagRef}, + want: []check{{RelationCanGet, "tag:team-a/v1"}}, + }, + { + name: "ListTags in an atespace", + fullMethod: ateapipb.Control_ListTags_FullMethodName, + req: &ateapipb.ListTagsRequest{Atespace: "team-a"}, + want: []check{{RelationCanListTags, "atespace:team-a"}}, + }, + { + name: "ListTags across all atespaces", + fullMethod: ateapipb.Control_ListTags_FullMethodName, + req: &ateapipb.ListTagsRequest{}, + want: []check{{RelationCanListTags, GlobalRootObject}}, + }, + { + name: "UpdateTag", + fullMethod: ateapipb.Control_UpdateTag_FullMethodName, + req: &ateapipb.UpdateTagRequest{Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "v1"}, + }}, + want: []check{{RelationCanUpdate, "tag:team-a/v1"}}, + }, + { + name: "DeleteTag", + fullMethod: ateapipb.Control_DeleteTag_FullMethodName, + req: &ateapipb.DeleteTagRequest{Tag: tagRef}, + want: []check{{RelationCanDelete, "tag:team-a/v1"}}, + }, } for _, tc := range tests { @@ -347,6 +444,26 @@ func TestDefaultRPCPermissions_MalformedIdentifierFailsClosed(t *testing.T) { }}, wantField: "actor.actor_template.atespace", }, + { + name: "CreateActor with a source tag missing its name", + fullMethod: ateapipb.Control_CreateActor_FullMethodName, + req: &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: actorMeta, + ActorTemplate: template, + SourceTag: &ateapipb.ObjectRef{Atespace: "team-b"}, + }}, + wantField: "actor.source_tag.name", + }, + { + name: "CreateActor with a source tag missing its atespace", + fullMethod: ateapipb.Control_CreateActor_FullMethodName, + req: &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: actorMeta, + ActorTemplate: template, + SourceTag: &ateapipb.ObjectRef{Name: "v1"}, + }}, + wantField: "actor.source_tag.atespace", + }, { name: "GetActor without a name", fullMethod: ateapipb.Control_GetActor_FullMethodName, @@ -467,12 +584,77 @@ func TestDefaultRPCPermissions_MalformedIdentifierFailsClosed(t *testing.T) { req: &ateapipb.DrainWorkerRequest{Worker: &ateapipb.ObjectRef{Name: strings.Repeat("w", 64)}}, wantField: "worker.name", }, + { + name: "MarkWorkerAvailable without a name", + fullMethod: ateapipb.Control_MarkWorkerAvailable_FullMethodName, + req: &ateapipb.MarkWorkerAvailableRequest{Worker: &ateapipb.ObjectRef{}}, + wantField: "worker.name", + }, + { + name: "MarkWorkerUnavailable with an invalid name", + fullMethod: ateapipb.Control_MarkWorkerUnavailable_FullMethodName, + req: &ateapipb.MarkWorkerUnavailableRequest{Worker: &ateapipb.ObjectRef{Name: "w/1"}}, + wantField: "worker.name", + }, { name: "ListWorkerActorAssignments without a name", fullMethod: ateapipb.Control_ListWorkerActorAssignments_FullMethodName, req: &ateapipb.ListWorkerActorAssignmentsRequest{Worker: &ateapipb.ObjectRef{}}, wantField: "worker.name", }, + { + name: "MintActorJWT without an actor", + fullMethod: ateapipb.Control_MintActorJwt_FullMethodName, + req: &ateapipb.MintActorJwtRequest{}, + wantField: "actor.atespace", + }, + { + name: "MintActorCertificate with an invalid name", + fullMethod: ateapipb.Control_MintActorCertificate_FullMethodName, + req: &ateapipb.MintActorCertificateRequest{Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "a/b"}}, + wantField: "actor.name", + }, + { + name: "CreateTag without a source actor", + fullMethod: ateapipb.Control_CreateTag_FullMethodName, + req: &ateapipb.CreateTagRequest{Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "v1"}, + }}, + wantField: "tag.source_actor.atespace", + }, + { + name: "CreateTag with an invalid source actor atespace", + fullMethod: ateapipb.Control_CreateTag_FullMethodName, + req: &ateapipb.CreateTagRequest{Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "v1"}, + SourceActor: &ateapipb.ObjectRef{Atespace: "Team-A", Name: "runner"}, + }}, + wantField: "tag.source_actor.atespace", + }, + { + name: "GetTag without an atespace", + fullMethod: ateapipb.Control_GetTag_FullMethodName, + req: &ateapipb.GetTagRequest{Tag: &ateapipb.ObjectRef{Name: "v1"}}, + wantField: "tag.atespace", + }, + { + name: "ListTags with an invalid atespace", + fullMethod: ateapipb.Control_ListTags_FullMethodName, + req: &ateapipb.ListTagsRequest{Atespace: "Team-A"}, + wantField: "atespace", + }, + { + name: "UpdateTag without metadata", + fullMethod: ateapipb.Control_UpdateTag_FullMethodName, + req: &ateapipb.UpdateTagRequest{Tag: &ateapipb.Tag{}}, + wantField: "tag.metadata.atespace", + }, + { + name: "DeleteTag without a name", + fullMethod: ateapipb.Control_DeleteTag_FullMethodName, + req: &ateapipb.DeleteTagRequest{Tag: &ateapipb.ObjectRef{Atespace: "team-a"}}, + wantField: "tag.name", + }, } for _, tc := range tests { @@ -493,8 +675,52 @@ func TestDefaultRPCPermissions_MalformedIdentifierFailsClosed(t *testing.T) { func TestDefaultRPCPermissions_UnexpectedRequestTypeFailsClosed(t *testing.T) { for fullMethod, r := range defaultRPCPermissions { + if r.authorizedInHandler != "" { + continue + } if _, err := r.extract("wrong-type"); apierror.Code(err) != codes.Internal { t.Errorf("%s: extract(wrong type) error = %v, want code Internal", fullMethod, err) } } } + +// Every RPC in ateapi.proto has exactly one of a permission rule or a reason +// its handler authorizes it, and the registry names no RPC that isn't in +// ateapi.proto. +func TestDefaultRPCPermissions_CoversEveryServedRPC(t *testing.T) { + served := map[string]bool{} + services := ateapipb.File_ateapi_proto.Services() + for i := range services.Len() { + sd := services.Get(i) + methods := sd.Methods() + for j := range methods.Len() { + m := methods.Get(j) + fullMethod := "/" + string(sd.FullName()) + "/" + string(m.Name()) + if m.IsStreamingClient() || m.IsStreamingServer() { + // The authz interceptor is unary-only, so a streaming RPC would go + // unchecked. + t.Errorf("%s is a streaming RPC, which the authz interceptor does not check", fullMethod) + } + served[fullMethod] = true + } + } + + for fullMethod := range served { + r, ok := defaultRPCPermissions[fullMethod] + if !ok { + t.Errorf("%s has no entry in defaultRPCPermissions; add a rule, or inHandler if its handler authorizes the caller", fullMethod) + continue + } + if (r.extract == nil) == (r.authorizedInHandler == "") { + t.Errorf("%s: want exactly one of a check extractor or authorizedInHandler", fullMethod) + } + if r.authorizedInHandler != "" && r.alwaysEnforce { + t.Errorf("%s: authorizedInHandler and alwaysEnforce are exclusive", fullMethod) + } + } + for fullMethod := range defaultRPCPermissions { + if !served[fullMethod] { + t.Errorf("defaultRPCPermissions has %s, which no served service has", fullMethod) + } + } +} diff --git a/cmd/ateapi/internal/controlapi/actor.go b/cmd/ateapi/internal/controlapi/actor.go index 52ba1ac464..75237457bf 100644 --- a/cmd/ateapi/internal/controlapi/actor.go +++ b/cmd/ateapi/internal/controlapi/actor.go @@ -553,10 +553,8 @@ func (s *RPCService) MintActorJwt(ctx context.Context, req *ateapipb.MintActorJw return nil, apierror.InvalidArgument("%v", errs.ToAggregate()) } - // TODO(authz): Authorization layer needs to check whether the caller has - // the mintActorJWT permission/relation with this actor. This could be an - // atelet (via the relationship of the atelet running the actor), or the - // egress gateway (via a cluster-level grant?) + // With authz enforced, the interceptor requires can_mint_actor_jwt on the + // actor, which only the egress gateway holds. // Verify that this actor exists in the store. It doesn't need to be // running, since we may need to issue JWTs during actor boot / resume. @@ -577,8 +575,9 @@ func (s *RPCService) MintActorJwt(ctx context.Context, req *ateapipb.MintActorJw now := time.Now().Truncate(time.Second) expiresAt := now.Add(time.Duration(req.GetExpirationSeconds()) * time.Second) actorClaims := &actoridjwt.Claims{ + Kind: actoridjwt.KindActor, Issuer: s.actorJWTIssuer, - Subject: fmt.Sprintf("actor/%s/%s", dbActor.GetMetadata().GetAtespace(), dbActor.GetMetadata().GetName()), + Subject: fmt.Sprintf("%s/%s/%s", actoridjwt.KindActor.SubjectPrefix(), dbActor.GetMetadata().GetAtespace(), dbActor.GetMetadata().GetName()), Audiences: req.GetAudiences(), Expiration: expiresAt, NotBefore: now.Add(-5 * time.Minute), @@ -608,10 +607,8 @@ func (s *RPCService) MintActorCertificate(ctx context.Context, req *ateapipb.Min return nil, apierror.InvalidArgument("%v", errs.ToAggregate()) } - // TODO(authz): Authorization layer needs to check whether the caller has - // the mintActorCertificate permission/relation with this actor. This - // could be an atelet (via the relationship of the atelet running the - // actor), or the egress gateway (via a cluster-level grant?) + // With authz enforced, the interceptor requires can_mint_actor_certificate + // on the actor, which only the egress gateway holds. // Check that the caller authenticated with a client certificate --- we // should not allow bootstrapping a proof-of-possession credential from a diff --git a/cmd/ateapi/internal/controlapi/actor_template_test.go b/cmd/ateapi/internal/controlapi/actor_template_test.go index a9b6dbbfc3..b1a765421c 100644 --- a/cmd/ateapi/internal/controlapi/actor_template_test.go +++ b/cmd/ateapi/internal/controlapi/actor_template_test.go @@ -47,7 +47,7 @@ func validActorTemplate(mutations ...func(*ateapipb.ActorTemplate)) *ateapipb.Ac StorageLocation: "gs://my-bucket/snapshots", PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, } for _, m := range mutations { m(template) @@ -91,19 +91,19 @@ func TestCreateActorTemplate_SandboxConfigChecks(t *testing.T) { wantCode codes.Code }{{ name: "named config exists and matches", - sandbox: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"}, + sandbox: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, wantCode: codes.OK, }, { name: "empty config_name is rejected", - sandbox: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM}, + sandbox: &ateapipb.SandboxConfig{SandboxClass: "microvm"}, wantCode: codes.InvalidArgument, }, { name: "named config missing", - sandbox: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "does-not-exist"}, + sandbox: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "does-not-exist"}, wantCode: codes.FailedPrecondition, }, { name: "named config class mismatch", - sandbox: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM, ConfigName: "gvisor-default"}, + sandbox: &ateapipb.SandboxConfig{SandboxClass: "microvm", ConfigName: "gvisor-default"}, wantCode: codes.FailedPrecondition, }} for i, tt := range tests { @@ -343,7 +343,7 @@ func seedSubstrateTemplate(t *testing.T, ctx context.Context, persistence store. StorageLocation: "gs://ate-snapshots/team-a/", }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor", }, }) diff --git a/cmd/ateapi/internal/controlapi/actor_test.go b/cmd/ateapi/internal/controlapi/actor_test.go index f5377c4169..ce5dbf3011 100644 --- a/cmd/ateapi/internal/controlapi/actor_test.go +++ b/cmd/ateapi/internal/controlapi/actor_test.go @@ -254,8 +254,8 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { // different sandbox class. dataVolume := &ateapipb.Volume{Name: "data", DurableDir: &ateapipb.DurableDirVolumeSource{}} scratchVolume := &ateapipb.Volume{Name: "scratch", DurableDir: &ateapipb.DurableDirVolumeSource{}} - gvisorConfig := &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"} - microvmConfig := &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM, ConfigName: "microvm"} + gvisorConfig := &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"} + microvmConfig := &ateapipb.SandboxConfig{SandboxClass: "microvm", ConfigName: "microvm"} templates := map[string]struct { mountPath string volumes []*ateapipb.Volume @@ -416,7 +416,7 @@ func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) { if _, err := persistence.CreateActorTemplate(ctx, &ateapipb.ActorTemplate{ Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: name}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: location}, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, }); err != nil { t.Fatalf("creating template %s: %v", name, err) } diff --git a/cmd/ateapi/internal/controlapi/converter.go b/cmd/ateapi/internal/controlapi/converter.go index f34b9aaeaf..e30d192a01 100644 --- a/cmd/ateapi/internal/controlapi/converter.go +++ b/cmd/ateapi/internal/controlapi/converter.go @@ -16,7 +16,6 @@ package controlapi import ( "github.com/agent-substrate/substrate/internal/proto/ateletpb" - atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -35,16 +34,3 @@ func fidelityToAtelet(in ateapipb.SnapshotFidelity) ateletpb.SnapshotFidelity { return ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED } } - -// sandboxClassString renders the proto enum in the CRD's lower-case string -// form, which the scheduler and the metric labels share. -func sandboxClassString(in ateapipb.SandboxClass) string { - switch in { - case ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR: - return string(atev1alpha1.SandboxClassGvisor) - case ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM: - return string(atev1alpha1.SandboxClassMicroVM) - default: - return "" - } -} diff --git a/cmd/ateapi/internal/controlapi/converter_test.go b/cmd/ateapi/internal/controlapi/converter_test.go index 4a0824230b..c9ef4c0e4a 100644 --- a/cmd/ateapi/internal/controlapi/converter_test.go +++ b/cmd/ateapi/internal/controlapi/converter_test.go @@ -65,21 +65,3 @@ func TestFidelityToAtelet(t *testing.T) { }) } } - -// TestSandboxClassString pins the label values the scheduler and metrics -// share with the CRD's lower-case enum. -func TestSandboxClassString(t *testing.T) { - tests := []struct { - in ateapipb.SandboxClass - expected string - }{ - {ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, "gvisor"}, - {ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM, "microvm"}, - {ateapipb.SandboxClass_SANDBOX_CLASS_UNSPECIFIED, ""}, - } - for _, tt := range tests { - if got := sandboxClassString(tt.in); got != tt.expected { - t.Errorf("sandboxClassString(%v) = %q, want %q", tt.in, got, tt.expected) - } - } -} diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go index 8a5034ebf7..9b09e2bee2 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go @@ -44,7 +44,7 @@ func TestActorTemplateCRUD(t *testing.T) { Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "1Gi"}}}, @@ -68,7 +68,7 @@ func TestActorTemplateCRUD(t *testing.T) { GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_ENABLED}, }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "1Gi"}}}, @@ -83,7 +83,7 @@ func TestActorTemplateCRUD(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "tmpl-a"}, Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, }, }) assertGrpcError(t, err, codes.AlreadyExists, "ActorTemplate "+testAtespace+"/tmpl-a already exists") @@ -139,7 +139,7 @@ func TestActorTemplateCRUD(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "tmpl-unnamed-config"}, Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, }, }) assertGrpcErrorRegex(t, err, codes.InvalidArgument, `sandbox_config\.config_name`) @@ -251,7 +251,7 @@ func TestActorLifecycleWithoutGoldenSnapshot(t *testing.T) { StorageLocation: testStorageLocation, GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_DISABLED}, }, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, Containers: []*ateapipb.Container{{Name: "main", Image: "main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", Command: []string{"/main"}}}, WorkerSelector: &ateapipb.Selector{MatchLabels: map[string]string{poolLabelKey: ns}}, }}) diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go index e045588b6e..24ae268e3c 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go @@ -20,6 +20,7 @@ import ( "encoding/json" "errors" "fmt" + "slices" "strings" "sync" "sync/atomic" @@ -148,9 +149,6 @@ func TestCreateActor_WithExternalVolumes(t *testing.T) { if vol.GetName() != "ext-vol-1" { t.Errorf("volume name = %q, want %q", vol.GetName(), "ext-vol-1") } - if vol.GetStatus() != ateapipb.ExternalVolume_STATUS_PENDING { - t.Errorf("volume status = %v, want %v", vol.GetStatus(), ateapipb.ExternalVolume_STATUS_PENDING) - } if vol.GetStorageVolumeId() != "" { t.Errorf("expected empty storageVolumeId before resume, got %q", vol.GetStorageVolumeId()) } @@ -165,8 +163,8 @@ func TestCreateActor_WithExternalVolumes(t *testing.T) { if len(getResp.GetStatus().GetExternalVolumes()) != 1 { t.Fatalf("expected 1 volume in GetActor response, got %d", len(getResp.GetStatus().GetExternalVolumes())) } - if getResp.GetStatus().GetExternalVolumes()[0].GetStatus() != ateapipb.ExternalVolume_STATUS_PENDING { - t.Errorf("GetActor status = %v, want %v", getResp.GetStatus().GetExternalVolumes()[0].GetStatus(), ateapipb.ExternalVolume_STATUS_PENDING) + if got := getResp.GetStatus().GetExternalVolumes()[0].GetStorageVolumeId(); got != "" { + t.Errorf("GetActor storageVolumeId = %q, want empty before resume", got) } } @@ -200,7 +198,7 @@ func TestCreateActor_SubstrateTemplateRef(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "sub-tmpl"}, Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: "gvisor-default"}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, }, }); err != nil { t.Fatalf("CreateActorTemplate failed: %v", err) @@ -323,7 +321,7 @@ func TestCreateActor_RejectsSnapshotWithExternalVolumes(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "tmpl1"}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Containers: []*ateapipb.Container{{ @@ -770,7 +768,7 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { }}, Volumes: tmpl.volumes, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, ConfigName: tmpl.configName}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: tmpl.configName}, }, }); err != nil { t.Fatalf("CreateActorTemplate %s failed: %v", name, err) @@ -1422,8 +1420,8 @@ func TestDeleteActor_MultipleVolumeDeletionFailures(t *testing.T) { Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ExternalVolumes: []*ateapipb.ExternalVolume{ - {Name: "vol1", StorageVolumeId: "storage-vol-1", Status: ateapipb.ExternalVolume_STATUS_CREATED, VolumeType: "substrate.io/mock"}, - {Name: "vol2", StorageVolumeId: "storage-vol-2", Status: ateapipb.ExternalVolume_STATUS_CREATED, VolumeType: "substrate.io/mock"}, + {Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "substrate.io/mock"}, + {Name: "vol2", StorageVolumeId: "storage-vol-2", VolumeType: "substrate.io/mock"}, }, }, } @@ -1469,13 +1467,13 @@ func (r *retryDeleteVolumePlugin) DeleteVolume(ctx context.Context, volumeID str } // TestDeleteActor_VolumeDeletionFailure_RetrySuccess tests that when volume deletion fails -// during DeleteActor, the actor transitions to ACTOR_STATE_DELETING and its volumes to -// ExternalVolume_STATUS_DELETING, and a subsequent retry of DeleteActor cleanly finalizes deletion. +// during DeleteActor, the actor transitions to ACTOR_STATE_DELETING keeping its volumes, +// and a subsequent retry of DeleteActor cleanly finalizes deletion. // // Workflow: // 1. Creates a suspended actor with a provisioned external volume. // 2. Calls DeleteActor, which fails because the CSI plugin returns an error on DeleteVolume. -// 3. Verifies that the actor is persisted in ACTOR_STATE_DELETING and the volume is marked STATUS_DELETING. +// 3. Verifies that the actor is persisted in ACTOR_STATE_DELETING and still records the volume. // 4. Clears the plugin error to simulate backend recovery. // 5. Retries DeleteActor, which re-attempts volume deletion and cleans up the actor from the store. // 6. Confirms GetActor returns NotFound. @@ -1498,7 +1496,7 @@ func TestDeleteActor_VolumeDeletionFailure_RetrySuccess(t *testing.T) { Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ExternalVolumes: []*ateapipb.ExternalVolume{ - {Name: "vol1", StorageVolumeId: "storage-vol-1", Status: ateapipb.ExternalVolume_STATUS_CREATED, VolumeType: "substrate.io/mock"}, + {Name: "vol1", StorageVolumeId: "storage-vol-1", VolumeType: "substrate.io/mock"}, }, }, } @@ -1514,7 +1512,7 @@ func TestDeleteActor_VolumeDeletionFailure_RetrySuccess(t *testing.T) { t.Fatalf("expected DeleteActor to fail, but got nil") } - // 3. Verify actor is persisted in ACTOR_STATE_DELETING with volume in ExternalVolume_STATUS_DELETING. + // 3. Verify actor is persisted in ACTOR_STATE_DELETING with its volume still recorded. getResp, err := tc.service.GetActor(context.Background(), &ateapipb.GetActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "delete-retry-actor"}, }) @@ -1524,8 +1522,8 @@ func TestDeleteActor_VolumeDeletionFailure_RetrySuccess(t *testing.T) { if getResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_DELETING { t.Errorf("actor state = %v, want ACTOR_STATE_DELETING", getResp.GetStatus().GetState()) } - if len(getResp.GetStatus().GetExternalVolumes()) != 1 || getResp.GetStatus().GetExternalVolumes()[0].GetStatus() != ateapipb.ExternalVolume_STATUS_DELETING { - t.Errorf("actor volume status = %v, want STATUS_DELETING", getResp.GetStatus().GetExternalVolumes()) + if len(getResp.GetStatus().GetExternalVolumes()) != 1 { + t.Errorf("actor volumes = %v, want the volume still recorded", getResp.GetStatus().GetExternalVolumes()) } // 4. Recover the CSI plugin to simulate backend recovery. @@ -1657,8 +1655,8 @@ func TestActorLifecycle_WithExternalVolumes(t *testing.T) { if createResp.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { t.Fatalf("expected initial state ACTOR_STATE_SUSPENDED, got %v", createResp.GetStatus().GetState()) } - if len(createResp.GetStatus().GetExternalVolumes()) != 1 || createResp.GetStatus().GetExternalVolumes()[0].GetStatus() != ateapipb.ExternalVolume_STATUS_PENDING { - t.Fatalf("expected 1 pending volume after CreateActor, got %v", createResp.GetStatus().GetExternalVolumes()) + if len(createResp.GetStatus().GetExternalVolumes()) != 1 || createResp.GetStatus().GetExternalVolumes()[0].GetStorageVolumeId() != "" { + t.Fatalf("expected 1 unprovisioned volume after CreateActor, got %v", createResp.GetStatus().GetExternalVolumes()) } // 2. ResumeActor @@ -1671,8 +1669,8 @@ func TestActorLifecycle_WithExternalVolumes(t *testing.T) { if resumeResp.GetActor().GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING { t.Fatalf("expected state ACTOR_STATE_RUNNING after resume, got %v", resumeResp.GetActor().GetStatus().GetState()) } - if len(resumeResp.GetActor().GetStatus().GetExternalVolumes()) != 1 || resumeResp.GetActor().GetStatus().GetExternalVolumes()[0].GetStatus() != ateapipb.ExternalVolume_STATUS_CREATED { - t.Fatalf("expected 1 created volume after ResumeActor, got %v", resumeResp.GetActor().GetStatus().GetExternalVolumes()) + if len(resumeResp.GetActor().GetStatus().GetExternalVolumes()) != 1 { + t.Fatalf("expected 1 volume after ResumeActor, got %v", resumeResp.GetActor().GetStatus().GetExternalVolumes()) } if resumeResp.GetActor().GetStatus().GetExternalVolumes()[0].GetStorageVolumeId() == "" { t.Fatalf("expected non-empty storageVolumeId after ResumeActor") @@ -1829,7 +1827,7 @@ func TestResumeActor_VolumeCreationFailure(t *testing.T) { t.Fatalf("expected non-empty UID on actor") } - // Verify that succ-vol1 was updated to CREATED with a storageVolumeId, and fail-vol2 is still PENDING + // Verify that succ-vol1 was given a storageVolumeId, and fail-vol2 still has none if len(getResp.GetStatus().GetExternalVolumes()) != 2 { t.Fatalf("expected 2 volumes on actor, got %d", len(getResp.GetStatus().GetExternalVolumes())) } @@ -1837,10 +1835,10 @@ func TestResumeActor_VolumeCreationFailure(t *testing.T) { for _, v := range getResp.GetStatus().GetExternalVolumes() { volsByName[v.GetName()] = v } - if v1, ok := volsByName["succ-vol1"]; !ok || v1.GetStatus() != ateapipb.ExternalVolume_STATUS_CREATED || v1.GetStorageVolumeId() == "" { + if v1, ok := volsByName["succ-vol1"]; !ok || v1.GetStorageVolumeId() == "" { t.Errorf("succ-vol1 unexpected state: %v", v1) } - if v2, ok := volsByName["fail-vol2"]; !ok || v2.GetStatus() != ateapipb.ExternalVolume_STATUS_PENDING { + if v2, ok := volsByName["fail-vol2"]; !ok || v2.GetStorageVolumeId() != "" { t.Errorf("fail-vol2 unexpected state: %v", v2) } @@ -1971,10 +1969,10 @@ func TestResumeActor_VolumeCreationRetrySuccess(t *testing.T) { for _, v := range getResp.GetStatus().GetExternalVolumes() { volsByName[v.GetName()] = v } - if v1, ok := volsByName["succ-vol1"]; !ok || v1.GetStatus() != ateapipb.ExternalVolume_STATUS_CREATED || v1.GetStorageVolumeId() == "" { + if v1, ok := volsByName["succ-vol1"]; !ok || v1.GetStorageVolumeId() == "" { t.Errorf("succ-vol1 unexpected state after first resume: %v", v1) } - if v2, ok := volsByName["retry-vol2"]; !ok || v2.GetStatus() != ateapipb.ExternalVolume_STATUS_PENDING { + if v2, ok := volsByName["retry-vol2"]; !ok || v2.GetStorageVolumeId() != "" { t.Errorf("retry-vol2 unexpected state after first resume: %v", v2) } @@ -1997,7 +1995,7 @@ func TestResumeActor_VolumeCreationRetrySuccess(t *testing.T) { t.Errorf("actor state after second resume = %v, want %v", getResp.GetStatus().GetState(), ateapipb.ActorState_ACTOR_STATE_RUNNING) } for _, v := range getResp.GetStatus().GetExternalVolumes() { - if v.GetStatus() != ateapipb.ExternalVolume_STATUS_CREATED || v.GetStorageVolumeId() == "" { + if v.GetStorageVolumeId() == "" { t.Errorf("volume %s unexpected state after second resume: %v", v.GetName(), v) } } @@ -2122,7 +2120,7 @@ func TestResumeActor_VolumeAttachFailureAndRetry(t *testing.T) { t.Fatalf("expected 1 volume on actor, got %d", len(getResp.GetStatus().GetExternalVolumes())) } vol := getResp.GetStatus().GetExternalVolumes()[0] - if vol.GetStatus() != ateapipb.ExternalVolume_STATUS_CREATED || vol.GetStorageVolumeId() == "" { + if vol.GetStorageVolumeId() == "" { t.Errorf("vol1 unexpected state after failed attach: %v", vol) } @@ -3101,7 +3099,7 @@ func createDataFidelityTemplate(t *testing.T, tc *testContext, ns string) *ateap PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Containers: []*ateapipb.Container{{ @@ -5311,8 +5309,12 @@ func TestMintActorJWT_Success(t *testing.T) { if want := "actor/" + testAtespace + "/id1"; claims.Subject != want { t.Errorf("sub = %q, want %q", claims.Subject, want) } - if want := createResp.GetMetadata().GetUid(); claims.Substrate.ActorUID != want { - t.Errorf("ate.dev.actorUID = %q, want the stored actor's %q", claims.Substrate.ActorUID, want) + kind, substrate, ok := claims.Substrate() + if !ok || kind != actoridjwt.KindActor { + t.Fatalf("token kind = %v (ok %v), want KindActor under actor.ate.dev", kind, ok) + } + if want := createResp.GetMetadata().GetUid(); substrate.ActorUID != want { + t.Errorf("actor.ate.dev.actorUID = %q, want the stored actor's %q", substrate.ActorUID, want) } assertActorJWTLifetime(t, mintResp, claims, 30*time.Minute) } @@ -5827,3 +5829,309 @@ func TestDeleteActor_Preconditions(t *testing.T) { _, err = tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: ref}) assertGrpcError(t, err, codes.NotFound, "Actor "+actorRef.String()+" not found") } + +// reportWorkerRuntime writes defaultRT and restorableRTs to workerName's status +// and waits for the worker cache to observe the update. +func reportWorkerRuntime(t *testing.T, tc *testContext, workerName string, defaultRT *ateapipb.SandboxRuntime, restorableRTs ...*ateapipb.SandboxRuntime) { + t.Helper() + ctx := context.Background() + cur, err := tc.persistence.GetWorker(ctx, workerName) + if err != nil { + t.Fatalf("GetWorker(%s): %v", workerName, err) + } + if _, err := tc.persistence.UpdateWorker(ctx, workerName, store.PreconditionFrom(cur), func(toUpdate *ateapipb.Worker) error { + toUpdate.Status.DefaultRuntime = proto.Clone(defaultRT).(*ateapipb.SandboxRuntime) + toUpdate.Status.RestorableRuntimes = nil + for _, r := range restorableRTs { + toUpdate.Status.RestorableRuntimes = append(toUpdate.Status.RestorableRuntimes, proto.Clone(r).(*ateapipb.SandboxRuntime)) + } + return nil + }); err != nil { + t.Fatalf("UpdateWorker(%s): %v", workerName, err) + } + if err := wait.PollUntilContextTimeout(ctx, 10*time.Millisecond, 5*time.Second, true, func(context.Context) (bool, error) { + got, err := tc.workerCache.Worker(workerName) + return err == nil && + proto.Equal(got.GetStatus().GetDefaultRuntime(), defaultRT) && + slices.EqualFunc(got.GetStatus().GetRestorableRuntimes(), restorableRTs, func(a, b *ateapipb.SandboxRuntime) bool { return proto.Equal(a, b) }), nil + }); err != nil { + t.Fatalf("worker %s runtime never reached worker cache: %v", workerName, err) + } +} + +// TestResumeActor_SandboxRuntimePlacementAndFallback exercises the end-to-end +// runtime stamping, placement, and fallback lifecycle across two workers +// advertising different SandboxRuntimes: +// 1. CreateActor -> ResumeActor -> SuspendActor stamps the worker's runtime +// onto WorkerAssignment and then onto the resulting Snapshot. +// 2. A subsequent ResumeActor places the actor onto the runtime-compatible +// worker and restores at full MEMORY fidelity when that worker has room. +// 3. When the compatible worker is full, ResumeActor falls back to the other +// worker, downgrades the restore to VOLUMES whether the snapshot is the +// actor's own or a borrowed golden, and re-stamps the new worker's runtime +// on the next SuspendActor. +// 4. When the other worker advertises the snapshot's runtime in +// restorable_runtimes, ResumeActor restores at full MEMORY fidelity, +// records the matched restorable runtime on WorkerAssignment, and preserves +// that stamp through PauseActor and SuspendActor-from-PAUSED. +func TestResumeActor_SandboxRuntimePlacementAndFallback(t *testing.T) { + ns := namespaceForTest("ns-runtime-fallback") + tc := setupTest(t, ns) + defer tc.cleanup() + + ctx := context.Background() + tmpl := createTemplate(t, tc, ns) + + rtA := &ateapipb.SandboxRuntime{ + SandboxClass: "gvisor", + Version: &ateapipb.VersionedSandboxCompat{ + SchemaVersion: "v1", + Attributes: []*ateapipb.AttributeEntry{ + {Key: "architecture", Value: "amd64"}, + {Key: "gvisor_asset_hash", Value: "hash-a"}, + }, + }, + } + rtB := &ateapipb.SandboxRuntime{ + SandboxClass: "gvisor", + Version: &ateapipb.VersionedSandboxCompat{ + SchemaVersion: "v1", + Attributes: []*ateapipb.AttributeEntry{ + {Key: "architecture", Value: "amd64"}, + {Key: "gvisor_asset_hash", Value: "hash-b"}, + }, + }, + } + + // Start with worker-a advertising rtA so the initial resume lands on rtA. + workerA := createWorkerPod(t, tc, ns, "worker-a", "node1", "pool1") + reportWorkerRuntime(t, tc, workerA, rtA) + + actorRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "actor-1"} + if _, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "actor-1"}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + }}); err != nil { + t.Fatalf("CreateActor failed: %v", err) + } + + resumed1, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef}) + if err != nil { + t.Fatalf("initial ResumeActor failed: %v", err) + } + if got := resumed1.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("initial WorkerAssignment.SandboxRuntime = %v, want %v", got, rtA) + } + + suspended1, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: actorRef}) + if err != nil { + t.Fatalf("first SuspendActor failed: %v", err) + } + waitForWorkerAvailable(t, tc, workerA) + if got := durableSnapshot(suspended1.GetActor().GetStatus()).GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("first Snapshot.SandboxRuntime = %v, want %v", got, rtA) + } + + // Add worker-b advertising an incompatible runtime (rtB). With both workers + // idle, resuming actor-1 must select worker-a and restore at MEMORY fidelity. + workerB := createWorkerPod(t, tc, ns, "worker-b", "node1", "pool1") + reportWorkerRuntime(t, tc, workerB, rtB) + + resumed2, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef}) + if err != nil { + t.Fatalf("second ResumeActor failed: %v", err) + } + if got := resumed2.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-a" { + t.Fatalf("second ResumeActor placed on %q, want worker-a", got) + } + if got := resumed2.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("second WorkerAssignment.SandboxRuntime = %v, want %v", got, rtA) + } + if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { + t.Fatalf("second RestoreRequest.Fidelity = %v, want MEMORY", got) + } + + if _, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: actorRef}); err != nil { + t.Fatalf("second SuspendActor failed: %v", err) + } + waitForWorkerAvailable(t, tc, workerA) + + // Tag actor-1's rtA snapshot and occupy worker-a with a blocker actor seeded + // from that tag so the rtA worker is at capacity. + tagRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tag-rt-a"} + tag, err := tc.client.CreateTag(ctx, &ateapipb.CreateTagRequest{ + Tag: &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "tag-rt-a"}, + SourceActor: actorRef, + Scope: ateapipb.TagScope_TAG_SCOPE_ATESPACE, + }, + }) + if err != nil { + t.Fatalf("CreateTag failed: %v", err) + } + if got := tag.GetStatus().GetSnapshot().GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("Tag.Snapshot.SandboxRuntime = %v, want %v", got, rtA) + } + + blockerRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "blocker"} + if _, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "blocker"}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + SourceTag: tagRef, + }}); err != nil { + t.Fatalf("CreateActor(blocker) failed: %v", err) + } + blockerResumed, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: blockerRef}) + if err != nil { + t.Fatalf("ResumeActor(blocker) failed: %v", err) + } + if got := blockerResumed.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-a" { + t.Fatalf("blocker placed on %q, want worker-a", got) + } + + // With worker-a full, resuming actor-1 must fall back to worker-b, downgrade + // the restore to VOLUMES fidelity, and re-stamp rtB on the next suspend. + resumed3, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actorRef}) + if err != nil { + t.Fatalf("fallback ResumeActor failed: %v", err) + } + if got := resumed3.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-b" { + t.Fatalf("fallback ResumeActor placed on %q, want worker-b", got) + } + if got := resumed3.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime(); !proto.Equal(got, rtB) { + t.Fatalf("fallback WorkerAssignment.SandboxRuntime = %v, want %v", got, rtB) + } + if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + t.Fatalf("fallback RestoreRequest.Fidelity = %v, want VOLUMES", got) + } + + suspended3, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: actorRef}) + if err != nil { + t.Fatalf("SuspendActor after fallback failed: %v", err) + } + waitForWorkerAvailable(t, tc, workerB) + if got := durableSnapshot(suspended3.GetActor().GetStatus()).GetSandboxRuntime(); !proto.Equal(got, rtB) { + t.Fatalf("Snapshot.SandboxRuntime after fallback suspend = %v, want %v", got, rtB) + } + + // Point the template's golden tag at a golden snapshot stamped with rtA and + // verify that a newly created actor borrowing the golden snapshot, when + // worker-a is full and it falls back onto worker-b, restores the golden at + // VOLUMES fidelity like any other durable snapshot rather than cold-booting. + goldenSnap := newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, tmpl.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t)) + goldenSnap.SandboxRuntime = proto.Clone(rtA).(*ateapipb.SandboxRuntime) + stampedGoldenTag, err := tc.persistence.CreateTag(ctx, &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: resources.GoldenActorAtespace, Name: "golden-rt-a"}, + SourceActor: &ateapipb.ObjectRef{Atespace: resources.GoldenActorAtespace, Name: tmpl.GetMetadata().GetUid()}, + Scope: ateapipb.TagScope_TAG_SCOPE_PUBLISHED, + Status: &ateapipb.TagStatus{ + Snapshot: goldenSnap, + ActorTemplateUid: tmpl.GetMetadata().GetUid(), + }, + }) + if err != nil { + t.Fatalf("CreateTag(golden-rt-a): %v", err) + } + curTmpl, err := tc.persistence.GetActorTemplate(ctx, resources.ActorTemplateRefFromActorTemplate(tmpl)) + if err != nil { + t.Fatalf("GetActorTemplate: %v", err) + } + if _, err := tc.persistence.UpdateActorTemplate(ctx, resources.ActorTemplateRefFromActorTemplate(curTmpl), store.PreconditionFrom(curTmpl), func(dbTemplate *ateapipb.ActorTemplate) error { + dbTemplate.Status.GoldenSnapshotStatus.GoldenTag = resources.TagRefFromTag(stampedGoldenTag).ToObjectRef() + return nil + }); err != nil { + t.Fatalf("UpdateActorTemplate(golden): %v", err) + } + + goldenActorRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "golden-actor"} + goldenCreated, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "golden-actor"}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + }}) + if err != nil { + t.Fatalf("CreateActor(golden-actor) failed: %v", err) + } + if got := durableSnapshot(goldenCreated.GetStatus()).GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("golden-actor initial Snapshot.SandboxRuntime = %v, want %v", got, rtA) + } + + tc.fakeAtelet.Lock.Lock() + tc.fakeAtelet.RunCalled = false + tc.fakeAtelet.RestoreCalled = false + tc.fakeAtelet.Lock.Unlock() + + goldenResumed, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: goldenActorRef}) + if err != nil { + t.Fatalf("ResumeActor(golden-actor) failed: %v", err) + } + if got := goldenResumed.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-b" { + t.Fatalf("golden-actor placed on %q, want worker-b", got) + } + if got := goldenResumed.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime(); !proto.Equal(got, rtB) { + t.Fatalf("golden-actor WorkerAssignment.SandboxRuntime = %v, want %v", got, rtB) + } + tc.fakeAtelet.Lock.Lock() + runCalled := tc.fakeAtelet.RunCalled + restoreCalled := tc.fakeAtelet.RestoreCalled + tc.fakeAtelet.Lock.Unlock() + if runCalled || !restoreCalled { + t.Fatalf("golden-actor fallback: RunCalled=%v RestoreCalled=%v, want RunCalled=false RestoreCalled=true", runCalled, restoreCalled) + } + goldenRestore := tc.fakeAtelet.lastRestoreRequest() + if got := goldenRestore.GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + t.Fatalf("golden-actor RestoreRequest.Fidelity = %v, want VOLUMES", got) + } + if got, want := goldenRestore.GetSnapshot().GetObject().GetSnapshotUri(), goldenSnapshotURI(t); got != want { + t.Fatalf("golden-actor RestoreRequest.SnapshotUri = %q, want golden %q", got, want) + } + + // Free worker-b and have it advertise rtA in restorable_runtimes while + // keeping rtB as its default. Resuming an actor whose snapshot requires rtA + // onto worker-b (with worker-a still full) must restore at MEMORY fidelity, + // record the matched restorable entry (rtA, not rtB) on WorkerAssignment, + // and preserve rtA through PauseActor and SuspendActor-from-PAUSED. + if _, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: goldenActorRef}); err != nil { + t.Fatalf("SuspendActor(golden-actor) failed: %v", err) + } + waitForWorkerAvailable(t, tc, workerB) + reportWorkerRuntime(t, tc, workerB, rtB, rtA) + + restorableRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "restorable-actor"} + if _, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "restorable-actor"}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + SourceTag: tagRef, + }}); err != nil { + t.Fatalf("CreateActor(restorable-actor) failed: %v", err) + } + restorableResumed, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: restorableRef}) + if err != nil { + t.Fatalf("ResumeActor(restorable-actor) failed: %v", err) + } + if got := restorableResumed.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-b" { + t.Fatalf("restorable-actor placed on %q, want worker-b", got) + } + if got := restorableResumed.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("restorable-actor WorkerAssignment.SandboxRuntime = %v, want matched restorable %v", got, rtA) + } + if got := tc.fakeAtelet.lastRestoreRequest().GetSnapshot().GetFidelity(); got != ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { + t.Fatalf("restorable-actor RestoreRequest.Fidelity = %v, want MEMORY", got) + } + + pausedResp, err := tc.client.PauseActor(ctx, &ateapipb.PauseActorRequest{Actor: restorableRef}) + if err != nil { + t.Fatalf("PauseActor(restorable-actor) failed: %v", err) + } + waitForWorkerAvailable(t, tc, workerB) + if got := localSnapshot(pausedResp.GetActor().GetStatus()).GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("paused local Snapshot.SandboxRuntime = %v, want %v", got, rtA) + } + + suspendedFromPaused, err := tc.client.SuspendActor(ctx, &ateapipb.SuspendActorRequest{Actor: restorableRef}) + if err != nil { + t.Fatalf("SuspendActor from PAUSED(restorable-actor) failed: %v", err) + } + if got := durableSnapshot(suspendedFromPaused.GetActor().GetStatus()).GetSandboxRuntime(); !proto.Equal(got, rtA) { + t.Fatalf("durable Snapshot.SandboxRuntime after SuspendActor from PAUSED = %v, want %v", got, rtA) + } +} diff --git a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go index 0b30cbef06..8a4cf14662 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go @@ -450,7 +450,7 @@ func createTemplateWithContainersAndVolumes(t *testing.T, tc *testContext, ns st StorageLocation: testStorageLocation, }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Containers: containers, @@ -658,7 +658,7 @@ func createTemplateWithSelector(t *testing.T, tc *testContext, name string, sele StorageLocation: testStorageLocation, }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Containers: []*ateapipb.Container{ @@ -741,6 +741,12 @@ func createWorkerPod(t *testing.T, tc *testContext, ns string, name string, node }); err != nil { t.Fatalf("failed to register worker: %v", err) } + // As the syncer does once the pod is Ready. + if _, err := tc.client.MarkWorkerAvailable(context.Background(), &ateapipb.MarkWorkerAvailableRequest{ + Worker: &ateapipb.ObjectRef{Name: string(createdPod.UID)}, + }); err != nil { + t.Fatalf("failed to mark worker available: %v", err) + } reportWorkerCapacity(t, tc, string(createdPod.UID), 1) // Wait for the worker to appear in worker cache. diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go index 4535709f02..200a5d5035 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go @@ -94,7 +94,8 @@ func containsWorker(workers []*ateapipb.Worker, name string) bool { } // TestCreateAndGetWorker registers a Worker and reads it back, checking that -// the server assigns the metadata and the ACTIVE status a new Worker starts in. +// the server assigns the metadata and the UNAVAILABLE status a new Worker +// starts in. func TestCreateAndGetWorker(t *testing.T) { ns := namespaceForTest("ns-worker-create-get") tc := setupTest(t, ns) @@ -104,7 +105,7 @@ func TestCreateAndGetWorker(t *testing.T) { want := newTestWorker(ns) want.Metadata.Version = 1 - want.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE} + want.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE} if diff := cmp.Diff(want, created, protocmp.Transform(), ignoreServerMetadata); diff != "" { t.Errorf("CreateWorker response mismatch (-want +got):\n%s", diff) } @@ -153,8 +154,9 @@ func TestListWorkers(t *testing.T) { { Metadata: &ateapipb.ResourceMetadata{ Name: podUID, - // Two writes: the registration, then the capacity report. - Version: 2, + // Three writes: the registration, marking it available, then + // the capacity report. + Version: 3, }, WorkerNamespace: ns, WorkerPool: "pool1", @@ -241,7 +243,7 @@ func TestUpdateWorker(t *testing.T) { want := newTestWorker(ns) want.Metadata.Version = 2 want.Labels = map[string]string{"tier": "batch"} - want.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE} + want.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE} if diff := cmp.Diff(want, updated, protocmp.Transform(), ignoreServerMetadata); diff != "" { t.Errorf("UpdateWorker response mismatch (-want +got):\n%s", diff) } @@ -256,7 +258,7 @@ func TestDrainWorker(t *testing.T) { defer tc.cleanup() registerWorker(t, tc, ns) - waitForWorkerAvailable(t, tc, testWorkerName) + waitForWorkerState(t, tc, testWorkerName, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) drained, err := tc.client.DrainWorker(context.Background(), &ateapipb.DrainWorkerRequest{Worker: workerRef(testWorkerName)}) if err != nil { @@ -274,6 +276,41 @@ func TestDrainWorker(t *testing.T) { waitForWorkerState(t, tc, testWorkerName, ateapipb.WorkerState_WORKER_STATE_DRAINING) } +// TestMarkWorkerAvailability moves a Worker between ACTIVE and UNAVAILABLE, and +// checks each state reaches the worker cache the scheduler places actors from. +func TestMarkWorkerAvailability(t *testing.T) { + ns := namespaceForTest("ns-worker-availability") + tc := setupTest(t, ns) + defer tc.cleanup() + ctx := context.Background() + + registerWorker(t, tc, ns) + waitForWorkerState(t, tc, testWorkerName, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) + + available, err := tc.client.MarkWorkerAvailable(ctx, &ateapipb.MarkWorkerAvailableRequest{Worker: workerRef(testWorkerName)}) + if err != nil { + t.Fatalf("MarkWorkerAvailable failed: %v", err) + } + want := newTestWorker(ns) + want.Metadata.Version = 2 + want.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE} + if diff := cmp.Diff(want, available.GetWorker(), protocmp.Transform(), ignoreServerMetadata); diff != "" { + t.Errorf("MarkWorkerAvailable response mismatch (-want +got):\n%s", diff) + } + waitForWorkerState(t, tc, testWorkerName, ateapipb.WorkerState_WORKER_STATE_ACTIVE) + + unavailable, err := tc.client.MarkWorkerUnavailable(ctx, &ateapipb.MarkWorkerUnavailableRequest{Worker: workerRef(testWorkerName)}) + if err != nil { + t.Fatalf("MarkWorkerUnavailable failed: %v", err) + } + want.Metadata.Version = 3 + want.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE} + if diff := cmp.Diff(want, unavailable.GetWorker(), protocmp.Transform(), ignoreServerMetadata); diff != "" { + t.Errorf("MarkWorkerUnavailable response mismatch (-want +got):\n%s", diff) + } + waitForWorkerState(t, tc, testWorkerName, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) +} + // TestDeleteWorker deregisters a Worker and checks it is gone. The delete // drains the Worker on its way out, so the record it returns is that one rather // than the one Create stored. Deregistering is not silently idempotent, so a diff --git a/cmd/ateapi/internal/controlapi/hosting.go b/cmd/ateapi/internal/controlapi/hosting.go new file mode 100644 index 0000000000..1fa19663c9 --- /dev/null +++ b/cmd/ateapi/internal/controlapi/hosting.go @@ -0,0 +1,80 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package controlapi + +import ( + "context" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// HostingNode is the node that holds the actor, and "" when no node does. The +// actor's state decides which status field to trust, since both can outlive +// the hosting they record: pause, suspend, crash and revert release the +// worker (delete its row in the store's worker assignments) before they clear +// status.worker_assignment, and a crash keeps status.assigned_node as a record +// of the last node. The rules: +// - SUSPENDED or SUSPENDING with no worker and a local-only latest +// snapshot, the shape of a paused actor: assigned_node, the node holding +// that snapshot. +// - RESUMING, RUNNING and SUSPENDING with status.worker_assignment set: the +// worker's node, but only while the store's worker assignment row binds +// that worker to this actor's UID; "" once it is released or the worker +// is gone. +// - Anything else: "". +// +// WorkerService uses it to refuse a node that asks for credentials for an +// actor it does not hold. +func HostingNode(ctx context.Context, st store.Interface, actor *ateapipb.Actor) (string, error) { + status := actor.GetStatus() + assignment := status.GetWorkerAssignment() + if assignment == nil { + switch status.GetState() { + case ateapipb.ActorState_ACTOR_STATE_SUSPENDED, ateapipb.ActorState_ACTOR_STATE_SUSPENDING: + if latestSnapshotIsLocalOnly(status) { + return status.GetAssignedNode(), nil + } + } + return "", nil + } + switch status.GetState() { + case ateapipb.ActorState_ACTOR_STATE_RESUMING, ateapipb.ActorState_ACTOR_STATE_RUNNING, + ateapipb.ActorState_ACTOR_STATE_SUSPENDING: + default: + return "", nil + } + workerName := assignment.GetWorker().GetName() + // The worker assignment, not the actor's status, binds the worker to the + // actor. The status keeps naming the worker after a release until the + // workflow that released it records the new state, and by then the worker + // may host another actor. + hosted, err := workerHostsActor(ctx, st, workerName, actor.GetMetadata().GetUid()) + if err != nil { + return "", err + } + if !hosted { + return "", nil + } + worker, err := st.GetWorker(ctx, workerName) + if errors.Is(err, store.ErrNotFound) { + return "", nil + } else if err != nil { + return "", fmt.Errorf("while retrieving worker %s: %w", workerName, err) + } + return worker.GetNodeName(), nil +} diff --git a/cmd/ateapi/internal/controlapi/hosting_test.go b/cmd/ateapi/internal/controlapi/hosting_test.go new file mode 100644 index 0000000000..585913fc95 --- /dev/null +++ b/cmd/ateapi/internal/controlapi/hosting_test.go @@ -0,0 +1,297 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package controlapi + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/scheduling" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storetest" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/workercache" + "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/grpc/codes" + "google.golang.org/protobuf/proto" +) + +// TestHostingNodeAfterWorkerRelease drives each workflow that releases an +// actor's worker before it rewrites the actor's status into the window +// between the two, by failing the second write. The status still names the +// worker there, and keeps doing so until the workflow is retried. The worker's +// node must not count as hosting the actor in that window, nor once the worker +// hosts another actor. +func TestHostingNodeAfterWorkerRelease(t *testing.T) { + const node = "node1" + gvisorTemplate := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}} + tests := []struct { + name string + state ateapipb.ActorState + // snapshot, when set, is the actor's latest snapshot. + snapshot func(t *testing.T) *ateapipb.Snapshot + // workerClass is the worker's sandbox class. A template asks for + // gVisor, so "microvm" makes the worker ineligible. + workerClass string + // finalize runs the step that releases the worker and then records + // the actor's new state against st. + finalize func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, actor *ateapipb.Actor) error + // wantRetry is the code of the retried finalize against the real + // store. + wantRetry codes.Code + // wantFinalState and wantFinalNode are the actor's state and + // HostingNode once the retry commits. + wantFinalState ateapipb.ActorState + wantFinalNode string + }{ + { + name: "pause", + state: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + snapshot: func(*testing.T) *ateapipb.Snapshot { + // A pause in progress: node-local and not yet captured. + return newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-snap-1", "") + }, + workerClass: "gvisor", + finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { + _, err := (&ActorWorkflow{store: st}).ensurePausedFinalized(ctx, actorRef) + return err + }, + wantRetry: codes.OK, + wantFinalState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, + // Paused: the node holding the local snapshot. + wantFinalNode: node, + }, + { + name: "suspend", + state: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + snapshot: func(t *testing.T) *ateapipb.Snapshot { + return newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot-1", someActorSnapshotURI(t, testStorageLocation, "team-a", "snapshot-1"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) + }, + workerClass: "gvisor", + finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { + _, err := (&ActorWorkflow{store: st}).ensureSuspendedFinalized(ctx, actorRef) + return err + }, + wantRetry: codes.OK, + wantFinalState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, + }, + { + name: "crash", + state: ateapipb.ActorState_ACTOR_STATE_RUNNING, + workerClass: "gvisor", + finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { + return crashActor(ctx, st, actorRef, ateattr.OperationResume, "test crash") + }, + wantRetry: codes.OK, + wantFinalState: ateapipb.ActorState_ACTOR_STATE_CRASHED, + }, + { + // A resume that finds its worker ineligible releases it, then + // crashes the actor. The retry finds the worker no longer hosts + // the actor and crashes it. + name: "resume onto an ineligible worker", + state: ateapipb.ActorState_ACTOR_STATE_RESUMING, + workerClass: "microvm", + finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, actor *ateapipb.Actor) error { + _, err := (&ActorWorkflow{store: st, scheduler: scheduling.New(nil)}).validateAssignedWorker(ctx, actorRef, actor, gvisorTemplate) + return err + }, + wantRetry: codes.Aborted, + wantFinalState: ateapipb.ActorState_ACTOR_STATE_CRASHED, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} + workerName := testWorkerUID("worker-pod-1") + if _, err := persistence.CreateWorker(ctx, &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: workerName}, + WorkerNamespace: "default", + WorkerPool: "pool1", + WorkerPod: "worker-pod-1", + WorkerPodUid: workerName, + NodeName: node, + SandboxClass: tc.workerClass, + Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}, + }); err != nil { + t.Fatalf("CreateWorker: %v", err) + } + status := &ateapipb.ActorStatus{ + State: tc.state, + AssignedNode: node, + WorkerAssignment: &ateapipb.WorkerAssignment{ + Worker: &ateapipb.ObjectRef{Name: workerName}, + WorkerNamespace: "default", + WorkerPool: "pool1", + WorkerPod: "worker-pod-1", + WorkerPodUid: workerName, + NodeName: node, + }, + LastAssignedGeneration: 1, + } + if tc.snapshot != nil { + status.Snapshots = []*ateapipb.Snapshot{tc.snapshot(t)} + } + created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Status: status, + }) + uid := created.GetMetadata().GetUid() + seedAssignment(t, persistence, workerName, &ateapipb.ActorAssignment{ + Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorUid: uid, + }) + + hostingNode := func(stage string) (string, *ateapipb.Actor) { + t.Helper() + actor, err := persistence.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + got, err := HostingNode(ctx, persistence, actor) + if err != nil { + t.Fatalf("%s: HostingNode: %v", stage, err) + } + t.Logf("%s: actor_state=%s status_worker_set=%t hosting_node=%q", + stage, actor.GetStatus().GetState(), actor.GetStatus().GetWorkerAssignment() != nil, got) + return got, actor + } + + got, actor := hostingNode("hosted") + if got != node { + t.Fatalf("HostingNode while the worker holds the actor = %q, want %q", got, node) + } + + // The release commits; recording the new state fails. + if err := tc.finalize(ctx, conflictingUpdateStore{Interface: persistence}, actorRef, actor); err == nil { + t.Fatal("finalize succeeded through a store that fails every actor update") + } + if _, err := persistence.GetWorkerAssignment(ctx, workerName, uid); !errors.Is(err, store.ErrNotFound) { + t.Fatalf("worker assignment after release: err %v, want ErrNotFound", err) + } + got, actor = hostingNode("released, status not finalized") + if actor.GetStatus().GetState() != tc.state || actor.GetStatus().GetWorkerAssignment() == nil { + t.Fatalf("actor after the failed update: state %v, worker assignment %v; want %v naming the worker", actor.GetStatus().GetState(), actor.GetStatus().GetWorkerAssignment(), tc.state) + } + if got != "" { + t.Errorf("HostingNode after the worker was released = %q, want \"\"", got) + } + + // The worker takes another actor. + seedAssignment(t, persistence, workerName, &ateapipb.ActorAssignment{ + Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: "actor-2"}, + ActorUid: "5c1d2e3f-4a5b-4c6d-8e7f-9a0b1c2d3e4f", + }) + if got, _ := hostingNode("worker rebooked for another actor"); got != "" { + t.Errorf("HostingNode after the worker was rebooked = %q, want \"\"", got) + } + + // A retry records the new state. + if err := tc.finalize(ctx, persistence, actorRef, actor); apierror.Code(err) != tc.wantRetry { + t.Fatalf("finalize retry: code %v, want %v (err: %v)", apierror.Code(err), tc.wantRetry, err) + } + got, actor = hostingNode("finalized") + if actor.GetStatus().GetState() != tc.wantFinalState { + t.Errorf("actor state once finalized = %v, want %v", actor.GetStatus().GetState(), tc.wantFinalState) + } + if got != tc.wantFinalNode { + t.Errorf("HostingNode once finalized = %q, want %q", got, tc.wantFinalNode) + } + }) + } +} + +// TestHostingNodeAfterResumeOntoNewWorker verifies that a resume moves the +// hosting node to the worker it binds: the bind commits before the status +// names the worker, so the worker assignment check never refuses the new +// worker's node, and the worker the actor was claimed by before is refused. +func TestHostingNodeAfterResumeOntoNewWorker(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + + actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, + }) + // stale-pod on node1 holds a claim from a failed attempt but is no longer + // eligible; free-pod on node2 is eligible and free. + stale := &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("stale-pod")}, + WorkerNamespace: "worker-ns", + WorkerPool: "pool-a", + WorkerPod: "stale-pod", + WorkerPodUid: testWorkerUID("stale-pod"), + NodeName: "node1", + SandboxClass: "microvm", + Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}, + } + free := &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("free-pod")}, + WorkerNamespace: "worker-ns", + WorkerPool: "pool-b", + WorkerPod: "free-pod", + WorkerPodUid: testWorkerUID("free-pod"), + NodeName: "node2", + SandboxClass: "gvisor", + Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}, + } + for _, w := range []*ateapipb.Worker{stale, free} { + if _, err := persistence.CreateWorker(ctx, w); err != nil { + t.Fatalf("CreateWorker(%s): %v", w.GetWorkerPod(), err) + } + } + seedAssignment(t, persistence, testWorkerUID("stale-pod"), &ateapipb.ActorAssignment{ + Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorUid: actor.GetMetadata().GetUid(), + }) + + cacheCtx, cancel := context.WithCancel(ctx) + defer cancel() + wc := workercache.New(persistence, time.Minute) + if err := wc.Start(cacheCtx); err != nil { + t.Fatalf("workercache.Start: %v", err) + } + w := &ActorWorkflow{store: persistence, workerCache: wc, scheduler: scheduling.New(wc)} + tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}} + resuming, _, err := w.assignWorkerAttempt(ctx, actorRef, actor, tmpl) + if err != nil { + t.Fatalf("assignWorkerAttempt: %v", err) + } + if got := resuming.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_RESUMING { + t.Fatalf("actor state after assignment = %v, want RESUMING", got) + } + + got, err := HostingNode(ctx, persistence, resuming) + if err != nil { + t.Fatalf("HostingNode: %v", err) + } + if got != "node2" { + t.Errorf("HostingNode after resuming onto free-pod = %q, want node2", got) + } + + // A status naming the released stale worker is refused. + onStale := proto.CloneOf(resuming) + onStale.Status.WorkerAssignment = &ateapipb.WorkerAssignment{Worker: &ateapipb.ObjectRef{Name: testWorkerUID("stale-pod")}} + if got, err := HostingNode(ctx, persistence, onStale); err != nil || got != "" { + t.Errorf("HostingNode naming the released stale worker = %q, %v; want \"\", nil", got, err) + } +} diff --git a/cmd/ateapi/internal/controlapi/metrics.go b/cmd/ateapi/internal/controlapi/metrics.go index e0c1a1405e..709b272a56 100644 --- a/cmd/ateapi/internal/controlapi/metrics.go +++ b/cmd/ateapi/internal/controlapi/metrics.go @@ -98,10 +98,10 @@ func RegisterWorkerCount(meter metric.Meter, workers func() ([]*ateapipb.Worker, } for _, w := range ws { state := workerState(w) - // CreateWorker does not validate the class, thus a worker can have an - // empty one. Report it as unknown, not as the pool's class: the - // scheduler puts no actor on a worker that offers no class, thus the - // pool's series must not count it as free capacity. + // CreateWorker accepts an empty class, as the field is optional. + // Report it as unknown, not as the pool's class: the scheduler puts + // no actor on a worker that offers no class, thus the pool's series + // must not count it as free capacity. class := ateattr.NormalizeSandboxClass(w.GetSandboxClass()) tally[key{w.GetWorkerNamespace(), w.GetWorkerPool(), state, class}]++ } @@ -219,7 +219,7 @@ func lifecycleOpAttrs(actor *ateapipb.Actor, template *ateapipb.ActorTemplate, s ass := actor.GetStatus().GetWorkerAssignment() attrs = append(attrs, ateattr.WorkerPoolAttributes(ass.GetWorkerNamespace(), ass.GetWorkerPool())...) if template != nil { - attrs = append(attrs, ateattr.SandboxClassAttribute(sandboxClassString(template.GetSandboxConfig().GetSandboxClass()))) + attrs = append(attrs, ateattr.SandboxClassAttribute(template.GetSandboxConfig().GetSandboxClass())) } if snapshotKind != "" { attrs = append(attrs, ateattr.SnapshotKindKey.String(snapshotKind)) @@ -245,7 +245,7 @@ func (i *Instruments) recordSchedulerAssignment(ctx context.Context, elapsed tim attrs = append(attrs, ateattr.SchedulerOutcomeKey.String(outcome)) attrs = append(attrs, ateattr.WorkerPoolAttributes(poolNamespace, pool)...) if class != "" { - attrs = append(attrs, ateattr.SandboxClassKey.String(class)) + attrs = append(attrs, ateattr.SandboxClassAttribute(class)) } if outcome == ateattr.SchedulerOutcomeError && err != nil { attrs = append(attrs, ateattr.ErrorTypeKey.String(apierror.Code(err).String())) diff --git a/cmd/ateapi/internal/controlapi/metrics_test.go b/cmd/ateapi/internal/controlapi/metrics_test.go index c0b191e427..6c67bfb96c 100644 --- a/cmd/ateapi/internal/controlapi/metrics_test.go +++ b/cmd/ateapi/internal/controlapi/metrics_test.go @@ -246,7 +246,7 @@ func TestLifecycleOpDurationShape(t *testing.T) { }, } template := &ateapipb.ActorTemplate{ - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, } inst.recordLifecycleOp(context.Background(), ateattr.OperationResume, time.Now(), nil, lifecycleOpAttrs(actor, template, ateattr.SnapshotKindLatest, ateattr.SnapshotFidelityVolumes)...) diff --git a/cmd/ateapi/internal/controlapi/sandbox_assets.go b/cmd/ateapi/internal/controlapi/sandbox_assets.go index b7aa641cce..ab3b838f54 100644 --- a/cmd/ateapi/internal/controlapi/sandbox_assets.go +++ b/cmd/ateapi/internal/controlapi/sandbox_assets.go @@ -40,7 +40,7 @@ func resolveTemplateSandboxConfig( if err != nil { return nil, fmt.Errorf("while getting SandboxConfig %q: %w", name, err) } - if class := sandboxClassString(templateSandbox.GetSandboxClass()); string(sc.Spec.SandboxClass) != class { + if class := templateSandbox.GetSandboxClass(); string(sc.Spec.SandboxClass) != class { return nil, apierror.FailedPrecondition( "SandboxConfig %q has class %q but sandbox_config.sandbox_class is %q", name, sc.Spec.SandboxClass, class) } @@ -56,8 +56,8 @@ func resolveSandboxAssets( sandboxConfigLister listersv1alpha1.SandboxConfigLister, templateSandbox *ateapipb.SandboxConfig, ) (*ateletpb.SandboxAssets, error) { - if sandboxClassString(templateSandbox.GetSandboxClass()) == "" { - return nil, fmt.Errorf("ActorTemplate names unrecognized sandbox_class %v", templateSandbox.GetSandboxClass()) + if templateSandbox.GetSandboxClass() == "" { + return nil, fmt.Errorf("ActorTemplate names no sandbox_config.sandbox_class") } if templateSandbox.GetConfigName() == "" { return nil, fmt.Errorf("ActorTemplate names no sandbox_config.config_name") diff --git a/cmd/ateapi/internal/controlapi/sandbox_assets_test.go b/cmd/ateapi/internal/controlapi/sandbox_assets_test.go index a9a98e3a8b..3cd70e2329 100644 --- a/cmd/ateapi/internal/controlapi/sandbox_assets_test.go +++ b/cmd/ateapi/internal/controlapi/sandbox_assets_test.go @@ -97,48 +97,52 @@ func TestResolveSandboxAssets(t *testing.T) { }{{ name: "named config", sandbox: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-custom", }, wantPauseImage: namedPause, }, { name: "named config without its default version", sandbox: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-no-default", }, wantErr: `SandboxConfig "gvisor-no-default" has no enabled version ""`, }, { name: "named config whose default version is disabled", sandbox: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-disabled-default", }, wantErr: `SandboxConfig "gvisor-disabled-default" has no enabled version "v1"`, }, { name: "named config class mismatch", sandbox: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM, + SandboxClass: "microvm", ConfigName: "gvisor-custom", }, wantErr: `has class "gvisor"`, }, { name: "missing named config", sandbox: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "does-not-exist", }, wantErr: `SandboxConfig "does-not-exist" not found`, }, { name: "unrecognized sandbox class", sandbox: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_UNSPECIFIED, + SandboxClass: "kata", ConfigName: "gvisor-custom", }, - wantErr: "unrecognized sandbox_class", + wantErr: `has class "gvisor" but sandbox_config.sandbox_class is "kata"`, + }, { + name: "empty sandbox class", + sandbox: &ateapipb.SandboxConfig{ConfigName: "gvisor-custom"}, + wantErr: "names no sandbox_config.sandbox_class", }, { name: "empty config name", - sandbox: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + sandbox: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, wantErr: "names no sandbox_config.config_name", }} for _, tt := range tests { diff --git a/cmd/ateapi/internal/controlapi/volumes.go b/cmd/ateapi/internal/controlapi/volumes.go index b2f8292599..22c65fbea7 100644 --- a/cmd/ateapi/internal/controlapi/volumes.go +++ b/cmd/ateapi/internal/controlapi/volumes.go @@ -29,7 +29,8 @@ import ( storagev1listers "k8s.io/client-go/listers/storage/v1" ) -// initialActorVolumes constructs initial volume objects in PENDING state before volume creation. +// initialActorVolumes constructs the actor's volume objects before volume +// creation. A volume has no storage volume ID until it is provisioned. func initialActorVolumes(ctx context.Context, scLister storagev1listers.StorageClassLister, template *ateapipb.ActorTemplate) ([]*ateapipb.ExternalVolume, error) { if template == nil { return nil, apierror.InvalidArgument("template is required") @@ -49,15 +50,14 @@ func initialActorVolumes(ctx context.Context, scLister storagev1listers.StorageC volumes = append(volumes, &ateapipb.ExternalVolume{ Name: vol.GetName(), VolumeType: sc.Provisioner, - Status: ateapipb.ExternalVolume_STATUS_PENDING, }) } } return volumes, nil } -// createActorVolumes provisions external volumes specified in volumesToCreate using the provided volume plugin. -// It returns the list of external volumes (with updated status and storage IDs), or an error if any creation fails. +// createActorVolumes provisions the external volumes in volumesToCreate that have no storage volume ID, using the provided volume plugin. +// It returns the list of external volumes (with storage IDs), or an error if any creation fails. // Any volumes processed before or during a failure are returned alongside the error so they can be persisted on the actor. func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLister storagev1listers.StorageClassLister, actorUID string, template *ateapipb.ActorTemplate, volumesToCreate []*ateapipb.ExternalVolume) (resultVolumes []*ateapipb.ExternalVolume, err error) { resultVolumes = make([]*ateapipb.ExternalVolume, 0, len(volumesToCreate)) @@ -86,16 +86,9 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi return resultVolumes, apierror.NotFound("volume %q not found in template", volName) } - switch vol.GetStatus() { - case ateapipb.ExternalVolume_STATUS_PENDING: - // proceed with volume creation - case ateapipb.ExternalVolume_STATUS_CREATED: + if vol.GetStorageVolumeId() != "" { resultVolumes = append(resultVolumes, vol) continue - case ateapipb.ExternalVolume_STATUS_DELETING: - return resultVolumes, apierror.FailedPrecondition("cannot create volume %q in DELETING status", volName) - default: - return resultVolumes, apierror.Internal("unexpected status %s for volume %q", vol.GetStatus(), volName) } actVolID := actorVolumeID(actorUID, volName) @@ -127,11 +120,15 @@ func createActorVolumes(ctx context.Context, registry VolumePluginRegistry, scLi return resultVolumes, apierror.Internal("failed to create volume %q: %v", specVol.GetName(), volErr) } + if resp.VolumeID == "" { + // An empty ID would read as not yet provisioned and be created again. + return resultVolumes, apierror.Internal("volume plugin returned no ID for volume %q", specVol.GetName()) + } + resultVolumes = append(resultVolumes, &ateapipb.ExternalVolume{ Name: volName, StorageVolumeId: resp.VolumeID, VolumeType: sc.Provisioner, - Status: ateapipb.ExternalVolume_STATUS_CREATED, VolumeContext: resp.VolumeContext, }) } @@ -222,7 +219,7 @@ func detachActorVolumes(ctx context.Context, registry VolumePluginRegistry, acto var errs []error for _, vol := range volumesToDetach { // StorageVolumeId is only populated once the volume is provisioned. - // Skip volumes that were never created (e.g. failed during PENDING state). + // Skip volumes that were never created (e.g. creation failed). if vol.GetStorageVolumeId() == "" { slog.WarnContext(ctx, "Volume has no storage volume ID, skipping detach", slog.String("volume_name", vol.GetName()), slog.String("actor_id", actor.GetMetadata().GetName())) continue diff --git a/cmd/ateapi/internal/controlapi/volumes_test.go b/cmd/ateapi/internal/controlapi/volumes_test.go index 5fb207c11d..e9be56827d 100644 --- a/cmd/ateapi/internal/controlapi/volumes_test.go +++ b/cmd/ateapi/internal/controlapi/volumes_test.go @@ -71,7 +71,6 @@ func TestActorVolumesStorageClassErrors(t *testing.T) { volumes := []*ateapipb.ExternalVolume{{ Name: "data-vol", VolumeType: "mock-standard", - Status: ateapipb.ExternalVolume_STATUS_PENDING, }} for _, tt := range []struct { @@ -116,7 +115,7 @@ func TestActorVolumesStorageClassErrors(t *testing.T) { } } -func TestInitialActorVolumes_PendingState(t *testing.T) { +func TestInitialActorVolumes(t *testing.T) { tmpl := &ateapipb.ActorTemplate{ Volumes: []*ateapipb.Volume{ { @@ -145,12 +144,10 @@ func TestInitialActorVolumes_PendingState(t *testing.T) { { Name: "data-vol-1", VolumeType: "mock-standard", - Status: ateapipb.ExternalVolume_STATUS_PENDING, }, { Name: "data-vol-2", VolumeType: "mock-fast", - Status: ateapipb.ExternalVolume_STATUS_PENDING, }, } @@ -224,19 +221,10 @@ func TestCreateActorVolumes(t *testing.T) { name: "partial failure returns error and preserves succeeded, failed, and remaining volumes", tmpl: multiVolTmpl, inputVolumes: []*ateapipb.ExternalVolume{ - { - Name: "vol1", - VolumeType: "mock-standard", - Status: ateapipb.ExternalVolume_STATUS_PENDING, - }, - { - Name: "vol2", - Status: ateapipb.ExternalVolume_STATUS_DELETING, - }, - { - Name: "vol3", - Status: ateapipb.ExternalVolume_STATUS_PENDING, - }, + {Name: "vol1", VolumeType: "mock-standard"}, + // Its type differs from its StorageClass's provisioner, so creating it fails. + {Name: "vol2", VolumeType: "mock-fast"}, + {Name: "vol3", VolumeType: "mock-standard"}, }, wantErr: true, wantRes: []*ateapipb.ExternalVolume{ @@ -244,26 +232,18 @@ func TestCreateActorVolumes(t *testing.T) { Name: "vol1", StorageVolumeId: "mock-vol-substrate-actor-uid-123-vol1", VolumeType: "mock-standard", - Status: ateapipb.ExternalVolume_STATUS_CREATED, - }, - { - Name: "vol2", - Status: ateapipb.ExternalVolume_STATUS_DELETING, - }, - { - Name: "vol3", - Status: ateapipb.ExternalVolume_STATUS_PENDING, }, + {Name: "vol2", VolumeType: "mock-fast"}, + {Name: "vol3", VolumeType: "mock-standard"}, }, }, { - name: "created volume status succeeds", + name: "volume with a storage volume id is not created again", tmpl: standardTmpl, inputVolumes: []*ateapipb.ExternalVolume{ { Name: "data-vol", StorageVolumeId: "existing-vol-id", - Status: ateapipb.ExternalVolume_STATUS_CREATED, }, }, wantErr: false, @@ -271,24 +251,6 @@ func TestCreateActorVolumes(t *testing.T) { { Name: "data-vol", StorageVolumeId: "existing-vol-id", - Status: ateapipb.ExternalVolume_STATUS_CREATED, - }, - }, - }, - { - name: "unspecified volume status returns error", - tmpl: standardTmpl, - inputVolumes: []*ateapipb.ExternalVolume{ - { - Name: "data-vol", - Status: ateapipb.ExternalVolume_STATUS_UNSPECIFIED, - }, - }, - wantErr: true, - wantRes: []*ateapipb.ExternalVolume{ - { - Name: "data-vol", - Status: ateapipb.ExternalVolume_STATUS_UNSPECIFIED, }, }, }, @@ -296,17 +258,11 @@ func TestCreateActorVolumes(t *testing.T) { name: "volume not found in template returns error", tmpl: &ateapipb.ActorTemplate{}, inputVolumes: []*ateapipb.ExternalVolume{ - { - Name: "missing-vol", - Status: ateapipb.ExternalVolume_STATUS_PENDING, - }, + {Name: "missing-vol"}, }, wantErr: true, wantRes: []*ateapipb.ExternalVolume{ - { - Name: "missing-vol", - Status: ateapipb.ExternalVolume_STATUS_PENDING, - }, + {Name: "missing-vol"}, }, }, { @@ -316,7 +272,6 @@ func TestCreateActorVolumes(t *testing.T) { { Name: "data-vol", VolumeType: "mock-standard", - Status: ateapipb.ExternalVolume_STATUS_PENDING, }, }, storageClasses: map[string]*storagev1.StorageClass{ @@ -335,7 +290,6 @@ func TestCreateActorVolumes(t *testing.T) { Name: "data-vol", StorageVolumeId: "mock-vol-substrate-actor-uid-123-data-vol", VolumeType: "mock-standard", - Status: ateapipb.ExternalVolume_STATUS_CREATED, VolumeContext: map[string]string{ "type": "pd-ssd", "csi.storage.k8s.io/fstype": "ext4", @@ -379,6 +333,40 @@ func TestCreateActorVolumes(t *testing.T) { } } +// emptyIDVolumePlugin provisions volumes without returning an ID. +type emptyIDVolumePlugin struct { + volume.VolumePluginControlPlane +} + +func (emptyIDVolumePlugin) CreateVolume(context.Context, volume.CreateVolumeRequest) (volume.CreateVolumeResponse, error) { + return volume.CreateVolumeResponse{}, nil +} + +func TestCreateActorVolumesRejectsEmptyPluginID(t *testing.T) { + tmpl := &ateapipb.ActorTemplate{ + Volumes: []*ateapipb.Volume{{ + Name: "data-vol", + ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard"}, + }}, + } + registry := &mockPluginRegistry{ + plugins: map[string]volume.VolumePluginControlPlane{"mock-standard": emptyIDVolumePlugin{}}, + } + scLister := &fakeStorageClassLister{storageClasses: map[string]*storagev1.StorageClass{ + "standard": {ObjectMeta: metav1.ObjectMeta{Name: "standard"}, Provisioner: "mock-standard"}, + }} + input := []*ateapipb.ExternalVolume{{Name: "data-vol", VolumeType: "mock-standard"}} + + res, err := createActorVolumes(context.Background(), registry, scLister, "actor-uid-123", tmpl, input) + if got := apierror.Code(err); got != codes.Internal { + t.Fatalf("createActorVolumes() code = %v, want %v; error = %v", got, codes.Internal, err) + } + // The volume stays unprovisioned, so a retry creates it again. + if diff := cmp.Diff(input, res, protocmp.Transform()); diff != "" { + t.Errorf("createActorVolumes() mismatch (-want +got):\n%s", diff) + } +} + type trackingVolumePlugin struct { volume.VolumePluginControlPlane deletedIDs []string @@ -409,10 +397,10 @@ func TestDeleteActorVolumes(t *testing.T) { wantErr: false, }, { - name: "falls back to actorVolumeID when storage volume ID is empty regardless of status", + name: "falls back to actorVolumeID when storage volume ID is empty", actorUID: "uid-abc", volumes: []*ateapipb.ExternalVolume{ - {Name: "vol1", StorageVolumeId: "", Status: ateapipb.ExternalVolume_STATUS_CREATED, VolumeType: "mock"}, + {Name: "vol1", StorageVolumeId: "", VolumeType: "mock"}, }, wantDeleted: []string{"substrate-uid-abc-vol1"}, wantErr: false, diff --git a/cmd/ateapi/internal/controlapi/worker.go b/cmd/ateapi/internal/controlapi/worker.go index 247376b4b5..0e836c4588 100644 --- a/cmd/ateapi/internal/controlapi/worker.go +++ b/cmd/ateapi/internal/controlapi/worker.go @@ -122,12 +122,11 @@ func (s *RPCService) CreateWorker(ctx context.Context, req *ateapipb.CreateWorke } func (s *ServiceImpl) CreateWorker(ctx context.Context, inWorker *ateapipb.Worker) (*ateapipb.Worker, error) { - // A Worker is registered only once its pod is Ready and has an IP, which - // makes ACTIVE the only state it can be born in. + // A Worker takes no Actors until MarkWorkerAvailable. outWorker := proto.CloneOf(inWorker) // A new Worker hosts no Actors, so none are left from an earlier epoch. outWorker.Status = &ateapipb.WorkerStatus{ - State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, + State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, ObservedEpoch: inWorker.GetEpoch(), } @@ -255,21 +254,7 @@ func (s *RPCService) DrainWorker(ctx context.Context, req *ateapipb.DrainWorkerR if errs := apivalidation.ValidateDrainWorkerRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } - name := req.GetWorker().GetName() - - // A DrainWorkerRequest names a worker and carries no guards, so the ones - // the store requires come from a read here rather than from the client. A - // write that lands in between is reported as a conflict for the caller to - // retry, the same as any other guarded update. - observed, err := s.impl.GetWorker(ctx, name) - if errors.Is(err, store.ErrNotFound) { - return nil, apierror.NotFound("Worker %s not found", name) - } - if err != nil { - return nil, fmt.Errorf("while getting worker to drain: %w", err) - } - - drained, err := s.mutateWorker(ctx, name, store.PreconditionFrom(observed), func(toUpdate *ateapipb.Worker) error { + drained, err := s.mutateNamedWorker(ctx, req.GetWorker().GetName(), func(toUpdate *ateapipb.Worker) error { if toUpdate.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_DRAINING { // already draining, do nothing return &workerUnchanged{worker: proto.Clone(toUpdate).(*ateapipb.Worker)} @@ -285,6 +270,66 @@ func (s *RPCService) DrainWorker(ctx context.Context, req *ateapipb.DrainWorkerR return &ateapipb.DrainWorkerResponse{Worker: drained}, nil } +func (s *RPCService) MarkWorkerAvailable(ctx context.Context, req *ateapipb.MarkWorkerAvailableRequest) (*ateapipb.MarkWorkerAvailableResponse, error) { + if errs := apivalidation.ValidateMarkWorkerAvailableRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) + } + marked, err := s.mutateNamedWorker(ctx, req.GetWorker().GetName(), func(toUpdate *ateapipb.Worker) error { + switch toUpdate.GetStatus().GetState() { + case ateapipb.WorkerState_WORKER_STATE_ACTIVE: + return &workerUnchanged{worker: proto.Clone(toUpdate).(*ateapipb.Worker)} + case ateapipb.WorkerState_WORKER_STATE_DRAINING: + return errWorkerDraining + } + toUpdate.Status.State = ateapipb.WorkerState_WORKER_STATE_ACTIVE + return nil + }) + if err != nil { + return nil, err + } + return &ateapipb.MarkWorkerAvailableResponse{Worker: marked}, nil +} + +func (s *RPCService) MarkWorkerUnavailable(ctx context.Context, req *ateapipb.MarkWorkerUnavailableRequest) (*ateapipb.MarkWorkerUnavailableResponse, error) { + if errs := apivalidation.ValidateMarkWorkerUnavailableRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) + } + marked, err := s.mutateNamedWorker(ctx, req.GetWorker().GetName(), func(toUpdate *ateapipb.Worker) error { + switch toUpdate.GetStatus().GetState() { + case ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE: + return &workerUnchanged{worker: proto.Clone(toUpdate).(*ateapipb.Worker)} + case ateapipb.WorkerState_WORKER_STATE_DRAINING: + return errWorkerDraining + } + toUpdate.Status.State = ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE + // The assignments are left alone, as for draining. + return nil + }) + if err != nil { + return nil, err + } + return &ateapipb.MarkWorkerUnavailableResponse{Worker: marked}, nil +} + +// errWorkerDraining refuses to move a draining Worker out of DRAINING, which +// is final. +var errWorkerDraining = errors.New("worker is draining") + +// mutateNamedWorker runs mutate against the named Worker for a request that +// carries no guards, so the ones the store requires come from a read here +// rather than from the client. A write that lands in between is reported as a +// conflict for the caller to retry, the same as any other guarded update. +func (s *RPCService) mutateNamedWorker(ctx context.Context, name string, mutate func(toUpdate *ateapipb.Worker) error) (*ateapipb.Worker, error) { + observed, err := s.impl.GetWorker(ctx, name) + if errors.Is(err, store.ErrNotFound) { + return nil, apierror.NotFound("Worker %s not found", name) + } + if err != nil { + return nil, fmt.Errorf("while getting worker %s: %w", name, err) + } + return s.mutateWorker(ctx, name, store.PreconditionFrom(observed), mutate) +} + // mutateWorker runs mutate against the named Worker and translates what comes // back into the RPC's result. A mutation that found nothing to do reports the // Worker it saw; anything else is a store error. @@ -299,6 +344,8 @@ func (s *RPCService) mutateWorker(ctx context.Context, name string, precondition return unchanged.worker, nil } switch { + case errors.Is(err, errWorkerDraining): + return nil, apierror.FailedPrecondition("Worker %s is draining", name) case errors.Is(err, store.ErrNotFound): return nil, apierror.NotFound("Worker %s not found", name) case errors.Is(err, store.ErrUIDConflict): diff --git a/cmd/ateapi/internal/controlapi/worker_test.go b/cmd/ateapi/internal/controlapi/worker_test.go index 78e8cddd10..dc395e12ea 100644 --- a/cmd/ateapi/internal/controlapi/worker_test.go +++ b/cmd/ateapi/internal/controlapi/worker_test.go @@ -232,10 +232,9 @@ func TestCreateWorker(t *testing.T) { if got.GetMetadata().GetUid() == "" { t.Error("created worker has no uid; the store is meant to assign one") } - // A Worker is registered only once its pod is Ready and has an IP, which - // makes ACTIVE the only state it can be born in. - if got.GetStatus().GetState() != ateapipb.WorkerState_WORKER_STATE_ACTIVE { - t.Errorf("created worker state = %v, want %v", got.GetStatus().GetState(), ateapipb.WorkerState_WORKER_STATE_ACTIVE) + // A Worker takes no Actors until it is marked available. + if got.GetStatus().GetState() != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + t.Errorf("created worker state = %v, want %v", got.GetStatus().GetState(), ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) } stored, err := persistence.GetWorker(ctx, apiWorkerName) @@ -260,7 +259,7 @@ func TestCreateWorker_IgnoresRequestStatus(t *testing.T) { if err != nil { t.Fatalf("CreateWorker() failed: %v", err) } - want := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE} + want := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE} if diff := cmp.Diff(want, got.GetStatus(), protocmp.Transform()); diff != "" { t.Errorf("created worker status mismatch (-want +got):\n%s", diff) } @@ -675,6 +674,127 @@ func TestDrainWorker_Errors(t *testing.T) { } } +// A created Worker is marked available, back unavailable, and available again. +// A repeated mark is a no-op and must not bump the version: the syncer re-drives +// it on every pod event. +func TestMarkWorkerAvailability(t *testing.T) { + ctx := context.Background() + svc, _ := newWorkerAPIService(t) + created, err := svc.CreateWorker(ctx, &ateapipb.CreateWorkerRequest{Worker: validWorker(apiWorkerName)}) + if err != nil { + t.Fatalf("CreateWorker() failed: %v", err) + } + + markAvailable := func() (*ateapipb.Worker, error) { + resp, err := svc.MarkWorkerAvailable(ctx, &ateapipb.MarkWorkerAvailableRequest{Worker: workerRef(apiWorkerName)}) + return resp.GetWorker(), err + } + markUnavailable := func() (*ateapipb.Worker, error) { + resp, err := svc.MarkWorkerUnavailable(ctx, &ateapipb.MarkWorkerUnavailableRequest{Worker: workerRef(apiWorkerName)}) + return resp.GetWorker(), err + } + version := created.GetMetadata().GetVersion() + for i, step := range []struct { + name string + mark func() (*ateapipb.Worker, error) + want ateapipb.WorkerState + changed bool + }{ + {"available", markAvailable, ateapipb.WorkerState_WORKER_STATE_ACTIVE, true}, + {"available again", markAvailable, ateapipb.WorkerState_WORKER_STATE_ACTIVE, false}, + {"unavailable", markUnavailable, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, true}, + {"unavailable again", markUnavailable, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, false}, + {"available once more", markAvailable, ateapipb.WorkerState_WORKER_STATE_ACTIVE, true}, + } { + got, err := step.mark() + if err != nil { + t.Fatalf("step %d (%s): failed: %v", i, step.name, err) + } + if state := got.GetStatus().GetState(); state != step.want { + t.Errorf("step %d (%s): state = %v, want %v", i, step.name, state, step.want) + } + if step.changed { + version++ + } + if v := got.GetMetadata().GetVersion(); v != version { + t.Errorf("step %d (%s): version = %d, want %d", i, step.name, v, version) + } + } +} + +// Marking leaves the bound Actor alone, as draining does. +func TestMarkWorkerUnavailable_KeepsAssignment(t *testing.T) { + ctx := context.Background() + svc, persistence := newWorkerAPIService(t) + seedAPIWorker(t, ctx, persistence, validWorker(apiWorkerName)) + assignAPIWorker(t, ctx, persistence, apiWorkerName, "actor-uid-1") + + got, err := svc.MarkWorkerUnavailable(ctx, &ateapipb.MarkWorkerUnavailableRequest{Worker: workerRef(apiWorkerName)}) + if err != nil { + t.Fatalf("MarkWorkerUnavailable() failed: %v", err) + } + if n := got.GetWorker().GetStatus().GetAllocated().GetActors(); n != 1 { + t.Errorf("unavailable worker hosts %d actors, want the 1 left in place", n) + } +} + +// DRAINING is final: marking a draining Worker either way is refused and +// leaves it as it is. +func TestMarkWorker_Draining(t *testing.T) { + ctx := context.Background() + svc, persistence := newWorkerAPIService(t) + seedAPIWorker(t, ctx, persistence, validWorker(apiWorkerName)) + drained, err := svc.DrainWorker(ctx, &ateapipb.DrainWorkerRequest{Worker: workerRef(apiWorkerName)}) + if err != nil { + t.Fatalf("DrainWorker() failed: %v", err) + } + + _, err = svc.MarkWorkerUnavailable(ctx, &ateapipb.MarkWorkerUnavailableRequest{Worker: workerRef(apiWorkerName)}) + if got := apierror.Code(err); got != codes.FailedPrecondition { + t.Errorf("MarkWorkerUnavailable() on a draining worker code = %v (err %v), want %v", got, err, codes.FailedPrecondition) + } + _, err = svc.MarkWorkerAvailable(ctx, &ateapipb.MarkWorkerAvailableRequest{Worker: workerRef(apiWorkerName)}) + if got := apierror.Code(err); got != codes.FailedPrecondition { + t.Errorf("MarkWorkerAvailable() on a draining worker code = %v (err %v), want %v", got, err, codes.FailedPrecondition) + } + + stored, err := persistence.GetWorker(ctx, apiWorkerName) + if err != nil { + t.Fatalf("GetWorker() failed: %v", err) + } + if diff := cmp.Diff(drained.GetWorker(), stored, protocmp.Transform()); diff != "" { + t.Errorf("refused marks changed a draining worker (-drained +stored):\n%s", diff) + } +} + +func TestMarkWorker_Errors(t *testing.T) { + ctx := context.Background() + svc, _ := newWorkerAPIService(t) + + refs := []struct { + name string + ref *ateapipb.ObjectRef + want codes.Code + }{ + {"absent", workerRef(apiWorkerName), codes.NotFound}, + {"no ref", nil, codes.InvalidArgument}, + {"no name", &ateapipb.ObjectRef{}, codes.InvalidArgument}, + {"atespace set", &ateapipb.ObjectRef{Atespace: "team-a", Name: apiWorkerName}, codes.InvalidArgument}, + } + for _, tc := range refs { + t.Run(tc.name, func(t *testing.T) { + _, err := svc.MarkWorkerAvailable(ctx, &ateapipb.MarkWorkerAvailableRequest{Worker: tc.ref}) + if got := apierror.Code(err); got != tc.want { + t.Errorf("MarkWorkerAvailable() code = %v (err %v), want %v", got, err, tc.want) + } + _, err = svc.MarkWorkerUnavailable(ctx, &ateapipb.MarkWorkerUnavailableRequest{Worker: tc.ref}) + if got := apierror.Code(err); got != tc.want { + t.Errorf("MarkWorkerUnavailable() code = %v (err %v), want %v", got, err, tc.want) + } + }) + } +} + // TestServiceImplUpdateWorker_ImmutableFields pins the immutable-field rule at // the layer that now owns it: declarative validation in ServiceImpl, which // every write path shares. It moved up from the store contract when the store diff --git a/cmd/ateapi/internal/controlapi/workflow_delete.go b/cmd/ateapi/internal/controlapi/workflow_delete.go index e97af85ba5..c4c7ef0c0c 100644 --- a/cmd/ateapi/internal/controlapi/workflow_delete.go +++ b/cmd/ateapi/internal/controlapi/workflow_delete.go @@ -304,9 +304,9 @@ func (w *ActorWorkflow) ensureWorkerReleased(ctx context.Context, actorRef resou return latestActor, nil } -// ensureMarkedDeleting transitions the actor and its volumes to DELETING and -// persists the change, returning the stored copy. Skips when a previous -// attempt already marked the actor. +// ensureMarkedDeleting transitions the actor to DELETING and persists the +// change, returning the stored copy. Skips when a previous attempt already +// marked the actor. func (w *ActorWorkflow) ensureMarkedDeleting(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, anyState bool) (updated *ateapipb.Actor, err error) { ctx, done := stepSpan(ctx, "MarkDeleting") defer func() { err = done(err) }() @@ -331,9 +331,6 @@ func (w *ActorWorkflow) ensureMarkedDeleting(ctx context.Context, actorRef resou storedActor, err := w.store.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error { toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_DELETING - for _, vol := range toUpdate.GetStatus().GetExternalVolumes() { - vol.Status = ateapipb.ExternalVolume_STATUS_DELETING - } return nil }) if err != nil { diff --git a/cmd/ateapi/internal/controlapi/workflow_pause.go b/cmd/ateapi/internal/controlapi/workflow_pause.go index e8370a402b..d71edf3158 100644 --- a/cmd/ateapi/internal/controlapi/workflow_pause.go +++ b/cmd/ateapi/internal/controlapi/workflow_pause.go @@ -28,6 +28,7 @@ import ( "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "go.opentelemetry.io/otel/attribute" + "google.golang.org/protobuf/proto" ) // PauseActor executes the workflow to pause a running actor. Idempotent: @@ -140,14 +141,15 @@ func (w *ActorWorkflow) ensureMarkedPausing(ctx context.Context, actorRef resour // Increment last_assigned_generation for the new Pause request. gen := toUpdate.Status.LastAssignedGeneration + 1 toUpdate.Status.LastAssignedGeneration = gen - toUpdate.Status.Snapshots = append(toUpdate.Status.Snapshots, - newLocalSnapshot( - gen, - actorTemplate.GetSnapshotConfig().GetPreferredFidelity(), - actorTemplate.GetMetadata().GetUid(), - snapshotUUID, - "", - )) + snap := newLocalSnapshot( + gen, + actorTemplate.GetSnapshotConfig().GetPreferredFidelity(), + actorTemplate.GetMetadata().GetUid(), + snapshotUUID, + "", + ) + snap.SandboxRuntime = proto.CloneOf(toUpdate.GetStatus().GetWorkerAssignment().GetSandboxRuntime()) + toUpdate.Status.Snapshots = append(toUpdate.Status.Snapshots, snap) return nil }) if err != nil { diff --git a/cmd/ateapi/internal/controlapi/workflow_pause_test.go b/cmd/ateapi/internal/controlapi/workflow_pause_test.go index b11cbf7ced..052322754b 100644 --- a/cmd/ateapi/internal/controlapi/workflow_pause_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_pause_test.go @@ -24,7 +24,9 @@ import ( "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/google/go-cmp/cmp" "google.golang.org/grpc/codes" + "google.golang.org/protobuf/testing/protocmp" ) // TestEnsurePausedFinalized_WorkerGone reproduces the scenario where the worker @@ -225,6 +227,15 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} workerName := testWorkerUID("worker-pod-1") + wantRuntime := &ateapipb.SandboxRuntime{ + SandboxClass: "gvisor", + Version: &ateapipb.VersionedSandboxCompat{ + SchemaVersion: "v1", + Attributes: []*ateapipb.AttributeEntry{ + {Key: "architecture", Value: "amd64"}, + }, + }, + } created := storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, Status: &ateapipb.ActorStatus{ @@ -236,6 +247,7 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { WorkerPool: "pool1", WorkerPod: "worker-pod-1", WorkerPodUid: workerName, + SandboxRuntime: wantRuntime, }, }, }) @@ -246,7 +258,7 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { WorkerPod: "worker-pod-1", WorkerPodUid: workerName, NodeName: "node1", - Status: &ateapipb.WorkerStatus{}, + SandboxClass: "gvisor", }); err != nil { t.Fatalf("CreateWorker: %v", err) } @@ -285,6 +297,9 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { if localSnap.GetLocality() != "node1" { t.Errorf("local Snapshot.Locality = %q, want %q", localSnap.GetLocality(), "node1") } + if diff := cmp.Diff(wantRuntime, localSnap.GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("Snapshot.SandboxRuntime mismatch (-want +got):\n%s", diff) + } if got.GetStatus().GetAssignedNode() != "node1" { t.Errorf("AssignedNode = %q, want %q", got.GetStatus().GetAssignedNode(), "node1") } diff --git a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go index f6705c7e40..1cf4652f2e 100644 --- a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go @@ -600,7 +600,7 @@ func TestAssignWorkerAttempt_StampsWorkerEpoch(t *testing.T) { } w := &ActorWorkflow{store: persistence, workerCache: wc, scheduler: scheduling.New(wc)} - tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}} + tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}} stored, _, err := w.assignWorkerAttempt(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}, actor, tmpl) if err != nil { t.Fatalf("assignWorkerAttempt: %v", err) diff --git a/cmd/ateapi/internal/controlapi/workflow_resume.go b/cmd/ateapi/internal/controlapi/workflow_resume.go index bb42bc03a0..e8d1ef990b 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume.go @@ -26,10 +26,12 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/hardware" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/volume" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/wait" @@ -48,6 +50,9 @@ type resumeSnapshotSource struct { // TemplateReplaced is true when the external snapshot's recorded template // UID differs from the actor's current template. TemplateReplaced bool + // RequiredRuntime is the runtime resolved from the snapshot's guest state + // (via requiredSnapshotRuntime), or nil when guest state is not restored. + RequiredRuntime *ateapipb.SandboxRuntime } // restoreTelemetry labels the restore operation for the resume lifecycle @@ -173,6 +178,7 @@ func (w *ActorWorkflow) loadActorForResume(ctx context.Context, actorRef resourc if err != nil { return nil, nil, src, err } + src.RequiredRuntime = requiredSnapshotRuntime(actor, actorTemplate) if snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED); snap != nil { st := snap.GetDurableSnapshot() if src.SnapshotURI, err = resources.ParseSnapshotURI(st.GetObject().GetSnapshotUri()); err != nil { @@ -187,24 +193,24 @@ func (w *ActorWorkflow) loadActorForResume(ctx context.Context, actorRef resourc return actor, actorTemplate, src, nil } -// ensureVolumesCreated provisions any initial actor volumes that are in -// PENDING state, persisting the resulting volume state (even when creation +// ensureVolumesCreated provisions any actor volumes that have no storage +// volume ID yet, persisting the resulting volume state (even when creation // partially failed, so progress is not lost) and returning the stored copy. +// An actor being deleted gets no new volumes. func (w *ActorWorkflow) ensureVolumesCreated(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate) (_ *ateapipb.Actor, err error) { ctx, done := stepSpan(ctx, "CreateVolumes") defer func() { err = done(err) }() - pending := false - for _, vol := range actor.GetStatus().GetExternalVolumes() { - if vol.GetStatus() == ateapipb.ExternalVolume_STATUS_PENDING { - pending = true - break - } - } + pending := slices.ContainsFunc(actor.GetStatus().GetExternalVolumes(), func(vol *ateapipb.ExternalVolume) bool { + return vol.GetStorageVolumeId() == "" + }) if !pending { markSkipped(ctx, "no volumes awaiting creation") return actor, nil } + if st := actor.GetStatus().GetState(); st == ateapipb.ActorState_ACTOR_STATE_DELETING { + return nil, apierror.FailedPrecondition("cannot create volumes for Actor %s in state %s", actorRef, st) + } volumes, createErr := createActorVolumes(ctx, w.pluginRegistry, w.storageClassLister, actor.GetMetadata().GetUid(), actorTemplate, actor.GetStatus().GetExternalVolumes()) // createActorVolumes reports the state it got to even when it fails, so both @@ -338,7 +344,7 @@ var assignmentBackoff = wait.Backoff{ func (w *ActorWorkflow) recordExhaustedAssignment(ctx context.Context, elapsed time.Duration, actorTemplate *ateapipb.ActorTemplate, err error) { class := "" if actorTemplate != nil { - class = sandboxClassString(actorTemplate.GetSandboxConfig().GetSandboxClass()) + class = actorTemplate.GetSandboxConfig().GetSandboxClass() } w.instruments.recordSchedulerAssignment(ctx, elapsed, ateattr.SchedulerOutcomeError, "", "", class, err) } @@ -346,7 +352,8 @@ func (w *ActorWorkflow) recordExhaustedAssignment(ctx context.Context, elapsed t // validateAssignedWorker checks a RESUMING actor's persisted assignment // against the current worker record. Every invalid outcome crashes the actor: // a RESUMING actor whose worker vanished, drained, was reassigned, or is no -// longer eligible can never make progress on its own. +// longer eligible can never make progress on its own. A worker that is only +// unavailable is reported as UNAVAILABLE for the caller to retry. func (w *ActorWorkflow) validateAssignedWorker(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate) (*ateapipb.Worker, error) { assignment := actor.GetStatus().GetWorkerAssignment() if assignment == nil { @@ -379,6 +386,11 @@ func (w *ActorWorkflow) validateAssignedWorker(ctx context.Context, actorRef res } return nil, apierror.Aborted("actor %s crashed", actorRef.String()) } + // An unavailable worker may come back with its Actors intact. If its ateom + // restarted instead, the raised epoch has this Actor crashed for it. + if worker.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + return nil, apierror.Unavailable("worker %s assigned to actor %s is unavailable", worker.GetMetadata().GetName(), actorRef) + } // Verify the worker is still hosting this Actor. hosted, err := workerHostsActor(ctx, w.store, worker.GetMetadata().GetName(), actor.GetMetadata().GetUid()) if err != nil { @@ -396,7 +408,7 @@ func (w *ActorWorkflow) validateAssignedWorker(ctx context.Context, actorRef res if err != nil { return nil, err } - if !w.scheduler.Applies(worker, constraints) { + if !w.appliesWithFallback(worker, constraints, canRelaxRuntime(actor, constraints)) { slog.ErrorContext(ctx, "crashing actor because previously assigned worker is not eligible anymore") // If that worker's pool is no longer eligible (e.g. the actor's // worker_selector was updated after the failed attempt), release it back @@ -421,7 +433,7 @@ func admittedResources(constraints scheduling.Constraints) *ateapipb.Resources { // workerHoldingStaleClaim recovers a claim written before the Actor update. // It releases the claim if the Worker is no longer eligible. -func (w *ActorWorkflow) workerHoldingStaleClaim(ctx context.Context, actor *ateapipb.Actor, constraints scheduling.Constraints) (*ateapipb.Worker, error) { +func (w *ActorWorkflow) workerHoldingStaleClaim(ctx context.Context, actor *ateapipb.Actor, constraints scheduling.Constraints, relaxable bool) (*ateapipb.Worker, error) { actorUID := actor.GetMetadata().GetUid() workerName, err := w.store.FindWorkerHostingActor(ctx, actorUID) if err != nil { @@ -439,7 +451,7 @@ func (w *ActorWorkflow) workerHoldingStaleClaim(ctx context.Context, actor *atea } // The Actor's allocation makes HasRoom unsuitable for an existing claim. - if w.scheduler.Applies(worker, constraints) { + if w.appliesWithFallback(worker, constraints, relaxable) { return worker, nil } @@ -470,7 +482,7 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour pool := "" class := "" if actorTemplate != nil { - class = sandboxClassString(actorTemplate.GetSandboxConfig().GetSandboxClass()) + class = actorTemplate.GetSandboxConfig().GetSandboxClass() } defer func() { if schedulerRecordable(err) { @@ -482,13 +494,19 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour if err != nil { return nil, nil, err } + snapRuntime := constraints.RequiredRuntime + relaxable := canRelaxRuntime(actor, constraints) - assignedWorker, err := w.workerHoldingStaleClaim(ctx, actor, constraints) + assignedWorker, err := w.workerHoldingStaleClaim(ctx, actor, constraints, relaxable) if err != nil { return nil, nil, err } if assignedWorker == nil { pickedWorker, err := w.scheduler.Schedule(ctx, constraints) + if errors.Is(err, scheduling.ErrNoCapacity) && relaxable { + constraints.RequiredRuntime = nil + pickedWorker, err = w.scheduler.Schedule(ctx, constraints) + } if err != nil { if errors.Is(err, scheduling.ErrNoCapacity) { outcome = ateattr.SchedulerOutcomeNoCapacity @@ -499,6 +517,10 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour assignedWorker = pickedWorker slog.InfoContext(ctx, "Picked worker", slog.Any("worker", pickedWorker.String())) + } else if relaxable && !hardware.WorkerMatches(assignedWorker.GetStatus(), snapRuntime) { + // The stale claim only passed the relaxed check, so bind under it: + // admit asks the same question again under the row lock. + constraints.RequiredRuntime = nil } assignment := &ateapipb.ActorAssignment{ @@ -515,10 +537,13 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour // longer eligible. The store re-asks under the Worker's row lock, where the // answer holds until the bind commits, so nothing here needs a fresh read // and two claims for the last place cannot both be admitted. + // The bind evaluates the fresh row; record the assignment from the same view. + bound := assignedWorker admit := func(fresh *ateapipb.Worker) error { if !w.scheduler.Applies(fresh, constraints) || !w.scheduler.HasRoom(fresh, constraints) { return errWorkerFilledUp } + bound = fresh return nil } if err := w.store.BindActorToWorker(ctx, assignedWorker.GetMetadata().GetName(), assignment, admit); err != nil { @@ -532,7 +557,7 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour return nil, nil, err } - newAssignment := workerAssignmentFrom(assignedWorker) + newAssignment := workerAssignmentFrom(bound, snapRuntime) // The cached Worker may predate a raised epoch; the bind read it under the // Worker's row lock. newAssignment.WorkerEpoch = assignment.GetWorkerEpoch() @@ -560,11 +585,11 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour return nil, nil, apierror.Aborted("actor %s is %s and can no longer be resumed", actorRef, fresh.GetStatus().GetState()) } } - poolNamespace = assignedWorker.GetWorkerNamespace() - pool = assignedWorker.GetWorkerPool() + poolNamespace = bound.GetWorkerNamespace() + pool = bound.GetWorkerPool() outcome = ateattr.SchedulerOutcomeAssigned logActorStateChanged(ctx, storedActor, ateattr.OperationResume) - return storedActor, assignedWorker, nil + return storedActor, bound, nil } // errWorkerFilledUp reports that the Worker the scheduler picked would not take @@ -572,7 +597,7 @@ func (w *ActorWorkflow) assignWorkerAttempt(ctx context.Context, actorRef resour // attempt re-runs scheduling. var errWorkerFilledUp = errors.New("picked worker no longer has room") -func workerAssignmentFrom(w *ateapipb.Worker) *ateapipb.WorkerAssignment { +func workerAssignmentFrom(w *ateapipb.Worker, snapRuntime *ateapipb.SandboxRuntime) *ateapipb.WorkerAssignment { return &ateapipb.WorkerAssignment{ Worker: &ateapipb.ObjectRef{Name: w.GetMetadata().GetName()}, WorkerNamespace: w.GetWorkerNamespace(), @@ -581,6 +606,19 @@ func workerAssignmentFrom(w *ateapipb.Worker) *ateapipb.WorkerAssignment { WorkerPodUid: w.GetWorkerPodUid(), WorkerPodIps: slices.Clone(w.GetIps()), NodeName: w.GetNodeName(), + // The runtime the actor is started or restored under. Snapshots taken + // during this assignment are stamped with it, so a worker that later + // changes its default does not relabel state captured on the old one. + // + // TODO: A restorable_runtimes match records the old runtime here, but + // resolveSandboxAssets still ships the SandboxConfig's default version + // and atelet restores with the assets in the request, so the restore + // runs on the new version: it fails, or the next checkpoint is stamped + // with a runtime that did not produce it. Resolve the assets for the + // matched version once SandboxConfig can name them, or treat a + // restorable-only match as incompatible until it can. Latent for now: + // ateom reports no restorable_runtimes (internal/ateom/register.go). + SandboxRuntime: proto.CloneOf(hardware.RuntimeFor(w.GetStatus(), snapRuntime)), } } @@ -616,10 +654,11 @@ func schedulingConstraints(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) requiredNodes = []string{node} } c := scheduling.Constraints{ - SandboxClass: sandboxClassString(tmpl.GetSandboxConfig().GetSandboxClass()), - ActorSelector: labels.SelectorFromSet(labels.Set(actor.GetWorkerSelector().GetMatchLabels())), - RequiredNodes: requiredNodes, - Limits: limits.Proto(), + SandboxClass: tmpl.GetSandboxConfig().GetSandboxClass(), + RequiredRuntime: requiredSnapshotRuntime(actor, tmpl), + ActorSelector: labels.SelectorFromSet(labels.Set(actor.GetWorkerSelector().GetMatchLabels())), + RequiredNodes: requiredNodes, + Limits: limits.Proto(), } if sel := tmpl.GetWorkerSelector(); sel != nil { c.TemplateSelector = labels.SelectorFromSet(labels.Set(sel.GetMatchLabels())) @@ -627,6 +666,53 @@ func schedulingConstraints(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) return c, nil } +// requiredSnapshotRuntime returns the SandboxRuntime stamped on the snapshot +// that ResumeActor will restore guest state from, or nil when the resume does +// not restore guest state (cold boot, VOLUMES fidelity, or replaced template). +func requiredSnapshotRuntime(actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) *ateapipb.SandboxRuntime { + if localSnap := findLatestLocalSnapshot(actor.GetStatus()); localSnap != nil { + if localSnap.GetFidelity() == ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + return nil + } + return localSnap.GetSandboxRuntime() + } + if snap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED); snap != nil { + if snap.GetFidelity() == ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + return nil + } + capturedUnder := snap.GetActorTemplateUid() + if capturedUnder != "" && capturedUnder != tmpl.GetMetadata().GetUid() { + return nil + } + return snap.GetSandboxRuntime() + } + return nil +} + +// canRelaxRuntime reports whether a resume may drop its RequiredRuntime +// constraint and fall back to a volumes-only restore when no +// runtime-compatible worker has capacity. A local pause checkpoint is not +// relaxed: the resume stays pinned to its node, the local restore has no +// fidelity downgrade, and nothing invalidates the checkpoint files if the actor +// boots elsewhere. Suspending the actor uploads it and lifts the pin. +func canRelaxRuntime(actor *ateapipb.Actor, constraints scheduling.Constraints) bool { + if constraints.RequiredRuntime == nil { + return false + } + return findLatestLocalSnapshot(actor.GetStatus()) == nil +} + +// appliesWithFallback reports whether worker satisfies constraints, or — when +// relaxable — satisfies constraints with RequiredRuntime relaxed. +func (w *ActorWorkflow) appliesWithFallback(worker *ateapipb.Worker, constraints scheduling.Constraints, relaxable bool) bool { + relaxed := constraints + relaxed.RequiredRuntime = nil + if !w.scheduler.Applies(worker, relaxed) { + return false + } + return relaxable || hardware.WorkerMatches(worker.GetStatus(), constraints.RequiredRuntime) +} + // ensureVolumesAttached attaches the actor's mounted external volumes to the // assigned worker's node and returns each driver's attachment metadata by // volume name, for the Restore call that follows. Attachment is idempotent, so @@ -698,6 +784,9 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou return tele, fmt.Errorf("while resolving sandbox assets: %w", err) } + // The assignment is what the next checkpoint is stamped from, so it is also + // what decides how this restore runs. + runtimeIncompatible := src.RequiredRuntime != nil && !hardware.Matches(assignment.GetSandboxRuntime(), src.RequiredRuntime) if localSnap := findLatestLocalSnapshot(actor.GetStatus()); localSnap != nil { slog.InfoContext(ctx, "Actor has snapshot; Restoring from snapshot") tele.SnapshotKind = ateattr.SnapshotKindLocal @@ -730,7 +819,10 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou slog.InfoContext(ctx, "Actor has durable snapshot; Restoring from snapshot") tele.SnapshotKind = ateattr.SnapshotKindLatest scope := fidelityToAtelet(src.Fidelity) - if src.TemplateReplaced { + if src.TemplateReplaced || runtimeIncompatible { + if runtimeIncompatible { + logRuntimeFallback(ctx, actorRef, assignment, src.RequiredRuntime) + } scope = ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES } tele.WireFidelity = ateattr.SnapshotFidelityValue(scope) @@ -782,6 +874,17 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou } } +// logRuntimeFallback records a resume that restores the durable volumes alone +// because the assigned worker does not run the runtime the snapshot's guest +// state was captured under. +func logRuntimeFallback(ctx context.Context, actorRef resources.ActorRef, assignment *ateapipb.WorkerAssignment, required *ateapipb.SandboxRuntime) { + slog.WarnContext(ctx, "Assigned worker cannot restore the snapshot's runtime; restoring volumes only", + slog.String("actor", actorRef.String()), + slog.String("worker", assignment.GetWorkerPod()), + slog.Any("required_runtime", required), + slog.Any("assigned_runtime", assignment.GetSandboxRuntime())) +} + func (w *ActorWorkflow) egressGateway() *ateletpb.EgressGateway { if w.egressGatewayAddress == "" { return nil diff --git a/cmd/ateapi/internal/controlapi/workflow_resume_test.go b/cmd/ateapi/internal/controlapi/workflow_resume_test.go index 48248f3aa7..236e842ade 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume_test.go @@ -46,6 +46,7 @@ import ( "google.golang.org/protobuf/proto" "google.golang.org/protobuf/testing/protocmp" corev1 "k8s.io/api/core/v1" + storagev1 "k8s.io/api/storage/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" ) @@ -161,7 +162,7 @@ func TestAssignWorkerAttempt_MissingSelectedWorkerIsRetried(t *testing.T) { actor, wc := seedAssignFixture(t, ctx, persistence) st := &bindErrorStore{Interface: persistence, err: store.ErrNotFound} w := &ActorWorkflow{store: st, workerCache: wc, scheduler: scheduling.New(wc)} - tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}} + tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}} _, _, err := w.assignWorkerAttempt(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}, actor, tmpl) if !errors.Is(err, store.ErrVersionConflict) { @@ -182,7 +183,7 @@ func TestEnsureWorkerAssigned_ConflictExhaustionIsRetryable(t *testing.T) { actor, wc := seedAssignFixture(t, ctx, persistence) st := &bindErrorStore{Interface: persistence, err: store.ErrVersionConflict} w := &ActorWorkflow{store: st, workerCache: wc, scheduler: scheduling.New(wc)} - tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}} + tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}} _, _, err := w.ensureWorkerAssigned(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}, actor, tmpl) if !errors.Is(err, store.ErrVersionConflict) { @@ -225,7 +226,7 @@ func TestAssignWorkerAttempt_SkipsWorkerAssignedInOtherAtespace(t *testing.T) { Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "shared", Uid: "actor-uid"}, } tmpl := &ateapipb.ActorTemplate{ - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, } _, _, err := w.assignWorkerAttempt(ctx, resources.ActorRef{Atespace: "team-a", Name: "shared"}, actor, tmpl) if apierror.Code(err) != codes.ResourceExhausted { @@ -293,7 +294,7 @@ func TestAssignWorkerAttempt_ReleasesIneligibleStaleWorker(t *testing.T) { w := &ActorWorkflow{store: persistence, workerCache: wc, scheduler: scheduling.New(wc)} tmpl := &ateapipb.ActorTemplate{ - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, } _, worker, err := w.assignWorkerAttempt(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}, actor, tmpl) if err != nil { @@ -329,6 +330,7 @@ func TestAssignWorkerAttempt_RetryAfterConflictPicksFreshWorker(t *testing.T) { SandboxClass: "gvisor", Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}, } + wantRuntime := testSandboxRuntime() fallback := &ateapipb.Worker{ Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("fallback-pod")}, WorkerNamespace: "worker-ns", @@ -336,7 +338,11 @@ func TestAssignWorkerAttempt_RetryAfterConflictPicksFreshWorker(t *testing.T) { WorkerPod: "fallback-pod", WorkerPodUid: testWorkerUID("fallback-pod"), SandboxClass: "gvisor", - Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}, + Status: &ateapipb.WorkerStatus{ + State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, + Capacity: &ateapipb.WorkerResources{Actors: 1}, + DefaultRuntime: wantRuntime, + }, } for _, w := range []*ateapipb.Worker{contested, fallback} { if _, err := persistence.CreateWorker(ctx, w); err != nil { @@ -365,7 +371,7 @@ func TestAssignWorkerAttempt_RetryAfterConflictPicksFreshWorker(t *testing.T) { w := &ActorWorkflow{store: persistence, workerCache: wc, scheduler: scheduling.New(wc)} tmpl := &ateapipb.ActorTemplate{ - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, } _, worker, err := w.assignWorkerAttempt(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}, actor, tmpl) if err != nil { @@ -394,6 +400,9 @@ func TestAssignWorkerAttempt_RetryAfterConflictPicksFreshWorker(t *testing.T) { if got := storedActor.GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "fallback-pod" { t.Errorf("stored actor WorkerAssignment.WorkerPod = %q, want %q", got, "fallback-pod") } + if gotRt := storedActor.GetStatus().GetWorkerAssignment().GetSandboxRuntime(); !proto.Equal(gotRt, wantRuntime) { + t.Errorf("stored actor WorkerAssignment.SandboxRuntime = %v, want %v", gotRt, wantRuntime) + } } // conflictInjectingStore wraps a store and runs inject exactly once, @@ -503,7 +512,7 @@ func TestAssignWorkerAttempt_ConflictRefreshesActor(t *testing.T) { w := &ActorWorkflow{store: st, workerCache: wc, scheduler: scheduling.New(wc)} tmpl := &ateapipb.ActorTemplate{ - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, } refreshed, _, err := w.assignWorkerAttempt(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}, actor, tmpl) @@ -735,6 +744,7 @@ func TestValidateAssignedWorker(t *testing.T) { } activeStatus := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}} drainingStatus := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_DRAINING, Capacity: &ateapipb.WorkerResources{Actors: 1}} + unavailableStatus := &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, Capacity: &ateapipb.WorkerResources{Actors: 1}} tests := []struct { name string @@ -770,6 +780,15 @@ func TestValidateAssignedWorker(t *testing.T) { wantCrashMessage: "resume failed: " + crashMessageWorkerDraining, wantAssignment: ownAssignment, }, + { + name: "leaves actor and worker untouched when worker is unavailable", + workerStatus: unavailableStatus, + sandboxClass: "gvisor", + assignment: ownAssignment, + wantCode: codes.Unavailable, + wantActorState: ateapipb.ActorState_ACTOR_STATE_RESUMING, + wantAssignment: ownAssignment, + }, { name: "crashes actor and leaves worker untouched when assigned to another actor", workerStatus: activeStatus, @@ -865,7 +884,7 @@ func TestValidateAssignedWorker(t *testing.T) { }, }, } - tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}} + tmpl := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}} _, err := w.validateAssignedWorker(ctx, resources.ActorRef{Atespace: "team-a", Name: "shared"}, resumingActor, tmpl) if got := apierror.Code(err); got != tt.wantCode { t.Fatalf("apierror.Code(err) = %v, want %v (err: %v)", got, tt.wantCode, err) @@ -1192,6 +1211,10 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { // template's store-assigned UID (unknown until runtime), "" leaves the // field unset, anything else mismatches (a repointed actor). tmplUID string + // sourceTag seeds Actor.SourceTag when testing explicit tag clones. + sourceTag *ateapipb.ObjectRef + // assignedRuntime seeds WorkerAssignment.SandboxRuntime. + assignedRuntime *ateapipb.SandboxRuntime } // templateSeed is the ActorTemplate configuration a row persists. type templateSeed struct { @@ -1216,6 +1239,14 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { scope ateletpb.SnapshotFidelity } + amd64RT := testSandboxRuntime() + arm64RT := testRuntime("arm64", "") + stampedExtSnap := func(owner ateapipb.SnapshotOwner, uri string, fidelity ateapipb.SnapshotFidelity, rt *ateapipb.SandboxRuntime) *ateapipb.Snapshot { + snap := newDurableSnapshot(1, owner, fidelity, "", durableSnapshotName, uri, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + snap.SandboxRuntime = rt + return snap + } + tests := []struct { name string actor actorSeed @@ -1440,6 +1471,60 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { tmpl: templateSeed{configName: "missing"}, want: restoreWant{code: codes.FailedPrecondition}, }, + { + name: "21 stamped Full durable snapshot on matching worker restores in Full", + actor: actorSeed{ + externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, actorURI, fullScope, amd64RT), + tmplUID: "current", + assignedRuntime: amd64RT, + }, + want: restoreWant{ + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + }, + }, + { + name: "22 stamped actor-owned Full durable snapshot on incompatible worker drops to Volumes", + actor: actorSeed{ + externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, actorURI, fullScope, amd64RT), + tmplUID: "current", + assignedRuntime: arm64RT, + }, + want: restoreWant{ + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + }, + }, + { + name: "23 stamped borrowed golden snapshot on incompatible worker drops to Volumes", + actor: actorSeed{ + externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, goldenURI, fullScope, amd64RT), + tmplUID: "current", + assignedRuntime: arm64RT, + }, + tmpl: templateSeed{golden: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, goldenURI, fullScope, amd64RT)}, + want: restoreWant{ + snapshotName: durableSnapshotName, + snapshotURI: goldenURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + }, + }, + { + name: "24 stamped explicit source_tag clone on incompatible worker drops to Volumes", + actor: actorSeed{ + externalSnapshot: stampedExtSnap(ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, actorURI, fullScope, amd64RT), + tmplUID: "current", + sourceTag: &ateapipb.ObjectRef{Atespace: "team-a", Name: "user-tag"}, + assignedRuntime: arm64RT, + }, + want: restoreWant{ + snapshotName: durableSnapshotName, + snapshotURI: actorURI, + scope: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + }, + }, } // Every request atelet receives, Run or Restore, carries the sandbox the @@ -1466,7 +1551,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { StorageLocation: testStorageLocation, }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: configName, }, } @@ -1497,7 +1582,9 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { // checkpoint is what makes the resume local. actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} seedWorkflowActor(t, ctx, persistence, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, func(a *ateapipb.Actor) { + a.SourceTag = tt.actor.sourceTag a.Status.WorkerAssignment = wireTestAssignment() + a.Status.WorkerAssignment.SandboxRuntime = proto.CloneOf(tt.actor.assignedRuntime) uid := tt.actor.tmplUID if uid == "current" { uid = createdTmpl.GetMetadata().GetUid() @@ -1591,8 +1678,8 @@ func TestEnsureVolumesAttached_ReturnsPublishContext(t *testing.T) { Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_RESUMING, ExternalVolumes: []*ateapipb.ExternalVolume{ - {Name: "mounted", StorageVolumeId: "storage-mounted", VolumeType: "mock", Status: ateapipb.ExternalVolume_STATUS_CREATED}, - {Name: "unmounted", StorageVolumeId: "storage-unmounted", VolumeType: "mock", Status: ateapipb.ExternalVolume_STATUS_CREATED}, + {Name: "mounted", StorageVolumeId: "storage-mounted", VolumeType: "mock"}, + {Name: "unmounted", StorageVolumeId: "storage-unmounted", VolumeType: "mock"}, }, }, }) @@ -1636,6 +1723,78 @@ func TestEnsureVolumesAttached_ReturnsPublishContext(t *testing.T) { } } +// createCountingVolumePlugin provisions volumes and counts the calls. +type createCountingVolumePlugin struct { + volume.VolumePluginControlPlane + creates int +} + +func (p *createCountingVolumePlugin) CreateVolume(ctx context.Context, req volume.CreateVolumeRequest) (volume.CreateVolumeResponse, error) { + p.creates++ + return volume.CreateVolumeResponse{VolumeID: "id-" + req.Name}, nil +} + +// An actor being deleted gets no volume provisioned by a resume. +func TestEnsureVolumesCreated(t *testing.T) { + tmpl := &ateapipb.ActorTemplate{ + Volumes: []*ateapipb.Volume{{ + Name: "data", + ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "sc"}, + }}, + } + tests := []struct { + name string + state ateapipb.ActorState + wantCode codes.Code + wantCreates int + }{ + {"suspended actor gets its volume", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, codes.OK, 1}, + {"deleting actor gets no volume", ateapipb.ActorState_ACTOR_STATE_DELETING, codes.FailedPrecondition, 0}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "id1"}, + Status: &ateapipb.ActorStatus{ + State: tt.state, + ExternalVolumes: []*ateapipb.ExternalVolume{{Name: "data", VolumeType: "mock"}}, + }, + }) + actor, err := persistence.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + + plugin := &createCountingVolumePlugin{} + w := &ActorWorkflow{ + store: persistence, + pluginRegistry: &mockPluginRegistry{plugins: map[string]volume.VolumePluginControlPlane{"mock": plugin}}, + storageClassLister: &fakeStorageClassLister{storageClasses: map[string]*storagev1.StorageClass{ + "sc": {ObjectMeta: metav1.ObjectMeta{Name: "sc"}, Provisioner: "mock"}, + }}, + } + + _, err = w.ensureVolumesCreated(ctx, actorRef, actor, tmpl) + if code := apierror.Code(err); code != tt.wantCode { + t.Fatalf("ensureVolumesCreated code = %v, want %v; error = %v", code, tt.wantCode, err) + } + if plugin.creates != tt.wantCreates { + t.Errorf("CreateVolume calls = %d, want %d", plugin.creates, tt.wantCreates) + } + stored, err := persistence.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + if provisioned := stored.GetStatus().GetExternalVolumes()[0].GetStorageVolumeId() != ""; provisioned != (tt.wantCreates > 0) { + t.Errorf("stored volume provisioned = %t, want %t", provisioned, tt.wantCreates > 0) + } + }) + } +} + // filledUpScheduler hands out workers in order and refuses room on the ones // named in full. It stands in for a stale worker cache: Schedule offers a // worker that looked free, and the admit callback that BindActorToWorker runs @@ -1827,7 +1986,7 @@ func newContendedWorkflow(t *testing.T, sched *filledUpScheduler) (*ActorWorkflo func gvisorTemplate() *ateapipb.ActorTemplate { return &ateapipb.ActorTemplate{ - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, } } @@ -2054,3 +2213,428 @@ func TestEnsureWorkerAssigned_ContextEndsInsideAttemptRecordsOne(t *testing.T) { t.Errorf("recorded outcome %q, want %q", outcomes[0], want) } } + +// testRuntime is a gvisor SandboxRuntime for arch under compatibility schema +// v1. A non-empty assetHash adds a gvisor_asset_hash attribute, so two +// runtimes can share an architecture and still differ, as two versions of +// one SandboxConfig do. +func testRuntime(arch, assetHash string) *ateapipb.SandboxRuntime { + attrs := []*ateapipb.AttributeEntry{{Key: "architecture", Value: arch}} + if assetHash != "" { + attrs = append(attrs, &ateapipb.AttributeEntry{Key: "gvisor_asset_hash", Value: assetHash}) + } + return &ateapipb.SandboxRuntime{ + SandboxClass: "gvisor", + Version: &ateapipb.VersionedSandboxCompat{SchemaVersion: "v1", Attributes: attrs}, + } +} + +// testSandboxRuntime is the amd64 gvisor runtime most tests report and stamp. +func testSandboxRuntime() *ateapipb.SandboxRuntime { + return testRuntime("amd64", "") +} + +// seedRuntimeWorker stores an ACTIVE gvisor worker on node with room for one +// actor, reporting def as its default runtime and restorable beyond it. +func seedRuntimeWorker(t *testing.T, ctx context.Context, st store.Interface, pod, node string, def *ateapipb.SandboxRuntime, restorable ...*ateapipb.SandboxRuntime) *ateapipb.Worker { + t.Helper() + worker, err := st.CreateWorker(ctx, &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID(pod)}, + WorkerNamespace: "worker-ns", + WorkerPool: "pool", + WorkerPod: pod, + WorkerPodUid: testWorkerUID(pod), + SandboxClass: "gvisor", + NodeName: node, + Status: &ateapipb.WorkerStatus{ + State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, + Capacity: &ateapipb.WorkerResources{Actors: 1}, + DefaultRuntime: def, + RestorableRuntimes: restorable, + }, + }) + if err != nil { + t.Fatalf("CreateWorker(%s): %v", pod, err) + } + return worker +} + +// durableMemorySnapshot is a completed, actor-owned MEMORY snapshot of +// generation 1 under team-a/id1's own prefix, stamped with runtime. +func durableMemorySnapshot(t *testing.T, runtime *ateapipb.SandboxRuntime) *ateapipb.Snapshot { + t.Helper() + snap := newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap-1", someActorSnapshotURI(t, testStorageLocation, "team-a", "snap-1"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + snap.SandboxRuntime = runtime + return snap +} + +// seedActorWithSnapshot stores actor team-a/id1 in state holding snap as its +// only snapshot. node is the AssignedNode a PAUSED actor keeps, which pins +// its resume to that node. +func seedActorWithSnapshot(t *testing.T, ctx context.Context, st store.Interface, state ateapipb.ActorState, node string, snap *ateapipb.Snapshot) *ateapipb.Actor { + t.Helper() + return storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "id1"}, + Status: &ateapipb.ActorStatus{ + State: state, + AssignedNode: node, + LastAssignedGeneration: snap.GetGeneration(), + Snapshots: []*ateapipb.Snapshot{snap}, + }, + }) +} + +// newSchedulingWorkflow is a workflow that schedules over a worker cache fed +// by st. The cache is started now and stopped when the test ends. +func newSchedulingWorkflow(t *testing.T, ctx context.Context, st store.Interface) *ActorWorkflow { + t.Helper() + cacheCtx, cancel := context.WithCancel(ctx) + t.Cleanup(cancel) + wc := workercache.New(st, time.Minute) + if err := wc.Start(cacheCtx); err != nil { + t.Fatalf("workercache.Start: %v", err) + } + return &ActorWorkflow{store: st, workerCache: wc, scheduler: scheduling.New(wc)} +} + +// assertAssignedRuntime checks the runtime recorded on actor's assignment, +// which is what its next checkpoint will be stamped with. +func assertAssignedRuntime(t *testing.T, actor *ateapipb.Actor, want *ateapipb.SandboxRuntime) { + t.Helper() + if diff := cmp.Diff(want, actor.GetStatus().GetWorkerAssignment().GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("WorkerAssignment.SandboxRuntime mismatch (-want +got):\n%s", diff) + } +} + +// relaxingTestScheduler stands in for a fleet where no runtime-compatible +// worker has room: a strict Schedule reports no capacity and a relaxed one +// returns relaxedWorker, as the scheduler's cache would. Applies and HasRoom +// are the real ones. +type relaxingTestScheduler struct { + scheduling.Scheduler + relaxedWorker *ateapipb.Worker +} + +func (s *relaxingTestScheduler) Schedule(_ context.Context, c scheduling.Constraints) (*ateapipb.Worker, error) { + if c.RequiredRuntime != nil { + return nil, scheduling.ErrNoCapacity + } + return s.relaxedWorker, nil +} + +func TestAssignWorkerAttempt_PrefersRuntimeMatchingWorkerAndFallsBack(t *testing.T) { + amd64RT := testSandboxRuntime() + arm64RT := testRuntime("arm64", "") + // Two runtimes of one architecture that differ only by binary: a + // SandboxConfig whose default moved on while the old version stays + // enabled for restores. + newDefaultRT := testRuntime("amd64", "new-default") + oldRestorableRT := testRuntime("amd64", "old-restorable") + actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + suspended := ateapipb.ActorState_ACTOR_STATE_SUSPENDED + + t.Run("places on runtime-compatible worker when available", func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + seedRuntimeWorker(t, ctx, st, "arm-pod", "node-arm", arm64RT) + seedRuntimeWorker(t, ctx, st, "amd-pod", "node-amd", amd64RT) + actor := seedActorWithSnapshot(t, ctx, st, suspended, "", durableMemorySnapshot(t, amd64RT)) + + stored, worker, err := newSchedulingWorkflow(t, ctx, st).assignWorkerAttempt(ctx, actorRef, actor, gvisorTemplate()) + if err != nil { + t.Fatalf("assignWorkerAttempt: %v", err) + } + if got := worker.GetWorkerPod(); got != "amd-pod" { + t.Errorf("assigned worker = %q, want %q", got, "amd-pod") + } + assertAssignedRuntime(t, stored, amd64RT) + }) + + t.Run("records matched restorable_runtimes entry on WorkerAssignment", func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + seedRuntimeWorker(t, ctx, st, "amd-pod", "node-amd", newDefaultRT, oldRestorableRT) + actor := seedActorWithSnapshot(t, ctx, st, suspended, "", durableMemorySnapshot(t, oldRestorableRT)) + + stored, _, err := newSchedulingWorkflow(t, ctx, st).assignWorkerAttempt(ctx, actorRef, actor, gvisorTemplate()) + if err != nil { + t.Fatalf("assignWorkerAttempt: %v", err) + } + // The actor restores under the old version, not the worker's default, + // and its next checkpoint must say so. + assertAssignedRuntime(t, stored, oldRestorableRT) + }) + + t.Run("falls back to incompatible worker when no runtime-matching worker has room", func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + seedRuntimeWorker(t, ctx, st, "arm-pod", "node-arm", arm64RT) + actor := seedActorWithSnapshot(t, ctx, st, suspended, "", durableMemorySnapshot(t, amd64RT)) + + stored, worker, err := newSchedulingWorkflow(t, ctx, st).assignWorkerAttempt(ctx, actorRef, actor, gvisorTemplate()) + if err != nil { + t.Fatalf("assignWorkerAttempt: %v", err) + } + if got := worker.GetWorkerPod(); got != "arm-pod" { + t.Errorf("assigned worker = %q, want %q", got, "arm-pod") + } + // The volumes-only restore runs under the fallback worker's default. + assertAssignedRuntime(t, stored, arm64RT) + }) + + t.Run("does not relax runtime constraint for local pause snapshot", func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + seedRuntimeWorker(t, ctx, st, "node1-incompatible", "node-1", arm64RT) + // A paused actor: SUSPENDED, pinned to its node by a local snapshot. + local := newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-1", "node-1") + local.SandboxRuntime = amd64RT + actor := seedActorWithSnapshot(t, ctx, st, suspended, "node-1", local) + + _, _, err := newSchedulingWorkflow(t, ctx, st).assignWorkerAttempt(ctx, actorRef, actor, gvisorTemplate()) + if got := apierror.Code(err); got != codes.ResourceExhausted { + t.Fatalf("apierror.Code(err) = %v, want %v (err: %v)", got, codes.ResourceExhausted, err) + } + // Nothing was written: the actor waits for its node, still paused, + // unassigned, and holding no claim anything would have to release. + after, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor: %v", err) + } + if got := after.GetStatus().GetState(); got != suspended { + t.Errorf("state = %v, want SUSPENDED", got) + } + if got := after.GetStatus().GetWorkerAssignment(); got != nil { + t.Errorf("WorkerAssignment = %v, want none", got) + } + if _, err := st.FindWorkerHostingActor(ctx, actor.GetMetadata().GetUid()); !errors.Is(err, store.ErrNotFound) { + t.Errorf("FindWorkerHostingActor() error = %v, want ErrNotFound", err) + } + }) + + t.Run("relaxed schedule landing on restorable worker records matched restorable runtime", func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + // A compatible worker freed up between the strict and the relaxed + // Schedule: the relaxed pick can restore the snapshot after all. + worker := seedRuntimeWorker(t, ctx, st, "amd-pod", "node-amd", newDefaultRT, oldRestorableRT) + actor := seedActorWithSnapshot(t, ctx, st, suspended, "", durableMemorySnapshot(t, oldRestorableRT)) + + w := &ActorWorkflow{store: st, scheduler: &relaxingTestScheduler{Scheduler: scheduling.New(nil), relaxedWorker: worker}} + stored, _, err := w.assignWorkerAttempt(ctx, actorRef, actor, gvisorTemplate()) + if err != nil { + t.Fatalf("assignWorkerAttempt: %v", err) + } + // Relaxing the constraint must not relabel the assignment: it is + // derived from the snapshot's stamp, not from the relaxed constraints. + assertAssignedRuntime(t, stored, oldRestorableRT) + }) + + t.Run("records runtime from row-locked worker when cached worker runtime is stale", func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + // The stored row no longer lists the old version; the cached copy the + // scheduler hands back still does. + fresh := seedRuntimeWorker(t, ctx, st, "amd-pod", "node-amd", newDefaultRT) + stale := proto.CloneOf(fresh) + stale.Status.RestorableRuntimes = []*ateapipb.SandboxRuntime{oldRestorableRT} + actor := seedActorWithSnapshot(t, ctx, st, suspended, "", durableMemorySnapshot(t, oldRestorableRT)) + + w := &ActorWorkflow{store: st, scheduler: &relaxingTestScheduler{Scheduler: scheduling.New(nil), relaxedWorker: stale}} + stored, bound, err := w.assignWorkerAttempt(ctx, actorRef, actor, gvisorTemplate()) + if err != nil { + t.Fatalf("assignWorkerAttempt: %v", err) + } + assertAssignedRuntime(t, stored, newDefaultRT) + if got := bound.GetStatus().GetRestorableRuntimes(); len(got) != 0 { + t.Errorf("returned worker RestorableRuntimes = %v, want the fresh row's none", got) + } + }) +} + +func TestValidateAssignedWorker_RuntimeFallback(t *testing.T) { + t.Run("accepts fallback worker on re-entry for durable snapshot", func(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + arm64RT := testRuntime("arm64", "") + seedRuntimeWorker(t, ctx, st, "arm-pod", "node-arm", arm64RT) + seedAssignment(t, st, testWorkerUID("arm-pod"), &ateapipb.ActorAssignment{ + Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "id1"}, + ActorUid: "own-actor-uid", + }) + + // A previous attempt bound the actor relaxed, under the worker's + // default, because no worker could restore its amd64 snapshot. + resumingActor := &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "id1", Uid: "own-actor-uid"}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_RESUMING, + AssignedNode: "node-arm", + LastAssignedGeneration: 1, + Snapshots: []*ateapipb.Snapshot{durableMemorySnapshot(t, testSandboxRuntime())}, + WorkerAssignment: &ateapipb.WorkerAssignment{ + Worker: &ateapipb.ObjectRef{Name: testWorkerUID("arm-pod")}, + WorkerNamespace: "worker-ns", + WorkerPool: "pool", + WorkerPod: "arm-pod", + WorkerPodUid: testWorkerUID("arm-pod"), + NodeName: "node-arm", + SandboxRuntime: arm64RT, + }, + }, + } + + w := &ActorWorkflow{store: st, scheduler: scheduling.New(nil)} + worker, err := w.validateAssignedWorker(ctx, resources.ActorRef{Atespace: "team-a", Name: "id1"}, resumingActor, gvisorTemplate()) + if err != nil { + t.Fatalf("validateAssignedWorker: %v", err) + } + if got := worker.GetWorkerPod(); got != "arm-pod" { + t.Errorf("validated worker = %q, want %q", got, "arm-pod") + } + }) +} + +// TestRequiredSnapshotRuntime pins which snapshot a resume takes its runtime +// constraint from, and when it takes none. +func TestRequiredSnapshotRuntime(t *testing.T) { + const tmplUID = "tmpl-uid" + tmpl := &ateapipb.ActorTemplate{Metadata: &ateapipb.ResourceMetadata{Uid: tmplUID}} + localRT := testRuntime("amd64", "paused-under") + durableRT := testRuntime("amd64", "suspended-under") + memory := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY + rootfs := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + volumes := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + completed := ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED + pending := ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING + + durable := func(gen int32, fidelity ateapipb.SnapshotFidelity, capturedUnder string, status ateapipb.SnapshotStorageStatus, rt *ateapipb.SandboxRuntime) *ateapipb.Snapshot { + snap := newDurableSnapshot(gen, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, fidelity, capturedUnder, "snap", "gs://bucket/snap", status) + snap.SandboxRuntime = rt + return snap + } + local := func(gen int32, fidelity ateapipb.SnapshotFidelity, rt *ateapipb.SandboxRuntime) *ateapipb.Snapshot { + snap := newLocalSnapshot(gen, fidelity, tmplUID, "local", "node-1") + snap.SandboxRuntime = rt + return snap + } + + tests := []struct { + name string + snapshots []*ateapipb.Snapshot + want *ateapipb.SandboxRuntime + }{ + {name: "no snapshot cold-boots unconstrained"}, + { + name: "completed MEMORY durable snapshot requires its runtime", + snapshots: []*ateapipb.Snapshot{durable(1, memory, tmplUID, completed, durableRT)}, + want: durableRT, + }, + { + name: "completed ROOTFS durable snapshot requires its runtime", + snapshots: []*ateapipb.Snapshot{durable(1, rootfs, tmplUID, completed, durableRT)}, + want: durableRT, + }, + { + name: "VOLUMES durable snapshot restores no guest state", + snapshots: []*ateapipb.Snapshot{durable(1, volumes, tmplUID, completed, durableRT)}, + }, + { + name: "durable snapshot captured under a replaced template is not restored", + snapshots: []*ateapipb.Snapshot{durable(1, memory, "old-tmpl-uid", completed, durableRT)}, + }, + { + name: "durable snapshot with no template uid keeps its runtime", + snapshots: []*ateapipb.Snapshot{durable(1, memory, "", completed, durableRT)}, + want: durableRT, + }, + { + name: "unstamped durable snapshot imposes no constraint", + snapshots: []*ateapipb.Snapshot{durable(1, memory, tmplUID, completed, nil)}, + }, + { + name: "PENDING durable snapshot is not a restore source", + snapshots: []*ateapipb.Snapshot{durable(1, memory, tmplUID, pending, durableRT)}, + }, + { + name: "local checkpoint requires its runtime", + snapshots: []*ateapipb.Snapshot{local(1, memory, localRT)}, + want: localRT, + }, + { + // ensureAteletRestored restores a local checkpoint before any + // durable snapshot, so the constraint follows it regardless of + // which is newer. + name: "local checkpoint takes precedence over a newer durable snapshot", + snapshots: []*ateapipb.Snapshot{ + local(1, memory, localRT), + durable(2, memory, tmplUID, completed, durableRT), + }, + want: localRT, + }, + { + name: "VOLUMES local checkpoint imposes no constraint beside a stamped durable snapshot", + snapshots: []*ateapipb.Snapshot{ + local(2, volumes, localRT), + durable(1, memory, tmplUID, completed, durableRT), + }, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + actor := &ateapipb.Actor{Status: &ateapipb.ActorStatus{Snapshots: tc.snapshots}} + if diff := cmp.Diff(tc.want, requiredSnapshotRuntime(actor, tmpl), protocmp.Transform()); diff != "" { + t.Errorf("requiredSnapshotRuntime mismatch (-want +got):\n%s", diff) + } + }) + } +} + +// TestCanRelaxRuntime pins when a resume may drop its runtime constraint: only +// when it has one, and never while a local pause checkpoint pins the actor. +func TestCanRelaxRuntime(t *testing.T) { + rt := testSandboxRuntime() + memory := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY + durable := func(gen int32) *ateapipb.Snapshot { + return newDurableSnapshot(gen, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, memory, "", "snap", "gs://bucket/snap", ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + } + local := func(gen int32) *ateapipb.Snapshot { + return newLocalSnapshot(gen, memory, "", "local", "node-1") + } + + tests := []struct { + name string + snapshots []*ateapipb.Snapshot + required *ateapipb.SandboxRuntime + want bool + }{ + { + name: "nothing to relax without a runtime constraint", + snapshots: []*ateapipb.Snapshot{durable(1)}, + }, + { + name: "durable snapshot may fall back", + snapshots: []*ateapipb.Snapshot{durable(1)}, + required: rt, + want: true, + }, + { + name: "local checkpoint stays pinned", + snapshots: []*ateapipb.Snapshot{local(1)}, + required: rt, + }, + { + name: "local checkpoint stays pinned beside a durable snapshot", + snapshots: []*ateapipb.Snapshot{durable(1), local(2)}, + required: rt, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + actor := &ateapipb.Actor{Status: &ateapipb.ActorStatus{Snapshots: tc.snapshots}} + if got := canRelaxRuntime(actor, scheduling.Constraints{RequiredRuntime: tc.required}); got != tc.want { + t.Errorf("canRelaxRuntime = %v, want %v", got, tc.want) + } + }) + } +} diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend.go b/cmd/ateapi/internal/controlapi/workflow_suspend.go index ff2cce1231..400b9ce376 100644 --- a/cmd/ateapi/internal/controlapi/workflow_suspend.go +++ b/cmd/ateapi/internal/controlapi/workflow_suspend.go @@ -28,6 +28,7 @@ import ( "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "go.opentelemetry.io/otel/attribute" + "google.golang.org/protobuf/proto" ) // SuspendActor executes the workflow to suspend a running or paused actor: @@ -176,16 +177,17 @@ func (w *ActorWorkflow) ensureMarkedSuspending(ctx context.Context, actorRef res // a pending snapshot for that generation. gen := toUpdate.Status.LastAssignedGeneration + 1 toUpdate.Status.LastAssignedGeneration = gen - toUpdate.Status.Snapshots = append(toUpdate.Status.Snapshots, - newDurableSnapshot( - gen, - ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, - preferredFidelity(actorRef.Atespace, actorTemplate), - actorTemplate.GetMetadata().GetUid(), - uri.Name(), - uri.String(), - ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING, - )) + snap := newDurableSnapshot( + gen, + ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, + preferredFidelity(actorRef.Atespace, actorTemplate), + actorTemplate.GetMetadata().GetUid(), + uri.Name(), + uri.String(), + ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING, + ) + snap.SandboxRuntime = proto.CloneOf(toUpdate.GetStatus().GetWorkerAssignment().GetSandboxRuntime()) + toUpdate.Status.Snapshots = append(toUpdate.Status.Snapshots, snap) return nil }) if err != nil { diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go index 5eaf7d2dfa..202d9169cc 100644 --- a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go @@ -29,17 +29,39 @@ import ( "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/google/go-cmp/cmp" "google.golang.org/grpc/codes" "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/testing/protocmp" "k8s.io/client-go/tools/cache" ) func TestEnsureMarkedSuspending_SnapshotURI(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + workerName := testWorkerUID("worker-pod-1") + wantRuntime := &ateapipb.SandboxRuntime{ + SandboxClass: "gvisor", + Version: &ateapipb.VersionedSandboxCompat{ + SchemaVersion: "v1", + Attributes: []*ateapipb.AttributeEntry{ + {Key: "architecture", Value: "amd64"}, + }, + }, + } actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + WorkerAssignment: &ateapipb.WorkerAssignment{ + Worker: &ateapipb.ObjectRef{Name: workerName}, + WorkerNamespace: "default", + WorkerPool: "pool1", + WorkerPod: "worker-pod-1", + WorkerPodUid: workerName, + SandboxRuntime: wantRuntime, + }, + }, }) tmpl := &ateapipb.ActorTemplate{ SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://bucket/root/"}, @@ -62,6 +84,9 @@ func TestEnsureMarkedSuspending_SnapshotURI(t *testing.T) { if uri.String() != want { t.Errorf("snapshot URI = %q, want %q", uri, want) } + if diff := cmp.Diff(wantRuntime, inProgressSnap.GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("Snapshot.SandboxRuntime mismatch (-want +got):\n%s", diff) + } } // TestEnsureMarkedSuspending_ReentryKeepsPersistedSnapshotLocation verifies a @@ -216,23 +241,23 @@ func TestEnsureMarkedSuspending_RejectsPauseInProgress(t *testing.T) { // TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot verifies the suspend // edge from a paused actor (SUSPENDED whose latest snapshot is node-local): // it takes the edge and promotes that snapshot in place to a pending durable -// one, keeping its generation and uuid, with the upload destination derived -// from the uuid. +// one, keeping its generation, uuid, and sandbox runtime stamp, with the +// upload destination derived from the uuid. func TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) w := &ActorWorkflow{store: persistence} actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + localSnap := newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap-1", "node-1") + localSnap.SandboxRuntime = testSandboxRuntime() actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, AssignedNode: "node-1", LastAssignedGeneration: 1, - Snapshots: []*ateapipb.Snapshot{ - newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap-1", "node-1"), - }, + Snapshots: []*ateapipb.Snapshot{localSnap}, }, }) @@ -263,6 +288,11 @@ func TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot(t *testing.T) { if pending.GetLocality() != "node-1" { t.Errorf("pending snapshot locality = %q, want node-1 kept until the upload finishes", pending.GetLocality()) } + // The durable copy is the pause checkpoint uploaded, so it keeps the + // runtime stamp the pause recorded. + if diff := cmp.Diff(testSandboxRuntime(), pending.GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("paused-origin pending Snapshot.SandboxRuntime mismatch (-want +got):\n%s", diff) + } if got := marked.GetStatus().GetLastAssignedGeneration(); got != 1 { t.Errorf("paused-origin LastAssignedGeneration = %d, want 1", got) } diff --git a/cmd/ateapi/internal/controlapi/workflow_tag.go b/cmd/ateapi/internal/controlapi/workflow_tag.go index af5bd847dc..cc4d9a6e67 100644 --- a/cmd/ateapi/internal/controlapi/workflow_tag.go +++ b/cmd/ateapi/internal/controlapi/workflow_tag.go @@ -23,6 +23,7 @@ import ( "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" ) // TagActorSnapshot tags the external snapshot held by the suspended actor the @@ -303,6 +304,7 @@ func (w *ActorWorkflow) ensureTagSnapshotRecorded(ctx context.Context, tag *atea dst.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING, ) + pending.SandboxRuntime = proto.CloneOf(snapshot.GetSandboxRuntime()) stored, err := w.store.UpdateTag(ctx, tagRef, store.PreconditionFrom(tag), func(toUpdate *ateapipb.Tag) error { toUpdate.Status.Snapshot = pending return nil @@ -360,6 +362,7 @@ func (w *ActorWorkflow) ensureTagFinalized(ctx context.Context, tag *ateapipb.Ta dst.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED, ) + finalSnapshot.SandboxRuntime = proto.CloneOf(snapshot.GetSandboxRuntime()) stored, err := w.store.UpdateTag(ctx, tagRef, store.PreconditionFrom(tag), func(toUpdate *ateapipb.Tag) error { toUpdate.Status.Snapshot = finalSnapshot return nil diff --git a/cmd/ateapi/internal/controlapi/workflow_tag_test.go b/cmd/ateapi/internal/controlapi/workflow_tag_test.go index 1f72b4ed75..c2de668b39 100644 --- a/cmd/ateapi/internal/controlapi/workflow_tag_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_tag_test.go @@ -29,6 +29,7 @@ import ( "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/google/go-cmp/cmp" "google.golang.org/grpc/codes" + "google.golang.org/protobuf/testing/protocmp" ) // seedTagSource stores a suspended actor whose external snapshot is made of @@ -45,9 +46,9 @@ func seedTagSource(t *testing.T, ctx context.Context, persistence store.Interfac objects.PutSnapshot(t, uri, objectNames...) actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 1 - s.Snapshots = []*ateapipb.Snapshot{ - newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, template.GetMetadata().GetUid(), uri.Name(), uri.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), - } + snap := newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, template.GetMetadata().GetUid(), uri.Name(), uri.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + snap.SandboxRuntime = testSandboxRuntime() + s.Snapshots = []*ateapipb.Snapshot{snap} }) return actor, uri } @@ -105,6 +106,9 @@ func TestTagActorSnapshot(t *testing.T) { if got, want := tag.GetStatus().GetSnapshot().GetFidelity(), ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY; got != want { t.Errorf("fidelity = %v, want the source's %v", got, want) } + if diff := cmp.Diff(testSandboxRuntime(), tag.GetStatus().GetSnapshot().GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("tag Snapshot.SandboxRuntime mismatch (-want +got):\n%s", diff) + } // Both prefixes hold the same objects: the tag copied rather than moved. wantObjects := []string{"manifest.json", "memory.zst"} diff --git a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go index 10470ca1b3..e7a33bddc3 100644 --- a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go @@ -53,7 +53,7 @@ func newTestActorWorkflow(t *testing.T, st store.Interface, tmplAtespace, tmplNa StorageLocation: "gs://snapshots", }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor", }, }); err != nil && !errors.Is(err, store.ErrAlreadyExists) { diff --git a/cmd/ateapi/internal/controlapi/workload_spec_test.go b/cmd/ateapi/internal/controlapi/workload_spec_test.go index f84f6f5e36..a1b640ef0f 100644 --- a/cmd/ateapi/internal/controlapi/workload_spec_test.go +++ b/cmd/ateapi/internal/controlapi/workload_spec_test.go @@ -608,7 +608,7 @@ func TestToAteletResources(t *testing.T) { func TestWorkloadSpecFromActorTemplatePropagatesResources(t *testing.T) { got, err := workloadSpecFromActorTemplate(&ateapipb.ActorTemplate{ Metadata: &ateapipb.ResourceMetadata{Atespace: "agent-ns", Name: "tmpl-limits"}, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "microvm"}, Containers: []*ateapipb.Container{ { Name: "limited", diff --git a/cmd/ateapi/internal/scheduling/scheduling.go b/cmd/ateapi/internal/scheduling/scheduling.go index 698e32dcea..d820911dca 100644 --- a/cmd/ateapi/internal/scheduling/scheduling.go +++ b/cmd/ateapi/internal/scheduling/scheduling.go @@ -22,6 +22,7 @@ import ( "math/rand" "slices" + "github.com/agent-substrate/substrate/internal/hardware" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "k8s.io/apimachinery/pkg/labels" @@ -33,6 +34,11 @@ type Constraints struct { // portable across sandbox classes, so this is never relaxed. SandboxClass string + // RequiredRuntime, when non-nil, restricts placement to workers whose + // DefaultRuntime or RestorableRuntimes can restore a snapshot stamped with + // this runtime. + RequiredRuntime *ateapipb.SandboxRuntime + // TemplateSelector and ActorSelector must both match the worker's labels. TemplateSelector labels.Selector ActorSelector labels.Selector @@ -222,15 +228,19 @@ func (s *scheduler) Applies(worker *ateapipb.Worker, constraints Constraints) bo return false } - set := labels.Set(worker.GetLabels()) - if constraints.TemplateSelector != nil && !constraints.TemplateSelector.Matches(set) { + if len(constraints.RequiredNodes) > 0 && !slices.Contains(constraints.RequiredNodes, worker.GetNodeName()) { return false } - if constraints.ActorSelector != nil && !constraints.ActorSelector.Matches(set) { + + if !hardware.WorkerMatches(worker.GetStatus(), constraints.RequiredRuntime) { return false } - return len(constraints.RequiredNodes) == 0 || slices.Contains(constraints.RequiredNodes, worker.GetNodeName()) + set := labels.Set(worker.GetLabels()) + if constraints.TemplateSelector != nil && !constraints.TemplateSelector.Matches(set) { + return false + } + return constraints.ActorSelector == nil || constraints.ActorSelector.Matches(set) } // HasRoom reports whether what the worker has left admits one more actor of diff --git a/cmd/ateapi/internal/scheduling/scheduling_test.go b/cmd/ateapi/internal/scheduling/scheduling_test.go index 9cbc5e6ff0..e1775d15bf 100644 --- a/cmd/ateapi/internal/scheduling/scheduling_test.go +++ b/cmd/ateapi/internal/scheduling/scheduling_test.go @@ -325,6 +325,31 @@ func TestSchedule(t *testing.T) { constraints: Constraints{SandboxClass: "gvisor", Limits: resources.CPUMemory(1000, 2<<30)}, wantPod: "w-busy-big", }, + { + name: "required runtime matches worker default runtime", + fleet: fleet{ + worker("w-arm", "gvisor", "node-a", tierTwo, withRuntimes(sandboxRuntime("gvisor", "arm64"))), + worker("w-amd", "gvisor", "node-b", tierTwo, withRuntimes(sandboxRuntime("gvisor", "amd64"))), + }, + constraints: Constraints{SandboxClass: "gvisor", RequiredRuntime: sandboxRuntime("gvisor", "amd64")}, + wantPod: "w-amd", + }, + { + name: "required runtime matches worker restorable runtime", + fleet: fleet{ + worker("w-upgraded", "gvisor", "node-a", tierTwo, + withRuntimes(sandboxRuntime("gvisor", "amd64", "new"), sandboxRuntime("gvisor", "amd64", "old"))), + }, + constraints: Constraints{SandboxClass: "gvisor", RequiredRuntime: sandboxRuntime("gvisor", "amd64", "old")}, + wantPod: "w-upgraded", + }, + { + name: "required runtime with no matching worker yields ErrNoCapacity", + fleet: fleet{ + worker("w-arm", "gvisor", "node-a", tierTwo, withRuntimes(sandboxRuntime("gvisor", "arm64"))), + }, + constraints: Constraints{SandboxClass: "gvisor", RequiredRuntime: sandboxRuntime("gvisor", "amd64")}, + }, } for _, tc := range tests { @@ -418,12 +443,37 @@ func TestApplies(t *testing.T) { constraints: Constraints{SandboxClass: "gvisor"}, want: false, }, + { + name: "skips unavailable worker", + worker: worker("w", "gvisor", "node-a", nil, withState(ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE)), + constraints: Constraints{SandboxClass: "gvisor"}, + want: false, + }, { name: "skips unspecified worker", worker: worker("w", "gvisor", "node-a", nil, withState(ateapipb.WorkerState_WORKER_STATE_UNSPECIFIED)), constraints: Constraints{SandboxClass: "gvisor"}, want: false, }, + { + name: "matches required runtime via default runtime", + worker: worker("w", "gvisor", "node-a", nil, withRuntimes(sandboxRuntime("gvisor", "amd64", "old"))), + constraints: Constraints{SandboxClass: "gvisor", RequiredRuntime: sandboxRuntime("gvisor", "amd64", "old")}, + want: true, + }, + { + name: "matches required runtime via restorable runtimes", + worker: worker("w", "gvisor", "node-a", nil, + withRuntimes(sandboxRuntime("gvisor", "amd64", "new"), sandboxRuntime("gvisor", "amd64", "old"))), + constraints: Constraints{SandboxClass: "gvisor", RequiredRuntime: sandboxRuntime("gvisor", "amd64", "old")}, + want: true, + }, + { + name: "skips worker when required runtime does not match", + worker: worker("w", "gvisor", "node-a", nil, withRuntimes(sandboxRuntime("gvisor", "arm64"))), + constraints: Constraints{SandboxClass: "gvisor", RequiredRuntime: sandboxRuntime("gvisor", "amd64")}, + want: false, + }, } for _, tc := range tests { @@ -455,6 +505,27 @@ func worker(pod, class, node string, lbls map[string]string, opts ...func(*ateap return w } +func sandboxRuntime(class, arch string, assetHash ...string) *ateapipb.SandboxRuntime { + attrs := []*ateapipb.AttributeEntry{{Key: "architecture", Value: arch}} + if len(assetHash) > 0 && assetHash[0] != "" { + attrs = append(attrs, &ateapipb.AttributeEntry{Key: "gvisor_asset_hash", Value: assetHash[0]}) + } + return &ateapipb.SandboxRuntime{ + SandboxClass: class, + Version: &ateapipb.VersionedSandboxCompat{ + SchemaVersion: "v1", + Attributes: attrs, + }, + } +} + +func withRuntimes(def *ateapipb.SandboxRuntime, restorable ...*ateapipb.SandboxRuntime) func(*ateapipb.Worker) { + return func(w *ateapipb.Worker) { + w.Status.DefaultRuntime = def + w.Status.RestorableRuntimes = restorable + } +} + func withState(state ateapipb.WorkerState) func(*ateapipb.Worker) { return func(w *ateapipb.Worker) { w.Status.State = state diff --git a/cmd/ateapi/internal/store/atepg/atepg.go b/cmd/ateapi/internal/store/atepg/atepg.go index 63db45d5f9..d5945f1f70 100644 --- a/cmd/ateapi/internal/store/atepg/atepg.go +++ b/cmd/ateapi/internal/store/atepg/atepg.go @@ -52,16 +52,24 @@ const ( // Migrations need one connection for Goose's session lock and one for // migration work. Outbox maintenance is serial after startup. ownerPoolMaxConns = 2 + // The readiness probe pings over one connection that nothing else + // borrows. With no minimum it dials on the first probe. + probePoolMaxConns = 1 + probePoolMinConns = 0 ) type Persistence struct { pool *pgxpool.Pool // watchPool serves WatchWorkers pollers and expired-lease cleanup. // ownerPool applies migrations and maintains outbox partitions. + // probePool serves only Ping, so request traffic can never hold the + // connection a readiness probe needs. watchPool *pgxpool.Pool ownerPool *pgxpool.Pool + probePool *pgxpool.Pool ownsWatchPool bool ownsOwnerPool bool + ownsProbePool bool policyManager *authz.PolicyManager leaseTTL time.Duration pollFailureCloseAfter time.Duration @@ -125,7 +133,8 @@ type ConnectConfig struct { PoolMaxConns int32 } -// Connect opens read/write and owner pools. It creates the schema and applies migrations. +// Connect opens the read/write, owner, watch, and probe pools. It creates the +// schema and applies migrations. func Connect(ctx context.Context, config ConnectConfig) (*Persistence, error) { if config.Schema == "" { return nil, fmt.Errorf("PostgreSQL schema must not be empty") @@ -184,18 +193,23 @@ func Connect(ctx context.Context, config ConnectConfig) (*Persistence, error) { return nil, err } - watchCfg := readWriteConfig.Copy() - watchCfg.MaxConns = watchPoolMaxConns - watchCfg.MinConns = watchPoolMinConns - watchPool, err := pgxpool.NewWithConfig(ctx, watchCfg) + watchPool, err := pgxpool.NewWithConfig(ctx, resizedPoolConfig(readWriteConfig, watchPoolMaxConns, watchPoolMinConns)) if err != nil { ownerPool.Close() pool.Close() return nil, fmt.Errorf("opening PostgreSQL watch pool: %w", err) } + probePool, err := pgxpool.NewWithConfig(ctx, resizedPoolConfig(readWriteConfig, probePoolMaxConns, probePoolMinConns)) + if err != nil { + watchPool.Close() + ownerPool.Close() + pool.Close() + return nil, fmt.Errorf("opening PostgreSQL probe pool: %w", err) + } - p, err := newPersistence(ctx, pool, watchPool, ownerPool) + p, err := newPersistence(ctx, pool, watchPool, ownerPool, probePool) if err != nil { + probePool.Close() watchPool.Close() ownerPool.Close() pool.Close() @@ -203,9 +217,20 @@ func Connect(ctx context.Context, config ConnectConfig) (*Persistence, error) { } p.ownsWatchPool = true p.ownsOwnerPool = true + p.ownsProbePool = true return p, nil } +// resizedPoolConfig returns a copy of base, the read/write pool config, with +// its own connection limits, so a dedicated pool gets the same role, search +// path, and TLS handling as the main one. +func resizedPoolConfig(base *pgxpool.Config, maxConns, minConns int32) *pgxpool.Config { + cfg := base.Copy() + cfg.MaxConns = maxConns + cfg.MinConns = minConns + return cfg +} + func createSchema(ctx context.Context, pool *pgxpool.Pool, schema string) error { tx, err := pool.Begin(ctx) if err != nil { @@ -268,12 +293,13 @@ func poolConfig(dsn, role string) (*pgxpool.Config, error) { // NewPersistence wraps an already-open pool, applying pending migrations. // Callers that already hold a pool (e.g. tests using testcontainers) use -// this directly instead of Connect; outbox watch traffic shares the given pool. +// this directly instead of Connect; outbox watch traffic and Ping share the +// given pool. func NewPersistence(ctx context.Context, pool *pgxpool.Pool) (*Persistence, error) { - return newPersistence(ctx, pool, pool, pool) + return newPersistence(ctx, pool, pool, pool, pool) } -func newPersistence(ctx context.Context, pool, watchPool, ownerPool *pgxpool.Pool) (*Persistence, error) { +func newPersistence(ctx context.Context, pool, watchPool, ownerPool, probePool *pgxpool.Pool) (*Persistence, error) { if err := applyMigrations(ctx, ownerPool); err != nil { return nil, err } @@ -282,6 +308,7 @@ func newPersistence(ctx context.Context, pool, watchPool, ownerPool *pgxpool.Poo pool: pool, watchPool: watchPool, ownerPool: ownerPool, + probePool: probePool, leaseTTL: defaultLeaseTTL, pollFailureCloseAfter: outboxPollFailureCloseAfter, stopMaintenance: stopMaintenance, @@ -319,6 +346,16 @@ func (p *Persistence) Close() { if p.ownsOwnerPool { p.ownerPool.Close() } + if p.ownsProbePool { + p.probePool.Close() + } +} + +// Ping reports whether PostgreSQL answers. It uses a connection reserved for +// it, so a saturated request pool cannot make it fail: it fails only when the +// database does not answer within ctx. +func (p *Persistence) Ping(ctx context.Context) error { + return p.probePool.Ping(ctx) } // Pool returns the underlying PostgreSQL connection pool. diff --git a/cmd/ateapi/internal/store/atepg/connect_test.go b/cmd/ateapi/internal/store/atepg/connect_test.go index ddf1b723c4..2ac97f4ebd 100644 --- a/cmd/ateapi/internal/store/atepg/connect_test.go +++ b/cmd/ateapi/internal/store/atepg/connect_test.go @@ -350,6 +350,13 @@ func TestConnectSeparatesRuntimeAndDDLPrivileges(t *testing.T) { if got := p.ownerPool.Config().MaxConns; got != ownerPoolMaxConns { t.Fatalf("owner pool MaxConns = %d, want %d", got, ownerPoolMaxConns) } + var probeRole string + if err := p.probePool.QueryRow(ctx, `SELECT current_user`).Scan(&probeRole); err != nil || probeRole != runtimeRole { + t.Fatalf("probe pool role = %q, %v; want the read/write role %q", probeRole, err, runtimeRole) + } + if got := p.probePool.Config().MaxConns; got != probePoolMaxConns { + t.Fatalf("probe pool MaxConns = %d, want %d", got, probePoolMaxConns) + } if _, err := p.CreateAtespace(ctx, newTestAtespace("runtime-write")); err != nil { t.Fatalf("read/write operation failed: %v", err) diff --git a/cmd/ateapi/internal/store/atepg/ping_test.go b/cmd/ateapi/internal/store/atepg/ping_test.go new file mode 100644 index 0000000000..dc6ea8c941 --- /dev/null +++ b/cmd/ateapi/internal/store/atepg/ping_test.go @@ -0,0 +1,206 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atepg + +import ( + "context" + "fmt" + "io" + "net" + "net/url" + "sync" + "testing" + "time" +) + +// connectForPing connects to the shared container with a fresh schema that is +// dropped when the test ends. The returned config lets a test adjust the DSN +// or pool size before connecting. +func connectForPing(t *testing.T, schema string, adjust func(*ConnectConfig)) *Persistence { + t.Helper() + admin := requirePool(t) + ctx := t.Context() + drop := fmt.Sprintf(`DROP SCHEMA IF EXISTS %q CASCADE`, schema) + if _, err := admin.Exec(ctx, drop); err != nil { + t.Fatalf("resetting PostgreSQL schema: %v", err) + } + t.Cleanup(func() { _, _ = admin.Exec(context.Background(), drop) }) + + cfg := testConnectConfig(schema) + adjust(&cfg) + p, err := Connect(ctx, cfg) + if err != nil { + t.Fatalf("Connect failed: %v", err) + } + t.Cleanup(func() { + p.Close() + p.pool.Close() + }) + return p +} + +// A saturated request pool must not fail the readiness probe: the probe cannot +// tell a busy replica from an unreachable store if it borrows from that pool. +func TestPingIgnoresSaturatedRequestPool(t *testing.T) { + p := connectForPing(t, "ping-saturated", func(cfg *ConnectConfig) { cfg.PoolMaxConns = 1 }) + ctx := t.Context() + + if p.probePool == p.pool { + t.Fatal("probe pool is the request pool") + } + if got := p.probePool.Config().MaxConns; got != probePoolMaxConns { + t.Errorf("probe pool MaxConns = %d, want %d", got, probePoolMaxConns) + } + if got := p.probePool.Config().MinConns; got != probePoolMinConns { + t.Errorf("probe pool MinConns = %d, want %d", got, probePoolMinConns) + } + + held, err := p.pool.Acquire(ctx) + if err != nil { + t.Fatalf("acquiring the only request connection: %v", err) + } + defer held.Release() + + budget := func() (context.Context, context.CancelFunc) { + return context.WithTimeout(ctx, 500*time.Millisecond) + } + + // The premise: the request pool really is exhausted, and a ping through it + // blocks past the readiness budget. + poolCtx, cancel := budget() + defer cancel() + if err := p.pool.Ping(poolCtx); err == nil { + t.Fatal("request pool ping succeeded while every connection was held; the pool is not saturated") + } + + probeCtx, cancel := budget() + defer cancel() + if err := p.Ping(probeCtx); err != nil { + t.Errorf("Ping with a saturated request pool = %v, want nil", err) + } +} + +func TestPingFailsAfterClose(t *testing.T) { + p := connectForPing(t, "ping-closed", func(*ConnectConfig) {}) + if err := p.Ping(t.Context()); err != nil { + t.Fatalf("Ping before Close = %v, want nil", err) + } + p.Close() + if err := p.Ping(t.Context()); err == nil { + t.Error("Ping after Close succeeded, want an error from the closed probe pool") + } +} + +// Ping fails while PostgreSQL does not answer and passes again once it does, +// with no restart of the pool. +func TestPingFollowsStoreReachability(t *testing.T) { + requirePool(t) + dsn, err := url.Parse(containerDSN) + if err != nil { + t.Fatalf("parsing PostgreSQL DSN: %v", err) + } + proxy := newTCPProxy(t, dsn.Host) + dsn.Host = proxy.addr + p := connectForPing(t, "ping-reachability", func(cfg *ConnectConfig) { cfg.ReadWriteDSN = dsn.String() }) + + ping := func() error { + ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) + defer cancel() + return p.Ping(ctx) + } + if err := ping(); err != nil { + t.Fatalf("Ping with the store reachable = %v, want nil", err) + } + proxy.stop() + if err := ping(); err == nil { + t.Error("Ping with the store unreachable succeeded, want an error") + } + proxy.start() + if err := ping(); err != nil { + t.Errorf("Ping after the store became reachable = %v, want nil", err) + } +} + +// tcpProxy forwards TCP connections to target until stopped, and start brings +// it back on the same address. +type tcpProxy struct { + t *testing.T + target string + addr string + + mu sync.Mutex + ln net.Listener + conns []net.Conn +} + +func newTCPProxy(t *testing.T, target string) *tcpProxy { + t.Helper() + p := &tcpProxy{t: t, target: target, addr: "127.0.0.1:0"} + p.start() + t.Cleanup(p.stop) + return p +} + +func (p *tcpProxy) start() { + p.t.Helper() + ln, err := net.Listen("tcp", p.addr) + if err != nil { + p.t.Fatalf("listening on %s: %v", p.addr, err) + } + p.mu.Lock() + p.ln = ln + p.addr = ln.Addr().String() + p.mu.Unlock() + go func() { + for { + client, err := ln.Accept() + if err != nil { + return + } + go p.forward(client) + } + }() +} + +func (p *tcpProxy) forward(client net.Conn) { + upstream, err := net.Dial("tcp", p.target) + if err != nil { + client.Close() + return + } + p.mu.Lock() + p.conns = append(p.conns, client, upstream) + p.mu.Unlock() + go func() { + io.Copy(upstream, client) //nolint:errcheck // Either side closing ends the copy. + upstream.Close() + }() + io.Copy(client, upstream) //nolint:errcheck // Either side closing ends the copy. + client.Close() +} + +// stop refuses new connections and drops the open ones. +func (p *tcpProxy) stop() { + p.mu.Lock() + defer p.mu.Unlock() + if p.ln != nil { + p.ln.Close() + p.ln = nil + } + for _, c := range p.conns { + c.Close() + } + p.conns = nil +} diff --git a/cmd/ateapi/internal/workerservice/certificate_test.go b/cmd/ateapi/internal/workerservice/certificate_test.go index 1bbeeeb084..6982d389ef 100644 --- a/cmd/ateapi/internal/workerservice/certificate_test.go +++ b/cmd/ateapi/internal/workerservice/certificate_test.go @@ -36,7 +36,30 @@ import ( type fakeActorStore struct { store.Interface - actors map[resources.ActorRef]*ateapipb.Actor + actors map[resources.ActorRef]*ateapipb.Actor + workers map[string]*ateapipb.Worker + // assignments maps an actor UID to the worker holding its assignment. + assignments map[string]string + // assignmentErr, when set, fails every GetWorkerAssignment. + assignmentErr error +} + +func (f *fakeActorStore) GetWorkerAssignment(_ context.Context, workerName, actorUID string) (*ateapipb.ActorAssignment, error) { + if f.assignmentErr != nil { + return nil, f.assignmentErr + } + if holder, ok := f.assignments[actorUID]; !ok || holder != workerName { + return nil, store.ErrNotFound + } + return &ateapipb.ActorAssignment{ActorUid: actorUID}, nil +} + +func (f *fakeActorStore) GetWorker(_ context.Context, name string) (*ateapipb.Worker, error) { + w, ok := f.workers[name] + if !ok { + return nil, store.ErrNotFound + } + return w, nil } func (f *fakeActorStore) GetActor(_ context.Context, actorRef resources.ActorRef) (*ateapipb.Actor, error) { @@ -59,13 +82,19 @@ func newFakeStoreWithActor(atespace, name, uid string) (*fakeActorStore, *ateapi Name: "template-1", }, Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + WorkerAssignment: &ateapipb.WorkerAssignment{Worker: &ateapipb.ObjectRef{Name: "worker-1"}}, }, } return &fakeActorStore{ actors: map[resources.ActorRef]*ateapipb.Actor{ {Atespace: atespace, Name: name}: actor, }, + // worker-1 runs on node-1, the node the tests' atelet certificate names. + workers: map[string]*ateapipb.Worker{ + "worker-1": {Metadata: &ateapipb.ResourceMetadata{Name: "worker-1"}, NodeName: "node-1"}, + }, + assignments: map[string]string{uid: "worker-1"}, }, actor } @@ -97,7 +126,7 @@ func newTestCAPool(t *testing.T) *localca.ConcretePool { func TestMintAteomActorCertificate(t *testing.T) { st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") caPool := newTestCAPool(t) - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool, "", nil) csr, key := generateTestCSR(t) @@ -149,7 +178,7 @@ func TestMintAteomActorCertificate(t *testing.T) { func TestMintAteomActorCertificate_Errors(t *testing.T) { st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") caPool := newTestCAPool(t) - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool, "", nil) csr, _ := generateTestCSR(t) authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1")) diff --git a/cmd/ateapi/internal/workerservice/jwt.go b/cmd/ateapi/internal/workerservice/jwt.go new file mode 100644 index 0000000000..131fe5af37 --- /dev/null +++ b/cmd/ateapi/internal/workerservice/jwt.go @@ -0,0 +1,118 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package workerservice + +import ( + "context" + "crypto/rand" + "errors" + "fmt" + "log/slog" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/apivalidation" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/ateletauth" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/controlapi" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/internal/actoridjwt" + "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/types/known/timestamppb" +) + +// ateomActorJWTLifetime is the fixed lifetime of an ateom-for-actor JWT. It +// is not chosen by the caller: the token's only consumers are node-side +// plugins that exchange it for short-lived credentials. +const ateomActorJWTLifetime = time.Hour + +func (s *Server) MintAteomActorJwt(ctx context.Context, req *ateapipb.MintAteomActorJwtRequest) (*ateapipb.MintAteomActorJwtResponse, error) { + if errs := apivalidation.ValidateMintAteomActorJwtRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToAPIError(errs) + } + + caller, err := ateletauth.Authenticate(ctx, s.ateletSPIFFEID) + if err != nil { + return nil, err + } + + // The actor doesn't need to be running: tokens are needed to fetch a + // snapshot before resume and to upload a checkpoint after pause. + dbActor, err := s.store.GetActor(ctx, resources.ActorRefFromObjectRef(req.GetActor())) + if errors.Is(err, store.ErrNotFound) { + return nil, apierror.NotFound("actor not found") + } else if err != nil { + return nil, fmt.Errorf("while retrieving actor: %w", err) + } + // The node check comes first so a node that does not host the actor gets + // NotFound whatever UID it sends, and cannot use the UID check to learn + // that the actor exists. + if err := s.checkActorOnNode(ctx, dbActor, caller.NodeName); err != nil { + return nil, err + } + // A stale UID is Aborted, as in MintAteomActorCertificate. + if dbActor.GetMetadata().GetUid() != req.GetActorUid() { + return nil, apierror.Aborted("conflict; actor has been deleted and recreated") + } + + // JWT timestamps have one-second resolution. Truncating keeps expire_time + // equal to the exp claim. + now := time.Now().Truncate(time.Second) + expiresAt := now.Add(ateomActorJWTLifetime) + meta := dbActor.GetMetadata() + claims := &actoridjwt.Claims{ + Kind: actoridjwt.KindAteomForActor, + Issuer: s.actorJWTIssuer, + Subject: fmt.Sprintf("%s/%s/%s", actoridjwt.KindAteomForActor.SubjectPrefix(), meta.GetAtespace(), meta.GetName()), + Audiences: req.GetAudiences(), + Expiration: expiresAt, + NotBefore: now.Add(-5 * time.Minute), + IssuedAt: now, + JTI: rand.Text(), + + Substrate: actoridjwt.SubstrateClaims{ + Atespace: meta.GetAtespace(), + ActorName: meta.GetName(), + ActorUID: meta.GetUid(), + }, + } + token, err := s.actorIDJWTPool.SignJWT(claims) + if err != nil { + return nil, fmt.Errorf("while signing ateom-for-actor JWT: %w", err) + } + return &ateapipb.MintAteomActorJwtResponse{ + ActorJwt: token, + ExpireTime: timestamppb.New(expiresAt), + }, nil +} + +// checkActorOnNode refuses a mint from an atelet whose node does not host the +// actor, as RequestActorSuspend does. Authoritative state decides, never the +// request. The refusal is NotFound, so a node learns nothing about actors it +// does not host. +func (s *Server) checkActorOnNode(ctx context.Context, actor *ateapipb.Actor, callerNode string) error { + hostingNode, err := controlapi.HostingNode(ctx, s.store, actor) + if err != nil { + return err + } + if hostingNode == "" || hostingNode != callerNode { + slog.WarnContext(ctx, "Refusing an ateom-for-actor JWT for an actor the caller's node does not host", + slog.String("actor", resources.ActorRefFromActor(actor).String()), + slog.String("hosting_node", hostingNode), + slog.String("caller_node", callerNode)) + return apierror.NotFound("actor not found") + } + return nil +} diff --git a/cmd/ateapi/internal/workerservice/jwt_test.go b/cmd/ateapi/internal/workerservice/jwt_test.go new file mode 100644 index 0000000000..0725f21306 --- /dev/null +++ b/cmd/ateapi/internal/workerservice/jwt_test.go @@ -0,0 +1,411 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package workerservice + +import ( + "context" + "encoding/base64" + "encoding/json" + "errors" + "strings" + "testing" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/ateletauth/ateletauthtest" + "github.com/agent-substrate/substrate/internal/actoridjwt" + "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/localjwtauthority" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/grpc/codes" +) + +const testActorJWTIssuer = "https://idp.ate-system.svc" + +func newTestJWTPool(t *testing.T) localjwtauthority.Pool { + t.Helper() + authority, err := localjwtauthority.GenerateAuthority("ES256", "1") + if err != nil { + t.Fatalf("generate JWT authority: %v", err) + } + return &localjwtauthority.ConcretePool{Authorities: []*localjwtauthority.Authority{authority}, ActiveForSigning: "1"} +} + +// decodeJWTClaims returns the payload of a compact JWS without verifying it. +// The signing path is covered by the pool's own tests. +func decodeJWTClaims(t *testing.T, token string) actoridjwt.WireClaims { + t.Helper() + parts := strings.Split(token, ".") + if len(parts) != 3 { + t.Fatalf("token has %d parts, want 3", len(parts)) + } + payload, err := base64.RawURLEncoding.DecodeString(parts[1]) + if err != nil { + t.Fatalf("decode payload: %v", err) + } + var claims actoridjwt.WireClaims + if err := json.Unmarshal(payload, &claims); err != nil { + t.Fatalf("unmarshal payload: %v", err) + } + return claims +} + +func TestMintAteomActorJwt(t *testing.T) { + st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t)) + + ctx := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1")) + before := time.Now().Truncate(time.Second) + resp, err := s.MintAteomActorJwt(ctx, &ateapipb.MintAteomActorJwtRequest{ + Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, + ActorUid: actor.GetMetadata().GetUid(), + Audiences: []string{"snapshot-plugin"}, + }) + if err != nil { + t.Fatalf("MintAteomActorJwt() failed: %v", err) + } + + claims := decodeJWTClaims(t, resp.GetActorJwt()) + if claims.Issuer != testActorJWTIssuer { + t.Errorf("iss = %q, want %q", claims.Issuer, testActorJWTIssuer) + } + if want := "ateom-for-actor/team-a/my-actor"; claims.Subject != want { + t.Errorf("sub = %q, want %q", claims.Subject, want) + } + var audiences []string + if err := json.Unmarshal(claims.Audiences, &audiences); err != nil { + t.Fatalf("aud is not a JSON array: %s", claims.Audiences) + } + if len(audiences) != 1 || audiences[0] != "snapshot-plugin" { + t.Errorf("aud = %v, want [snapshot-plugin]", audiences) + } + kind, substrate, ok := claims.Substrate() + if !ok || kind != actoridjwt.KindAteomForActor { + t.Fatalf("token kind = %v (ok %v), want KindAteomForActor under ateom-for-actor.ate.dev", kind, ok) + } + if got, want := *substrate, (actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "my-actor", ActorUID: actor.GetMetadata().GetUid()}); got != want { + t.Errorf("ateom-for-actor.ate.dev = %+v, want %+v", got, want) + } + + expiresAt := resp.GetExpireTime().AsTime() + if got := time.Unix(int64(claims.Expiration), 0); !got.Equal(expiresAt) { + t.Errorf("exp = %v, expire_time = %v; want equal", got, expiresAt) + } + lifetime := expiresAt.Sub(time.Unix(int64(claims.IssuedAt), 0)) + if lifetime != ateomActorJWTLifetime { + t.Errorf("exp - iat = %v, want %v", lifetime, ateomActorJWTLifetime) + } + if expiresAt.Before(before.Add(ateomActorJWTLifetime)) { + t.Errorf("expire_time = %v is earlier than %v + lifetime", expiresAt, before) + } +} + +// pausedStatus is how a paused actor looks: SUSPENDED, no worker, assigned_node +// on the node holding its local snapshot, and a latest generation that is +// local-only. The latest generation is the one last_assigned_generation names. +func pausedStatus(state ateapipb.ActorState, node string) *ateapipb.ActorStatus { + return &ateapipb.ActorStatus{ + State: state, + AssignedNode: node, + LastAssignedGeneration: 1, + Snapshots: []*ateapipb.Snapshot{{Uuid: "snap-1", Generation: 1, Owner: ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, Locality: node}}, + } +} + +func TestMintAteomActorJwt_Errors(t *testing.T) { + st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t)) + authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1")) + + valid := func() *ateapipb.MintAteomActorJwtRequest { + return &ateapipb.MintAteomActorJwtRequest{ + Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, + ActorUid: actor.GetMetadata().GetUid(), + Audiences: []string{"snapshot-plugin"}, + } + } + const otherActorUID = "9a8b7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d" + // onWorker is an actor status naming worker-1, which runs on node-1. + onWorker := func(state ateapipb.ActorState) *ateapipb.ActorStatus { + return &ateapipb.ActorStatus{State: state, WorkerAssignment: &ateapipb.WorkerAssignment{Worker: &ateapipb.ObjectRef{Name: "worker-1"}}} + } + // pausingOnWorker is a pause in progress: SUSPENDING on worker-1 with a + // latest snapshot that is node-local and not yet captured. + pausingOnWorker := func() *ateapipb.ActorStatus { + status := onWorker(ateapipb.ActorState_ACTOR_STATE_SUSPENDING) + status.AssignedNode = "node-1" + status.LastAssignedGeneration = 1 + status.Snapshots = []*ateapipb.Snapshot{{Uuid: "snap-1", Generation: 1, Owner: ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR}} + return status + } + released := map[string]string{} + rebooked := map[string]string{otherActorUID: "worker-1"} + tests := []struct { + name string + ctx context.Context + mutate func(*ateapipb.MintAteomActorJwtRequest) + // status, when set, replaces the actor's status for the case. + status *ateapipb.ActorStatus + // assignments, when set, replaces the store's worker assignments + // (actor UID to worker) for the case. By default worker-1 holds the + // actor. + assignments map[string]string + // assignmentErr, when set, fails the store's worker assignment + // lookup. + assignmentErr error + want codes.Code + }{ + { + name: "running actor on its worker's node", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RUNNING), + want: codes.OK, + }, + { + name: "resuming actor on its worker's node", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RESUMING), + want: codes.OK, + }, + { + // Suspending a running actor uploads its checkpoint from its worker. + name: "running actor being suspended on its worker's node", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_SUSPENDING), + want: codes.OK, + }, + { + name: "running actor being paused on its worker's node", + ctx: authed, + status: pausingOnWorker(), + want: codes.OK, + }, + { + // Pause, suspend and crash release the worker before they clear the + // actor's status, which keeps naming the worker until they do. + name: "running actor whose worker was released", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RUNNING), + assignments: released, + want: codes.NotFound, + }, + { + // A resume that finds its worker ineligible releases it, then + // crashes the actor. + name: "resuming actor whose worker was released", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RESUMING), + assignments: released, + want: codes.NotFound, + }, + { + name: "suspending actor whose worker was released", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_SUSPENDING), + assignments: released, + want: codes.NotFound, + }, + { + name: "pausing actor whose worker was released", + ctx: authed, + status: pausingOnWorker(), + assignments: released, + want: codes.NotFound, + }, + { + // The released worker took another actor. Its node hosts that one, + // not this one. + name: "running actor whose worker was rebooked for another actor", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RUNNING), + assignments: rebooked, + want: codes.NotFound, + }, + { + name: "resuming actor whose worker was rebooked for another actor", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RESUMING), + assignments: rebooked, + want: codes.NotFound, + }, + { + name: "suspending actor whose worker was rebooked for another actor", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_SUSPENDING), + assignments: rebooked, + want: codes.NotFound, + }, + { + name: "pausing actor whose worker was rebooked for another actor", + ctx: authed, + status: pausingOnWorker(), + assignments: rebooked, + want: codes.NotFound, + }, + { + // The store binds the actor to a different worker than its status + // names, so the status's worker does not host it. + name: "actor held by a worker other than the one its status names", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RUNNING), + assignments: map[string]string{actor.GetMetadata().GetUid(): "worker-2"}, + want: codes.NotFound, + }, + { + // A failed lookup refuses the mint rather than trusting the status. + name: "worker assignment lookup fails", + ctx: authed, + status: onWorker(ateapipb.ActorState_ACTOR_STATE_RUNNING), + assignmentErr: errors.New("connection reset"), + want: codes.Internal, + }, + { + name: "unauthenticated", + ctx: ateletauthtest.ContextWith(nil), + want: codes.Unauthenticated, + }, + { + name: "missing actor", + ctx: authed, + mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Actor = nil }, + want: codes.InvalidArgument, + }, + { + name: "missing actor uid", + ctx: authed, + mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.ActorUid = "" }, + want: codes.InvalidArgument, + }, + { + name: "missing audiences", + ctx: authed, + mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Audiences = nil }, + want: codes.InvalidArgument, + }, + { + name: "empty audience", + ctx: authed, + mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Audiences = []string{""} }, + want: codes.InvalidArgument, + }, + { + // The actor's worker runs on node-1 and the caller proves it is on node-2. + name: "actor hosted on another node", + ctx: ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-2")), + want: codes.NotFound, + }, + { + // A foreign node must not learn that the actor exists from the UID + // check answering Aborted: the node check runs first. + name: "actor hosted on another node, stale uid", + ctx: ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-2")), + mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.ActorUid = "7d1c0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a" }, + want: codes.NotFound, + }, + { + name: "paused actor on this node", + ctx: authed, + status: pausedStatus(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-1"), + want: codes.OK, + }, + { + name: "paused actor on another node", + ctx: authed, + status: pausedStatus(ateapipb.ActorState_ACTOR_STATE_SUSPENDED, "node-2"), + want: codes.NotFound, + }, + { + // assigned_node without a local-only snapshot is not a paused actor, + // whatever left it set. + name: "suspended actor with a stale assigned_node", + ctx: authed, + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, AssignedNode: "node-1"}, + want: codes.NotFound, + }, + { + // Suspending a paused actor uploads its checkpoint from the node that + // holds it, with no worker assigned. + name: "paused actor being suspended on this node", + ctx: authed, + status: pausedStatus(ateapipb.ActorState_ACTOR_STATE_SUSPENDING, "node-1"), + want: codes.OK, + }, + { + // A crash clears the worker but leaves assigned_node on the last + // node. Nothing is hosted there, so nothing may be minted. + name: "crashed actor on its last node", + ctx: authed, + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_CRASHED, AssignedNode: "node-1"}, + want: codes.NotFound, + }, + { + // A revert releases the worker but keeps the assignment until it + // finalizes. The node no longer runs the actor. + name: "reverting actor with a stale assignment", + ctx: authed, + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_REVERTING, WorkerAssignment: &ateapipb.WorkerAssignment{Worker: &ateapipb.ObjectRef{Name: "worker-1"}}}, + want: codes.NotFound, + }, + { + // A suspended actor is hosted nowhere, so no node may mint for it. + name: "actor hosted by no node", + ctx: authed, + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, + want: codes.NotFound, + }, + { + name: "actor assigned to a worker the store has lost", + ctx: authed, + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING, WorkerAssignment: &ateapipb.WorkerAssignment{Worker: &ateapipb.ObjectRef{Name: "worker-gone"}}}, + want: codes.NotFound, + }, + { + name: "actor not found", + ctx: authed, + mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Actor.Name = "nonexistent" }, + want: codes.NotFound, + }, + { + // A recreated actor has a new UID, and a token must not be minted + // for the old incarnation. Aborted, as MintAteomActorCertificate answers. + name: "stale actor uid", + ctx: authed, + mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.ActorUid = "7d1c0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a" }, + want: codes.Aborted, + }, + } + hostedStatus := actor.GetStatus() + hostedAssignments := st.assignments + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + actor.Status = hostedStatus + if tc.status != nil { + actor.Status = tc.status + } + st.assignments = hostedAssignments + if tc.assignments != nil { + st.assignments = tc.assignments + } + st.assignmentErr = tc.assignmentErr + req := valid() + if tc.mutate != nil { + tc.mutate(req) + } + _, err := s.MintAteomActorJwt(tc.ctx, req) + if got := apierror.Code(err); got != tc.want { + t.Errorf("MintAteomActorJwt() code = %v, want %v (err: %v)", got, tc.want, err) + } + }) + } +} diff --git a/cmd/ateapi/internal/workerservice/register_test.go b/cmd/ateapi/internal/workerservice/register_test.go index 52e0cc15be..de99e677a1 100644 --- a/cmd/ateapi/internal/workerservice/register_test.go +++ b/cmd/ateapi/internal/workerservice/register_test.go @@ -61,7 +61,7 @@ func setRequest(actors int32) *ateapipb.RegisterWorkerRequest { func TestRegisterWorker(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil) seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(2000, 0)}) got, err := s.RegisterWorker(ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)), setRequest(4094)) @@ -85,7 +85,7 @@ func TestRegisterWorker(t *testing.T) { func TestRegisterWorker_Runtimes(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil) authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)) gvisorAMD64, gvisorARM64 := testRuntime("gvisor", "amd64"), testRuntime("gvisor", "arm64") @@ -220,7 +220,7 @@ func TestRegisterWorker_Runtimes(t *testing.T) { func TestRegisterWorker_OtherNodeIsNotFound(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil) seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1}) _, err := s.RegisterWorker(ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "some-other-node")), setRequest(4094)) @@ -244,7 +244,7 @@ func TestRegisterWorker_OtherNodeIsNotFound(t *testing.T) { func TestRegisterWorker_UnchangedDoesNotWrite(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil) seeded := seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 4094}) for range 3 { @@ -267,7 +267,7 @@ func TestRegisterWorker_UnchangedDoesNotWrite(t *testing.T) { func TestRegisterWorker_SortsAttributes(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil) seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1}) authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)) @@ -310,7 +310,7 @@ func TestRegisterWorker_SortsAttributes(t *testing.T) { func TestRegisterWorker_Errors(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil) seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1}) authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)) @@ -355,7 +355,7 @@ func TestRegisterWorker_Errors(t *testing.T) { func TestRegisterWorker_RejectsNonsense(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() - s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil) + s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, "", nil) seeded := seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 4094}) authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)) diff --git a/cmd/ateapi/internal/workerservice/service.go b/cmd/ateapi/internal/workerservice/service.go index 4087818380..7ea05eb484 100644 --- a/cmd/ateapi/internal/workerservice/service.go +++ b/cmd/ateapi/internal/workerservice/service.go @@ -21,6 +21,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/localca" + "github.com/agent-substrate/substrate/internal/localjwtauthority" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -41,6 +42,12 @@ type Server struct { ateletSPIFFEID string actorIDCAPool localca.Pool + + // actorJWTIssuer is copied verbatim into the iss claim of every + // ateom-for-actor JWT, and actorIDJWTPool signs them. Both are the ones + // Control.MintActorJwt uses. + actorJWTIssuer string + actorIDJWTPool localjwtauthority.Pool } // actorSuspender is the in-process slice of the Control service this package @@ -51,11 +58,13 @@ type actorSuspender interface { var _ ateapipb.WorkerServiceServer = (*Server)(nil) -func New(store store.Interface, suspender actorSuspender, ateletSPIFFEID string, actorIDCAPool localca.Pool) *Server { +func New(store store.Interface, suspender actorSuspender, ateletSPIFFEID string, actorIDCAPool localca.Pool, actorJWTIssuer string, actorIDJWTPool localjwtauthority.Pool) *Server { return &Server{ store: store, suspender: suspender, ateletSPIFFEID: ateletSPIFFEID, actorIDCAPool: actorIDCAPool, + actorJWTIssuer: actorJWTIssuer, + actorIDJWTPool: actorIDJWTPool, } } diff --git a/cmd/ateapi/internal/workerservice/suspend_test.go b/cmd/ateapi/internal/workerservice/suspend_test.go index 1423755d04..408432e2ea 100644 --- a/cmd/ateapi/internal/workerservice/suspend_test.go +++ b/cmd/ateapi/internal/workerservice/suspend_test.go @@ -79,7 +79,7 @@ func TestRequestActorSuspend(t *testing.T) { Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, } suspender := &fakeSuspender{resp: &ateapipb.SuspendActorResponse{Actor: suspended}} - s := New(st, suspender, testAteletSPIFFEID, nil) + s := New(st, suspender, testAteletSPIFFEID, nil, "", nil) got, err := s.RequestActorSuspend( ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)), @@ -127,7 +127,7 @@ func TestRequestActorSuspend_OnlyForHostedActors(t *testing.T) { seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1}) actor := seedHostedActor(t, st, tc.assignedWorker) suspender := &fakeSuspender{} - s := New(st, suspender, testAteletSPIFFEID, nil) + s := New(st, suspender, testAteletSPIFFEID, nil, "", nil) uid := tc.requestedUID if uid == "" { @@ -153,7 +153,7 @@ func TestRequestActorSuspend_Errors(t *testing.T) { actor := seedHostedActor(t, st, testWorkerName) uid := actor.GetMetadata().GetUid() suspender := &fakeSuspender{} - s := New(st, suspender, testAteletSPIFFEID, nil) + s := New(st, suspender, testAteletSPIFFEID, nil, "", nil) authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)) tests := []struct { @@ -232,7 +232,7 @@ func TestRequestActorSuspend_GoldenActorCannotSelfSuspend(t *testing.T) { }, }) suspender := &fakeSuspender{} - s := New(st, suspender, testAteletSPIFFEID, nil) + s := New(st, suspender, testAteletSPIFFEID, nil, "", nil) // Everything else about the request is in order: the Worker really does // host this actor, so being golden is the only reason it is refused. @@ -260,7 +260,7 @@ func TestRequestActorSuspend_PassesThroughTheWorkflowsRefusal(t *testing.T) { seedReportedWorker(t, st, testNode, &ateapipb.WorkerResources{Actors: 1}) actor := seedHostedActor(t, st, testWorkerName) suspender := &fakeSuspender{err: apierror.FailedPrecondition("Actor is RESUMING")} - s := New(st, suspender, testAteletSPIFFEID, nil) + s := New(st, suspender, testAteletSPIFFEID, nil, "", nil) _, err := s.RequestActorSuspend( ateletauthtest.ContextWith(ateletauthtest.CertOn(t, testNode)), diff --git a/cmd/ateapi/main.go b/cmd/ateapi/main.go index f2bcd717fa..4293d38dd7 100644 --- a/cmd/ateapi/main.go +++ b/cmd/ateapi/main.go @@ -80,7 +80,7 @@ var ( postgresReadWriteRole = pflag.String("postgres-read-write-role", "", "Required PostgreSQL role assumed by read/write connections.") postgresOwnerRole = pflag.String("postgres-owner-role", "", "Required PostgreSQL role assumed by owner connections.") postgresSchema = pflag.String("postgres-schema", "substrate", "PostgreSQL schema for Substrate tables. This overrides a search_path connection parameter.") - postgresPoolMaxConns = pflag.Int32("postgres-pool-max-conns", 0, "Maximum connections in the shared Substrate and OpenFGA read/write PostgreSQL pool. Does not affect the owner or watch pools. The DSN or pgx default is used when unset.") + postgresPoolMaxConns = pflag.Int32("postgres-pool-max-conns", 0, "Maximum connections in the shared Substrate and OpenFGA read/write PostgreSQL pool. Does not affect the owner, watch, or readiness probe pools. The DSN or pgx default is used when unset.") experimentalEnableAuthz = pflag.Bool("experimental-enable-authz", false, "Enforce OpenFGA authorization checks on all registered RPCs (experimental). AccessPolicy RPCs are always checked.") // TODO: Move the authz settings into the hot-reloadable config proto // (agent-substrate/substrate#2021) once it lands, so bootstrap owner @@ -372,9 +372,14 @@ func main() { ) reflection.Register(mux) ateapipb.RegisterControlServer(mux, controlSrv) - ateapipb.RegisterWorkerServiceServer(mux, workerservice.New(persistence, controlSrv, ateletSPIFFEID, actorIDCAPool)) + ateapipb.RegisterWorkerServiceServer(mux, workerservice.New(persistence, controlSrv, ateletSPIFFEID, actorIDCAPool, *actorJWTIssuer, actorIDJWTAuthorityPool)) + // Every RPC needs the store, so a replica that cannot reach it leaves the + // Service until it can. Liveness stays green: the pool reconnects on its own. + // The check pings over its own connection, so a busy request pool does not + // fail it. readiness := &serverboot.Readiness{} + readiness.AddCheck("store", persistence.Ping) go serverboot.StartHealthServer(ctx, *healthListenAddr, readiness) drainDone := drainOnShutdown(shutdownCtx, mux, readiness) diff --git a/cmd/atecontroller/internal/controllers/workerpool_apply.go b/cmd/atecontroller/internal/controllers/workerpool_apply.go index afa2a3edd5..c282f71bdb 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_apply.go +++ b/cmd/atecontroller/internal/controllers/workerpool_apply.go @@ -99,7 +99,7 @@ const ( // Deployment managed by a WorkerPool. Only fields owned by this controller // are declared here. otel, when it carries an endpoint, is propagated to the // ateom container so it pushes telemetry to that collector. -func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettings, systemNamespace, ateletServiceAccount, routerServiceAccount string) *appsv1ac.DeploymentApplyConfiguration { +func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettings, systemNamespace, ateletServiceAccount, routerServiceAccount, dnsEgressPolicy string) *appsv1ac.DeploymentApplyConfiguration { labels := map[string]string{} annotations := map[string]string{} if wp.Spec.Template != nil { @@ -126,15 +126,13 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettin "--atunnel-client-identity=" + installdefaults.SPIFFEID(systemNamespace, routerServiceAccount), } - // --atunnel-broker-identity is newer than the oldest ateom a rolling - // upgrade still has running. docs/upgrade.md keeps the outgoing worker pool - // serving alongside the new one, and that pool's Deployment is reconciled - // by this controller while still pinned to its old image, which exits on an - // unrecognized flag. An ateom without the flag hardcodes the canonical - // identity, and an ateom with it defaults to the same, so omitting the flag - // when it carries that value is equivalent for both and keeps the upgrade - // intact. A relocated or renamed install passes something else and needs an - // image new enough to accept it, which it necessarily has. + // --atunnel-broker-identity is newer than the oldest ateom a WorkerPool + // may still pin, and an ateom exits on an unrecognized flag. An ateom + // without the flag hardcodes the canonical identity, and an ateom with it + // defaults to the same, so omitting the flag when it carries that value is + // equivalent for both. A relocated or renamed install passes something + // else and needs an image new enough to accept it, which it necessarily + // has. if brokerIdentity := installdefaults.SPIFFEID(systemNamespace, ateletServiceAccount); brokerIdentity != installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace) { args = append(args, "--atunnel-broker-identity="+brokerIdentity) } @@ -142,6 +140,7 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettin args = append(args, "--atunnel-egress-listen-address=0.0.0.0:15001", "--atunnel-egress-trust-bundle="+atunnelEgressTrustMountPath+"/trust-bundle.pem", + "--dns-egress-policy="+dnsEgressPolicy, ) containerAC := corev1ac.Container(). diff --git a/cmd/atecontroller/internal/controllers/workerpool_apply_test.go b/cmd/atecontroller/internal/controllers/workerpool_apply_test.go index d5bd9283ec..94bc08954e 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_apply_test.go +++ b/cmd/atecontroller/internal/controllers/workerpool_apply_test.go @@ -32,6 +32,7 @@ import ( "github.com/agent-substrate/substrate/internal/ateom" "github.com/agent-substrate/substrate/internal/deviceplugin" + "github.com/agent-substrate/substrate/internal/egresspolicy" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/nodepath" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" @@ -222,7 +223,7 @@ func TestBuildDeploymentApplyConfig(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - got := buildDeploymentApplyConfig(tt.wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount) + got := buildDeploymentApplyConfig(tt.wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) if diff := cmp.Diff(tt.want, got); diff != "" { t.Fatalf("buildDeploymentApplyConfig() mismatch (-want +got):\n%s", diff) } @@ -242,7 +243,7 @@ func TestBuildDeploymentApplyConfigMetadata(t *testing.T) { }, }) - got := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount) + got := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) wantLabels := map[string]string{ "project": "agent-substrate", "team": "compute", @@ -300,7 +301,7 @@ func TestSandboxClassToleration(t *testing.T) { }}, }) wp.Spec.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClass{{Name: tt.class}} - ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount).Spec.Template.Spec + ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec var classValues []string hasTemplateToleration := false @@ -348,7 +349,7 @@ func TestMicroVMPodShape(t *testing.T) { t.Run(tt.name, func(t *testing.T) { wp := testWorkerPoolApplyConfig(nil) wp.Spec.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClass{{Name: tt.class}} - ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount).Spec.Template.Spec + ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec // /dev/kvm must come from the device plugin, never a hostPath: a // hostPath mount carries no cgroup device allow rule, and the @@ -435,7 +436,7 @@ func TestMicroVMDeviceRequestsPreserveTemplateResources(t *testing.T) { }, }) wp.Spec.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClass{{Name: atev1alpha1.SandboxClassMicroVM}} - c := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount).Spec.Template.Spec.Containers[0] + c := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec.Containers[0] if got, ok := deviceLimit(c, string(corev1.ResourceMemory)); !ok || got != "2Gi" { t.Errorf("memory limit = %q (present=%v), want 2Gi", got, ok) @@ -500,7 +501,7 @@ func TestAteomSecurityContextByClass(t *testing.T) { // TestTerminationGracePeriodSeconds asserts the pod's grace period is hardcoded to 3600s. func TestTerminationGracePeriodSeconds(t *testing.T) { wp := testWorkerPoolApplyConfig(nil) - ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount).Spec.Template.Spec + ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec if ps.TerminationGracePeriodSeconds == nil { t.Fatalf("TerminationGracePeriodSeconds not set") } @@ -514,7 +515,7 @@ func TestTerminationGracePeriodSeconds(t *testing.T) { // actors' drain window. func TestRolloutStrategy(t *testing.T) { wp := testWorkerPoolApplyConfig(nil) - spec := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount).Spec + spec := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec if spec.Strategy == nil || spec.Strategy.RollingUpdate == nil { t.Fatalf("Strategy.RollingUpdate not set") } @@ -548,7 +549,7 @@ func TestBuildDeploymentApplyConfigOTelEndpoint(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{Endpoint: tt.endpoint}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{Endpoint: tt.endpoint}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] env := envByName(c.Env) @@ -636,7 +637,7 @@ func TestBuildDeploymentApplyConfigMetricExportTuning(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] env := envByName(c.Env) for _, k := range []string{"OTEL_METRIC_EXPORT_INTERVAL", "OTEL_METRIC_EXPORT_TIMEOUT"} { @@ -693,7 +694,7 @@ func TestBuildDeploymentApplyConfigTracesSamplerPropagation(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] env := envByName(c.Env) for _, k := range []string{"OTEL_TRACES_SAMPLER", "OTEL_TRACES_SAMPLER_ARG"} { @@ -831,6 +832,7 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf "--atunnel-client-identity="+installdefaults.RouterSPIFFEID(installdefaults.SystemNamespace), "--atunnel-egress-listen-address=0.0.0.0:15001", "--atunnel-egress-trust-bundle="+atunnelEgressTrustMountPath+"/trust-bundle.pem", + "--dns-egress-policy="+egresspolicy.DNSEgressPolicyEnforceInAtunnel, ). WithPorts(corev1ac.ContainerPort(). WithName("https"). @@ -941,7 +943,7 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf func TestBuildDeploymentAtunnelIdentitiesRelocatedNamespace(t *testing.T) { const relocated = "substrate-test" - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, relocated, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, relocated, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] want := map[string]string{ @@ -976,7 +978,7 @@ func TestBuildDeploymentAtunnelIdentitiesPrefixedServiceAccounts(t *testing.T) { router = "kagent-atenet-router" ) - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, namespace, atelet, router). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, namespace, atelet, router, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] want := map[string]string{ @@ -997,15 +999,12 @@ func TestBuildDeploymentAtunnelIdentitiesPrefixedServiceAccounts(t *testing.T) { } // TestBuildDeploymentOmitsBrokerIdentityForCanonicalInstall pins the flag's -// absence, which is what keeps a rolling upgrade working. docs/upgrade.md runs -// the outgoing worker pool alongside the new one, and this controller -// reconciles that pool's Deployment while it is still pinned to its old image. -// An ateom from before --atunnel-broker-identity existed exits on the -// unrecognized flag, so passing it would crashloop every old worker the moment -// the control plane rolled out. +// absence. A WorkerPool may still pin an ateom from before +// --atunnel-broker-identity existed, which exits on the unrecognized flag, so +// passing it would crashloop that pool's workers. func TestBuildDeploymentOmitsBrokerIdentityForCanonicalInstall(t *testing.T) { c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, - installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount). + installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] for _, arg := range c.Args { @@ -1031,7 +1030,7 @@ func TestBuildDeploymentOmitsBrokerIdentityForCanonicalInstall(t *testing.T) { // pool from the label the controller puts on the pod, so the usage records // name the pool the pod belongs to. func TestBuildDeploymentApplyConfigWorkerPoolEnv(t *testing.T) { - d := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount) + d := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) c := d.Spec.Template.Spec.Containers[0] var path string for _, e := range c.Env { @@ -1061,7 +1060,7 @@ func TestBuildDeploymentApplyConfigLogsExporter(t *testing.T) { {"ignored without endpoint", ateomOTelSettings{LogsExporter: "otlp"}, ""}, } { t.Run(tt.name, func(t *testing.T) { - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] got, ok := envByName(c.Env)["OTEL_LOGS_EXPORTER"] if ok != (tt.want != "") || (ok && got.value != tt.want) { diff --git a/cmd/atecontroller/internal/controllers/workerpool_controller.go b/cmd/atecontroller/internal/controllers/workerpool_controller.go index f6b6637f79..f306704e92 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_controller.go +++ b/cmd/atecontroller/internal/controllers/workerpool_controller.go @@ -79,6 +79,8 @@ type WorkerPoolReconciler struct { SystemNamespace string AteletServiceAccount string RouterServiceAccount string + // DNSEgressPolicy is the --dns-egress-policy mode propagated to ateom pods. + DNSEgressPolicy string // Recorder emits a Normal event each time a pool's SandboxConfig // changes, and a Warning event when a class stops resolving. Recorder events.EventRecorder @@ -322,7 +324,7 @@ func (r *WorkerPoolReconciler) applyDeployment(ctx context.Context, wp *atev1alp TracesSampler: r.OTelTracesSampler, TracesSamplerArg: r.OTelTracesSamplerArg, LogsExporter: r.OTelLogsExporter, - }, r.SystemNamespace, r.AteletServiceAccount, r.RouterServiceAccount) + }, r.SystemNamespace, r.AteletServiceAccount, r.RouterServiceAccount, r.DNSEgressPolicy) if err := r.Apply(ctx, depAC, client.FieldOwner(workerPoolFieldOwner), client.ForceOwnership); err != nil { return fmt.Errorf("failed to apply Deployment: %w", err) } diff --git a/cmd/atecontroller/internal/workersync/fakecontrol_test.go b/cmd/atecontroller/internal/workersync/fakecontrol_test.go index 06fbb31ef2..b614a9ae31 100644 --- a/cmd/atecontroller/internal/workersync/fakecontrol_test.go +++ b/cmd/atecontroller/internal/workersync/fakecontrol_test.go @@ -33,8 +33,9 @@ import ( // // It reproduces the behavior the syncer is written against and nothing more: // server-assigned uid and version, the uid+version precondition every update -// carries, the NOT_FOUND / ALREADY_EXISTS / ABORTED codes, a DrainWorker that -// is idempotent down to leaving the version alone, and paged ListWorkers. +// carries, the NOT_FOUND / ALREADY_EXISTS / ABORTED codes, Workers born +// UNAVAILABLE, state RPCs that are idempotent down to leaving the version +// alone, and paged ListWorkers. // // Request validation is not mirrored — that is the server's own contract, and // duplicating it here would only test the copy. A test that needs a rejection @@ -48,6 +49,9 @@ type fakeControl struct { mu sync.Mutex workers map[string]*ateapipb.Worker uidSeq int + // history holds every Worker written, in order, so a test can check the + // states in between the ones it reads back. + history []*ateapipb.Worker // createHook, when set, decides CreateWorker's outcome: a non-nil error is // returned and nothing is registered. @@ -103,10 +107,26 @@ func (f *fakeControl) putLocked(w *ateapipb.Worker) *ateapipb.Worker { if stored.GetStatus() == nil { stored.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE} } - f.workers[stored.GetMetadata().GetName()] = stored + f.storeLocked(stored) return proto.Clone(stored).(*ateapipb.Worker) } +// storeLocked writes w and records it in the write history. +func (f *fakeControl) storeLocked(w *ateapipb.Worker) { + f.workers[w.GetMetadata().GetName()] = w + f.history = append(f.history, proto.Clone(w).(*ateapipb.Worker)) +} + +// writes returns every Worker written since the history was last taken, in +// order, and clears it. +func (f *fakeControl) writes() []*ateapipb.Worker { + f.mu.Lock() + defer f.mu.Unlock() + history := f.history + f.history = nil + return history +} + func (f *fakeControl) setCreateHook(hook func(*ateapipb.Worker) error) { f.mu.Lock() defer f.mu.Unlock() @@ -157,10 +177,9 @@ func (f *fakeControl) CreateWorker(_ context.Context, in *ateapipb.CreateWorkerR if _, ok := f.workers[name]; ok { return nil, status.Errorf(codes.AlreadyExists, "Worker %s already exists", name) } - // status is output-only: a Worker is registered only once its pod has an - // IP, so ACTIVE is the only state it can be born in. + // status is output-only: every Worker is born UNAVAILABLE. created := proto.Clone(in.GetWorker()).(*ateapipb.Worker) - created.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE} + created.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE} return f.putLocked(created), nil } @@ -210,7 +229,7 @@ func (f *fakeControl) UpdateWorker(_ context.Context, in *ateapipb.UpdateWorkerR } updated.Metadata.Version++ - f.workers[md.GetName()] = updated + f.storeLocked(updated) return proto.Clone(updated).(*ateapipb.Worker), nil } @@ -243,10 +262,49 @@ func (f *fakeControl) DrainWorker(_ context.Context, in *ateapipb.DrainWorkerReq drained := proto.Clone(stored).(*ateapipb.Worker) drained.Status.State = ateapipb.WorkerState_WORKER_STATE_DRAINING drained.Metadata.Version++ - f.workers[name] = drained + f.storeLocked(drained) return &ateapipb.DrainWorkerResponse{Worker: proto.Clone(drained).(*ateapipb.Worker)}, nil } +func (f *fakeControl) MarkWorkerAvailable(_ context.Context, in *ateapipb.MarkWorkerAvailableRequest, _ ...grpc.CallOption) (*ateapipb.MarkWorkerAvailableResponse, error) { + w, err := f.markWorker(in.GetWorker().GetName(), ateapipb.WorkerState_WORKER_STATE_ACTIVE) + if err != nil { + return nil, err + } + return &ateapipb.MarkWorkerAvailableResponse{Worker: w}, nil +} + +func (f *fakeControl) MarkWorkerUnavailable(_ context.Context, in *ateapipb.MarkWorkerUnavailableRequest, _ ...grpc.CallOption) (*ateapipb.MarkWorkerUnavailableResponse, error) { + w, err := f.markWorker(in.GetWorker().GetName(), ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) + if err != nil { + return nil, err + } + return &ateapipb.MarkWorkerUnavailableResponse{Worker: w}, nil +} + +// markWorker moves a Worker between ACTIVE and UNAVAILABLE as the Mark RPCs +// do: idempotent down to leaving the version alone, with DRAINING final. +func (f *fakeControl) markWorker(name string, state ateapipb.WorkerState) (*ateapipb.Worker, error) { + f.mu.Lock() + defer f.mu.Unlock() + + stored, ok := f.workers[name] + if !ok { + return nil, status.Errorf(codes.NotFound, "Worker %s not found", name) + } + switch stored.GetStatus().GetState() { + case state: + return proto.Clone(stored).(*ateapipb.Worker), nil + case ateapipb.WorkerState_WORKER_STATE_DRAINING: + return nil, status.Errorf(codes.FailedPrecondition, "Worker %s is draining", name) + } + marked := proto.Clone(stored).(*ateapipb.Worker) + marked.Status.State = state + marked.Metadata.Version++ + f.storeLocked(marked) + return proto.Clone(marked).(*ateapipb.Worker), nil +} + func (f *fakeControl) ListWorkers(_ context.Context, in *ateapipb.ListWorkersRequest, _ ...grpc.CallOption) (*ateapipb.ListWorkersResponse, error) { f.mu.Lock() defer f.mu.Unlock() diff --git a/cmd/atecontroller/internal/workersync/syncer.go b/cmd/atecontroller/internal/workersync/syncer.go index a3551ed17b..48587e094b 100644 --- a/cmd/atecontroller/internal/workersync/syncer.go +++ b/cmd/atecontroller/internal/workersync/syncer.go @@ -263,10 +263,6 @@ func (s *WorkerPoolSyncer) reconcile(ctx context.Context, key workerKey) error { slog.InfoContext(ctx, "Syncer: deregistering worker (pod replaced)", key.logAttrs()...) return s.reconcileDeadWorker(ctx, key) } - // Checked before eligibility: draining works off the registered record by name - // and never reads the pod IP, while a Terminating pod can legitimately report - // no IP once its sandbox is torn down. Gating on the IP first would drop the - // transition and leave the worker schedulable for as long as the pod lingers. if pod.DeletionTimestamp != nil { // The pod has entered Terminating: mark the worker DRAINING so the // scheduler stops routing new actors to it. We deliberately do NOT touch @@ -275,18 +271,12 @@ func (s *WorkerPoolSyncer) reconcile(ctx context.Context, key workerKey) error { // Deleted event. return s.markWorkerDraining(ctx, key) } - // Checked before eligibility for the same reason: a terminal pod is never - // Ready, so the eligibility gate would leave its Worker, and the Actors bound - // to it, registered for as long as the pod object lingers. + // A terminal pod is never Ready again, so left alone its Worker would stay + // UNAVAILABLE, and the Actors bound to it registered, for as long as the pod + // object lingers. if isPodTerminal(pod) { return s.deleteTerminalPod(ctx, key, pod) } - if !isWorkerEligible(pod) { - // The pod has no IP or is not Ready yet; a later update event re-enqueues - // it. A registered Worker still takes a raised epoch: an ateom that is - // restarting is not Ready, but its Actors are already lost. - return s.raiseEpoch(ctx, key, pod) - } return s.createOrUpdateWorker(ctx, key, pod) } @@ -304,31 +294,11 @@ func podEpoch(pod *corev1.Pod) int64 { return 0 } -// raiseEpoch writes the pod's epoch to its registered Worker if it is higher -// than the one recorded there. A pod that is not registered is left to -// createOrUpdateWorker. -func (s *WorkerPoolSyncer) raiseEpoch(ctx context.Context, key workerKey, pod *corev1.Pod) error { - epoch := podEpoch(pod) - if epoch == 0 { - return nil - } - w, err := s.client.GetWorker(ctx, &ateapipb.GetWorkerRequest{Worker: key.workerRef()}) - if status.Code(err) == codes.NotFound { - return nil - } - if err != nil { - return fmt.Errorf("getting worker: %w", err) - } - if epoch <= w.GetEpoch() { - return nil - } - slog.InfoContext(ctx, "Syncer: updating worker (ateom restarted)", - append(key.logAttrs(), slog.Int64("epoch", epoch))...) - w.Epoch = epoch - _, err = s.client.UpdateWorker(ctx, &ateapipb.UpdateWorkerRequest{Worker: w}) - return err -} - +// createOrUpdateWorker registers the pod's Worker, or brings the registered +// one in line with the pod. A Worker is registered UNAVAILABLE and marked +// available only while its pod is Ready, and only once its epoch matches the +// ateom running, so nothing is placed on it under the epoch of an ateom that +// has since restarted. func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerKey, pod *corev1.Pod) error { poolName := pod.Labels[workerPodLabel] poolObject, exists, err := s.workerPoolInformer.GetIndexer().GetByKey(key.namespace + "/" + poolName) @@ -343,8 +313,15 @@ func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerK return fmt.Errorf("getting WorkerPool %s/%s: unexpected object type %T", key.namespace, poolName, poolObject) } + available := isWorkerAvailable(pod) + w, err := s.client.GetWorker(ctx, &ateapipb.GetWorkerRequest{Worker: key.workerRef()}) if status.Code(err) == codes.NotFound { + // ips and node_name are required and immutable, so the pod is registered + // once it has an IP; the update that reports it re-enqueues the pod. + if len(pod.Status.PodIPs) == 0 { + return nil + } slog.InfoContext(ctx, "Syncer: registering worker", key.logAttrs()...) worker := &ateapipb.Worker{ // Workers are global-scoped, so the name carries no atespace. See @@ -364,18 +341,31 @@ func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerK // that report lands, CreateWorker's reified ceiling holds the // Worker to a single Actor. } - // status is output-only: CreateWorker sets STATE_ACTIVE itself. + // status is output-only: CreateWorker registers the Worker UNAVAILABLE. // // ALREADY_EXISTS means we lost a create race; requeue and converge via // the update path. INVALID_ARGUMENT is terminal — see // processNextWorkItem. - _, err := s.client.CreateWorker(ctx, &ateapipb.CreateWorkerRequest{Worker: worker}) - return err - } - if err != nil { + w, err = s.client.CreateWorker(ctx, &ateapipb.CreateWorkerRequest{Worker: worker}) + if err != nil { + return err + } + } else if err != nil { return fmt.Errorf("getting worker: %w", err) } + // Each write below returns the Worker at its new version, which the next + // one carries as its precondition. ABORTED requeues the key; the retry + // re-fetches the worker. + if !available && w.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_ACTIVE { + slog.InfoContext(ctx, "Syncer: marking worker unavailable (pod not Ready)", key.logAttrs()...) + marked, err := s.client.MarkWorkerUnavailable(ctx, &ateapipb.MarkWorkerUnavailableRequest{Worker: key.workerRef()}) + if err != nil { + return err + } + w = marked.GetWorker() + } + // UpdateWorker replaces the whole resource, so the mutable fields are // edited onto the Worker as it was read and the rest is sent back unchanged // — anything else altered here, including a field cleared by omission, is @@ -404,7 +394,7 @@ func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerK slog.DebugContext(ctx, "Syncer: registered worker sandbox class predates its pool", append(key.logAttrs(), slog.String("registered", w.GetSandboxClass()), slog.String("pool", string(pool.Spec.DefaultSandboxClass())))...) } - if ips := podIPs(pod); !slices.Equal(w.GetIps(), ips) { + if ips := podIPs(pod); len(ips) > 0 && !slices.Equal(w.GetIps(), ips) { // TODO: I don't think this is possible, but handling this case so we can // log it just in case we can reproduce it. It is logged rather than // repaired because ips is immutable on a registered Worker: writing the @@ -412,18 +402,26 @@ func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerK slog.WarnContext(ctx, "Syncer: registered worker IPs disagree with its pod", append(key.logAttrs(), slog.Any("registered", w.GetIps()), slog.Any("pod_ips", ips))...) } - if !changed { - return nil + if changed { + w, err = s.client.UpdateWorker(ctx, &ateapipb.UpdateWorkerRequest{Worker: w}) + if err != nil { + return err + } } - // w carries the uid and version it was read at, which the API requires as the - // update's precondition. ABORTED requeues the key; the retry re-fetches the - // worker at its new version. - _, err = s.client.UpdateWorker(ctx, &ateapipb.UpdateWorkerRequest{Worker: w}) - return err + // Last, so the Worker takes Actors only at the epoch written above. A + // DRAINING Worker is left alone: its pod is on its way out. + if available && w.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + slog.InfoContext(ctx, "Syncer: marking worker available (pod Ready)", key.logAttrs()...) + _, err = s.client.MarkWorkerAvailable(ctx, &ateapipb.MarkWorkerAvailableRequest{Worker: key.workerRef()}) + return err + } + return nil } -func isWorkerEligible(pod *corev1.Pod) bool { +// isWorkerAvailable reports whether the pod is Ready with an IP, and so can +// take Actors. +func isWorkerAvailable(pod *corev1.Pod) bool { if len(pod.Status.PodIPs) == 0 { return false } diff --git a/cmd/atecontroller/internal/workersync/syncer_restart_test.go b/cmd/atecontroller/internal/workersync/syncer_restart_test.go index a308a737c6..19045786cf 100644 --- a/cmd/atecontroller/internal/workersync/syncer_restart_test.go +++ b/cmd/atecontroller/internal/workersync/syncer_restart_test.go @@ -18,6 +18,7 @@ import ( "context" "testing" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" corev1 "k8s.io/api/core/v1" ) @@ -103,7 +104,107 @@ func TestSyncer_RaisesEpochOnAteomRestart(t *testing.T) { } } -func TestSyncer_NotReadyUnregisteredPodIsNotRegistered(t *testing.T) { +// TestSyncer_NoPlacementUnderAnEarlierEpoch walks an ateom restart through +// the pod states kubelet reports, and pins that no write leaves the Worker +// ACTIVE while its epoch trails the ateom that is running. An Actor placed +// then would be stamped with the earlier epoch and crashed with the Actors the +// restart lost. +func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { + ctx := context.Background() + ns, podName, poolName := "ns-syncer-placement", "worker-placement-1", "pool1" + notReady := func(pod *corev1.Pod) *corev1.Pod { + pod.Status.Conditions = []corev1.PodCondition{{Type: corev1.PodReady, Status: corev1.ConditionFalse}} + return pod + } + + for _, tc := range []struct { + name string + steps []*corev1.Pod + want []ateapipb.WorkerState + }{ + { + name: "every transition observed", + steps: []*corev1.Pod{ + // ateom exited; kubelet has not started it again. + notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)), + // ateom running again, readiness probe not yet passed. + notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1)), + withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + }, + want: []ateapipb.WorkerState{ + ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, + ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, + ateapipb.WorkerState_WORKER_STATE_ACTIVE, + }, + }, + { + // The informer coalesces updates, so the syncer may only see the + // pod Ready again after the restart. + name: "not Ready never observed", + steps: []*corev1.Pod{ + withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + }, + want: []ateapipb.WorkerState{ateapipb.WorkerState_WORKER_STATE_ACTIVE}, + }, + { + // The epoch has to be raised before the Worker is marked available. + name: "Ready again under a new epoch", + steps: []*corev1.Pod{ + notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)), + withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + }, + want: []ateapipb.WorkerState{ + ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, + ateapipb.WorkerState_WORKER_STATE_ACTIVE, + }, + }, + { + // The restart is first seen with ateom running again but not Ready. + name: "not Ready under a new epoch", + steps: []*corev1.Pod{ + notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1)), + withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + }, + want: []ateapipb.WorkerState{ + ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, + ateapipb.WorkerState_WORKER_STATE_ACTIVE, + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + api := newFakeControl() + s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) + key := seedPod(t, pods, withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)) + mustReconcile(t, ctx, s, key) + if state := api.get(testPodUID).GetStatus().GetState(); state != ateapipb.WorkerState_WORKER_STATE_ACTIVE { + t.Fatalf("state after registering = %v, want %v", state, ateapipb.WorkerState_WORKER_STATE_ACTIVE) + } + api.writes() + + for i, pod := range tc.steps { + if err := pods.Update(pod); err != nil { + t.Fatalf("step %d: updating pod: %v", i, err) + } + mustReconcile(t, ctx, s, key) + if state := api.get(testPodUID).GetStatus().GetState(); state != tc.want[i] { + t.Errorf("step %d: state = %v, want %v", i, state, tc.want[i]) + } + for j, w := range api.writes() { + if w.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_ACTIVE && w.GetEpoch() < podEpoch(pod) { + t.Errorf("step %d, write %d: worker ACTIVE at epoch %d while the pod reports epoch %d", i, j, w.GetEpoch(), podEpoch(pod)) + } + } + } + if got := api.get(testPodUID).GetEpoch(); got != 2 { + t.Errorf("epoch after the restart = %d, want 2", got) + } + }) + } +} + +// A pod first seen not Ready after ateom restarts registers unavailable at +// the epoch of the ateom it is running. +func TestSyncer_NotReadyPodRegistersAtItsEpoch(t *testing.T) { ctx := context.Background() ns, poolName := "ns-syncer-notready-restart", "pool1" @@ -114,8 +215,12 @@ func TestSyncer_NotReadyUnregisteredPodIsNotRegistered(t *testing.T) { key := seedPod(t, pods, pod) mustReconcile(t, ctx, s, key) - if got := api.names(); len(got) != 0 { - t.Errorf("registry holds %v for a not-Ready pod, want it empty", got) + got := api.get(testPodUID) + if got.GetStatus().GetState() != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + t.Errorf("state = %v, want %v", got.GetStatus().GetState(), ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) + } + if got.GetEpoch() != 4 { + t.Errorf("epoch = %d, want 4", got.GetEpoch()) } } diff --git a/cmd/atecontroller/internal/workersync/syncer_test.go b/cmd/atecontroller/internal/workersync/syncer_test.go index 29b301d295..0bec94613f 100644 --- a/cmd/atecontroller/internal/workersync/syncer_test.go +++ b/cmd/atecontroller/internal/workersync/syncer_test.go @@ -50,7 +50,7 @@ const ( ) // workerPod builds a pod the informer's label selector matches. An empty ip -// leaves the pod ineligible, the shape a pod has before its sandbox is up. +// leaves the pod unregistrable, the shape a pod has before its sandbox is up. func workerPod(ns, name, poolName, uid, ip string) *corev1.Pod { pod := &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ @@ -69,7 +69,7 @@ func workerPod(ns, name, poolName, uid, ip string) *corev1.Pod { Phase: corev1.PodRunning, PodIP: ip, PodIPs: []corev1.PodIP{{IP: ip}}, - // Eligibility requires Ready in addition to an IP: readiness is + // Availability requires Ready in addition to an IP: readiness is // what says ateom is actually serving, not just that the sandbox // got an address. Conditions: []corev1.PodCondition{{ @@ -210,8 +210,8 @@ func TestSyncer_Lifecycle(t *testing.T) { t.Fatalf("while checking a pod with no IP stays unregistered: %v", err) } - // Once the pod reports an IP it becomes eligible and is registered, with the - // fields the syncer copies off its pool. + // Once the pod reports an IP it is registered, with the fields the syncer + // copies off its pool. if _, err := fakeK8s.CoreV1().Pods(ns).Update(ctx, workerPod(ns, podName, poolName, testPodUID, "127.0.0.1"), metav1.UpdateOptions{}); err != nil { t.Fatalf("update pod: %v", err) } @@ -321,8 +321,7 @@ func TestSyncer_SoftDelete_MarksDraining(t *testing.T) { } // TestSyncer_SoftDelete_NoPodIP pins the draining mark for a Terminating pod -// reporting no IP, the shape a pod takes once its sandbox is torn down. It -// fails if the isWorkerEligible gate moves back ahead of the deletion check. +// reporting no IP, the shape a pod takes once its sandbox is torn down. func TestSyncer_SoftDelete_NoPodIP(t *testing.T) { ctx := context.Background() ns, poolName, podName := "ns-drain-noip", "pool1", "worker-drain-noip" @@ -891,10 +890,10 @@ func TestSyncer_PodRecreatedWithNewUID(t *testing.T) { } } -// TestSyncer_DeleteNeverEligiblePod verifies that deleting a pod that never got -// an IP (and so was never registered) is a no-op rather than an error, which is +// TestSyncer_DeletePodWithoutIP verifies that deleting a pod that never got an +// IP (and so was never registered) is a no-op rather than an error, which is // what keeps it from error-looping. -func TestSyncer_DeleteNeverEligiblePod(t *testing.T) { +func TestSyncer_DeletePodWithoutIP(t *testing.T) { ctx := context.Background() ns, podName, poolName := "ns-syncer-neverip", "worker-noip-1", "pool1" @@ -917,12 +916,12 @@ func TestSyncer_DeleteNeverEligiblePod(t *testing.T) { } } -// TestSyncer_PodWithIPButNotReadyIsNotRegistered pins the readiness half of -// the eligibility gate: an IP alone no longer registers a worker, because the -// sandbox having an address says nothing about ateom serving yet (#1106). -// Both not-Ready shapes are covered — condition absent (kubelet hasn't probed -// yet) and condition explicitly False (probe failing). -func TestSyncer_PodWithIPButNotReadyIsNotRegistered(t *testing.T) { +// TestSyncer_PodWithIPButNotReadyIsUnavailable pins that an IP alone registers +// a worker that takes no Actors, because the sandbox having an address says +// nothing about ateom serving yet. Both not-Ready shapes are covered — +// condition absent (kubelet hasn't probed yet) and condition explicitly False +// (probe failing). +func TestSyncer_PodWithIPButNotReadyIsUnavailable(t *testing.T) { ctx := context.Background() ns, poolName := "ns-syncer-notready", "pool1" @@ -945,17 +944,73 @@ func TestSyncer_PodWithIPButNotReadyIsNotRegistered(t *testing.T) { key := seedPod(t, pods, pod) mustReconcile(t, ctx, s, key) - if got := api.names(); len(got) != 0 { - t.Errorf("registry holds %v for a not-Ready pod, want it empty", got) + got := api.get(testPodUID) + if got == nil { + t.Fatalf("worker not registered for a not-Ready pod with an IP; registry holds %v", api.names()) + } + if state := got.GetStatus().GetState(); state != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + t.Errorf("state = %v, want %v", state, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) } }) } } -// TestSyncer_ReadinessFlapDoesNotDeregister pins that eligibility only gates -// registration: once a worker exists, a readiness blip must not remove it — +// A pod that got an IP but never turned Ready is registered unavailable, and +// deregistered when it goes. +func TestSyncer_DeleteNeverReadyPod(t *testing.T) { + ctx := context.Background() + ns, podName, poolName := "ns-syncer-neverready", "worker-neverready-1", "pool1" + + api := newFakeControl() + s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) + pod := workerPod(ns, podName, poolName, testPodUID, "10.0.0.9") + pod.Status.Conditions = nil + key := seedPod(t, pods, pod) + + mustReconcile(t, ctx, s, key) + if got := api.get(testPodUID); got.GetStatus().GetState() != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + t.Fatalf("worker = %v, want it registered %v", got, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) + } + if err := pods.Delete(pod); err != nil { + t.Fatalf("deleting pod: %v", err) + } + mustReconcile(t, ctx, s, key) + if got := api.names(); len(got) != 0 { + t.Errorf("registry holds %v, want it empty", got) + } +} + +// A registered worker whose pod loses its IP, as when its sandbox is +// recreated, is unavailable until the pod reports one again. +func TestSyncer_RegisteredPodWithoutIPIsUnavailable(t *testing.T) { + ctx := context.Background() + ns, podName, poolName := "ns-syncer-lostip", "worker-lostip-1", "pool1" + + api := newFakeControl() + s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) + key := seedPod(t, pods, withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.9"), 0)) + mustReconcile(t, ctx, s, key) + + if err := pods.Update(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, ""), 1)); err != nil { + t.Fatalf("updating pod: %v", err) + } + mustReconcile(t, ctx, s, key) + got := api.get(testPodUID) + if got.GetStatus().GetState() != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + t.Errorf("state = %v, want %v", got.GetStatus().GetState(), ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) + } + if got.GetEpoch() != 2 { + t.Errorf("epoch = %d, want 2", got.GetEpoch()) + } + if want := []string{"10.0.0.9"}; !slices.Equal(got.GetIps(), want) { + t.Errorf("ips = %v, want the registered %v kept", got.GetIps(), want) + } +} + +// TestSyncer_ReadinessFlapDoesNotDeregister pins that once a worker exists, a +// readiness blip must not remove it — // a bound actor keeps running through a failed probe, and deregistering would -// strand it. +// strand it. The worker is only unavailable until the pod is Ready again. func TestSyncer_ReadinessFlapDoesNotDeregister(t *testing.T) { ctx := context.Background() ns, podName, poolName := "ns-syncer-flap", "worker-flap-1", "pool1" @@ -981,8 +1036,17 @@ func TestSyncer_ReadinessFlapDoesNotDeregister(t *testing.T) { if got == nil { t.Fatalf("worker deregistered on a readiness flap; registry holds %v", api.names()) } - if got.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_DRAINING { - t.Errorf("worker marked DRAINING on a readiness flap, want it left ACTIVE") + if state := got.GetStatus().GetState(); state != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + t.Errorf("state = %v on a readiness flap, want %v", state, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) + } + + if err := pods.Update(pod); err != nil { + t.Fatalf("updating pod: %v", err) + } + mustReconcile(t, ctx, s, key) + got = api.get(testPodUID) + if state := got.GetStatus().GetState(); state != ateapipb.WorkerState_WORKER_STATE_ACTIVE { + t.Errorf("state = %v once Ready again, want %v", state, ateapipb.WorkerState_WORKER_STATE_ACTIVE) } } diff --git a/cmd/atecontroller/main.go b/cmd/atecontroller/main.go index 87ce83a43a..174ad5c2c5 100644 --- a/cmd/atecontroller/main.go +++ b/cmd/atecontroller/main.go @@ -22,6 +22,7 @@ import ( "github.com/agent-substrate/substrate/cmd/atecontroller/internal/controllers" "github.com/agent-substrate/substrate/cmd/atecontroller/internal/workersync" "github.com/agent-substrate/substrate/internal/ateapiauth" + "github.com/agent-substrate/substrate/internal/egresspolicy" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/version" @@ -86,6 +87,8 @@ var ( ateapiClientCert = pflag.String("ateapi-client-cert", "", "Credential bundle presented as the client certificate when dialing ateapi. Required.") workerSyncConcurrency = pflag.Int("worker-sync-concurrency", 2, "Number of concurrent worker goroutines reconciling worker pods into the Worker registry. Defaults to 2.") + + dnsEgressPolicy = pflag.String("dns-egress-policy", "", "DNS egress policy enforcement mode propagated to ateom worker pods (must be "+egresspolicy.DNSEgressPolicyEnforceInAtunnel+").") ) func init() { @@ -154,6 +157,10 @@ func main() { setupLog.Error(nil, "invalid flag", "flag", "--worker-sync-concurrency", "reason", "must be positive", "value", *workerSyncConcurrency) os.Exit(1) } + if err := egresspolicy.ValidateDNSEgressPolicy(*dnsEgressPolicy); err != nil { + setupLog.Error(err, "invalid flag", "flag", "--dns-egress-policy", "value", *dnsEgressPolicy) + os.Exit(1) + } // Both providers must be registered before the ateapi client below: // otelgrpc.NewClientHandler captures the global tracer and meter providers at @@ -241,6 +248,7 @@ func main() { SystemNamespace: systemNamespace, AteletServiceAccount: *ateletServiceAccount, RouterServiceAccount: *routerServiceAccount, + DNSEgressPolicy: *dnsEgressPolicy, Recorder: mgr.GetEventRecorder("workerpool-controller"), APIReader: mgr.GetAPIReader(), }).SetupWithManager(mgr); err != nil { diff --git a/cmd/atecontroller/main_test.go b/cmd/atecontroller/main_test.go index 0068d361b5..0301f916b0 100644 --- a/cmd/atecontroller/main_test.go +++ b/cmd/atecontroller/main_test.go @@ -20,6 +20,7 @@ import ( "strings" "testing" + "github.com/spf13/pflag" prombridge "go.opentelemetry.io/contrib/bridges/prometheus" corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" @@ -28,6 +29,17 @@ import ( metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" ) +func TestDNSEgressPolicyFlagDefault(t *testing.T) { + t.Parallel() + f := pflag.Lookup("dns-egress-policy") + if f == nil { + t.Fatal("no --dns-egress-policy flag registered") + } + if f.DefValue != "" { + t.Errorf("--dns-egress-policy default = %q, want empty", f.DefValue) + } +} + func TestNewControllerRuntimeLoggerVerbosity(t *testing.T) { const probe = "verbosity-probe" diff --git a/cmd/atelet/ateomsupport.go b/cmd/atelet/ateomsupport.go index c6e778a91a..ad3e8f5d3f 100644 --- a/cmd/atelet/ateomsupport.go +++ b/cmd/atelet/ateomsupport.go @@ -34,6 +34,7 @@ import ( type ateomSupportServer struct { ateletpb.UnimplementedAteomSupportServer workers ateapipb.WorkerServiceClient + control ateapipb.ControlClient } func (b *ateomSupportServer) MintActorCertificate(ctx context.Context, req *ateletpb.MintActorCertificateRequest) (*ateletpb.MintActorCertificateResponse, error) { @@ -223,3 +224,48 @@ func (s *ateomSupportServer) RequestActorSuspend(ctx context.Context, req *atele slog.String("actor_uid", req.GetActorUid())) return &ateletpb.RequestActorSuspendResponse{}, nil } + +// GetActorEgressPolicy reads the egress policy of an actor. +func (s *ateomSupportServer) GetActorEgressPolicy( + ctx context.Context, + req *ateletpb.GetActorEgressPolicyRequest, +) (*ateletpb.GetActorEgressPolicyResponse, error) { + workerIdentity, err := authenticatedWorkerIdentity(ctx) + if err != nil { + return nil, err + } + // Validate request. + if errs := apivalidation.ValidateGetActorEgressPolicyRequest(ctx, req); len(errs) > 0 { + return nil, resources.ToGRPCStatusError(errs) + } + + // TODO(bowei): add validation for which Actor requested this. + + policy, err := s.control.GetActorEgressPolicy(ctx, &ateapipb.GetActorEgressPolicyRequest{ + Actor: &ateapipb.ObjectRef{ + Atespace: req.GetActorAtespace(), + Name: req.GetActorName(), + }, + Name: resources.SingletonName, + }) + + // Note: we do not check that the EgressPolicy is directly attached to the + // Actor: Policies are applied by (user-given) name as opposed to linked + // explicitly to the internal Actor UUID. + + // NotFound is treated as empty policy (block all). + if status.Code(err) == codes.NotFound { + return &ateletpb.GetActorEgressPolicyResponse{}, nil + } + + if err != nil { + return nil, err + } + + slog.DebugContext(ctx, "Fetched an actor egress policy", + slog.String("pod_uid", workerIdentity.PodUID), + slog.String("actor_atespace", req.GetActorAtespace()), + slog.String("actor_name", req.GetActorName())) + + return &ateletpb.GetActorEgressPolicyResponse{EgressPolicy: toAteletEgressPolicy(policy)}, nil +} diff --git a/cmd/atelet/ateomsupport_test.go b/cmd/atelet/ateomsupport_test.go index d6f096be19..124ab8bccd 100644 --- a/cmd/atelet/ateomsupport_test.go +++ b/cmd/atelet/ateomsupport_test.go @@ -365,3 +365,108 @@ func TestMintActorCertificateRequiresCertificate(t *testing.T) { t.Errorf("unauthenticated request still forwarded %v", workers.mintGot) } } + +type fakeEgressPolicyControl struct { + ateapipb.ControlClient + + got []*ateapipb.GetActorEgressPolicyRequest + resp *ateapipb.EgressPolicy + err error +} + +func (c *fakeEgressPolicyControl) GetActorEgressPolicy(_ context.Context, in *ateapipb.GetActorEgressPolicyRequest, _ ...grpc.CallOption) (*ateapipb.EgressPolicy, error) { + c.got = append(c.got, in) + if c.err != nil { + return nil, c.err + } + return c.resp, nil +} + +func TestGetActorEgressPolicy(t *testing.T) { + policy := &ateapipb.EgressPolicy{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default"}, + Rules: []*ateapipb.EgressRule{{Http: &ateapipb.HttpRule{ + Hostnames: []string{"api.example.com"}, + Ports: &ateapipb.Ports{Numbers: []int32{80}}, + }}}, + } + // The copy toAteletEgressPolicy makes; TestToAteletEgressPolicy covers it. + copied := &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{Http: &ateletpb.EgressDestinations{ + Hostnames: []string{"api.example.com"}, + Ports: &ateletpb.EgressPorts{Numbers: []int32{80}}, + }}}} + req := &ateletpb.GetActorEgressPolicyRequest{ActorAtespace: "team-a", ActorName: "actor-1"} + forwarded := []*ateapipb.GetActorEgressPolicyRequest{{ + Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "actor-1"}, + Name: resources.SingletonName, + }} + + tests := []struct { + name string + // unauthenticated drops the peer certificate from the context. + unauthenticated bool + req *ateletpb.GetActorEgressPolicyRequest + // resp and serviceErr are what the control plane answers with. + resp *ateapipb.EgressPolicy + serviceErr error + wantCode codes.Code + wantForwarded []*ateapipb.GetActorEgressPolicyRequest + // wantPolicy is the policy the worker gets back; nil means unset. + wantPolicy *ateletpb.EgressPolicy + }{{ + name: "returns the policy", + req: req, + resp: policy, + wantForwarded: forwarded, + wantPolicy: copied, + }, { + name: "leaves no policy unset", + req: req, + serviceErr: status.Error(codes.NotFound, "EgressPolicy for actor team-a/actor-1 not found"), + wantForwarded: forwarded, + }, { + name: "requires a certificate", + unauthenticated: true, + req: req, + wantCode: codes.Unauthenticated, + }, { + name: "rejects an invalid request without forwarding", + req: &ateletpb.GetActorEgressPolicyRequest{ActorAtespace: "team-a"}, + wantCode: codes.InvalidArgument, + }, { + // A failure is not "no policy": the worker must be able to tell them + // apart, and refuses DNS until a fetch succeeds. + name: "surfaces the control plane's failure", + req: req, + serviceErr: status.Error(codes.Unavailable, "ateapi unavailable"), + wantCode: codes.Unavailable, + wantForwarded: forwarded, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + control := &fakeEgressPolicyControl{resp: tt.resp, err: tt.serviceErr} + svc := &ateomSupportServer{control: control} + ctx := workerContext(t, "pod-a") + if tt.unauthenticated { + ctx = context.Background() + } + + resp, err := svc.GetActorEgressPolicy(ctx, tt.req) + if got := status.Code(err); got != tt.wantCode { + t.Fatalf("GetActorEgressPolicy() code = %v (%v), want %v", got, err, tt.wantCode) + } + if diff := cmp.Diff(tt.wantForwarded, control.got, protocmp.Transform()); diff != "" { + t.Errorf("forwarded requests mismatch (-want +got):\n%s", diff) + } + if err != nil { + return + } + if (resp.GetEgressPolicy() == nil) != (tt.wantPolicy == nil) { + t.Fatalf("EgressPolicy = %v, want %v", resp.GetEgressPolicy(), tt.wantPolicy) + } + if diff := cmp.Diff(tt.wantPolicy, resp.GetEgressPolicy(), protocmp.Transform()); diff != "" { + t.Errorf("returned policy mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/cmd/atelet/egresspolicy.go b/cmd/atelet/egresspolicy.go new file mode 100644 index 0000000000..97769f38f7 --- /dev/null +++ b/cmd/atelet/egresspolicy.go @@ -0,0 +1,55 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// toAteletEgressPolicy copies the destinations each rule of the control +// plane's policy allows. Metadata and rule effects are dropped: the worker +// does not apply them, and they name credentials it has no need to see. +func toAteletEgressPolicy(in *ateapipb.EgressPolicy) *ateletpb.EgressPolicy { + if in == nil { + return nil + } + out := &ateletpb.EgressPolicy{Rules: make([]*ateletpb.EgressRule, 0, len(in.GetRules()))} + for _, r := range in.GetRules() { + rule := &ateletpb.EgressRule{} + if h := r.GetHttp(); h != nil { + rule.Http = toEgressDestinations(h.GetHostnames(), h.GetPorts()) + } + if h := r.GetHttps(); h != nil { + rule.Https = toEgressDestinations(h.GetHostnames(), h.GetPorts()) + } + if t := r.GetTlsPassthrough(); t != nil { + rule.TlsPassthrough = toEgressDestinations(t.GetHostnames(), t.GetPorts()) + } + out.Rules = append(out.Rules, rule) + } + return out +} + +func toEgressDestinations(hostnames []string, ports *ateapipb.Ports) *ateletpb.EgressDestinations { + out := &ateletpb.EgressDestinations{Hostnames: hostnames} + if ports != nil { + out.Ports = &ateletpb.EgressPorts{Numbers: ports.GetNumbers()} + if ports.GetAll() != nil { + out.Ports.All = &ateletpb.EgressAllPorts{} + } + } + return out +} diff --git a/cmd/atelet/egresspolicy_test.go b/cmd/atelet/egresspolicy_test.go new file mode 100644 index 0000000000..a1dcec0744 --- /dev/null +++ b/cmd/atelet/egresspolicy_test.go @@ -0,0 +1,106 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "testing" + + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/google/go-cmp/cmp" + "google.golang.org/protobuf/testing/protocmp" +) + +func TestToAteletEgressPolicy(t *testing.T) { + effects := &ateapipb.HttpRuleEffects{ReplaceHeaders: []*ateapipb.CredentialHeader{{ + Header: "Authorization", + CredentialUri: "secret://team-a/token", + }}} + for _, tc := range []struct { + name string + in *ateapipb.EgressPolicy + want *ateletpb.EgressPolicy + }{{ + name: "no policy", + }, { + name: "no rules", + in: &ateapipb.EgressPolicy{Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default"}}, + want: &ateletpb.EgressPolicy{}, + }, { + // Metadata and effects are the gateway's, and never reach the worker. + name: "every protocol", + in: &ateapipb.EgressPolicy{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default", Uid: "uid"}, + Rules: []*ateapipb.EgressRule{{ + Http: &ateapipb.HttpRule{ + Hostnames: []string{"api.example.com"}, + Ports: &ateapipb.Ports{Numbers: []int32{80, 8080}}, + Effects: effects, + }, + }, { + Https: &ateapipb.HttpsRule{ + Hostnames: []string{"*.example.com"}, + Ports: &ateapipb.Ports{All: &ateapipb.AllPorts{}}, + Effects: effects, + }, + }, { + TlsPassthrough: &ateapipb.TlsPassthroughRule{ + Hostnames: []string{"db.example.com", "cache.example.com"}, + Ports: &ateapipb.Ports{Numbers: []int32{5432}}, + }, + }}, + }, + want: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Http: &ateletpb.EgressDestinations{ + Hostnames: []string{"api.example.com"}, + Ports: &ateletpb.EgressPorts{Numbers: []int32{80, 8080}}, + }, + }, { + Https: &ateletpb.EgressDestinations{ + Hostnames: []string{"*.example.com"}, + Ports: &ateletpb.EgressPorts{All: &ateletpb.EgressAllPorts{}}, + }, + }, { + TlsPassthrough: &ateletpb.EgressDestinations{ + Hostnames: []string{"db.example.com", "cache.example.com"}, + Ports: &ateletpb.EgressPorts{Numbers: []int32{5432}}, + }, + }}}, + }, { + name: "rule without ports", + in: &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{ + TlsPassthrough: &ateapipb.TlsPassthroughRule{Hostnames: []string{"db.example.com"}}, + }}}, + want: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + TlsPassthrough: &ateletpb.EgressDestinations{Hostnames: []string{"db.example.com"}}, + }}}, + }, { + // A rule of a protocol the copy does not carry keeps its place, and + // allows nothing. + name: "rule without a protocol", + in: &ateapipb.EgressPolicy{Rules: []*ateapipb.EgressRule{{}}}, + want: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{}}}, + }} { + t.Run(tc.name, func(t *testing.T) { + got := toAteletEgressPolicy(tc.in) + if (got == nil) != (tc.want == nil) { + t.Fatalf("toAteletEgressPolicy() = %v, want %v", got, tc.want) + } + if diff := cmp.Diff(tc.want, got, protocmp.Transform()); diff != "" { + t.Errorf("toAteletEgressPolicy() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/cmd/atelet/internal/apivalidation/validate.go b/cmd/atelet/internal/apivalidation/validate.go index f3a0b3de32..63f9f143b2 100644 --- a/cmd/atelet/internal/apivalidation/validate.go +++ b/cmd/atelet/internal/apivalidation/validate.go @@ -20,7 +20,6 @@ import ( "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" - atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -30,6 +29,12 @@ func ValidateRequestActorSuspendRequest(ctx context.Context, req *ateletpb.Reque return Validate_RequestActorSuspendRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) } +// ValidateGetActorEgressPolicyRequest runs the generated validation for +// req. +func ValidateGetActorEgressPolicyRequest(ctx context.Context, req *ateletpb.GetActorEgressPolicyRequest) field.ErrorList { + return Validate_GetActorEgressPolicyRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) +} + // ValidateMintActorCertificateRequest runs the generated validation for req. func ValidateMintActorCertificateRequest(ctx context.Context, req *ateletpb.MintActorCertificateRequest) field.ErrorList { return Validate_MintActorCertificateRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) @@ -76,16 +81,12 @@ func ValidateCustom_EgressGateway_Address(_ context.Context, _ operation.Operati return resources.ValidateHostPort(fldPath, *value) } -// ValidateCustom_SandboxAssets_SandboxClass allows the sandbox classes of the -// SandboxConfig CRD. func ValidateCustom_SandboxAssets_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { - switch atev1alpha1.SandboxClass(*value) { - case atev1alpha1.SandboxClassGvisor, atev1alpha1.SandboxClassMicroVM: - return nil - } - return field.ErrorList{field.NotSupported(fldPath, *value, []string{ - string(atev1alpha1.SandboxClassGvisor), string(atev1alpha1.SandboxClassMicroVM), - })} + return resources.ValidateSandboxClass(fldPath, *value) +} + +func ValidateCustom_SandboxRuntime_SandboxClass(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { + return resources.ValidateSandboxClass(fldPath, *value) } func ValidateCustom_SandboxAssets_PauseImage(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { diff --git a/cmd/atelet/internal/apivalidation/validation_test.go b/cmd/atelet/internal/apivalidation/validation_test.go index ef855df94a..6e020b666a 100644 --- a/cmd/atelet/internal/apivalidation/validation_test.go +++ b/cmd/atelet/internal/apivalidation/validation_test.go @@ -127,6 +127,49 @@ func TestValidateRequestActorSuspendRequest(t *testing.T) { } } +func TestValidateGetActorEgressPolicyRequest(t *testing.T) { + valid := func(mutate ...func(*ateletpb.GetActorEgressPolicyRequest)) *ateletpb.GetActorEgressPolicyRequest { + r := &ateletpb.GetActorEgressPolicyRequest{ + ActorAtespace: "team-a", + ActorName: "actor-1", + } + for _, m := range mutate { + m(r) + } + return r + } + + tests := []struct { + name string + obj *ateletpb.GetActorEgressPolicyRequest + want field.ErrorList + }{{ + name: "valid", + obj: valid(), + }, { + name: "missing actor_atespace", + obj: valid(func(r *ateletpb.GetActorEgressPolicyRequest) { r.ActorAtespace = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_atespace"), "")}, + }, { + name: "invalid actor_atespace: uppercase", + obj: valid(func(r *ateletpb.GetActorEgressPolicyRequest) { r.ActorAtespace = "Team-A" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_atespace"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing actor_name", + obj: valid(func(r *ateletpb.GetActorEgressPolicyRequest) { r.ActorName = "" }), + want: field.ErrorList{field.Required(field.NewPath("actor_name"), "")}, + }, { + name: "invalid actor_name: trailing dash", + obj: valid(func(r *ateletpb.GetActorEgressPolicyRequest) { r.ActorName = "actor-" }), + want: field.ErrorList{field.Invalid(field.NewPath("actor_name"), nil, "").WithOrigin("format=k8s-short-name")}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assertValidateErr(t, ValidateGetActorEgressPolicyRequest(context.Background(), tt.obj), tt.want) + }) + } +} + func TestValidateMintActorCertificateRequest(t *testing.T) { valid := func(mutate ...func(*ateletpb.MintActorCertificateRequest)) *ateletpb.MintActorCertificateRequest { r := &ateletpb.MintActorCertificateRequest{ @@ -1060,7 +1103,20 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { Version: testRuntime().Version, }, }, - want: field.ErrorList{field.TooLong(defaultRuntimePath.Child("sandbox_class"), "", 63).WithOrigin("maxLength")}, + want: field.ErrorList{ + field.TooLong(defaultRuntimePath.Child("sandbox_class"), "", 63).WithOrigin("maxLength"), + field.NotSupported[string](defaultRuntimePath.Child("sandbox_class"), nil, nil), + }, + }, { + name: "unsupported default_runtime.sandbox_class", + obj: &ateletpb.RegisterWorkerRequest{ + Capacity: &ateletpb.WorkerResources{}, + DefaultRuntime: &ateletpb.SandboxRuntime{ + SandboxClass: "kata", + Version: testRuntime().Version, + }, + }, + want: field.ErrorList{field.NotSupported[string](defaultRuntimePath.Child("sandbox_class"), nil, nil)}, }, { name: "missing default_runtime.version", obj: &ateletpb.RegisterWorkerRequest{ @@ -1200,6 +1256,17 @@ func TestValidateRegisterWorkerRequest(t *testing.T) { RestorableRuntimes: []*ateletpb.SandboxRuntime{{SandboxClass: "gvisor"}}, }, want: field.ErrorList{field.Required(restorablePath.Index(0).Child("version"), "")}, + }, { + name: "unsupported restorable_runtimes sandbox_class", + obj: &ateletpb.RegisterWorkerRequest{ + Capacity: &ateletpb.WorkerResources{}, + DefaultRuntime: testRuntime(), + RestorableRuntimes: []*ateletpb.SandboxRuntime{{ + SandboxClass: "GVISOR", + Version: testRuntime().Version, + }}, + }, + want: field.ErrorList{field.NotSupported[string](restorablePath.Index(0).Child("sandbox_class"), nil, nil)}, }, { name: "restorable_runtimes at the bound", obj: &ateletpb.RegisterWorkerRequest{ diff --git a/cmd/atelet/internal/apivalidation/zz_generated.validation.go b/cmd/atelet/internal/apivalidation/zz_generated.validation.go index 4cf3036e60..fb86928436 100644 --- a/cmd/atelet/internal/apivalidation/zz_generated.validation.go +++ b/cmd/atelet/internal/apivalidation/zz_generated.validation.go @@ -1367,6 +1367,79 @@ func Validate_ExternalVolumeSource( return errs } +// Validate_GetActorEgressPolicyRequest validates an instance of GetActorEgressPolicyRequest according +// to declarative validation rules in the API schema. +func Validate_GetActorEgressPolicyRequest( + ctx context.Context, op operation.Operation, fldPath *field.Path, + obj, oldObj *ateletpb.GetActorEgressPolicyRequest) (errs field.ErrorList) { + + { // field ateletpb.GetActorEgressPolicyRequest.ActorAtespace + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.GetActorEgressPolicyRequest) *string { + return &oldObj.ActorAtespace + }) + errs = append(errs, fn(fldPath.Child("actor_atespace"), &obj.ActorAtespace, oldVal, oldObj != nil)...) + } + + { // field ateletpb.GetActorEgressPolicyRequest.ActorName + fn := func( + fldPath *field.Path, + obj, oldObj *string, + oldValueCorrelated bool) (errs field.ErrorList) { + // don't revalidate unchanged data + if oldValueCorrelated && op.Type == operation.Update { + if obj == oldObj || (obj != nil && oldObj != nil && *obj == *oldObj) { + return nil + } + } + // call field-attached validations + earlyReturn := false + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if earlyReturn { + return // do not proceed + } + if e := validate.ShortName(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } + return + } + oldVal := safe.Field(oldObj, + func(oldObj *ateletpb.GetActorEgressPolicyRequest) *string { + return &oldObj.ActorName + }) + errs = append(errs, fn(fldPath.Child("actor_name"), &obj.ActorName, oldVal, oldObj != nil)...) + } + + return errs +} + // Validate_HttpGetAction validates an instance of HttpGetAction according // to declarative validation rules in the API schema. func Validate_HttpGetAction( @@ -3210,6 +3283,10 @@ func Validate_SandboxRuntime( if earlyReturn { return // do not proceed } + // custom validation + if e := ValidateCustom_SandboxRuntime_SandboxClass(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } if e := validate.MaxLength(ctx, op, fldPath, obj, oldObj, 63); len(e) != 0 { errs = append(errs, e...) } diff --git a/cmd/atelet/lifecycle_test.go b/cmd/atelet/lifecycle_test.go index 7d272cfd1e..1d12c0cf57 100644 --- a/cmd/atelet/lifecycle_test.go +++ b/cmd/atelet/lifecycle_test.go @@ -17,6 +17,7 @@ package main import ( "context" "crypto/sha256" + "errors" "fmt" "net" "os" @@ -27,11 +28,13 @@ import ( "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" "github.com/agent-substrate/substrate/cmd/atelet/internal/trustbundle" + "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/proto/ateompb" "github.com/agent-substrate/substrate/internal/resources" "google.golang.org/grpc" + "google.golang.org/grpc/codes" "google.golang.org/protobuf/proto" ) @@ -244,7 +247,7 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { t.Fatalf("pause did not write the local snapshot: %v", err) } - // Resume: restores from that local snapshot. + // Resume: no Snapshot.Object; atelet must find that local snapshot. if _, err := s.Restore(ctx, &ateletpb.RestoreRequest{ Atespace: atespace, ActorName: actorName, @@ -454,7 +457,7 @@ func TestActivationFailureBeforeRegistration(t *testing.T) { {name: "Run partial reset removes registration", failure: "reset", wantErr: "while removing volume dir"}, {name: "Run record failure removes registration", failure: "record", wantErr: "while recording sandbox assets"}, {name: "Restore partial reset removes registration", restore: true, failure: "reset", wantErr: "while removing volume dir"}, - {name: "Restore manifest failure removes registration", restore: true, failure: "manifest", wantErr: "while reading local snapshot manifest"}, + {name: "Restore manifest failure removes registration", restore: true, failure: "manifest", wantErr: "no snapshot.object to fall back to"}, {name: "Restore asset failure removes registration", restore: true, failure: "asset", wantErr: "asset unavailable"}, } { t.Run(tc.name, func(t *testing.T) { @@ -677,6 +680,234 @@ func TestRestoreFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { } } +// TestRestorePrefersLocalAndFallsBackToObjectStorage checks that Restore reads +// a usable node-local checkpoint in place even when snapshot.object is also +// populated, and falls back to staging from snapshot.object when the local copy +// is absent or unusable (corrupt manifest, missing listed file, or non-regular +// file). +func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) { + const ( + atespace = "ate-demo" + actorName = "counter" + actorUID = "01234567-89ab-cdef-0123-456789abcdef" + workerPodUID = "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f" + snapshotName = "pause-snap-1" + ) + + ateom := &fakeAteom{snapshotFiles: map[string]string{"checkpoint.img": ""}} + serveFakeAteom(t, ateom) + + host := imageVolumeTestRegistry(t) + pinnedImage := pushTestImage(t, host+"/actor:v1", singleFileLayer(t, "bin/app", "app")) + + runsc := []byte("runsc binary") + assetHash := fmt.Sprintf("%x", sha256.Sum256(runsc)) + sandboxAssets := &ateletpb.SandboxAssets{ + SandboxClass: "gvisor", + PauseImage: pinnedImage, + Assets: map[string]*ateletpb.ArchAssets{ + runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{ + runscAssetName: { + Url: "gs://test-bucket/runsc", + Sha256: assetHash, + }, + }}, + }, + } + spec := &ateletpb.WorkloadSpec{ + Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}}, + } + manifestRec := sandboxAssetsRecord{ + SandboxClass: "gvisor", + PauseImage: pinnedImage, + Assets: map[string]assetEntry{runscAssetName: {URL: "gs://test-bucket/runsc", SHA256: assetHash}}, + SnapshotFiles: []string{"checkpoint.img"}, + } + + // Upload a committed copy to object storage once; every subtest restores + // with snapshot.object pointing at it. + objStore := &recordingObjectStorage{} + s := newPluginHerder(t, objStore) + s.ateomDialer = newAteomDialer(1) + s.imageCache = newImageVolumeStore(t) + s.anonGCSClient = fakeObjectStorage{data: runsc} + s.systemInfoVolumes = newSystemInfoVolumeRefresher(nil, nil) + + uri, err := resources.ParseSnapshotURI(pausedSnapshotURI) + if err != nil { + t.Fatal(err) + } + remoteSrcDir := t.TempDir() + writeLocalSnapshot(t, remoteSrcDir, manifestRec, map[string]string{"checkpoint.img": "object-storage-memory"}) + if err := s.uploadSnapshot(t.Context(), uri, remoteSrcDir, &manifestRec, "default", "counter"); err != nil { + t.Fatalf("uploadSnapshot: %v", err) + } + + for _, tc := range []struct { + name string + prepLocal func(t *testing.T, localDir string) + wantLocal bool + }{{ + name: "usable local copy is preferred over object storage", + prepLocal: func(t *testing.T, localDir string) { + writeLocalSnapshot(t, localDir, manifestRec, map[string]string{"checkpoint.img": "local-memory"}) + }, + wantLocal: true, + }, { + name: "absent local copy falls back to object storage", + prepLocal: func(*testing.T, string) {}, + wantLocal: false, + }, { + name: "local copy with missing listed file falls back to object storage", + prepLocal: func(t *testing.T, localDir string) { + writeLocalSnapshot(t, localDir, manifestRec, nil) + }, + wantLocal: false, + }, { + name: "local copy with corrupt manifest falls back to object storage", + prepLocal: func(t *testing.T, localDir string) { + if err := os.MkdirAll(localDir, 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(localDir, sandboxManifestName), []byte("{not-json"), 0o600); err != nil { + t.Fatal(err) + } + }, + wantLocal: false, + }, { + name: "local copy with symlinked snapshot file falls back to object storage", + prepLocal: func(t *testing.T, localDir string) { + writeLocalSnapshot(t, localDir, manifestRec, nil) + target := filepath.Join(t.TempDir(), "outside.img") + if err := os.WriteFile(target, []byte("outside"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.Symlink(target, filepath.Join(localDir, "checkpoint.img")); err != nil { + t.Fatal(err) + } + }, + wantLocal: false, + }} { + t.Run(tc.name, func(t *testing.T) { + useTempNodeDirs(t) + localDir := ateletpath.LocalSnapshotDir(actorUID, snapshotName) + tc.prepLocal(t, localDir) + + if _, err := s.Restore(t.Context(), &ateletpb.RestoreRequest{ + Atespace: atespace, + ActorName: actorName, + ActorUid: actorUID, + ActorTemplateAtespace: "default", + ActorTemplateName: "counter", + WorkerPodUid: workerPodUID, + SandboxAssets: sandboxAssets, + Spec: spec, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: pausedSnapshotURI}, + }, + }); err != nil { + t.Fatalf("Restore: %v", err) + } + + wantContent := "object-storage-memory" + wantRestoreDir := ateletpath.RestoreStateDir(actorUID) + if tc.wantLocal { + wantContent = "local-memory" + wantRestoreDir = localDir + } + if got := ateom.restored["checkpoint.img"]; got != wantContent { + t.Errorf("restored checkpoint.img = %q, want %q", got, wantContent) + } + if ateom.preserveRestoreDir != tc.wantLocal { + t.Errorf("RestoreWorkload preserve_restore_dir = %v, want %v", ateom.preserveRestoreDir, tc.wantLocal) + } + if got := ateom.actorDirs["RestoreWorkload"].GetRestoreDir(); got != wantRestoreDir { + t.Errorf("RestoreWorkload restore_dir = %q, want %q", got, wantRestoreDir) + } + if tc.wantLocal { + if entries, err := os.ReadDir(ateletpath.RestoreStateDir(actorUID)); err != nil || len(entries) != 0 { + t.Errorf("RestoreStateDir entries = %v, err = %v; want empty on local restore", entries, err) + } + } + }) + } +} + +// TestRestoreWithoutUsableLocalCopy checks that a Restore with no usable local +// copy and nothing to fall back to fails as NotFound and wraps the local +// copy's problem, whether the request has no snapshot.object or names one that +// was never uploaded. +func TestRestoreWithoutUsableLocalCopy(t *testing.T) { + const ( + actorUID = "0f6c2b0e-1b7e-4b3f-9a6a-1d2c3e4f5a65" + snapshotName = "pause-snap-1" + ) + assetHash := fmt.Sprintf("%x", sha256.Sum256([]byte("runsc binary"))) + assets := &ateletpb.SandboxAssets{ + SandboxClass: "gvisor", PauseImage: testPauseImage, + Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{ + runscAssetName: {Url: "gs://test-bucket/runsc", Sha256: assetHash}, + }}}, + } + for _, tc := range []struct { + name string + // unusableLocal writes a manifest that lists a file that is not there. + unusableLocal bool + object *ateletpb.ObjectStorage + wantErr string + wantCause error + }{{ + name: "no local copy and no object", + wantErr: "no snapshot.object to fall back to", + wantCause: os.ErrNotExist, + }, { + name: "unusable local copy and no object", + unusableLocal: true, + wantErr: "no snapshot.object to fall back to", + wantCause: errUnusableLocalSnapshot, + }, { + name: "unusable local copy and an object never uploaded", + unusableLocal: true, + object: &ateletpb.ObjectStorage{SnapshotUri: pausedSnapshotURI}, + wantErr: "no uploaded copy", + wantCause: errUnusableLocalSnapshot, + }} { + t.Run(tc.name, func(t *testing.T) { + useTempNodeDirs(t) + if tc.unusableLocal { + writeLocalSnapshot(t, ateletpath.LocalSnapshotDir(actorUID, snapshotName), sandboxAssetsRecord{ + SandboxClass: "gvisor", PauseImage: testPauseImage, SnapshotFiles: []string{"checkpoint.img"}, + }, nil) + } + // The object store is empty: nothing was ever uploaded. + s := newPluginHerder(t, &recordingObjectStorage{}) + s.systemInfoVolumes = newSystemInfoVolumeRefresher(nil, nil) + _, err := s.Restore(t.Context(), &ateletpb.RestoreRequest{ + Atespace: "team-a", ActorName: "actor-1", ActorUid: actorUID, + WorkerPodUid: "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f", + SandboxAssets: assets, + Spec: &ateletpb.WorkloadSpec{}, + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: snapshotName, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: tc.object, + }, + }) + if code := apierror.Code(err); code != codes.NotFound { + t.Fatalf("Restore error = %v (code %v), want NotFound", err, code) + } + if !strings.Contains(err.Error(), tc.wantErr) { + t.Errorf("Restore error = %v, want it to say %q", err, tc.wantErr) + } + if !errors.Is(err, tc.wantCause) { + t.Errorf("Restore error = %v, want it to wrap %v", err, tc.wantCause) + } + }) + } +} + // TestTerminateWithoutWorkerPodUID verifies that Terminate with an empty // WorkerPodUid (used when cleaning up the assigned node for a paused or // crashed actor that no longer has a worker pod) skips dialing ateom while diff --git a/cmd/atelet/main.go b/cmd/atelet/main.go index af81336599..c5e9e74670 100644 --- a/cmd/atelet/main.go +++ b/cmd/atelet/main.go @@ -375,6 +375,7 @@ func main() { ateletpb.RegisterAteomSupportServer(ateomFacingSrv, &ateomSupportServer{ workers: ateapipb.NewWorkerServiceClient(ateapiConn), + control: ateapipb.NewControlClient(ateapiConn), }) go func() { if err := ateomFacingSrv.Serve(ateomFacingLis); err != nil { @@ -797,8 +798,8 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe // Do not move this above CheckpointWorkload to keep MergeDeltaIntoBase on its // in-place path: that leaves the whole checkpoint window with no local snapshot // while LocalSnapshotInfo still names the pruned one, and a crash there strands - // the actor for good (resume never falls back to object storage, RequiredNodes - // pins it to this node, nothing clears the field). + // the actor for good (a paused actor has no uploaded copy for resume to fall + // back to, RequiredNodes pins it to this node, nothing clears the field). // // Best-effort: if this fail, the actor's terminate prunes again. if err := pruneLocalCheckpoints(ctx, actorUID); err != nil { @@ -1087,6 +1088,33 @@ func (s *AteomHerder) uploadLocalCheckpointDir(ctx context.Context, req *ateletp return rec.SandboxClass, pluginsocket.CallError(s.uploadSnapshot(ctx, uri, localDir, rec, req.GetActorTemplateAtespace(), req.GetActorTemplateName())) } +// errUnusableLocalSnapshot marks a node-local snapshot that is present on disk +// (not simply absent) but cannot be read or validated: its manifest cannot be +// read or parsed, or lists a file that is missing or not a regular file, as an +// interrupted prune can leave. File contents and sizes are not checked. +var errUnusableLocalSnapshot = errors.New("unusable local snapshot") + +// readLocalSnapshot returns the manifest of the node-local snapshot in +// localDir once every file it lists is present. A missing manifest reports +// os.ErrNotExist, and any other failure wraps errUnusableLocalSnapshot. +func readLocalSnapshot(localDir string) (*sandboxAssetsRecord, error) { + manifest, err := readSnapshotManifest(localDir) + if errors.Is(err, os.ErrNotExist) { + return nil, err + } + if err != nil { + return nil, fmt.Errorf("%w: %w", errUnusableLocalSnapshot, err) + } + rec, err := unmarshalSandboxRecord(manifest) + if err != nil { + return nil, fmt.Errorf("%w: %w", errUnusableLocalSnapshot, err) + } + if err := checkLocalSnapshotFiles(localDir, rec.SnapshotFiles); err != nil { + return nil, fmt.Errorf("%w: %w", errUnusableLocalSnapshot, err) + } + return rec, nil +} + func readSnapshotManifest(dir string) ([]byte, error) { root, err := os.OpenRoot(dir) if err != nil { @@ -1164,14 +1192,14 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) } checkpointDir := ateletpath.RestoreStateDir(actorUID) - directLocal := req.GetSnapshot().GetObject() == nil - if directLocal { - checkpointDir = ateletpath.LocalSnapshotDir(actorUID, req.GetSnapshot().GetSnapshotUuid()) - } + localDir := ateletpath.LocalSnapshotDir(actorUID, req.GetSnapshot().GetSnapshotUuid()) + directLocal := false // Fetch the snapshot manifest stored beside the checkpoint images // first: it lists the checkpoint files to download and records the actor - // identity used to label the restore's metrics. + // identity used to label the restore's metrics. Try the node-local + // checkpoint directory first and fall back to object storage if the local + // copy is absent or unusable. tManifest := time.Now() manifestDone := false defer func() { @@ -1180,16 +1208,27 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) } }() var sandboxRec *sandboxAssetsRecord - if directLocal { - manifest, err := readSnapshotManifest(checkpointDir) - if err != nil { - return nil, wrapFileSystemErr("while reading local snapshot manifest", err) + localRec, localErr := readLocalSnapshot(localDir) + durable := req.GetSnapshot().GetObject() + if localErr == nil { + directLocal = true + checkpointDir = localDir + sandboxRec = localRec + } else { + if durable == nil { + return nil, apierror.NotFound("snapshot %q has no usable local copy and no snapshot.object to fall back to: %w", + req.GetSnapshot().GetSnapshotUuid(), localErr) } - if sandboxRec, err = unmarshalSandboxRecord(manifest); err != nil { - return nil, fmt.Errorf("while unmarshalling sandbox record: %w", err) + if errors.Is(localErr, errUnusableLocalSnapshot) { + // The next checkpoint or terminate prunes the unusable copy. + slog.WarnContext(ctx, "Local snapshot is unusable; restoring from object storage", + slog.String("local_dir", localDir), slog.Any("err", localErr)) + } + manifest, err := s.fetchManifest(ctx, durable.GetSnapshotUri()) + if status.Code(err) == codes.NotFound { + return nil, apierror.NotFound("snapshot %q has no usable local copy and no uploaded copy: %w", + req.GetSnapshot().GetSnapshotUuid(), localErr) } - } else { - manifest, err := s.fetchManifest(ctx, req.GetSnapshot().GetObject().GetSnapshotUri()) if err != nil { return nil, fmt.Errorf("while fetching snapshot manifest: %w", err) } @@ -1203,7 +1242,7 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) // The manifest is what tells a golden restore from a latest one, so the // snapshot kind only becomes knowable here. - op.kind = restoreSnapshotKind(req, sandboxRec) + op.kind = restoreSnapshotKind(directLocal, sandboxRec) var registration *registeredActor // Undo the Register if the restore fails. The errgroup join below happens @@ -1234,10 +1273,11 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) downloadErr = err }() if directLocal { - // Restore in place from LocalSnapshotDir; no staging. - return checkLocalSnapshotFiles(checkpointDir, sandboxRec.SnapshotFiles) + // Restore in place from LocalSnapshotDir, whose files + // readLocalSnapshot checked; no staging. + return nil } - return s.downloadExternalCheckpoint(gctx, req.GetSnapshot().GetObject().GetSnapshotUri(), checkpointDir, sandboxRec.SnapshotFiles) + return s.downloadExternalCheckpoint(gctx, durable.GetSnapshotUri(), checkpointDir, sandboxRec.SnapshotFiles) }) g.Go(func() (err error) { defer func() { prepErr = err }() diff --git a/cmd/atelet/metrics.go b/cmd/atelet/metrics.go index 804020545f..7433497b8b 100644 --- a/cmd/atelet/metrics.go +++ b/cmd/atelet/metrics.go @@ -153,13 +153,13 @@ func assetsAfterCollateral(prepFailedPhase string, assets time.Duration) time.Du return assets } -// restoreSnapshotKind classifies which snapshot a restore reads. A local -// restore is evident from the wire; golden and latest both arrive as an external -// URI prefix, so they are told apart by the identity the manifest records for -// the actor that wrote the snapshot. An empty result means the manifest has not -// been read yet, so the kind is not knowable. -func restoreSnapshotKind(req *ateletpb.RestoreRequest, rec *sandboxAssetsRecord) string { - if req.GetSnapshot().GetObject() == nil { +// restoreSnapshotKind classifies which snapshot a restore reads: local when it +// is served from the node-local copy, and otherwise golden or latest, told +// apart by the identity the manifest records for the actor that wrote the +// snapshot. An empty result means the manifest has not been read yet, so the +// kind is not knowable. +func restoreSnapshotKind(directLocal bool, rec *sandboxAssetsRecord) string { + if directLocal { return ateattr.SnapshotKindLocal } if rec == nil { diff --git a/cmd/atelet/metrics_test.go b/cmd/atelet/metrics_test.go index 8aaf437113..92f5ef34d2 100644 --- a/cmd/atelet/metrics_test.go +++ b/cmd/atelet/metrics_test.go @@ -280,59 +280,48 @@ func TestAssetsAfterCollateral(t *testing.T) { } func TestRestoreSnapshotKind(t *testing.T) { - localSnapshot := &ateletpb.Snapshot{ - SnapshotUuid: "snap-1", - } - externalSnapshot := &ateletpb.Snapshot{ - SnapshotUuid: "snap-1", - Object: &ateletpb.ObjectStorage{SnapshotUri: "gs://bucket/snap"}, - } tests := []struct { - name string - req *ateletpb.RestoreRequest - rec *sandboxAssetsRecord - want string + name string + directLocal bool + rec *sandboxAssetsRecord + want string }{ { - name: "local pause snapshot", - req: &ateletpb.RestoreRequest{Snapshot: localSnapshot}, - rec: &sandboxAssetsRecord{Atespace: "team-a"}, - want: ateattr.SnapshotKindLocal, + name: "restored from the local copy", + directLocal: true, + rec: &sandboxAssetsRecord{Atespace: "team-a"}, + want: ateattr.SnapshotKindLocal, }, { - name: "local restore is classifiable before the manifest is read", - req: &ateletpb.RestoreRequest{Snapshot: localSnapshot}, - rec: nil, - want: ateattr.SnapshotKindLocal, + name: "a golden actor's snapshot served locally is still local", + directLocal: true, + rec: &sandboxAssetsRecord{Atespace: resources.GoldenActorAtespace}, + want: ateattr.SnapshotKindLocal, }, { name: "external snapshot written by a golden actor", - req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: &sandboxAssetsRecord{Atespace: resources.GoldenActorAtespace}, want: ateattr.SnapshotKindGolden, }, { name: "external snapshot written by a tenant actor", - req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: &sandboxAssetsRecord{Atespace: "team-a"}, want: ateattr.SnapshotKindLatest, }, { name: "manifest predating the identity fields", - req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: &sandboxAssetsRecord{}, want: ateattr.SnapshotKindLatest, }, { name: "external kind is unknowable until the manifest is read", - req: &ateletpb.RestoreRequest{Snapshot: externalSnapshot}, rec: nil, want: "", }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - if got := restoreSnapshotKind(tt.req, tt.rec); got != tt.want { + if got := restoreSnapshotKind(tt.directLocal, tt.rec); got != tt.want { t.Errorf("restoreSnapshotKind() = %q, want %q", got, tt.want) } }) diff --git a/cmd/atelet/sandbox_assets.go b/cmd/atelet/sandbox_assets.go index 8b70d56ab1..3e0189e071 100644 --- a/cmd/atelet/sandbox_assets.go +++ b/cmd/atelet/sandbox_assets.go @@ -295,11 +295,11 @@ func (s *AteomHerder) fetchGVisorRelease(ctx context.Context, entry assetEntry) // first, with the plugin's credentials; atelet holds none of its own and does // not know which bucket holds snapshots. A gs:// asset is then read // anonymously, which serves public buckets such as gs://gvisor, but only when -// no plugin is configured or the plugin reports it has no such object or -// cannot serve it, as a plugin on a backend other than GCS does for every -// gs:// URI (see anonymousFallbackCodes). A refusal on policy or content, and a canceled -// request, are final. atelet verifies the bytes it caches whichever path -// fetched them, since it executes them. +// no plugin is configured or the plugin reports it has no such object, cannot +// serve it (as a plugin on a backend other than GCS does for every gs:// URI), +// or failed unexpectedly (see anonymousFallbackCodes). A refusal on policy or +// content, and a canceled request, are final. atelet verifies the bytes it +// caches whichever path fetched them, since it executes them. func (s *AteomHerder) downloadVerified(ctx context.Context, entry assetEntry, tmpPrefix string) (string, error) { wantSum, err := hex.DecodeString(entry.SHA256) if err != nil { @@ -348,7 +348,14 @@ func (s *AteomHerder) downloadVerified(ctx context.Context, entry assetEntry, tm if ctxErr := ctx.Err(); ctxErr != nil { return "", fmt.Errorf("sandbox asset %s: %w", shown, ctxErr) } - slog.InfoContext(ctx, "Object-store plugin did not serve sandbox asset; downloading it anonymously", + // Internal and Unavailable mean the plugin failed rather than declined + // the URL, so falling back must not hide them: warn. The plugin logs + // why it failed; atelet logs only the code (see fetchFromPlugin). + level := slog.LevelInfo + if pluginCode == codes.Internal || pluginCode == codes.Unavailable { + level = slog.LevelWarn + } + slog.Log(ctx, level, "Object-store plugin did not serve sandbox asset; downloading it anonymously", slog.String("url", shown), slog.String("code", pluginCode.String())) } else if !anonFallback { return "", apierror.FailedPrecondition("sandbox asset %s: no object-store plugin is configured to fetch it", shown) @@ -366,10 +373,11 @@ func (s *AteomHerder) downloadVerified(ctx context.Context, entry assetEntry, tm // anonymousFallbackCodes are the plugin codes after which atelet reads a gs:// // asset anonymously: the plugin has no such object, cannot serve the URL, does -// not implement AssetProvider, or cannot be reached. PermissionDenied (the URL -// is under a snapshot location) and FailedPrecondition (wrong size or sha256) -// are refusals another source must not override. -var anonymousFallbackCodes = []codes.Code{codes.NotFound, codes.InvalidArgument, codes.Unimplemented, codes.Unavailable} +// not implement AssetProvider, cannot be reached, or failed with Internal. The +// in-tree plugin returns Internal for any error it does not recognize, such as +// GCS refusing its credentials for a public bucket (see #2613), so this falls +// back on any unexpected failure of that plugin. Any other code is final. +var anonymousFallbackCodes = []codes.Code{codes.NotFound, codes.InvalidArgument, codes.Unimplemented, codes.Unavailable, codes.Internal} // downloadAnonymous streams the gs:// asset entry.URL into the existing file // at path with no credentials, replacing anything a failed plugin fetch left diff --git a/cmd/atelet/sandbox_assets_plugin_test.go b/cmd/atelet/sandbox_assets_plugin_test.go index 15ebfe7418..049cd8c8ff 100644 --- a/cmd/atelet/sandbox_assets_plugin_test.go +++ b/cmd/atelet/sandbox_assets_plugin_test.go @@ -373,7 +373,7 @@ func TestFetchAssetAnonymousFallback(t *testing.T) { // Pinned here rather than read from anonymousFallbackCodes, so dropping a // code from production fails this test. - for _, code := range []codes.Code{codes.NotFound, codes.InvalidArgument, codes.Unimplemented, codes.Unavailable} { + for _, code := range []codes.Code{codes.NotFound, codes.InvalidArgument, codes.Unimplemented, codes.Unavailable, codes.Internal} { t.Run(code.String()+" falls back", func(t *testing.T) { logs := captureLogs(t) f := newAssetHerderFixture(t, "s3") @@ -393,6 +393,21 @@ func TestFetchAssetAnonymousFallback(t *testing.T) { if !strings.Contains(logs.String(), `"code":"`+code.String()+`"`) || strings.Contains(logs.String(), "SECRET") { t.Errorf("logs do not carry the plugin code %s alone:\n%s", code, logs) } + // A plugin that failed, rather than declined the URL, must stay + // visible even though the anonymous read succeeded. + wantLevel := `"level":"INFO"` + if code == codes.Internal || code == codes.Unavailable { + wantLevel = `"level":"WARN"` + } + var fallbackLine string + for l := range strings.Lines(logs.String()) { + if strings.Contains(l, "downloading it anonymously") { + fallbackLine = l + } + } + if !strings.Contains(fallbackLine, wantLevel) { + t.Errorf("fallback log line is not at %s:\n%s", wantLevel, fallbackLine) + } }) t.Run(code.String()+" then anonymous failure reports both", func(t *testing.T) { @@ -412,6 +427,8 @@ func TestFetchAssetAnonymousFallback(t *testing.T) { wantCode := codes.FailedPrecondition if code == codes.Unavailable { wantCode = codes.Unavailable + } else if code == codes.Internal { + wantCode = codes.Internal } if st.Code() != wantCode { t.Errorf("code = %s, want %s (%v)", st.Code(), wantCode, err) @@ -445,7 +462,7 @@ func TestFetchAssetAnonymousFallback(t *testing.T) { } } - for _, code := range []codes.Code{codes.PermissionDenied, codes.FailedPrecondition, codes.Internal, codes.Unknown, codes.ResourceExhausted, codes.Canceled, codes.DeadlineExceeded} { + for _, code := range []codes.Code{codes.PermissionDenied, codes.FailedPrecondition, codes.Unknown, codes.ResourceExhausted, codes.Canceled, codes.DeadlineExceeded} { t.Run(code.String()+" does not fall back", func(t *testing.T) { f := newAssetHerderFixture(t, "s3") f.anon.objects[object] = content diff --git a/cmd/atelet/statspoller.go b/cmd/atelet/statspoller.go index cdd9aade76..359e57378a 100644 --- a/cmd/atelet/statspoller.go +++ b/cmd/atelet/statspoller.go @@ -325,7 +325,7 @@ func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggr key := templateKey{ templateNamespace: sample.GetActorTemplateAtespace(), templateName: sample.GetActorTemplateName(), - sandboxClass: ateomstats.SandboxClassLabel(sample.GetSandboxClass()), + sandboxClass: ateattr.NormalizeSandboxClass(sample.GetSandboxClass()), source: ateomstats.StatsSourceLabel(sample.GetSource()), workerPool: pools[podUID], } diff --git a/cmd/atelet/statspoller_test.go b/cmd/atelet/statspoller_test.go index f6f6f9636f..0a8e3a6c31 100644 --- a/cmd/atelet/statspoller_test.go +++ b/cmd/atelet/statspoller_test.go @@ -61,7 +61,7 @@ func (f *fakeStatsAteom) GetActiveWorkloadStats(ctx context.Context, req *ateomp } // executingResponse builds the sample an executing ateom would echo. -func executingResponse(templateNS, templateName string, class ateompb.SandboxClass, source ateompb.StatsSource, current, workingSet uint64) *ateompb.GetActiveWorkloadStatsResponse { +func executingResponse(templateNS, templateName string, class string, source ateompb.StatsSource, current, workingSet uint64) *ateompb.GetActiveWorkloadStatsResponse { return &ateompb.GetActiveWorkloadStatsResponse{ Samples: []*ateompb.WorkloadStatsSample{{ ActorTemplateAtespace: templateNS, @@ -80,7 +80,7 @@ func measuredSample(actorUID, templateNS, templateName string, current, workingS ActorUid: actorUID, ActorTemplateAtespace: templateNS, ActorTemplateName: templateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: current, MemoryWorkingSetBytes: workingSet, @@ -95,7 +95,7 @@ func pendingSample(actorUID, templateNS, templateName string) *ateompb.WorkloadS ActorUid: actorUID, ActorTemplateAtespace: templateNS, ActorTemplateName: templateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", } } @@ -128,7 +128,7 @@ func pendingResponse(actorUID string) *ateompb.GetActiveWorkloadStatsResponse { ActorUid: actorUID, ActorTemplateAtespace: "ns-a", ActorTemplateName: "tmpl-a", - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", }}, } } @@ -193,9 +193,9 @@ func TestStatsPollerCollectAggregates(t *testing.T) { // worker, one mid-boot: the same-template pair sums, the others contribute // nothing. fakes := map[string]*fakeStatsAteom{ - "uid-1": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, - "uid-2": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 500, 300)}, - "uid-3": {resp: executingResponse("ns-b", "tmpl-b", ateompb.SandboxClass_SANDBOX_CLASS_MICROVM, ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT, 42, 40)}, + "uid-1": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, + "uid-2": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 500, 300)}, + "uid-3": {resp: executingResponse("ns-b", "tmpl-b", "microvm", ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT, 42, 40)}, "uid-4": {resp: availableResponse()}, "uid-5": {resp: pendingResponse("uid-5-actor")}, } @@ -231,7 +231,7 @@ func TestStatsPollerCollectAggregates(t *testing.T) { // The healthy ateom's sample must still be aggregated. func TestStatsPollerCollectSkipsFailures(t *testing.T) { fakes := map[string]*fakeStatsAteom{ - "uid-healthy": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, + "uid-healthy": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, "uid-stale": nil, // directory with no reachable socket: dial fails "uid-broken": {err: errors.New("rpc error: connection refused")}, } @@ -343,7 +343,7 @@ func cpuResponse(actorUID string, cpuUsec uint64) *ateompb.GetActiveWorkloadStat ActorUid: actorUID, ActorTemplateAtespace: "ns-a", ActorTemplateName: "tmpl-a", - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, CpuUsageUsec: cpuUsec, }}, @@ -396,8 +396,8 @@ func TestStatsPollerCPUDeltas(t *testing.T) { // than vanishing, and the two never merge. func TestStatsPollerWorkerPoolLabels(t *testing.T) { fakes := map[string]*fakeStatsAteom{ - "uid-pooled": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, - "uid-unpooled": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, + "uid-pooled": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, + "uid-unpooled": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, } p, _ := newPollerFixture(t, fakes) p.fetchWorkerPools = func(context.Context) map[string]workerPoolRef { @@ -425,8 +425,8 @@ func TestStatsPollerWorkerPoolLabels(t *testing.T) { func TestStatsPollerCollectSaturatesCorruptSamples(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} fakes := map[string]*fakeStatsAteom{ - "uid-1": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, math.MaxUint64, math.MaxUint64)}, - "uid-2": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, + "uid-1": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, math.MaxUint64, math.MaxUint64)}, + "uid-2": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, } p, _ := newPollerFixture(t, fakes) @@ -495,7 +495,7 @@ func TestNewWorkerPoolFetcher(t *testing.T) { // that feeds the monotonic counter and can never be re-attributed -- on the // pooled label set. func TestStatsPollerPoolCacheSurvivesListFlap(t *testing.T) { - resp := executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp.GetSamples()[0].ActorUid = "uid-a" resp.GetSamples()[0].CpuUsageUsec = 1000 fake := &fakeStatsAteom{resp: resp} @@ -519,7 +519,7 @@ func TestStatsPollerPoolCacheSurvivesListFlap(t *testing.T) { // Sweep 2: the list fails AND the actor consumed CPU. Both the sample and // its delta must still group under the pool. listOK = false - resp2 := executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp2 := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp2.GetSamples()[0].ActorUid = "uid-a" resp2.GetSamples()[0].CpuUsageUsec = 1600 fake.resp = resp2 @@ -564,7 +564,7 @@ func TestStatsPollerPoolCachePrunes(t *testing.T) { // labels (the residual, documented case) and heals on the next good list. func TestStatsPollerPoolCacheMissDuringOutage(t *testing.T) { fakes := map[string]*fakeStatsAteom{ - "uid-new": {resp: executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, + "uid-new": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, } p, _ := newPollerFixture(t, fakes) p.fetchWorkerPools = func(context.Context) map[string]workerPoolRef { return nil } @@ -583,7 +583,7 @@ func TestStatsPollerPoolCacheMissDuringOutage(t *testing.T) { // all-nil flap test above -- a whole-map fallback would pass that test and // fail this one. func TestStatsPollerPoolCachePartialListFallsBack(t *testing.T) { - resp := executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp.GetSamples()[0].ActorUid = "uid-a" resp.GetSamples()[0].CpuUsageUsec = 1000 fake := &fakeStatsAteom{resp: resp} @@ -606,7 +606,7 @@ func TestStatsPollerPoolCachePartialListFallsBack(t *testing.T) { } full = false - resp2 := executingResponse("ns-a", "tmpl-a", ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp2 := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp2.GetSamples()[0].ActorUid = "uid-a" resp2.GetSamples()[0].CpuUsageUsec = 1250 fake.resp = resp2 diff --git a/cmd/atenet/README.md b/cmd/atenet/README.md index 43fa5a4482..8f17a3855c 100644 --- a/cmd/atenet/README.md +++ b/cmd/atenet/README.md @@ -37,9 +37,11 @@ likely be split in the future for better scalability.) Envoy container's `preStop` hook waits for the router's drain-complete marker on a pod-shared emptyDir — so established connections and parked requests finish instead of resetting. The whole sequence must fit within - `terminationGracePeriodSeconds` (see the manifest comments). Upgrades are - whole-system swaps (#473) rather than per-Deployment rolling updates; the - drain is what makes the old system's termination lossless. + `terminationGracePeriodSeconds` (see the manifest comments). An upgrade + rolls the router as an ordinary Deployment update (see + [docs/upgrade.md](../../docs/upgrade.md)). The drain lets each old pod's + requests finish, except one still open about 15 seconds after the drain + starts, which is cut. RBAC permissions: * get, list, watch on ate-system EndpointSlices diff --git a/cmd/atenet/internal/router/cmd.go b/cmd/atenet/internal/router/cmd.go index 966bd73597..7bd65c2d3d 100644 --- a/cmd/atenet/internal/router/cmd.go +++ b/cmd/atenet/internal/router/cmd.go @@ -82,10 +82,10 @@ func bindRouterFlags(fs *pflag.FlagSet, cfg *routerConfig) { "Ingress authentication and authorization mode. `deprecated-insecure` (both http and https served, client certificates optional on https) or `static-mtls` (only https served, client certificates required).", ) - fs.StringVar(&cfg.UpstreamCredentialBundlePath, "upstream-credential-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "PEM credential bundle (cert+key) the router presents as the client cert when dialing the actor's atunnel ingress server over mTLS. Empty disables upstream mTLS (legacy plaintext pod-IP:80).") + fs.StringVar(&cfg.UpstreamCredentialBundlePath, "upstream-credential-bundle", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "PEM credential bundle (cert+key) the router presents as the client cert when dialing the actor's atunnel ingress server over mTLS. Empty disables upstream mTLS (legacy plaintext pod-IP:80) and needs --upstream-connect=false.") fs.StringVar(&cfg.UpstreamTrustBundlePath, "upstream-trust-bundle", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "PEM trust bundle used to validate the actor's atunnel ingress server certificate.") fs.StringVar(&cfg.UpstreamSpiffePrefix, "upstream-spiffe-prefix", "spiffe://cluster.local/", "SPIFFE URI SAN prefix (trust domain) the actor's atunnel server cert must match. Empty falls back to default SAN check against the dialed pod IP (which SPIFFE-only certs never match).") - fs.BoolVar(&cfg.UpstreamConnect, "upstream-connect", false, "Reach atunnel through mTLS CONNECT tunnels on port 8443 instead of HTTPS on 443 (Envoy dataplane only). Requires --upstream-credential-bundle") + fs.BoolVar(&cfg.UpstreamConnect, "upstream-connect", true, "Reach atunnel through mTLS CONNECT tunnels on port 8443; false reaches it over HTTPS on 443 instead (Envoy dataplane only). Requires --upstream-credential-bundle") fs.StringVar(&cfg.ActorIdentityCAFile, "actor-identity-ca-file", "", "PEM trust bundle for the actor-identity CA, used to verify the actor client certificates presented on egress CONNECTs. Required by the egress gateway's ext_proc sidecar; empty (the default) leaves egress authentication unconfigured and every egress CONNECT is denied.") diff --git a/cmd/atenet/internal/router/config.go b/cmd/atenet/internal/router/config.go index 6d4e60c4b8..55c9cb00da 100644 --- a/cmd/atenet/internal/router/config.go +++ b/cmd/atenet/internal/router/config.go @@ -323,8 +323,9 @@ func (c routerConfig) validate() error { return fmt.Errorf("--extproc-max-requests (%d) must be >= --parked-request-max (%d): a circuit breaker below the parking lot silently truncates it with Envoy-generated 503s", c.ExtProcMaxRequests, c.ParkedRequest.Max) } - if c.UpstreamConnect && c.UpstreamCredentialBundlePath == "" { - return fmt.Errorf("--upstream-connect requires --upstream-credential-bundle: atunnel's CONNECT listener accepts only mTLS") + // Only the Envoy ingress router uses this flag; agentgateway always uses CONNECT. + if c.UpstreamConnect && c.UpstreamCredentialBundlePath == "" && c.Mode.ServesIngress() && c.atenetRouter() == atenetRouterEnvoy { + return fmt.Errorf("--upstream-connect (on by default) requires --upstream-credential-bundle: atunnel's CONNECT listener accepts only mTLS. Set --upstream-connect=false to dial atunnel without it") } if c.EgressPolicyCacheTTL < 0 { return fmt.Errorf("--egress-policy-cache-ttl must not be negative, got %s (0 disables the cache)", c.EgressPolicyCacheTTL) diff --git a/cmd/atenet/internal/router/config_test.go b/cmd/atenet/internal/router/config_test.go index 99a738c2a4..16ecb54bef 100644 --- a/cmd/atenet/internal/router/config_test.go +++ b/cmd/atenet/internal/router/config_test.go @@ -112,7 +112,11 @@ func TestRouterConfigValidate(t *testing.T) { { name: "upstream-connect without the upstream credential bundle rejected", cfg: routerConfig{Mode: ModeIngress, UpstreamConnect: true}, - wantErr: "--upstream-connect requires --upstream-credential-bundle", + wantErr: "requires --upstream-credential-bundle", + }, + { + name: "egress mode ignores upstream-connect", + cfg: routerConfig{Mode: ModeEgress, UpstreamConnect: true}, }, { name: "negative drain-delay rejected", diff --git a/cmd/atenet/internal/router/dataplane.go b/cmd/atenet/internal/router/dataplane.go index c3c4f0d9ab..7bae70d171 100644 --- a/cmd/atenet/internal/router/dataplane.go +++ b/cmd/atenet/internal/router/dataplane.go @@ -63,6 +63,29 @@ func (s *RouterServer) startDataplane(ctx context.Context, g *errgroup.Group, pa } func (s *RouterServer) startEnvoyDataplane(ctx context.Context, g *errgroup.Group, parkCfg ingress.ParkedRequestConfig, traceRootSamplingPercent float64) error { + xdsSrv, err := s.newEnvoyXdsServer(ctx, parkCfg, traceRootSamplingPercent) + if err != nil { + return err + } + + // Envoy receives all routing configuration from the local xDS server. + g.Go(func() error { + addr := xdsListenAddress(s.cfg.XdsPort) + slog.InfoContext(ctx, "Starting Envoy xDS Server", slog.String("address", addr)) + lis, err := net.Listen("tcp", addr) + if err != nil { + return fmt.Errorf("failed to listen on %s: %w", addr, err) + } + defer lis.Close() + + return xdsSrv.Serve(ctx, lis) + }) + return nil +} + +// newEnvoyXdsServer applies the router configuration to a new xDS server and +// builds its snapshot. +func (s *RouterServer) newEnvoyXdsServer(ctx context.Context, parkCfg ingress.ParkedRequestConfig, traceRootSamplingPercent float64) (*XdsServer, error) { xdsSrv := NewXdsServer(s.cfg.XdsPort) xdsSrv.SetConfig(s.cfg.HttpPort, s.cfg.ExtprocPort, s.cfg.ExtprocAddr) xdsSrv.SetConnectPorts(s.cfg.ConnectPlainTextPort, s.cfg.ConnectTLSPort) @@ -87,7 +110,7 @@ func (s *RouterServer) startEnvoyDataplane(ctx context.Context, g *errgroup.Grou // one as a listener that never warms. Parse it here so the router // fails startup instead. if _, err := credbundle.ParsePool(s.cfg.IngressAuth.ClientCAFile); err != nil { - return fmt.Errorf("loading --ingress-client-ca-file: %w", err) + return nil, fmt.Errorf("loading --ingress-client-ca-file: %w", err) } require := s.cfg.IngressAuth.mode() == IngressAuthStaticMTLS xdsSrv.SetDownstreamClientAuth(s.cfg.IngressAuth.ClientCAFile, s.cfg.IngressAuth.AllowedSPIFFEIDs, require) @@ -106,22 +129,9 @@ func (s *RouterServer) startEnvoyDataplane(ctx context.Context, g *errgroup.Grou // is built exactly once; an Envoy that connects later is served from the // xDS cache. if err := xdsSrv.UpdateSnapshot(); err != nil { - return fmt.Errorf("building the xDS snapshot: %w", err) + return nil, fmt.Errorf("building the xDS snapshot: %w", err) } - - // Envoy receives all routing configuration from the local xDS server. - g.Go(func() error { - addr := xdsListenAddress(s.cfg.XdsPort) - slog.InfoContext(ctx, "Starting Envoy xDS Server", slog.String("address", addr)) - lis, err := net.Listen("tcp", addr) - if err != nil { - return fmt.Errorf("failed to listen on %s: %w", addr, err) - } - defer lis.Close() - - return xdsSrv.Serve(ctx, lis) - }) - return nil + return xdsSrv, nil } // xdsListenAddress is where the router serves xDS to the Envoy in its own pod. diff --git a/cmd/atenet/internal/router/xds_upstream_connect_test.go b/cmd/atenet/internal/router/xds_upstream_connect_test.go index 7bea556e82..c243c3777a 100644 --- a/cmd/atenet/internal/router/xds_upstream_connect_test.go +++ b/cmd/atenet/internal/router/xds_upstream_connect_test.go @@ -16,10 +16,13 @@ package router import ( "regexp" + "slices" + "strings" "testing" clusterv3 "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3" listenerv3 "github.com/envoyproxy/go-control-plane/envoy/config/listener/v3" + routev3 "github.com/envoyproxy/go-control-plane/envoy/config/route/v3" setfilterstatecommonv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/common/set_filter_state/v3" setfilterstatev3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/http/set_filter_state/v3" hcmv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/http_connection_manager/v3" @@ -29,6 +32,7 @@ import ( httpv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/upstreams/http/v3" cachev3 "github.com/envoyproxy/go-control-plane/pkg/cache/v3" resourcev3 "github.com/envoyproxy/go-control-plane/pkg/resource/v3" + "github.com/spf13/pflag" "github.com/agent-substrate/substrate/cmd/atenet/internal/router/extproc" "github.com/agent-substrate/substrate/internal/atenet" @@ -58,6 +62,91 @@ func upstreamConnectSnapshot(t *testing.T) *cachev3.Snapshot { return snap } +// Parses real flags so the default and the wiring into the xDS server are +// tested together. +func TestUpstreamConnectFlag(t *testing.T) { + tests := []struct { + name string + args []string + wantErr string // substring from validate; empty means valid + wantTunnels bool + }{ + { + name: "default tunnels to atunnel", + wantTunnels: true, + }, + { + name: "false reaches atunnel over HTTPS", + args: []string{"--upstream-connect=false"}, + }, + { + name: "false without upstream mTLS accepted", + args: []string{"--upstream-connect=false", "--upstream-credential-bundle="}, + }, + { + name: "default without upstream mTLS rejected", + args: []string{"--upstream-credential-bundle="}, + wantErr: "--upstream-connect=false", + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + var cfg routerConfig + fs := pflag.NewFlagSet("router", pflag.ContinueOnError) + bindRouterFlags(fs, &cfg) + if err := fs.Parse(append([]string{"--mode=ingress", "--otlp-collector-address="}, tc.args...)); err != nil { + t.Fatalf("parsing flags: %v", err) + } + err := cfg.validate() + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("validate() = %v, want an error containing %q", err, tc.wantErr) + } + return + } + if err != nil { + t.Fatalf("validate() = %v, want valid", err) + } + + s := &RouterServer{cfg: cfg} + x, err := s.newEnvoyXdsServer(t.Context(), cfg.ParkedRequest.Normalized(), 0) + if err != nil { + t.Fatalf("newEnvoyXdsServer: %v", err) + } + res, err := x.snapshot.GetSnapshot(NodeID) + if err != nil { + t.Fatalf("Failed to get snapshot: %v", err) + } + snap := res.(*cachev3.Snapshot) + if err := snap.Consistent(); err != nil { + t.Fatalf("Integrity check failed on snapshot: %v", err) + } + + var routed []string + rc := snap.GetResources(resourcev3.RouteType)[RouteName].(*routev3.RouteConfiguration) + for _, r := range rc.GetVirtualHosts()[0].GetRoutes() { + routed = append(routed, r.GetRoute().GetCluster()) + } + want := []string{OriginalDstClusterName} + if tc.wantTunnels { + want = []string{ActorTunnelHTTP2ClusterName, ActorTunnelHTTP1ClusterName} + } + if !slices.Equal(routed, want) { + t.Errorf("routes send to %v, want %v", routed, want) + } + clusters := snap.GetResources(resourcev3.ClusterType) + for _, name := range routed { + if _, ok := clusters[name]; !ok { + t.Errorf("a route sends to cluster %q, which the snapshot lacks", name) + } + } + if _, ok := snap.GetResources(resourcev3.ListenerType)[ActorConnectName]; ok != tc.wantTunnels { + t.Errorf("listener %q present = %v, want %v", ActorConnectName, ok, tc.wantTunnels) + } + }) + } +} + func TestXdsServer_UpstreamConnectResources(t *testing.T) { snap := upstreamConnectSnapshot(t) diff --git a/cmd/ateom-gvisor/hosted.go b/cmd/ateom-gvisor/hosted.go index 45b06e35e3..81f6793317 100644 --- a/cmd/ateom-gvisor/hosted.go +++ b/cmd/ateom-gvisor/hosted.go @@ -89,6 +89,7 @@ func (s *AteomService) hostActor(ctx context.Context, attribution resources.Acto ActorUID: uid, Veth: true, EgressPort: s.tunnel.EgressPort, + DNSPolicy: s.tunnel.ActorEgressPolicy(attribution), }, s.tunnel.Egress, s.tunnel.DNSRelay) if err != nil { s.actorsMu.Lock() diff --git a/cmd/ateom-gvisor/main.go b/cmd/ateom-gvisor/main.go index 8feeb9b00b..6457e2e72b 100644 --- a/cmd/ateom-gvisor/main.go +++ b/cmd/ateom-gvisor/main.go @@ -40,11 +40,11 @@ import ( "github.com/agent-substrate/substrate/internal/ateinterceptors" "github.com/agent-substrate/substrate/internal/ateom" "github.com/agent-substrate/substrate/internal/ateomcgroup" + "github.com/agent-substrate/substrate/internal/ateominit" "github.com/agent-substrate/substrate/internal/ateomnet" "github.com/agent-substrate/substrate/internal/ateomphaselog" "github.com/agent-substrate/substrate/internal/ateomstats" "github.com/agent-substrate/substrate/internal/ateomtunnel" - "github.com/agent-substrate/substrate/internal/childreap" "github.com/agent-substrate/substrate/internal/imagecache" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/ocispec" @@ -85,9 +85,6 @@ var ( otlpRelaySocket = pflag.String("otlp-relay-socket", nodepath.AteletOTLPSocketPath(), "Unix socket of atelet's OTLP relay to export telemetry through, keeping it off the pod network. Empty, or absent at startup, exports directly to OTEL_EXPORTER_OTLP_ENDPOINT instead.") - - // reaper collects children orphaned in the pod PID namespace. - reaper = childreap.New() ) // minUsageSampleInterval is the floor of --usage-sample-interval, the same as @@ -101,6 +98,7 @@ const minUsageSampleInterval = 50 * time.Second const workloadGracePeriod = 30 * time.Minute func main() { + ateominit.RunIfPID1() pflag.Parse() if *showVersion { fmt.Println(version.String()) @@ -203,9 +201,6 @@ func do(ctx context.Context) error { return fmt.Errorf("while setting up cgroup delegation: %w", err) } - go reaper.Run(ctx) - slog.InfoContext(ctx, "Child process reaper launched") - // Clean up any old socket. sockPath := nodepath.AteomSocketPath(*podUID) if err := os.RemoveAll(sockPath); err != nil { @@ -663,10 +658,16 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload return nil, fmt.Errorf("while composing pause rootfs: %w", err) } containersToDelete = append(containersToDelete, ocispec.PauseContainer) - if err := rcmd.cmdCreate(ctx, os.Stdout, ocispec.PauseContainer, nil); err != nil { + // Attribute runsc CLI output and the pause container's stdio to the actor. + pw, err := s.actorLogger.StartJSONLogPipe(attribution, ocispec.PauseContainer) + if err != nil { + return nil, fmt.Errorf("while starting pause log pipe: %w", err) + } + defer pw.Close() + if err := rcmd.cmdCreate(ctx, pw, ocispec.PauseContainer, nil); err != nil { return nil, fmt.Errorf("while creating pause container: %w", err) } - if err := rcmd.cmdStart(ctx, os.Stdout, ocispec.PauseContainer); err != nil { + if err := rcmd.cmdStart(ctx, pw, ocispec.PauseContainer); err != nil { return nil, fmt.Errorf("while starting pause container: %w", err) } @@ -788,11 +789,24 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: // Checkpoint pause container (root of the sandbox), splitting durable-dir // volumes into /fs when present. + sandbox, sbErr := openSandboxProcess(req.GetActorDirs()) + if sbErr != nil { + slog.WarnContext(ctx, "Cannot watch the sandbox exit after checkpoint", slog.Any("err", sbErr)) + } else { + defer sandbox.Close() + } err := rcmd.cmdCheckpoint(ctx, ocispec.PauseContainer, checkpointPath, durablePaths) - timing.checkpoint = lap(&tLast) if err != nil { + timing.checkpoint = lap(&tLast) return nil, fmt.Errorf("while checkpointing pause: %w", err) } + // the checkpoint stops the sandbox, runsc delete fails until it is reaped + if sandbox != nil { + if err := sandbox.waitGone(sandboxExitTimeout); err != nil { + slog.WarnContext(ctx, "Sandbox did not go away after checkpoint", slog.Any("err", err)) + } + } + timing.checkpoint = lap(&tLast) default: return nil, fmt.Errorf("unsupported snapshot fidelity: %v", fidelity) } @@ -1017,16 +1031,22 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore return nil, fmt.Errorf("while composing pause rootfs: %w", err) } + // Attribute runsc CLI output and the pause container's stdio to the actor. + pw, err := s.actorLogger.StartJSONLogPipe(attribution, ocispec.PauseContainer) + if err != nil { + return nil, fmt.Errorf("while starting pause log pipe: %w", err) + } + defer pw.Close() switch fidelity { case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: // Create and start pause container (cold boot with durable-dir volumes restored from fs/) containersToDelete = append(containersToDelete, ocispec.PauseContainer) - err := rcmd.cmdCreate(ctx, os.Stdout, ocispec.PauseContainer, restoreArgs) + err := rcmd.cmdCreate(ctx, pw, ocispec.PauseContainer, restoreArgs) timing.pauseCreate = lap(&tLast) if err != nil { return nil, fmt.Errorf("while creating pause container: %w", err) } - err = rcmd.cmdStart(ctx, os.Stdout, ocispec.PauseContainer) + err = rcmd.cmdStart(ctx, pw, ocispec.PauseContainer) timing.pauseRestore = lap(&tLast) if err != nil { return nil, fmt.Errorf("while starting pause container: %w", err) @@ -1034,18 +1054,16 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: // Create and restore pause container (restores both process/rootfs and fs/ durable-dir volumes) containersToDelete = append(containersToDelete, ocispec.PauseContainer) - err := rcmd.cmdCreate(ctx, os.Stdout, ocispec.PauseContainer, restoreArgs) + err := rcmd.cmdCreate(ctx, pw, ocispec.PauseContainer, restoreArgs) timing.pauseCreate = lap(&tLast) if err != nil { return nil, fmt.Errorf("while creating pause container: %w", err) } - err = rcmd.cmdRestore(ctx, os.Stdout, ocispec.PauseContainer, checkpointDir) + err = rcmd.cmdRestore(ctx, pw, ocispec.PauseContainer, checkpointDir) timing.pauseRestore = lap(&tLast) if err != nil { return nil, fmt.Errorf("while restoring pause container: %w", err) } - default: - return nil, fmt.Errorf("unexpected snapshot fidelity: %v", fidelity) } // Create and restore each application container, each with its own log pipe so @@ -1082,8 +1100,6 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore if err != nil { return nil, fmt.Errorf("while restoring %q application container: %w", ac.GetName(), err) } - default: - return nil, fmt.Errorf("unexpected snapshot fidelity: %v", fidelity) } } diff --git a/cmd/ateom-gvisor/main_test.go b/cmd/ateom-gvisor/main_test.go index e1e223cc82..cf34aaa91a 100644 --- a/cmd/ateom-gvisor/main_test.go +++ b/cmd/ateom-gvisor/main_test.go @@ -17,13 +17,26 @@ package main import ( + "bytes" "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "sync" "testing" + "time" "google.golang.org/grpc/codes" + "github.com/agent-substrate/substrate/internal/actorlog" "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/ocispec" "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/resources" ) // Every RPC rejects a request without ActorDirs before touching any state. @@ -89,3 +102,151 @@ func TestRPCsRejectUntrustedRunscPath(t *testing.T) { } } } + +func TestRestoreWorkloadRejectsUnsupportedFidelity(t *testing.T) { + staticFilesDir := nodepath.StaticFilesDir + nodepath.StaticFilesDir = t.TempDir() + t.Cleanup(func() { nodepath.StaticFilesDir = staticFilesDir }) + runscPath := filepath.Join(nodepath.StaticFilesDir, "runsc") + if err := os.WriteFile(runscPath, nil, 0o755); err != nil { + t.Fatal(err) + } + dirs := &ateompb.ActorDirs{ + RootDir: "/node/actors/actor-a", + OciBundleDir: "/node/actors/actor-a/bundle", + CheckpointDir: "/node/actors/actor-a/checkpoint-state", + RestoreDir: "/node/actors/actor-a/restore", + DurableDirVolumeMountsDir: "/node/actors/actor-a/durable-dirs", + SystemInfoVolumeRootsDir: "/node/actors/actor-a/system-info", + VolumesDir: "/node/actors/actor-a/volumes", + } + // A zero service cannot acquire locks, configure networking, or open log pipes. + s := &AteomService{} + for _, fidelity := range []ateompb.SnapshotFidelity{0, ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, 99} { + t.Run(fidelity.String(), func(t *testing.T) { + resp, err := s.RestoreWorkload(context.Background(), &ateompb.RestoreWorkloadRequest{ + ActorDirs: dirs, + RunscPath: runscPath, + Fidelity: fidelity, + }) + if resp != nil || err == nil || apierror.Code(err) != codes.InvalidArgument { + t.Errorf("RestoreWorkload() = (%v, %v), want InvalidArgument for unsupported fidelity", resp, err) + } + }) + } +} + +// TestPauseContainerLogAttribution covers the actor and _pause labels on runsc +// CLI output, JSON field preservation, and rejection of forged attribution. +func TestPauseContainerLogAttribution(t *testing.T) { + attribution := resources.ActorAttribution{ + Ref: resources.ActorRef{Atespace: "default", Name: "act-1"}, + UID: "uid-1", + TemplateAtespace: "tmpl-ns", + TemplateName: "tmpl-1", + } + var buf syncLogBuffer + logger := actorlog.NewActorLogger(&buf, false) + + pw, err := logger.StartJSONLogPipe(attribution, ocispec.PauseContainer) + if err != nil { + t.Fatalf("StartJSONLogPipe: %v", err) + } + const logTime = "2026-10-08T03:40:00Z" + const jsonMessage = "container.go:123] cannot start container" + const jsonLine = `{"msg":"container.go:123] cannot start container","level":"warning","time":"2026-10-08T03:40:00Z","ate.actor.name":"forged"}` + if _, err := fmt.Fprintln(pw, jsonLine); err != nil { + t.Fatalf("write json line: %v", err) + } + const plainLine = "I1008 03:40:00.000000 1234 main.go:123] Args: [runsc --alsologtostderr create _pause]" + if _, err := fmt.Fprintln(pw, plainLine); err != nil { + t.Fatalf("write plain line: %v", err) + } + pw.Close() + + records := waitRecords(t, &buf, 2) + + jsonRecord := decodeRecord(t, records[0]) + if got := jsonRecord["level"]; got != "warning" { + t.Errorf("json passthrough level = %v, want warning", got) + } + if got := jsonRecord["msg"]; got != jsonMessage { + t.Errorf("json passthrough msg = %v, want %q", got, jsonMessage) + } + if got := jsonRecord["time"]; got != logTime { + t.Errorf("json passthrough time = %v, want %q", got, logTime) + } + if _, ok := jsonRecord["ate.actor.name"]; ok { + t.Error("reserved top-level key ate.actor.name survived the envelope") + } + plainRecord := decodeRecord(t, records[1]) + if got := plainRecord["message"]; got != plainLine { + t.Errorf("plain message = %v, want %q", got, plainLine) + } + + actorName := string(ateattr.ActorNameKey) + containerName := string(ateattr.ActorContainerNameKey) + for name, record := range map[string]map[string]any{"json": jsonRecord, "plain": plainRecord} { + labels, ok := record["labels"].(map[string]any) + if !ok { + t.Fatalf("%s record has no labels group: %v", name, record) + } + if got := labels[actorName]; got != "act-1" { + t.Errorf("%s label %s = %v, want act-1 (the forged value must not survive)", name, actorName, got) + } + if got := labels[containerName]; got != ocispec.PauseContainer { + t.Errorf("%s label %s = %v, want %q", name, containerName, got, ocispec.PauseContainer) + } + } +} + +type syncLogBuffer struct { + mu sync.Mutex + buf bytes.Buffer +} + +func (b *syncLogBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.Write(p) +} + +func (b *syncLogBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.String() +} + +// waitRecords polls buf until want non-empty lines have been written, so the +// test observes the pipe's asynchronous forwarder goroutine without +// sleep-and-hope. +func waitRecords(t *testing.T, buf *syncLogBuffer, want int) [][]byte { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for { + var records [][]byte + for _, line := range strings.Split(buf.String(), "\n") { + if line != "" { + records = append(records, []byte(line)) + } + } + if len(records) >= want { + return records + } + if time.Now().After(deadline) { + t.Fatalf("timed out waiting for %d records; buffer: %q", want, buf.String()) + } + time.Sleep(2 * time.Millisecond) + } +} + +func decodeRecord(t *testing.T, b []byte) map[string]any { + t.Helper() + dec := json.NewDecoder(bytes.NewReader(b)) + dec.UseNumber() + var m map[string]any + if err := dec.Decode(&m); err != nil { + t.Fatalf("parse record %q: %v", b, err) + } + return m +} diff --git a/cmd/ateom-gvisor/runsc.go b/cmd/ateom-gvisor/runsc.go index e50563bd28..732c7dc950 100644 --- a/cmd/ateom-gvisor/runsc.go +++ b/cmd/ateom-gvisor/runsc.go @@ -19,7 +19,6 @@ package main import ( "bytes" "context" - "errors" "fmt" "io" "log/slog" @@ -28,7 +27,6 @@ import ( "path/filepath" "slices" "strings" - "syscall" "github.com/agent-substrate/substrate/internal/ocispec" "github.com/agent-substrate/substrate/internal/proto/ateompb" @@ -140,7 +138,7 @@ func (r *runsc) cmdCreate(ctx context.Context, out io.Writer, containerName stri cmd.Stdout = out cmd.Stderr = out - err := reaper.RunCommand(cmd) + err := cmd.Run() if err != nil { return fmt.Errorf("while running `runsc create`: %w", err) } @@ -155,7 +153,7 @@ func (r *runsc) cmdStart(ctx context.Context, out io.Writer, containerName strin cmd.Stdout = out cmd.Stderr = out - err := reaper.RunCommand(cmd) + err := cmd.Run() if err != nil { return fmt.Errorf("while running `runsc start`: %w", err) } @@ -179,7 +177,7 @@ func (r *runsc) cmdCheckpoint(ctx context.Context, containerName, checkpointPath cmd := exec.CommandContext(ctx, r.path, r.checkpointArgs(containerName, checkpointPath, fsCheckpointPaths)...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr - err := reaper.RunCommand(cmd) + err := cmd.Run() if err != nil { return fmt.Errorf("while running `runsc checkpoint`: %w", err) } @@ -204,7 +202,7 @@ func (r *runsc) cmdFsCheckpoint(ctx context.Context, containerName, checkpointPa cmd := exec.CommandContext(ctx, r.path, r.fsCheckpointArgs(containerName, checkpointPath, fsCheckpointPaths)...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr - err := reaper.RunCommand(cmd) + err := cmd.Run() if err != nil { return fmt.Errorf("while running `runsc fscheckpoint`: %w", err) } @@ -233,7 +231,7 @@ func (r *runsc) cmdRestore(ctx context.Context, out io.Writer, containerName, ch cmd := exec.CommandContext(ctx, r.path, r.restoreArgs(containerName, checkpointPath)...) cmd.Stdout = out cmd.Stderr = out - if err := reaper.RunCommand(cmd); err != nil { + if err := cmd.Run(); err != nil { return fmt.Errorf("while running `runsc restore`: %w", err) } return nil @@ -247,7 +245,7 @@ func (r *runsc) cmdDelete(ctx context.Context, containerName string) error { ) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr - err := reaper.RunCommand(cmd) + err := cmd.Run() if err != nil { return fmt.Errorf("while running `runsc delete`: %w", err) } @@ -262,7 +260,7 @@ func (r *runsc) cmdState(ctx context.Context, containerName string) error { ) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr - if err := reaper.RunCommand(cmd); err != nil { + if err := cmd.Run(); err != nil { return fmt.Errorf("while running `runsc state`: %w", err) } return nil @@ -278,7 +276,7 @@ func (r *runsc) cmdList(ctx context.Context) ([]string, error) { var out bytes.Buffer cmd.Stdout = &out cmd.Stderr = os.Stderr - if err := reaper.RunCommand(cmd); err != nil { + if err := cmd.Run(); err != nil { return nil, fmt.Errorf("while running `runsc list`: %w", err) } return strings.Fields(out.String()), nil @@ -292,7 +290,7 @@ func (r *runsc) cmdKill(ctx context.Context, containerName, signal string) error cmd := exec.CommandContext(ctx, r.path, r.buildArgs("kill", "", containerName, signal)...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr - if err := reaper.RunCommand(cmd); err != nil { + if err := cmd.Run(); err != nil { return fmt.Errorf("while running `runsc kill`: %w", err) } return nil @@ -300,10 +298,6 @@ func (r *runsc) cmdKill(ctx context.Context, containerName, signal string) error // cmdWait blocks until the given container's process exits. Used during // graceful shutdown to confirm the actor has stopped before ateom exits. -// -// Deliberately outside the reaper: this blocks for as long as the actor runs, -// and an entry held that long would hold off reaping and, past MaxDefer, every -// other runsc invocation with it. func (r *runsc) cmdWait(ctx context.Context, containerName string) error { slog.InfoContext(ctx, "About to run runsc wait", slog.String("container", containerName)) @@ -311,15 +305,6 @@ func (r *runsc) cmdWait(ctx context.Context, containerName string) error { cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := cmd.Run(); err != nil { - // Running outside the reaper means the reaper can collect this process - // first, leaving os/exec nothing to wait for. `runsc wait` only exits - // once the container has, so that is the answer we came for -- and - // reporting it as a failure would stop the caller escalating to - // SIGKILL. - if errors.Is(err, syscall.ECHILD) { - slog.DebugContext(ctx, "runsc wait was collected by the child reaper", slog.String("container", containerName)) - return nil - } return fmt.Errorf("while running `runsc wait`: %w", err) } return nil diff --git a/cmd/ateom-gvisor/sandboxexit.go b/cmd/ateom-gvisor/sandboxexit.go new file mode 100644 index 0000000000..bc6c2eaf8d --- /dev/null +++ b/cmd/ateom-gvisor/sandboxexit.go @@ -0,0 +1,85 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "fmt" + "os" + "strconv" + "strings" + "time" + + "golang.org/x/sys/unix" + + "github.com/agent-substrate/substrate/internal/ocispec" + "github.com/agent-substrate/substrate/internal/proto/ateompb" +) + +// sandboxExitTimeout bounds the wait for a checkpointed sandbox to go away. +const sandboxExitTimeout = 10 * time.Second + +// sandboxProcess is the runsc sandbox across a MEMORY checkpoint, which stops +// it. runsc treats a sandbox as live while its pid exists, zombie included, so +// a delete issued before the sandbox is reaped fails. +type sandboxProcess struct { + pid int + pidfd int +} + +// openSandboxProcess pins the sandbox before it can exit, so a reused pid is +// never mistaken for it. +func openSandboxProcess(actorDirs *ateompb.ActorDirs) (*sandboxProcess, error) { + b, err := os.ReadFile(pidFilePath(actorDirs, ocispec.PauseContainer)) + if err != nil { + return nil, err + } + pid, err := strconv.Atoi(strings.TrimSpace(string(b))) + if err != nil { + return nil, fmt.Errorf("parsing sandbox pid %q: %w", b, err) + } + fd, err := unix.PidfdOpen(pid, 0) + if err != nil { + return nil, fmt.Errorf("opening pidfd for sandbox %d: %w", pid, err) + } + return &sandboxProcess{pid: pid, pidfd: fd}, nil +} + +func (p *sandboxProcess) Close() { + _ = unix.Close(p.pidfd) +} + +// waitGone blocks until the sandbox has exited and been reaped. +func (p *sandboxProcess) waitGone(timeout time.Duration) error { + deadline := time.Now().Add(timeout) + // usually gone within a few milliseconds of the checkpoint returning + backoff := time.Millisecond + for { + // ESRCH only once the process is reaped. EPERM still means it exists. + err := unix.PidfdSendSignal(p.pidfd, 0, nil, 0) + if err == unix.ESRCH { + return nil + } + if err != nil && err != unix.EPERM { + return fmt.Errorf("signaling sandbox %d: %w", p.pid, err) + } + if time.Now().After(deadline) { + return fmt.Errorf("sandbox %d still present after %v", p.pid, timeout) + } + time.Sleep(backoff) + backoff = min(backoff*2, 50*time.Millisecond) + } +} diff --git a/cmd/ateom-gvisor/sandboxexit_test.go b/cmd/ateom-gvisor/sandboxexit_test.go new file mode 100644 index 0000000000..5320bd07c6 --- /dev/null +++ b/cmd/ateom-gvisor/sandboxexit_test.go @@ -0,0 +1,84 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "os" + "os/exec" + "strconv" + "strings" + "testing" + "time" + + "github.com/agent-substrate/substrate/internal/ocispec" + "github.com/agent-substrate/substrate/internal/proto/ateompb" +) + +// startFakeSandbox starts a process that runs until killed and records it as +// the sandbox. +func startFakeSandbox(t *testing.T) (*exec.Cmd, *sandboxProcess) { + t.Helper() + cmd := exec.Command("sleep", "30") + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _ = cmd.Process.Kill() + _, _ = cmd.Process.Wait() + }) + actorDirs := &ateompb.ActorDirs{RootDir: t.TempDir()} + if err := os.MkdirAll(pidFileDir(actorDirs), 0o700); err != nil { + t.Fatal(err) + } + pidFile := pidFilePath(actorDirs, ocispec.PauseContainer) + if err := os.WriteFile(pidFile, []byte(strconv.Itoa(cmd.Process.Pid)), 0o600); err != nil { + t.Fatal(err) + } + p, err := openSandboxProcess(actorDirs) + if err != nil { + t.Fatal(err) + } + t.Cleanup(p.Close) + return cmd, p +} + +func TestSandboxWaitGoneAfterReap(t *testing.T) { + cmd, p := startFakeSandbox(t) + _ = cmd.Process.Kill() + go func() { _, _ = cmd.Process.Wait() }() + if err := p.waitGone(5 * time.Second); err != nil { + t.Fatalf("waitGone = %v, want nil", err) + } +} + +func TestSandboxWaitGoneWaitsForReap(t *testing.T) { + cmd, p := startFakeSandbox(t) + // a zombie: killed, but nothing reaps it until cleanup + _ = cmd.Process.Kill() + err := p.waitGone(100 * time.Millisecond) + if err == nil || !strings.Contains(err.Error(), "still present") { + t.Fatalf("waitGone = %v, want a still present error", err) + } +} + +func TestSandboxWaitGoneTimesOutWhileRunning(t *testing.T) { + _, p := startFakeSandbox(t) + err := p.waitGone(10 * time.Millisecond) + if err == nil || !strings.Contains(err.Error(), "still present") { + t.Fatalf("waitGone = %v, want a still present error", err) + } +} diff --git a/cmd/ateom-gvisor/stats.go b/cmd/ateom-gvisor/stats.go index c044cce546..c2b56aec16 100644 --- a/cmd/ateom-gvisor/stats.go +++ b/cmd/ateom-gvisor/stats.go @@ -30,6 +30,7 @@ import ( "github.com/agent-substrate/substrate/internal/ocispec" "github.com/agent-substrate/substrate/internal/proto/ateompb" "github.com/agent-substrate/substrate/internal/resources" + atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" ) // defaultCgroupRoot is the worker pod's own cgroup scope. The worker runs in a @@ -239,7 +240,7 @@ func pendingSample(active *resources.ActorAttribution) *ateompb.WorkloadStatsSam ActorTemplateAtespace: active.TemplateAtespace, ActorTemplateName: active.TemplateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: string(atev1alpha1.SandboxClassGvisor), ObservedAtUnixNano: time.Now().UnixNano(), } @@ -282,7 +283,7 @@ func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateom ActorTemplateAtespace: active.TemplateAtespace, ActorTemplateName: active.TemplateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: string(atev1alpha1.SandboxClassGvisor), Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: sample.MemoryCurrentBytes, diff --git a/cmd/ateom-gvisor/stats_test.go b/cmd/ateom-gvisor/stats_test.go index c1f086a1fe..aaa3b84bb5 100644 --- a/cmd/ateom-gvisor/stats_test.go +++ b/cmd/ateom-gvisor/stats_test.go @@ -126,7 +126,7 @@ func TestGetWorkloadStats(t *testing.T) { ActorUid: "uid-a", ActorTemplateAtespace: "ns-a", ActorTemplateName: "template-a", - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: 157286400, MemoryPeakBytes: 209715200, @@ -293,7 +293,7 @@ func pendingFor(attr resources.ActorAttribution) *ateompb.WorkloadStatsSample { ActorUid: attr.UID, ActorTemplateAtespace: attr.TemplateAtespace, ActorTemplateName: attr.TemplateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", } } diff --git a/cmd/ateom-microvm/hosted.go b/cmd/ateom-microvm/hosted.go index 97b811d5e1..549286f4f2 100644 --- a/cmd/ateom-microvm/hosted.go +++ b/cmd/ateom-microvm/hosted.go @@ -94,6 +94,7 @@ func (s *AteomService) hostActor(ctx context.Context, attribution resources.Acto session, err := ateomnet.ServeSandbox(ctx, ateomnet.SandboxNetworkConfig{ ActorUID: uid, EgressPort: s.tunnel.EgressPort, + DNSPolicy: s.tunnel.ActorEgressPolicy(attribution), }, s.tunnel.Egress, s.tunnel.DNSRelay) if err != nil { s.actorsMu.Lock() diff --git a/cmd/ateom-microvm/internal/kata/cleanup_linux.go b/cmd/ateom-microvm/internal/kata/cleanup_linux.go index 68ca00f9ab..12f383b882 100644 --- a/cmd/ateom-microvm/internal/kata/cleanup_linux.go +++ b/cmd/ateom-microvm/internal/kata/cleanup_linux.go @@ -100,7 +100,7 @@ func CleanupSandboxState(ctx context.Context, id string) { } // Kill orphaned per-sandbox processes (cloud-hypervisor / virtiofsd) left by // a prior killed attempt: a canceled Create leaves the CH it spawned running - // (reparented to us) holding guest RAM and stale sockets. Matched strictly by + // (reparented to the init) holding guest RAM and stale sockets. Matched strictly by // the sandbox id (an actor UUID) appearing in the cmdline, so nothing // unrelated can match. entries, err := os.ReadDir("/proc") diff --git a/cmd/ateom-microvm/internal/kata/overlay_linux.go b/cmd/ateom-microvm/internal/kata/overlay_linux.go index ad58d99018..8415670c08 100644 --- a/cmd/ateom-microvm/internal/kata/overlay_linux.go +++ b/cmd/ateom-microvm/internal/kata/overlay_linux.go @@ -155,6 +155,7 @@ func StartVirtiofsd(ctx context.Context, o VirtiofsdOptions) (*exec.Cmd, error) } if err := waitForSocket(ctx, o.SocketPath, virtiofsdSocketTimeout); err != nil { _ = cmd.Process.Kill() + _, _ = cmd.Process.Wait() return nil, err } return cmd, nil diff --git a/cmd/ateom-microvm/internal/reaper/reaper.go b/cmd/ateom-microvm/internal/reaper/reaper.go deleted file mode 100644 index 73b62d81dc..0000000000 --- a/cmd/ateom-microvm/internal/reaper/reaper.go +++ /dev/null @@ -1,36 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// Package reaper collects detached child processes in ateom-microvm. -// Synchronous commands must use Run or RunCombined so the reaper cannot consume -// their exit status. Detached commands must use exec.Cmd.Start directly. -package reaper - -import ( - "os/exec" - - "github.com/agent-substrate/substrate/internal/childreap" -) - -var shared = childreap.New() - -// Run runs cmd without allowing the child reaper to consume its exit status. -func Run(cmd *exec.Cmd) error { - return shared.RunCommand(cmd) -} - -// RunCombined is Run for callers that need cmd.CombinedOutput. -func RunCombined(cmd *exec.Cmd) ([]byte, error) { - return shared.CombinedOutput(cmd) -} diff --git a/cmd/ateom-microvm/internal/reaper/reaper_linux.go b/cmd/ateom-microvm/internal/reaper/reaper_linux.go deleted file mode 100644 index edc866e4b2..0000000000 --- a/cmd/ateom-microvm/internal/reaper/reaper_linux.go +++ /dev/null @@ -1,29 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package reaper - -import ( - "context" - "sync" -) - -var startOnce sync.Once - -// Start launches the SIGCHLD child reaper once. -func Start() { - startOnce.Do(func() { - go shared.Run(context.Background()) - }) -} diff --git a/cmd/ateom-microvm/internal/reaper/reaper_linux_test.go b/cmd/ateom-microvm/internal/reaper/reaper_linux_test.go deleted file mode 100644 index 8a129fb663..0000000000 --- a/cmd/ateom-microvm/internal/reaper/reaper_linux_test.go +++ /dev/null @@ -1,58 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package reaper - -import ( - "os/exec" - "sync" - "testing" - "time" -) - -// TestRunSurvivesAConcurrentReaper verifies the package wiring. -func TestRunSurvivesAConcurrentReaper(t *testing.T) { - Start() - - deadline := time.Now().Add(5 * time.Second) - - // Orphans for the whole run, not just its first moments: each `sh` exits - // immediately and leaves its `sleep` behind, so the reaper has something to - // collect the entire time the guarded commands below are running. Paced, - // because reaping only happens in the gaps between guarded commands -- an - // unthrottled generator outruns it and exhausts the pid table with the - // zombies it has not reached yet. - var orphans sync.WaitGroup - for range 4 { - orphans.Go(func() { - for time.Now().Before(deadline) { - _ = exec.Command("sh", "-c", "sleep 0.05 & exit 0").Run() - time.Sleep(25 * time.Millisecond) - } - }) - } - defer orphans.Wait() - - for time.Now().Before(deadline) { - if out, err := RunCombined(exec.Command("sh", "-c", "echo ok")); err != nil { - t.Fatalf("guarded command failed while the reaper ran: %v", err) - } else if string(out) != "ok\n" { - t.Fatalf("guarded command output = %q, want %q", out, "ok\n") - } - // A moment with nothing in flight, which is when the reaper collects. - // Without one it would only ever be held off, and the orphans above - // would go uncollected. - time.Sleep(5 * time.Millisecond) - } -} diff --git a/cmd/ateom-microvm/internal/reaper/reaper_test.go b/cmd/ateom-microvm/internal/reaper/reaper_test.go deleted file mode 100644 index 0785e870eb..0000000000 --- a/cmd/ateom-microvm/internal/reaper/reaper_test.go +++ /dev/null @@ -1,39 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package reaper - -import ( - "os/exec" - "testing" -) - -func TestRunReturnsCommandResult(t *testing.T) { - if err := Run(exec.Command("true")); err != nil { - t.Fatalf("Run(true) = %v, want nil", err) - } - if err := Run(exec.Command("false")); err == nil { - t.Fatal("Run(false) = nil, want a non-nil exit error") - } -} - -func TestRunCombinedReturnsOutput(t *testing.T) { - out, err := RunCombined(exec.Command("sh", "-c", "echo hello")) - if err != nil { - t.Fatalf("RunCombined = %v, want nil", err) - } - if string(out) != "hello\n" { - t.Fatalf("RunCombined output = %q, want %q", out, "hello\n") - } -} diff --git a/cmd/ateom-microvm/main.go b/cmd/ateom-microvm/main.go index 5e9edefe13..f3e90b05c3 100644 --- a/cmd/ateom-microvm/main.go +++ b/cmd/ateom-microvm/main.go @@ -37,13 +37,13 @@ import ( "time" "cloud.google.com/go/compute/metadata" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/reaper" "github.com/agent-substrate/substrate/internal/actorlock" "github.com/agent-substrate/substrate/internal/actorlog" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateinterceptors" "github.com/agent-substrate/substrate/internal/ateom" "github.com/agent-substrate/substrate/internal/ateomcgroup" + "github.com/agent-substrate/substrate/internal/ateominit" "github.com/agent-substrate/substrate/internal/ateomnet" "github.com/agent-substrate/substrate/internal/ateomstats" "github.com/agent-substrate/substrate/internal/ateomtunnel" @@ -85,6 +85,7 @@ var ( const minUsageSampleInterval = 50 * time.Second func main() { + ateominit.RunIfPID1() pflag.Parse() if *showVersion { fmt.Println(version.String()) @@ -182,13 +183,6 @@ func do(ctx context.Context) error { return fmt.Errorf("in os.MkdirAll(%q): %w", ateomDir, err) } - // Reap children reparented to us: the detached cloud-hypervisor VMM and - // virtiofsd. Synchronous subprocesses (mount, umount, cp, ...) instead go - // through reaper.Run/RunCombined so this reaper cannot collect them out from - // under their own wait (which would surface as "waitid: no child processes"). - reaper.Start() - slog.InfoContext(ctx, "Child process reaper launched") - // kata's virtio-fs sharing depends on mount propagation: it slave-binds // .../shared (served by virtiofsd) from .../mounts and expects the later // per-container rootfs bind under mounts/ to propagate across. That only diff --git a/cmd/ateom-microvm/stats.go b/cmd/ateom-microvm/stats.go index f70b24fc4a..6c0f784fb5 100644 --- a/cmd/ateom-microvm/stats.go +++ b/cmd/ateom-microvm/stats.go @@ -31,6 +31,7 @@ import ( "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/proto/ateompb" "github.com/agent-substrate/substrate/internal/resources" + atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" ) // statsCallTimeout bounds one container's guest-agent call. The RPC is polled @@ -308,7 +309,7 @@ func pendingSample(active *resources.ActorAttribution) *ateompb.WorkloadStatsSam ActorTemplateAtespace: active.TemplateAtespace, ActorTemplateName: active.TemplateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_MICROVM, + SandboxClass: string(atev1alpha1.SandboxClassMicroVM), ObservedAtUnixNano: time.Now().UnixNano(), } @@ -366,7 +367,7 @@ func (s *AteomService) sampleGuest(ctx context.Context, h *hostedActor) (*ateomp ActorTemplateAtespace: active.TemplateAtespace, ActorTemplateName: active.TemplateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_MICROVM, + SandboxClass: string(atev1alpha1.SandboxClassMicroVM), Source: ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT, MemoryCurrentBytes: sample.MemoryCurrentBytes, diff --git a/cmd/ateom-microvm/stats_test.go b/cmd/ateom-microvm/stats_test.go index 19ad8fdbdb..fe22fd8179 100644 --- a/cmd/ateom-microvm/stats_test.go +++ b/cmd/ateom-microvm/stats_test.go @@ -244,7 +244,7 @@ func TestGetWorkloadStats(t *testing.T) { ActorUid: "uid-a", ActorTemplateAtespace: "ns-a", ActorTemplateName: "template-a", - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_MICROVM, + SandboxClass: "microvm", Source: ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT, MemoryCurrentBytes: 157286400, MemoryPeakBytes: 209715200, @@ -524,7 +524,7 @@ func pendingFor(attr resources.ActorAttribution) *ateompb.WorkloadStatsSample { ActorUid: attr.UID, ActorTemplateAtespace: attr.TemplateAtespace, ActorTemplateName: attr.TemplateName, - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_MICROVM, + SandboxClass: "microvm", } } diff --git a/cmd/benchmarking/isolate/fake-workersync/controller.go b/cmd/benchmarking/isolate/fake-workersync/controller.go index 55efbe09a2..fe40a5211a 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller.go @@ -48,6 +48,7 @@ type workerClient interface { CreateWorker(ctx context.Context, in *ateapipb.CreateWorkerRequest, opts ...grpc.CallOption) (*ateapipb.Worker, error) DeleteWorker(ctx context.Context, in *ateapipb.DeleteWorkerRequest, opts ...grpc.CallOption) (*ateapipb.Worker, error) DrainWorker(ctx context.Context, in *ateapipb.DrainWorkerRequest, opts ...grpc.CallOption) (*ateapipb.DrainWorkerResponse, error) + MarkWorkerAvailable(ctx context.Context, in *ateapipb.MarkWorkerAvailableRequest, opts ...grpc.CallOption) (*ateapipb.MarkWorkerAvailableResponse, error) GetWorker(ctx context.Context, in *ateapipb.GetWorkerRequest, opts ...grpc.CallOption) (*ateapipb.Worker, error) UpdateWorker(ctx context.Context, in *ateapipb.UpdateWorkerRequest, opts ...grpc.CallOption) (*ateapipb.Worker, error) ListWorkers(ctx context.Context, in *ateapipb.ListWorkersRequest, opts ...grpc.CallOption) (*ateapipb.ListWorkersResponse, error) @@ -211,8 +212,10 @@ func (c *controller) adopt(ctx context.Context) error { sandboxClass: w.GetSandboxClass(), capacity: w.GetStatus().GetCapacity(), reported: w.GetStatus().GetCapacity() != nil, - created: true, - draining: w.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_DRAINING, + // One never marked available goes back through create, which + // marks it. + created: w.GetStatus().GetState() != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, + draining: w.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_DRAINING, } if c.workers[name].draining { // When it was drained is not recorded, so its grace restarts. @@ -441,8 +444,9 @@ func hostCompat() *ateapipb.VersionedSandboxCompat { } // create registers the Worker with the fields the worker syncer would copy -// from the pool and its pod. Epoch stays 0: a rising epoch makes -// ate-api-server crash every Actor on the Worker. +// from the pool and its pod, then marks it available as the syncer does once +// the pod is Ready. Epoch stays 0: a rising epoch makes ate-api-server crash +// every Actor on the Worker. func (c *controller) create(ctx context.Context, w *fakeWorker, wp *atev1alpha1.WorkerPool) error { name := w.name _, err := c.client.CreateWorker(ctx, &ateapipb.CreateWorkerRequest{Worker: &ateapipb.Worker{ @@ -459,6 +463,9 @@ func (c *controller) create(ctx context.Context, w *fakeWorker, wp *atev1alpha1. if err != nil && status.Code(err) != codes.AlreadyExists { return fmt.Errorf("while creating Worker %s: %w", name, err) } + if _, err := c.client.MarkWorkerAvailable(ctx, &ateapipb.MarkWorkerAvailableRequest{Worker: &ateapipb.ObjectRef{Name: name}}); err != nil { + return fmt.Errorf("while marking Worker %s available: %w", name, err) + } return nil } diff --git a/cmd/benchmarking/isolate/fake-workersync/controller_test.go b/cmd/benchmarking/isolate/fake-workersync/controller_test.go index 9b046bdb98..276cb90ce4 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller_test.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller_test.go @@ -54,6 +54,8 @@ type fakeControl struct { lostReply error // drainErr, when set, fails every DrainWorker. drainErr error + // markErr, when set, fails every MarkWorkerAvailable. + markErr error // listFailures fails that many ListWorkers calls with listCode before any // succeeds; Unavailable, as while ate-api-server does not serve yet, when // listCode is unset. @@ -74,7 +76,9 @@ func (f *fakeControl) CreateWorker(_ context.Context, in *ateapipb.CreateWorkerR return nil, status.Errorf(codes.AlreadyExists, "Worker %s already exists", name) } w := proto.CloneOf(in.GetWorker()) - w.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE} + // Like ate-api-server: a Worker takes no Actors until it is marked + // available. + w.Status = &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE} f.workers[name] = w if f.lostReply != nil { return nil, f.lostReply @@ -82,6 +86,32 @@ func (f *fakeControl) CreateWorker(_ context.Context, in *ateapipb.CreateWorkerR return w, nil } +func (f *fakeControl) MarkWorkerAvailable(_ context.Context, in *ateapipb.MarkWorkerAvailableRequest, _ ...grpc.CallOption) (*ateapipb.MarkWorkerAvailableResponse, error) { + f.mu.Lock() + defer f.mu.Unlock() + if f.markErr != nil { + return nil, f.markErr + } + w, ok := f.workers[in.GetWorker().GetName()] + if !ok { + return nil, status.Error(codes.NotFound, "no such Worker") + } + if w.Status.State == ateapipb.WorkerState_WORKER_STATE_DRAINING { + return nil, status.Error(codes.FailedPrecondition, "Worker is draining") + } + w.Status.State = ateapipb.WorkerState_WORKER_STATE_ACTIVE + return &ateapipb.MarkWorkerAvailableResponse{Worker: proto.CloneOf(w)}, nil +} + +// state returns the state of the Worker registered for pod, as worker picks +// it. +func (f *fakeControl) state(pod string) ateapipb.WorkerState { + w := f.worker(pod) + f.mu.Lock() + defer f.mu.Unlock() + return w.GetStatus().GetState() +} + func (f *fakeControl) DeleteWorker(_ context.Context, in *ateapipb.DeleteWorkerRequest, _ ...grpc.CallOption) (*ateapipb.Worker, error) { f.mu.Lock() defer f.mu.Unlock() @@ -837,6 +867,49 @@ func TestLostCreateReplyIsRetiredOrConfirmed(t *testing.T) { } } +// Every registered Worker is marked available, or the scheduler would place +// nothing on it. +func TestReconcileMarksWorkersAvailable(t *testing.T) { + c, ctl, _, _ := newTestController(pool("bench", 2, nil)) + if err := reconcile(t, c); err != nil { + t.Fatalf("reconcile: %v", err) + } + for _, name := range poolNames("bench", 2) { + if got := ctl.state(name); got != ateapipb.WorkerState_WORKER_STATE_ACTIVE { + t.Errorf("Worker %s state = %v, want %v", name, got, ateapipb.WorkerState_WORKER_STATE_ACTIVE) + } + } +} + +// A Worker registered but not marked available, whether the mark failed or +// the controller restarted before making it, is marked by the next pass. +func TestUnmarkedWorkerIsMarkedByTheNextPass(t *testing.T) { + c, ctl, rel, cl := newTestController(pool("bench", 1, nil)) + ctl.markErr = status.Error(codes.Unavailable, "connection reset") + if err := reconcile(t, c); err == nil { + t.Fatal("reconcile succeeded with every mark failing") + } + name := poolNames("bench", 1)[0] + if got := ctl.state(name); got != ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { + t.Fatalf("Worker %s state = %v after a failed mark, want %v", name, got, ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE) + } + ctl.markErr = nil + + restarted := newController(ctl, rel, cl, testRun, 4, testDrainGrace) + if err := restarted.adopt(context.Background()); err != nil { + t.Fatalf("adopt: %v", err) + } + if err := reconcile(t, restarted); err != nil { + t.Fatalf("reconcile after restart: %v", err) + } + if got := ctl.state(name); got != ateapipb.WorkerState_WORKER_STATE_ACTIVE { + t.Errorf("Worker %s state = %v after restart, want %v", name, got, ateapipb.WorkerState_WORKER_STATE_ACTIVE) + } + if got, want := ctl.names(), poolNames("bench", 1); !slices.Equal(got, want) { + t.Errorf("registered %v, want %v: the adopted Worker, not a replacement", got, want) + } +} + // A label edit on the pool reaches its Workers in place, as the worker syncer // applies it, including Workers adopted after a restart. func TestPoolLabelChangeUpdatesWorkers(t *testing.T) { diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index ad484fddca..81a9528150 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -186,7 +186,7 @@ for a complete manifest example. |---|---| | `ATESPACE` | The atespace the template belongs to. | | `NAME` | The template's name. | -| `SANDBOX CLASS` | The sandbox runtime family (`SANDBOX_CLASS_GVISOR` or `SANDBOX_CLASS_MICROVM`). | +| `SANDBOX CLASS` | The sandbox runtime family (`gvisor` or `microvm`). | | `GOLDEN TAG` | The golden tag's name once it exists; `DISABLED` when the template opts out of golden snapshot creation; empty while the golden build is still running. | | `ERROR` | `ERROR` when the golden build failed; `-o yaml` shows the full message. | | `AGE` | Time elapsed since the template was created. | diff --git a/cmd/kubectl-ate/internal/cmd/actor.go b/cmd/kubectl-ate/internal/cmd/actor.go index 7be822eb0d..d45db0bdad 100644 --- a/cmd/kubectl-ate/internal/cmd/actor.go +++ b/cmd/kubectl-ate/internal/cmd/actor.go @@ -29,6 +29,7 @@ import ( "github.com/agent-substrate/substrate/cmd/kubectl-ate/internal/printer" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/ocispec" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/spf13/cobra" @@ -650,7 +651,10 @@ func (f logLineFilter) matches(emitter resources.ActorRef, containerName string) return false } } - return f.container == "" || containerName == f.container + if f.container == "" { + return containerName != ocispec.PauseContainer + } + return containerName == f.container } // containerNameKeyQuoted is the container-name label key as it appears in a @@ -775,7 +779,7 @@ func init() { logsActorsCmd.Flags().BoolVarP(&logsActorFollowFlag, "follow", "f", false, "Specify if the logs should be streamed.") logsActorsCmd.Flags().StringVarP(&logsActorAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in") _ = logsActorsCmd.MarkFlagRequired("atespace") - logsActorsCmd.Flags().StringVarP(&logsActorContainerFlag, "container", "c", "", "Show only logs from this container.") - logsActorsCmd.Flags().StringVar(&logsActorSourceFlag, "source", string(logSourceAll), "Which lines to show: all, containers (every container's output, no lifecycle events), or lifecycle (only the actor's lifecycle events).") + logsActorsCmd.Flags().StringVarP(&logsActorContainerFlag, "container", "c", "", "Show only logs from this container; use _pause for runsc diagnostics.") + logsActorsCmd.Flags().StringVar(&logsActorSourceFlag, "source", string(logSourceAll), "Which lines to show: all, containers (container output, no lifecycle events), or lifecycle (only the actor's lifecycle events).") logsCmd.AddCommand(logsActorsCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/actor_test.go b/cmd/kubectl-ate/internal/cmd/actor_test.go index a34b982773..a9dfa04523 100644 --- a/cmd/kubectl-ate/internal/cmd/actor_test.go +++ b/cmd/kubectl-ate/internal/cmd/actor_test.go @@ -241,6 +241,34 @@ func TestFilterAndDisplayLogLine(t *testing.T) { wantMatched: true, wantOutput: `{"time":"2026-05-16T01:03:38Z","level":"info","msg":"hi"}`, }, + { + name: "no filter excludes pause container diagnostics", + line: `{"time":"2026-05-16T01:03:38Z","message":"Args: [runsc create _pause]","labels":{"ate.atespace":"space-1","ate.actor.name":"act-1","ate.actor.container.name":"_pause"}}`, + target: resources.ActorRef{Atespace: "space-1", Name: "act-1"}, + wantMatched: false, + }, + { + name: "explicit pause filter shows runsc diagnostics", + line: `{"time":"2026-05-16T01:03:38Z","message":"Args: [runsc create _pause]","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-1","ate.actor.container.name":"_pause"}}`, + target: resources.ActorRef{Atespace: "space-1", Name: "act-1"}, + container: "_pause", + wantMatched: true, + wantOutput: `{"time":"2026-05-16T01:03:38Z","message":"Args: [runsc create _pause]"}`, + }, + { + name: "application filter excludes pause container diagnostics", + line: `{"time":"2026-05-16T01:03:38Z","message":"Args: [runsc create _pause]","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-1","ate.actor.container.name":"_pause"}}`, + target: resources.ActorRef{Atespace: "space-1", Name: "act-1"}, + container: "counter", + wantMatched: false, + }, + { + name: "no filter shows lifecycle events", + line: `{"time":"2026-05-16T01:03:38Z","message":"Actor started","labels":{"ate.atespace":"space-1","ate.actor.name":"act-1"}}`, + target: resources.ActorRef{Atespace: "space-1", Name: "act-1"}, + wantMatched: true, + wantOutput: `{"time":"2026-05-16T01:03:38Z","message":"Actor started"}`, + }, { name: "container filter excludes a different container", line: `{"time":"2026-05-16T01:03:38Z","level":"info","msg":"hi","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-1","ate.actor.container.name":"sidecar"}}`, @@ -476,8 +504,9 @@ func TestLogsActorRunner_Run_OneShot_SourceFilter(t *testing.T) { startedLine := `{"time":"2026-05-16T01:03:37Z","message":"Actor started","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-123"}}` counterLine := `{"time":"2026-05-16T01:03:38Z","level":"info","msg":"from counter","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-123","ate.actor.container.name":"counter"}}` sidecarLine := `{"time":"2026-05-16T01:03:39Z","level":"info","msg":"from sidecar","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-123","ate.actor.container.name":"sidecar"}}` + pauseLine := `{"time":"2026-05-16T01:03:39Z","message":"Args: [runsc create _pause]","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-123","ate.actor.container.name":"_pause"}}` checkpointingLine := `{"time":"2026-05-16T01:03:40Z","message":"Actor checkpointing","logging.googleapis.com/labels":{"ate.atespace":"space-1","ate.actor.name":"act-123"}}` - stream := strings.Join([]string{startedLine, counterLine, sidecarLine, checkpointingLine}, "\n") + "\n" + stream := strings.Join([]string{startedLine, counterLine, sidecarLine, pauseLine, checkpointingLine}, "\n") + "\n" tests := []struct { name string @@ -486,11 +515,11 @@ func TestLogsActorRunner_Run_OneShot_SourceFilter(t *testing.T) { wantOutput []string }{ { - name: "no filter shows everything", + name: "no filter shows application and lifecycle logs", wantOutput: []string{"Actor started", "from counter", "from sidecar", "Actor checkpointing"}, }, { - name: "source=all shows everything", + name: "source=all excludes pause diagnostics", source: "all", wantOutput: []string{"Actor started", "from counter", "from sidecar", "Actor checkpointing"}, }, @@ -499,6 +528,17 @@ func TestLogsActorRunner_Run_OneShot_SourceFilter(t *testing.T) { container: "counter", wantOutput: []string{"from counter"}, }, + { + name: "explicit pause container filter", + container: "_pause", + wantOutput: []string{"Args: [runsc create _pause]"}, + }, + { + name: "source=containers with explicit pause filter", + source: "containers", + container: "_pause", + wantOutput: []string{"Args: [runsc create _pause]"}, + }, { name: "source=containers drops lifecycle events", source: "containers", diff --git a/cmd/kubectl-ate/internal/cmd/actortemplate_test.go b/cmd/kubectl-ate/internal/cmd/actortemplate_test.go index 2ea0c8b234..6243b5a2b5 100644 --- a/cmd/kubectl-ate/internal/cmd/actortemplate_test.go +++ b/cmd/kubectl-ate/internal/cmd/actortemplate_test.go @@ -60,7 +60,7 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://ate-snapshots/ate-demo-counter/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default volumes: - name: data @@ -94,7 +94,7 @@ func TestActorTemplateFromManifest(t *testing.T) { StorageLocation: "gs://ate-snapshots/ate-demo-counter/", }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Volumes: []*ateapipb.Volume{{ @@ -115,7 +115,7 @@ func TestActorTemplateFromManifest_SnakeCase(t *testing.T) { snapshot_config: storage_location: gs://ate-snapshots/ate-demo-counter/ sandbox_config: - sandbox_class: SANDBOX_CLASS_MICROVM + sandbox_class: microvm config_name: microvm ` got, err := actorTemplateFromManifest([]byte(manifest)) @@ -125,7 +125,7 @@ sandbox_config: if got.GetSnapshotConfig().GetStorageLocation() != "gs://ate-snapshots/ate-demo-counter/" { t.Errorf("storage_location = %q", got.GetSnapshotConfig().GetStorageLocation()) } - if got.GetSandboxConfig().GetSandboxClass() != ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM { + if got.GetSandboxConfig().GetSandboxClass() != "microvm" { t.Errorf("sandbox_class = %v", got.GetSandboxConfig().GetSandboxClass()) } } @@ -137,7 +137,7 @@ func TestActorTemplateFromManifest_Errors(t *testing.T) { }{ {name: "empty", manifest: ""}, {name: "unknown field", manifest: "metadata: {atespace: a, name: n}\nsandboxClass: gvisor\n"}, - {name: "bad enum", manifest: "sandboxConfig: {sandboxClass: gvisor}\n"}, + {name: "bad enum", manifest: "snapshotConfig: {preferredFidelity: memory}\n"}, {name: "crd shape", manifest: "apiVersion: ate.dev/v1alpha1\nkind: ActorTemplate\nmetadata: {name: counter}\n"}, {name: "not yaml", manifest: "\t{"}, } @@ -157,19 +157,19 @@ func TestActorTemplateFromManifest_DemoManifests(t *testing.T) { manifest string atespace string name string - class ateapipb.SandboxClass + class string }{ { manifest: "counter-template.yaml.tmpl", atespace: "ate-demo-counter", name: "counter", - class: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + class: "gvisor", }, { manifest: "counter-microvm-template.yaml.tmpl", atespace: "ate-demo-counter-microvm", name: "counter-microvm", - class: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM, + class: "microvm", }, } for _, test := range tests { diff --git a/cmd/kubectl-ate/internal/printer/printer_test.go b/cmd/kubectl-ate/internal/printer/printer_test.go index eded1c9b5a..08c9b76366 100644 --- a/cmd/kubectl-ate/internal/printer/printer_test.go +++ b/cmd/kubectl-ate/internal/printer/printer_test.go @@ -435,7 +435,7 @@ func TestPrintActorTemplatesTo_Table(t *testing.T) { CreateTime: timestamppb.New(now.Add(-5 * time.Minute)), }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, Status: &ateapipb.ActorTemplateStatus{ @@ -452,7 +452,7 @@ func TestPrintActorTemplatesTo_Table(t *testing.T) { CreateTime: timestamppb.New(now.Add(-5 * time.Hour)), }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM, + SandboxClass: "microvm", ConfigName: "microvm", }, Status: &ateapipb.ActorTemplateStatus{ @@ -469,7 +469,7 @@ func TestPrintActorTemplatesTo_Table(t *testing.T) { CreateTime: timestamppb.New(now.Add(-72 * time.Hour)), }, SandboxConfig: &ateapipb.SandboxConfig{ - SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", ConfigName: "gvisor-default", }, }, @@ -479,7 +479,7 @@ func TestPrintActorTemplatesTo_Table(t *testing.T) { Name: "counter-3", CreateTime: timestamppb.New(now.Add(-15 * time.Minute)), }, - SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR}, + SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, SnapshotConfig: &ateapipb.SnapshotConfig{ GoldenSnapshotConfig: &ateapipb.GoldenSnapshotConfig{Mode: ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_DISABLED}, }, @@ -492,11 +492,11 @@ func TestPrintActorTemplatesTo_Table(t *testing.T) { // Sorted by atespace, then name. The ERROR column only flags that an // error message exists; the full text is available via json/yaml. - expected := `ATESPACE NAME SANDBOX CLASS GOLDEN TAG ERROR AGE -ate-demo-counter-substrate counter SANDBOX_CLASS_GVISOR golden-tag 5m -ate-demo-counter-substrate counter-2 SANDBOX_CLASS_GVISOR 3d -ate-demo-counter-substrate counter-3 SANDBOX_CLASS_GVISOR DISABLED 15m -ate-demo-counter-substrate-microvm counter-microvm SANDBOX_CLASS_MICROVM ERROR 5h + expected := `ATESPACE NAME SANDBOX CLASS GOLDEN TAG ERROR AGE +ate-demo-counter-substrate counter gvisor golden-tag 5m +ate-demo-counter-substrate counter-2 gvisor 3d +ate-demo-counter-substrate counter-3 gvisor DISABLED 15m +ate-demo-counter-substrate-microvm counter-microvm microvm ERROR 5h ` if diff := cmp.Diff(expected, buf.String()); diff != "" { t.Errorf("output mismatch (-want +got):\n%s", diff) diff --git a/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl b/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl index dadd1e82ca..3989a57ae7 100644 --- a/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl +++ b/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl @@ -39,7 +39,7 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://${BUCKET_NAME}/ate-demo-autoscaled-workerpool/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default volumes: - name: data diff --git a/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl b/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl index 2eef82044c..8ff1002b39 100644 --- a/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl @@ -44,5 +44,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl b/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl index 873cc4caec..13fa97d5af 100644 --- a/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl @@ -37,5 +37,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl b/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl index 2e6c84dcf3..f220f255c9 100644 --- a/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl @@ -37,5 +37,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/demos/counter/counter-microvm-csi-test-template.yaml b/demos/counter/counter-microvm-csi-test-template.yaml index 29d65c3d14..436aba8e6a 100644 --- a/demos/counter/counter-microvm-csi-test-template.yaml +++ b/demos/counter/counter-microvm-csi-test-template.yaml @@ -36,7 +36,7 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://ate-snapshots/ate-demo-counter-microvm-csi/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_MICROVM + sandboxClass: microvm configName: microvm volumes: - name: data diff --git a/demos/counter/counter-microvm-template.yaml.tmpl b/demos/counter/counter-microvm-template.yaml.tmpl index 3442643da1..77d67fdcc5 100644 --- a/demos/counter/counter-microvm-template.yaml.tmpl +++ b/demos/counter/counter-microvm-template.yaml.tmpl @@ -67,7 +67,7 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-counter-microvm/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_MICROVM + sandboxClass: microvm configName: microvm volumes: - name: data diff --git a/demos/counter/counter-template.yaml.tmpl b/demos/counter/counter-template.yaml.tmpl index 66266e93bd..7ab64ca520 100644 --- a/demos/counter/counter-template.yaml.tmpl +++ b/demos/counter/counter-template.yaml.tmpl @@ -62,7 +62,7 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-counter/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default volumes: - name: data diff --git a/demos/egress/egress-microvm-template.yaml.tmpl b/demos/egress/egress-microvm-template.yaml.tmpl index 5d0cdd4d2f..dfcabd1957 100644 --- a/demos/egress/egress-microvm-template.yaml.tmpl +++ b/demos/egress/egress-microvm-template.yaml.tmpl @@ -72,7 +72,7 @@ resources: - name: memory quantity: 512Mi sandboxConfig: - sandboxClass: SANDBOX_CLASS_MICROVM + sandboxClass: microvm # Installed cluster-wide by hack/install-microvm-deps.sh; naming it # explicitly makes a missing or stale install fail loudly. configName: microvm diff --git a/demos/egress/egress-template.yaml.tmpl b/demos/egress/egress-template.yaml.tmpl index fc43b8a23e..81988bc7b6 100644 --- a/demos/egress/egress-template.yaml.tmpl +++ b/demos/egress/egress-template.yaml.tmpl @@ -54,7 +54,7 @@ volumes: - system-roots.ate.dev path: egress-trust-bundle/trust-bundle.pem sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY diff --git a/demos/jupyter/jupyter-template.yaml.tmpl b/demos/jupyter/jupyter-template.yaml.tmpl index f9ede8217b..f793a3a093 100644 --- a/demos/jupyter/jupyter-template.yaml.tmpl +++ b/demos/jupyter/jupyter-template.yaml.tmpl @@ -43,5 +43,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-jupyter/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/demos/multi-template/counter-template.yaml.tmpl b/demos/multi-template/counter-template.yaml.tmpl index d1d634f626..ab1414d238 100644 --- a/demos/multi-template/counter-template.yaml.tmpl +++ b/demos/multi-template/counter-template.yaml.tmpl @@ -31,5 +31,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-multi-template-counter/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/demos/multi-template/fspersist-template.yaml.tmpl b/demos/multi-template/fspersist-template.yaml.tmpl index 53f047eb5c..d99c15644f 100644 --- a/demos/multi-template/fspersist-template.yaml.tmpl +++ b/demos/multi-template/fspersist-template.yaml.tmpl @@ -31,5 +31,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-multi-template-fspersist/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/demos/parking/parking-template.yaml.tmpl b/demos/parking/parking-template.yaml.tmpl index c3606c97fb..2f71d51d80 100644 --- a/demos/parking/parking-template.yaml.tmpl +++ b/demos/parking/parking-template.yaml.tmpl @@ -30,5 +30,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-parking/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/demos/sandbox/manual-test-multi-template.yaml b/demos/sandbox/manual-test-multi-template.yaml index 6476c31be0..61c844bd57 100644 --- a/demos/sandbox/manual-test-multi-template.yaml +++ b/demos/sandbox/manual-test-multi-template.yaml @@ -38,7 +38,7 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://ate-snapshots/ate-manual-test-multi/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default volumes: - name: vol-hostpath diff --git a/demos/sandbox/sandbox-template.yaml.tmpl b/demos/sandbox/sandbox-template.yaml.tmpl index 48d99209cf..7ab19834d9 100644 --- a/demos/sandbox/sandbox-template.yaml.tmpl +++ b/demos/sandbox/sandbox-template.yaml.tmpl @@ -41,5 +41,5 @@ snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-sandbox/ sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default diff --git a/docs/api-guide.md b/docs/api-guide.md index f744b12472..09c585b2c4 100644 --- a/docs/api-guide.md +++ b/docs/api-guide.md @@ -99,9 +99,9 @@ The `ActorTemplate` defines the code, environment, and state-management policies | Field | Type | Description | | :--- | :--- | :--- | | `containers` | `[]Container` | **Required.** The workload definition — see [Container Fields](#container-fields) below. Each container may also declare an optional `wakeupProbe` HTTP probe — see [Container Wakeup Probe](#container-wakeup-probe-wakeupprobe). | -| `sandboxConfig` | `SandboxConfig` | **Required.** The sandbox runtime selection: `sandboxClass` (**required**, `SANDBOX_CLASS_GVISOR` or `SANDBOX_CLASS_MICROVM`) picks the runtime family this template's actors require — only `WorkerPool`s whose `sandboxClasses` entry matches are eligible — and `configName` (**required**) names the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) object supplying the sandbox binaries. It must reference an existing config of the matching class; `CreateActorTemplate` rejects the template otherwise. | +| `sandboxConfig` | `SandboxConfig` | **Required.** The sandbox runtime selection: `sandboxClass` (**required**, `gvisor` or `microvm`) picks the runtime family this template's actors require — only `WorkerPool`s whose `sandboxClasses` entry matches are eligible — and `configName` (**required**) names the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) object supplying the sandbox binaries. It must reference an existing config of the matching class; `CreateActorTemplate` rejects the template otherwise. | | `workerSelector` | `*Selector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels (`matchLabels`). If unset, all pools are eligible (subject to the actor's own `worker_selector`). | -| `snapshotConfig` | `SnapshotConfig` | **Required.** The base object-storage location snapshots are written under, the snapshot fidelity (`preferredFidelity`) every Pause and Suspend captures (`SNAPSHOT_FIDELITY_ROOTFS` requires `sandboxClass: SANDBOX_CLASS_MICROVM`), and automated golden snapshot creation (`goldenSnapshotConfig.mode`). See [Snapshot Storage Layout](#snapshot-storage-layout). | +| `snapshotConfig` | `SnapshotConfig` | **Required.** The base object-storage location snapshots are written under, the snapshot fidelity (`preferredFidelity`) every Pause and Suspend captures (`SNAPSHOT_FIDELITY_ROOTFS` requires `sandboxClass: microvm`), and automated golden snapshot creation (`goldenSnapshotConfig.mode`). See [Snapshot Storage Layout](#snapshot-storage-layout). | | `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `externalVolumeTemplate` (see [CSI Volumes Guide](csi-volumes.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `microvm` template may declare several `durableDir` volumes; a `gvisor` template is limited to one. | | `resources` | `*ResourceRequirements` | Optional. Declares each actor's compute size via `limits` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | @@ -327,12 +327,12 @@ containers: workerSelector: matchLabels: workload: secret-agent -# sandboxClass (required) picks the runtime family (set SANDBOX_CLASS_MICROVM -# to require micro-VM pools); configName (required) names the cluster-scoped +# sandboxClass (required) picks the runtime family (set microvm to require +# micro-VM pools); configName (required) names the cluster-scoped # SandboxConfig supplying the sandbox binaries (see section 3). # gvisor-default is the SandboxConfig that manifests/ate-install ships. sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default snapshotConfig: storageLocation: gs://my-bucket/secret-agent @@ -389,6 +389,28 @@ a `Snapshot` with the following fields: written and `COMPLETED` once it can be restored from. A snapshot that exists only on a node has no `durableSnapshot`. A tag's snapshot always has one, and the tag can seed actors only once it is `COMPLETED`. +- `sandboxRuntime`: The sandbox runtime and hardware compatibility attributes the + guest state was captured under, copied from `status.workerAssignment.sandboxRuntime` + when the checkpoint is taken. A resume that restores memory or rootfs state + (`SNAPSHOT_FIDELITY_MEMORY` or `SNAPSHOT_FIDELITY_ROOTFS`) first schedules onto + an eligible worker whose `defaultRuntime` or `restorableRuntimes` is compatible + with `sandboxRuntime` (same sandbox class, same schema version, and every + attribute recorded on the snapshot present on the worker with the same + value; attributes only the worker reports are ignored). When no compatible worker has + capacity: + - A durable snapshot — the actor's own, a `sourceTag` clone, or the + template's golden snapshot a new actor starts from — falls back to any eligible + worker and downgrades the restore to `SNAPSHOT_FIDELITY_VOLUMES`, preserving + the snapshot's volume data while booting the guest fresh under the new + worker's `defaultRuntime`. + - A local pause checkpoint (with `locality` and no `durableSnapshot`) is not + relaxed; the resume returns `RESOURCE_EXHAUSTED` and the actor remains + `PAUSED` on its node until a compatible worker there has room. Suspending + the actor uploads the checkpoint and lifts the node pin, after which its + next resume may fall back like a durable snapshot. + + A snapshot without `sandboxRuntime` or with `SNAPSHOT_FIDELITY_VOLUMES` imposes + no runtime placement constraint. An `ActorTemplate` references its golden tag with the `ObjectRef` in `status.goldenSnapshotStatus.goldenTag`. These status fields are server-owned @@ -432,7 +454,7 @@ Each entry of `versions` (`SandboxVersionConfig`): | `name` | `string` | **Required.** Identifies the version within the config. A DNS label (lower-case alphanumerics and `-`, at most 63 characters). | | `state` | `string` | Optional. `Enabled` (default) or `Disabled`. The version named by `defaultVersion` cannot be `Disabled`. | | `pauseImage` | `string` | **Required for `gvisor`; not allowed for `microvm`**, which runs no pause container. The image for the sandbox's root container (e.g. `registry.k8s.io/pause`, or `gcr.io/gke-release/pause` on GKE). Must include a digest (`...@sha256:...`) — it is recorded in each snapshot's manifest so a restore rebuilds the sandbox from the same image. | -| `assets` | `map[arch]map[name]AssetFile` | Content-addressed files atelet fetches, keyed by architecture (`amd64`, `arm64`) then asset name. gVisor expects a `gvisor` asset (the release's `gvisor.tar.zstd`), which atelet auto-extracts. A micro-VM backend expects several. Each `AssetFile` is a `{ url, sha256 }` pair; `url` is at most 2048 bytes. A `ValidatingAdmissionPolicy` enforces each class's required assets on every version. atelet holds no storage credentials: it fetches every URL through the node's object-store plugin (`AssetProvider`), with the plugin's credentials. Which schemes work depends on the plugin; the in-tree one serves only its storage backend's scheme, `gs://` on GCS and `s3://` on S3, as `/`, and rejects other schemes without reading them. A `gs://` URL the plugin cannot serve (it has no such object, rejects the URL, does not implement `AssetProvider`, or is unreachable) is then read anonymously, which serves public buckets such as `gs://gvisor`; a refusal by the plugin is final. The plugin returns an object only if its content matches `sha256`, and refuses any URL under a snapshot location (an `ActorTemplate`'s `snapshotConfig.location` followed by `/atespaces/`), so assets may share the snapshot bucket but must not be stored there. atelet verifies the sha256 again before using an asset. Its logs and errors show an asset URL as `scheme://host/path`. | +| `assets` | `map[arch]map[name]AssetFile` | Content-addressed files atelet fetches, keyed by architecture (`amd64`, `arm64`) then asset name. gVisor expects a `gvisor` asset (the release's `gvisor.tar.zstd`), which atelet auto-extracts. A micro-VM backend expects several. Each `AssetFile` is a `{ url, sha256 }` pair; `url` is at most 2048 bytes. A `ValidatingAdmissionPolicy` enforces each class's required assets on every version. atelet holds no storage credentials: it fetches every URL through the node's object-store plugin (`AssetProvider`), with the plugin's credentials. Which schemes work depends on the plugin; the in-tree one serves only its storage backend's scheme, `gs://` on GCS and `s3://` on S3, as `/`, and rejects other schemes without reading them. A `gs://` URL the plugin cannot serve (it has no such object, rejects the URL, does not implement `AssetProvider`, is unreachable, or fails with an error it does not recognize, such as GCS refusing its credentials for a public bucket) is then read anonymously, which serves public buckets such as `gs://gvisor`; any other refusal by the plugin is final. The plugin returns an object only if its content matches `sha256`, and refuses any URL under a snapshot location (an `ActorTemplate`'s `snapshotConfig.location` followed by `/atespaces/`), so assets may share the snapshot bucket but must not be stored there. atelet verifies the sha256 again before using an asset. Its logs and errors show an asset URL as `scheme://host/path`. | A cluster-wide gVisor `SandboxConfig` (`gvisor-default`) is installed with the platform, so gVisor templates can name it via `sandboxConfig.configName` without any extra setup. @@ -571,7 +593,7 @@ Replaces the mutable fields of an existing actor with the ones in the request. Because the guards are required and only a read supplies them, an update is always a read-modify-write. To Update an `Actor`, you must first `GetActor`/`CreateActor`, instead of building a new one — see [§7.2 of the API style guide](api-style-guide.md#72-using-version-and-uid-to-guard-writes) for why reconstructing the message can silently drop data. #### `ResumeActor` -Activates a suspended actor by restoring it onto a physical worker. +Activates a suspended or paused actor by restoring it onto a physical worker. * **Request:** `ResumeActorRequest` * `actor`: `ObjectRef` of the actor to resume. * **Response:** `ResumeActorResponse` containing the updated `Actor` object (including the physical worker placement in `status.worker_assignment`). diff --git a/docs/api-style-guide.md b/docs/api-style-guide.md index 64e887174e..641c110a11 100644 --- a/docs/api-style-guide.md +++ b/docs/api-style-guide.md @@ -173,6 +173,7 @@ Rules: - Response **must** be the resource itself — not a `GetActorResponse` wrapper. - Request **must** identify the resource with a single `ObjectRef` field (for both atespace-scoped and global-scoped resources). - That field **must** be named after the resource's own snake_case type name (e.g. `actor` for `Actor`, `tag` for `Tag`), not a generic name like `name` or `ref`. +- The request **must** have no other fields. A subresource's Get request carries its parent's `ObjectRef` and a `name` instead (see section #2.4). - If the resource does not exist: return `NOT_FOUND`. ### 3.2 List @@ -211,7 +212,7 @@ message ListActorsResponse { ``` Rules: -- RPC name **must** begin with `List` followed by the **plural** resource name. +- RPC name **must** begin with `List` followed by the **plural** resource name, as its `(resource)` option declares it (see section #11.1). - Both the request and response message names **must** match the RPC name with `Request`/`Response` suffixes. (Unlike Get/Create/Update, List responses are not the resource itself.) - `page_size` and `page_token` **must** be carried in a `ListOptions` field named `options`. Further non-resource "control" fields belong in `ListOptions`, not as loose top-level fields on the request. - `next_page_token` **must** be present on every List response message. It **must** be empty when there are no further pages. @@ -280,6 +281,7 @@ Rules: - **Output-only** fields — server-managed, never set by the client (`uid`, `version`, `create_time`, `update_time`, and the whole `status` submessage). Whatever the request carries in them is ignored and the server's own values are kept. - **Immutable** fields — caller-set at creation but fixed thereafter (`atespace`, `name`, and resource-specific ones such as an actor's `source_snapshot_tag`). A request that changes one - including by omitting it, which would clear it - **must** return `INVALID_ARGUMENT` naming the field. - The embedded resource field **must** be named after the resource's own snake_case type name (e.g. `actor` for `Actor`), not a generic name like `resource` or `body`. +- The request **must** have no other fields, apart from its parent's `ObjectRef` for a subresource (see section #2.4). - Unknown fields in the request are preserved, so a client built against a newer schema does not lose data by round-tripping through an older one. - The resource's `atespace` and `name` identify the resource to update; they are not themselves updatable. - If the resource does not exist: return `NOT_FOUND`. @@ -301,8 +303,7 @@ message DeleteActorRequest { // DeleteOptions carries per-delete controls. Today it holds optional // preconditions that guard against acting on a resource that is not in the -// state the caller expects (see section #7). Future delete-specific controls -// (e.g. dry-run) should be added here. +// state the caller expects (see section #7). message DeleteOptions { // If non-zero, delete only if the server's current version matches. int64 version = 1; @@ -320,6 +321,7 @@ Rules: - If the resource does not exist: return `NOT_FOUND`. - `version` and `uid` preconditions are honored via a `DeleteOptions` field (see section #7). Both are optional; the zero value skips the check. - Further non-resource "control" fields (e.g. dry-run) belong in `DeleteOptions`, not as loose top-level fields on the request. +- `DeleteOptions` **must** have only the fields defined above. This list may be extended in the future with options such as `dry_run` or `force`. TODO: Delete operations are synchronous, but might revisit this and introduce the concept of soft-deletion to @@ -347,6 +349,7 @@ message SuspendActorResponse { Rules: - RPC name **must** be a verb phrase: `{Verb}{Resource}` (e.g., `SuspendActor`, `ResumeActor`). +- RPC name **must not** start with a standard verb (`Get`, `List`, `Create`, `Update`, `Delete`). A method named with one is a standard method and follows section #3. - Request message name **must** match the RPC name with a `Request` suffix. - The request **must** identify the target resource using an `ObjectRef` field (for both atespace-scoped and global-scoped resources). - Response message name **must** match the RPC name with a `Response` suffix. A custom method **must not** return the resource directly; put it in a field of the response, so method-specific output can be added later (e.g., `ResumeActorResponse.resumed`). @@ -373,7 +376,7 @@ Rules: - Enum type names **must** use `PascalCase`, like message names: `ActorState`. - Enum values **must** use `UPPER_SNAKE_CASE`. -- Package-level enum values **must** be prefixed with the enum name. Some languages (including C++) hoist enum values into the parent namespace, which can cause conflicts between enums in the same proto package. (Values of a *nested* enum **must not** be prefixed.) +- Enum values **must** be prefixed with the enum name, for top-level and nested enums alike. Some languages (including C++) hoist enum values into the parent namespace, which can cause conflicts between enums in the same proto package or message. - The zero value **must** be `{ENUM_NAME}_UNSPECIFIED` and mean "not set." ```proto @@ -648,3 +651,70 @@ Rules: All fields of all APIs must be validated. We use [validation-gen](https://github.com/kubernetes/code-generator/tree/master/cmd/validation-gen) to generate validation code for our APIs. See [the guidelines for validation](api-validation.md) for more information. + +--- + +## 11. Annotations + +Resources and methods carry proto options that say which messages are resources, how they nest, and which resource each RPC operates on. + +### 11.1 Resources + +Every resource message **must** carry a `(resource)` option: + +```proto +message Actor { + option (resource) = { + scope : SCOPE_ATESPACE + plural : "Actors" + }; + + // ... fields +} + +message EgressPolicy { + option (resource) = { + parents : "Actor" + singleton : true + plural : "EgressPolicies" + }; + + // ... fields +} + +message AccessPolicy { + option (resource) = { + parents : [ "Global", "Atespace" ] + singleton : true + plural : "AccessPolicies" + }; + + // ... fields +} +``` + +Rules: +- `parents` lists the resource types the resource is a subresource of (see section #2.4): another resource's message name, or `Global`. A top-level resource has no `parents`. +- `singleton` is `true` for a singleton (see section #2.5). +- `scope` **must** be set on a top-level resource: `SCOPE_ATESPACE` or `SCOPE_GLOBAL` (see section #2.1). A subresource **must not** set it, because it has its parent's scope. +- `plural` **must** be set on every resource, singletons included. It is the plural that List methods and their repeated resource field use (see section #3). +- A message that is not a resource, such as a request, a status message, or `ObjectRef`, carries no `(resource)` option. +- When you add, remove, or re-parent a resource, update [the resource model](resource-model.md) too. + +### 11.2 Methods + +Every RPC of the `Control` service **must** carry a `(method)` option naming the resource it operates on, for standard and custom methods alike: + +```proto +rpc GetActorEgressPolicy(GetActorEgressPolicyRequest) returns (EgressPolicy) { + option (method).resource = "EgressPolicy"; +} + +rpc SuspendActor(SuspendActorRequest) returns (SuspendActorResponse) { + option (method).resource = "Actor"; +} +``` + +Rules: +- `resource` is the resource's message name, and that message **must** carry a `(resource)` option. +- A subresource's method names the subresource, not its parent: `EgressPolicy` for `GetActorEgressPolicy`. diff --git a/docs/architecture.md b/docs/architecture.md index f313ab117d..39b11d0018 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -349,11 +349,11 @@ Handles actor-aware routing and automatic re-animation. not select the Actor. * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an - authenticated TLS tunnel to the worker's `atunnel` listener on port 443. + mTLS `CONNECT` tunnel to the worker's `atunnel` listener on port 8443. `atunnel`, hosted by `ateom`, forwards the request to the active Actor over its private veth interface. Worker pod port 80 is not a direct Actor ingress - path. With `--upstream-connect`, the router instead reaches `atunnel` - through mTLS `CONNECT` tunnels on port 8443. + path. With `--upstream-connect=false`, the Envoy router instead reaches + `atunnel` over HTTPS on port 443. * **Arbitrary-Port Ingress**: A client reaches a port on the Actor other than its default (80) with an HTTP `CONNECT` request naming the port in the diff --git a/docs/csi-deployment.md b/docs/csi-deployment.md index de46c86edf..ec9a2e4862 100644 --- a/docs/csi-deployment.md +++ b/docs/csi-deployment.md @@ -38,6 +38,7 @@ When exposing the CSI Controller over the network, communication should be secur * `enabled: true`: Enables TLS/mTLS for gRPC communication between `ateapi` and the CSI Controller service. If `false` (or omitted), communication falls back to unencrypted plaintext gRPC. * `usePodIdentity: true`: Instructs `ateapi` to authenticate using Substrate's SPIFFE Pod Identity client certificate (`/run/podidentity.podcert.ate.dev/credential-bundle.pem`) and verify the controller's server certificate using Substrate's dynamic Service DNS CA trust bundle (`/run/servicedns.podcert.ate.dev/trust-bundle.pem`). +* `serverName: `: Specifies the expected Subject Alternative Name (SAN) of the controller for TLS certificate verification. Required when `enabled` is `true`. > [!IMPORTANT] > **What happens if `usePodIdentity` is `false`?** diff --git a/docs/csi-volumes.md b/docs/csi-volumes.md index ae170b3f25..7aca525fa2 100644 --- a/docs/csi-volumes.md +++ b/docs/csi-volumes.md @@ -51,7 +51,7 @@ spec: | :--- | :--- | :--- | | `enabled` | `bool` | **Required.** Enables TLS/mTLS for the gRPC connection. | | `usePodIdentity` | `bool` | **Optional.** When `true`, reuses Substrate's SPIFFE Pod Identity certificates for mutual TLS (mTLS) with dynamic CA trust bundle verification and rotation. Must be `true` when `enabled` is `true`. | -| `serverName` | `string` | **Optional.** Server name override for TLS certificate verification. | +| `serverName` | `string` | **Required.** Expected server name (SAN) for TLS certificate verification when TLS is enabled. | > [!NOTE] > For details on exposing CSI controller endpoints over the network and configuring CSI node DaemonSets with required mount propagations, see the [CSI Driver Deployment Guide](csi-deployment.md). @@ -180,7 +180,7 @@ containers: path: /readyz port: 8080 sandboxConfig: - sandboxClass: SANDBOX_CLASS_GVISOR + sandboxClass: gvisor configName: gvisor-default snapshotConfig: storageLocation: gs://my-snapshots-bucket/stateful-agent diff --git a/docs/dev/cloud-sql-scaling-guide.md b/docs/dev/cloud-sql-scaling-guide.md index 048c3a33be..383382d311 100644 --- a/docs/dev/cloud-sql-scaling-guide.md +++ b/docs/dev/cloud-sql-scaling-guide.md @@ -45,16 +45,16 @@ connections ≈ QPS × mean latency in seconds ≈ 10,000 req/s × 0.006 s ≈ 60 active connections ``` -Provision ~2× headroom for bursts. This setting does not affect the owner and -watch pools; their fixed limits add up to 5 connections per replica beyond the -read/write pool limit (e.g. 4 replicas with +Provision ~2× headroom for bursts. This setting does not affect the owner, +watch, and readiness probe pools; their fixed limits add up to 6 connections per +replica beyond the read/write pool limit (e.g. 4 replicas with `ATE_API_POSTGRES_POOL_MAX_CONNS=32` can open up to -4 × (32 + 2 + 3) = 148 connections). +4 × (32 + 2 + 3 + 1) = 152 connections). An undersized pool causes client-side queuing inside `pgx` rather than database errors. Ensure total connections across all replicas stay within Cloud SQL's limit: ``` -replicas × (pool_max_conns + 5) ≤ max_connections − slack (superuser, maintenance) +replicas × (pool_max_conns + 6) ≤ max_connections − slack (superuser, maintenance) ``` Exceeding diff --git a/docs/dev/microvm-local.md b/docs/dev/microvm-local.md index 687fcbccbb..4af3632028 100644 --- a/docs/dev/microvm-local.md +++ b/docs/dev/microvm-local.md @@ -77,8 +77,8 @@ Expected: NAMESPACE NAME DESIRED READY AVAILABLE ate-demo-counter-microvm workerpool.ate.dev/counter-microvm 1 1 1 -ATESPACE NAME SANDBOX CLASS GOLDEN SNAPSHOT ERROR AGE -ate-demo-counter-microvm counter-microvm SANDBOX_CLASS_MICROVM b9f6bd93-3c5a-4b64-9d5e-2f8a1c7d0e42 1m +ATESPACE NAME SANDBOX CLASS GOLDEN SNAPSHOT ERROR AGE +ate-demo-counter-microvm counter-microvm microvm b9f6bd93-3c5a-4b64-9d5e-2f8a1c7d0e42 1m ``` The template is ready once the GOLDEN SNAPSHOT column is non-empty (the diff --git a/docs/egress-traffic.md b/docs/egress-traffic.md index 6f63d47bcc..f25c7d8853 100644 --- a/docs/egress-traffic.md +++ b/docs/egress-traffic.md @@ -1,9 +1,16 @@ This document specifies the supported egress traffic for [GA](https://github.com/agent-substrate/substrate/milestone/3). -Actor TCP egress (except DNS traffic on port 53) is redirected to atunnel, +Actor TCP egress (except DNS to the sandbox's gateway) is redirected to atunnel, which opens a CONNECT tunnel to the egress gateway. Egress gateway applies policy. -DNS-over-TCP, UDP and other traffic is filtered by nftables and never reaches the gateway. +DNS over UDP and TCP goes to the sandbox's DNS relay, which forwards a query +to the worker pod's resolvers only when the queried hostname matches a host or +host wildcard in the actor's egress policy. Queries from an actor without a +policy, or for a hostname not allowed by any rule, are answered `REFUSED`. +A policy created, updated, or deleted while the actor runs applies to its DNS +within about 5 seconds. + +Other UDP and other traffic is filtered by nftables and never reaches the gateway. This applies to an actor's own telemetry too: an OpenTelemetry SDK inside the actor cannot reach the collector until the actor has an `http` egress rule for it. See @@ -16,7 +23,7 @@ cannot reach the collector until the actor has an `http` egress rule for it. See | any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden` | | any | WebSocket | Blocked | n/a | `403 Forbidden` | | any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | n/a | `403 Forbidden` | -| 53 | DNS | Allowed via netfilter rules | nftables -> node-configured DNS | n/a | +| 53 | DNS to the sandbox's gateway | Allowed for hostnames matching the actor's egress policy | sandbox DNS relay -> worker pod's resolvers | `REFUSED` | | any | Any other TCP | Blocked | n/a | The connection is accepted and then closed with no bytes returned. There is no status code. atunnel logs the failure. | ## UDP @@ -24,7 +31,7 @@ cannot reach the collector until the actor has an `http` egress rule for it. See | Port | Traffic | Behavior | Path | What the actor sees when refused | | :---- | :---- | :---- | :---- | :---- | -| 53 | DNS | Allowed via netfilter rules | nftables -> node-configured DNS | n/a | +| 53 | DNS to the sandbox's gateway | Allowed for hostnames matching the actor's egress policy | sandbox DNS relay -> worker pod's resolvers | `REFUSED` | | any other | Any other UDP | Blocked | n/a | Packets are dropped, not rejected: no ICMP port-unreachable is sent, so the client hangs until its own timeout. | ## Other protocols diff --git a/docs/egress-trust-bundle.md b/docs/egress-trust-bundle.md index 276dde86bd..c426723e04 100644 --- a/docs/egress-trust-bundle.md +++ b/docs/egress-trust-bundle.md @@ -30,9 +30,11 @@ CA. Substrate can automatically inject the root certificates for this CA into the actor's filesystem using a SystemInfo volume with a TrustBundle data source. Note: DNS traffic from the actor (both TCP and UDP) is not forwarded to the -gateway, and thus cannot be affected by egress policies. Instead, the DNS -traffic is allowed to directly exit the worker pod and be answered by the host -cluster's configured DNS server. +gateway. Instead, the sandbox's DNS relay forwards a query to the worker pod's +resolvers, which are the host cluster's configured DNS servers, only when the +queried hostname matches a host or host wildcard in the actor's egress policy: +queries from an actor without a policy, or for a hostname not matched by any +rule, are answered `REFUSED`. ## Configuring a workload to trust public and man-in-the-middle CAs diff --git a/docs/metrics/registry/metrics.yaml b/docs/metrics/registry/metrics.yaml index 36892ee8a1..f9382534ba 100644 --- a/docs/metrics/registry/metrics.yaml +++ b/docs/metrics/registry/metrics.yaml @@ -704,17 +704,6 @@ groups: type: string brief: The RPC system. otelgrpc always sets the value grpc. examples: ["grpc"] - - id: server.address - stability: stable - type: string - brief: > - The address of the server that a gRPC client called. otelgrpc reads it - from the remote address of the connection, so it holds an IP address - and not the name in the dial target. For a unix socket it holds the - path of the socket. atelet calls each ateom on a socket path that holds - the UID of the worker pod, so on atelet this key has one value for - each worker pod. - examples: ["10.8.2.14", "/var/lib/ate/ateoms//ateom.sock"] - id: server.port stability: stable type: int @@ -1321,7 +1310,10 @@ groups: This is the client half of rpc.server.call.duration. Compare the two to find the time that the network and the queues add. If the client duration is much larger than the server duration of the same method, the delay is - not in the handler. + not in the handler. otelgrpc also sets server.address, but serverboot + removes it with a View. The key holds the IP address of one server pod, + or the ateom socket path with the pod UID, so it has one value for each + pod. annotations: substrate: emitted_by: [ateapi, atelet, atecontroller, atenet-router] @@ -1337,9 +1329,6 @@ groups: requirement_level: required - ref: rpc.response.status_code requirement_level: required - - ref: server.address - requirement_level: - conditionally_required: The client sent the request headers on a connection. - ref: server.port requirement_level: conditionally_required: The client called the server over TCP. A call on a unix socket has no port. diff --git a/docs/metrics/substrate.yaml b/docs/metrics/substrate.yaml index 3313e3782f..66fb64e709 100644 --- a/docs/metrics/substrate.yaml +++ b/docs/metrics/substrate.yaml @@ -26,9 +26,9 @@ registry_version: 1 # The version of the upstream semantic conventions that Substrate borrows from. -# Substrate takes error.type, file.name, the rpc.* attributes, server.address -# and server.port from it. This is the only version that the code imports under -# cmd/, internal/ and pkg/. +# Substrate takes error.type, file.name, the rpc.* attributes and server.port +# from it. This is the only version that the code imports under cmd/, internal/ +# and pkg/. # It is not the version of this registry: manifest.yaml holds that. upstream_semconv_version: 1.40.0 diff --git a/docs/network-egress.md b/docs/network-egress.md index 950749ce6c..05ec840efc 100644 --- a/docs/network-egress.md +++ b/docs/network-egress.md @@ -1,6 +1,6 @@ # Network Egress Contract -Last updated: 09/18/2026 +Last updated: 10/09/2026 ## Overview @@ -8,7 +8,7 @@ Agent Substrate enforces that all outbound traffic from an actor, except traffic ## Transport -At present, actor TCP traffic exits the sandbox through [`atunnel`](./glossary.md#components) and is forwarded on to the PEP, except traffic to destination port 53; traffic using TCP or UDP destination port 53 is allowed to bypass the PEP for DNS, while all other protocols are blocked. `atunnel` is configured with the PEP's network address and port at startup and uses that address to open an mTLS HTTP/1.1 CONNECT tunnel to the PEP. The CONNECT request target and `Host` header are both set to the original destination IP and port that `atunnel` derived from the actor's TCP connection. Actor networking is currently IPv4-only. +At present, actor TCP traffic exits the sandbox through [`atunnel`](./glossary.md#components) and is forwarded on to the PEP. The exception is DNS: TCP and UDP traffic to port 53 on the sandbox's gateway bypasses the PEP and is answered by the sandbox's DNS relay, which forwards a query to the worker pod's resolvers only when the queried hostname matches a host or host wildcard in the actor's egress policy; queries from an actor without a policy, or for a hostname not matched by any rule, are refused. All other protocols are blocked. `atunnel` is configured with the PEP's network address and port at startup and uses that address to open an mTLS HTTP/1.1 CONNECT tunnel to the PEP. The CONNECT request target and `Host` header are both set to the original destination IP and port that `atunnel` derived from the actor's TCP connection. Actor networking is currently IPv4-only. Before traffic reaches the egress PEP, ateom rejects packets destined for the IPv4 link-local range `169.254.0.0/16` or the IPv6 link-local range `fe80::/10`. The internal actor network `169.254.17.0/30` is exempt because it carries traffic for the actor gateway, DNS relay, ingress, and wakeup probes. Localhost traffic remains inside the actor sandbox and does not traverse the PEP. Access to non-link-local destinations remains controlled by substrate egress policy and, at the `WorkerPool` boundary, Kubernetes NetworkPolicy. diff --git a/docs/observability.md b/docs/observability.md index c451ebf633..8674f7f601 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -12,7 +12,7 @@ To make underlying infrastructure transitions transparent, Agent Substrate estab * `ate.actor.uid`: Server-assigned UID of the actor, unique to the lifetime of an actor. * `ate.template.name`: The name of the actor's ActorTemplate (e.g., `counter`). * `ate.template.atespace`: The atespace of the actor's ActorTemplate (e.g., `ate-demo-counter`). -* `ate.actor.container.name`: The name of the container within the actor that produced the log line (e.g., `counter`), so a multi-container actor's logs can be demultiplexed by container. Absent on the synthetic lifecycle records (`Actor starting`, `Actor restored`, …): those are about the actor, so no container produced them. +* `ate.actor.container.name`: The name of the container within the actor that produced the log line (e.g., `counter`), so a multi-container actor's logs can be demultiplexed by container. Absent on the synthetic lifecycle records (`Actor starting`, `Actor restored`, …): those are about the actor, so no container produced them. The `runsc create`, `start`, and `restore` CLI output and the pause container's stdio are wrapped under `_pause`. This does not capture sentry or gofer output, which runsc directs to `/dev/null` when debug logging is disabled. Currently, Agent Substrate automatically wraps container output and injects these metadata labels into **container logs**. For metrics and distributed tracing, Agent Substrate provides foundational system telemetry and on-demand request tracing, with roadmap plans to fully integrate actor-level correlation. @@ -68,13 +68,19 @@ Actor is currently running on pod ate-demo-counter/counter-ab123-x4y5z ``` #### Example 4: Filtering by Container or Source -An actor can run several containers. By default every line is shown, including the synthetic lifecycle events (`Actor starting`, `Actor started`, `Actor checkpointing`, `Actor checkpointed`, `Actor restoring`, `Actor restored`, `Actor terminated`). `--container` (short form `-c`) restricts the output to the named container's logs: +An actor can run several containers. By default application container logs and synthetic lifecycle events (`Actor starting`, `Actor started`, `Actor checkpointing`, `Actor checkpointed`, `Actor restoring`, `Actor restored`, `Actor terminated`) are shown. The `_pause` logs are excluded so runsc startup diagnostics do not mix with application output. `--container` (short form `-c`) restricts the output to the named container's logs: ```bash kubectl ate logs actor -a -c ``` -`--source` selects a class of lines instead of a container. `containers` shows every container's output with the lifecycle events removed; `lifecycle` shows only the lifecycle events, which is the quickest way to see how often an actor has been suspended and restored on its current worker (a migration moves the actor to another pod, whose log holds none of the earlier workers' events): +To view the pause container's runsc diagnostics explicitly: + +```bash +kubectl ate logs actors -a -c _pause +``` + +`--source` selects a class of lines instead of a container. `containers` shows application container output with the lifecycle events removed; `_pause` remains excluded unless selected with `-c _pause`. `lifecycle` shows only the lifecycle events, which is the quickest way to see how often an actor has been suspended and restored on its current worker (a migration moves the actor to another pod, whose log holds none of the earlier workers' events): ```bash $ kubectl ate logs actor test -a demo --source=lifecycle @@ -458,7 +464,7 @@ Verbatim forwarding is restricted to known ateom sources and refuses anything el ### Actor telemetry needs an egress policy -An actor's own OpenTelemetry SDK does not use the relay. It dials the collector over the network like any other outbound connection, and that connection is subject to the actor egress lockdown described in [Network Egress](network-egress.md): every actor TCP connection is redirected into `atunnel` and carried to the egress gateway, which refuses anything the actor's `EgressPolicy` does not allow. An actor with no policy, or a policy with no matching rule, cannot reach the collector at all. The SDK sees `Unavailable: connection reset by peer` on each export, and the gateway logs `egress denied: actor has no egress policy`. +An actor's own OpenTelemetry SDK does not use the relay. It dials the collector over the network like any other outbound connection, and that connection is subject to the actor egress lockdown described in [Network Egress](network-egress.md): every actor TCP connection is redirected into `atunnel` and carried to the egress gateway, which refuses anything the actor's `EgressPolicy` does not allow. An actor with no policy, or a policy with no matching rule, cannot reach the collector at all. With no policy, or a policy whose rules do not include the collector's hostname, the actor cannot even resolve the collector's name: the sandbox's DNS relay answers its queries `REFUSED`, so each export fails before it connects. If a rule allows the hostname on another protocol or port (for example, `*` on port `443`), DNS resolves the name, the SDK sees `Unavailable: connection reset by peer` on each export, and the gateway logs why it denied the connection. To let an actor export, give it an `http` rule naming the collector's Service and the OTLP ports. It must be an `http` rule, not `https`: the OTLP/gRPC exporter is gRPC over cleartext HTTP/2 (h2c) on `4317`, which an `http` rule admits, and the OTLP/HTTP exporter is cleartext HTTP/1.1 on `4318`. On Kind the collector is `opentelemetry-collector.otel-system.svc`; on GKE with the managed collector it is `opentelemetry-collector.gke-managed-otel.svc.cluster.local`. Whatever the name, it has to match the host the actor's `OTEL_EXPORTER_OTLP_ENDPOINT` names, because the rule is matched on the request authority. @@ -487,7 +493,7 @@ An actor that already has a policy for other destinations keeps those rules and Three things to know when applying this: * **It is per actor.** The policy is a resource under the actor, and nothing applies one by default today. Every actor that exports telemetry needs its own, created after the actor and before, or shortly after, it is resumed. A template-level default policy is proposed in [#1558](https://github.com/agent-substrate/substrate/pull/1558); the underlying ask is [#1430](https://github.com/agent-substrate/substrate/issues/1430). -* **A running actor recovers slowly.** If the policy is created while the actor is already exporting and failing, the SDK's gRPC client is in reconnect backoff and the gateway caches the missing policy for `--egress-policy-cache-ttl` (10s by default). Expect the first successful export 45s to two minutes after the policy appears. An actor whose policy exists before it starts has no such delay. +* **A running actor recovers slowly.** If the policy is created while the actor is already exporting and failing, the SDK's gRPC client is in reconnect backoff, the sandbox's DNS relay takes up to 5s to notice the policy, and the gateway caches the missing policy for `--egress-policy-cache-ttl` (10s by default). Expect the first successful export 45s to two minutes after the policy appears. An actor whose policy exists before it starts has no such delay. * **The collector sees the gateway, not the actor.** The TCP connection to the collector comes from the egress gateway pod, so a collector that enriches by source IP (the `k8sattributes` processor, which the GKE managed collector runs) labels the actor's spans and metrics with the gateway's `k8s.pod.name`, `k8s.deployment.name=atenet-egress`, and `k8s.namespace.name`. The actor's `service.name` and anything it puts in `OTEL_RESOURCE_ATTRIBUTES` survive. Set the actor's identity there yourself if you need it on the backend; the relay-based path that will do this for actors is tracked in [#853](https://github.com/agent-substrate/substrate/issues/853) and [#761](https://github.com/agent-substrate/substrate/issues/761). **Logs written to stdout and stderr need none of this.** ateom reads a container's output stream inside the worker pod and writes it to the pod's own stdout, which is where [`kubectl ate logs`](#active-actor-inspection-via-cli) and a node log agent pick it up. That path never leaves the sandbox over the network, so it is untouched by the egress lockdown and works for an actor with no policy at all. The policy is only for telemetry an SDK inside the actor pushes itself: traces, metrics, and OTLP logs if `OTEL_LOGS_EXPORTER=otlp` is set in the actor. diff --git a/docs/postgres.md b/docs/postgres.md index 91aa347917..bf4fed5348 100644 --- a/docs/postgres.md +++ b/docs/postgres.md @@ -86,7 +86,7 @@ Set `ATE_API_POSTGRES_OWNER_ROLE` and `ATE_API_POSTGRES_READ_WRITE_ROLE` to the User credentials are passed as connection strings through `ATE_API_POSTGRES_OWNER_CONNECTION_STRING` and `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING`. When running `ateapi` directly, pass the DSNs as flags or use `@env` flags to read these environment variables; the installer manifest already uses `@env`. -`ATE_API_POSTGRES_POOL_MAX_CONNS` (or `--postgres-pool-max-conns` when running `ateapi` directly) sets the read/write pool limit after the connection string is loaded. When unset, a `pool_max_conns` value in the DSN or the pgxpool default applies. This setting does not affect the owner or watch pools; they are capped at 2 and 3 connections, respectively. +`ATE_API_POSTGRES_POOL_MAX_CONNS` (or `--postgres-pool-max-conns` when running `ateapi` directly) sets the read/write pool limit after the connection string is loaded. When unset, a `pool_max_conns` value in the DSN or the pgxpool default applies. This setting does not affect the owner, watch, or readiness probe pools; they are capped at 2, 3, and 1 connections, respectively. `ateapi` uses the same `@env` flag pattern for role names and schema. diff --git a/docs/threat-model.md b/docs/threat-model.md index a956839504..92ef6db759 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -38,7 +38,7 @@ Substrate is an early, fast moving product. It is full of debate and subject to * **ateom:** Per-worker Pod sidecar, running inside the worker Pod. Ateom sets up "interior" sandboxes in the worker Pod and manages sandbox lifecycle, including image pulls. It currently uses gvisor but Substrate will support multiple microvm solutions. * **Worker:** Preprovisioned Pods that actors get scheduled to. * **Actor:** The core compute primitive, gets scheduled to/from worker via Run for cold start and Resume for snapshot resume. -* **Actor Network:** `ateom` creates a private point-to-point veth network for the active Actor inside a worker Pod. The Actor is not served directly from the worker Pod's port 80; ingress enters through `atunnel`'s authenticated listener on port 443. +* **Actor Network:** `ateom` creates a private point-to-point veth network for the active Actor inside a worker Pod. The Actor is not served directly from the worker Pod's port 80; ingress enters through `atunnel`'s authenticated listeners: mTLS `CONNECT` on port 8443, which the router uses by default, and HTTPS on port 443. * **Actor Routing:** Requests sent to `atenet-router` identify the target Actor with `ate-target-actor: /`. Host and authority values are application metadata and do not select the Actor. * **atenet-router:** Substrate runs Envoy with an `ext_proc` external processor to handle Actor ingress. With `--ingress-auth-mode=static-mtls`, Envoy accepts connections only on its TLS listeners and only from clients presenting a certificate that chains to the configured client CA bundle and carries one of a static list of SPIFFE IDs; any listed client may reach any Actor. The default, `deprecated-insecure`, rejects a TLS client whose certificate fails those checks but accepts one that presents none, and also serves plaintext listeners that accept any client. The ext_proc reads and validates `ate-target-actor`, calls the Substrate API to resume the Actor and obtain its current worker assignment, and selects that worker as a dynamic backend. It overwrites the routing header before forwarding. The router then connects with mTLS to `atunnel`; `atunnel` validates the router identity and authorizes the header against the Actor currently assigned to that worker. * **Object Storage:** Used to store actor snapshots. @@ -74,7 +74,7 @@ While it's probably unlikely that Substrate, which is expected to be an internal | T-05 | High | Privilege escalation via access to sensitive labels. | If Substrate offers its own resource labeling mechanism, it must also offer a way to authorize label updates on a per-label basis. | Substrate authorization system requires explicit authorization to update metadata, separate from updating resource body. Substrate authorization system supports per-label authorization rules. | Plenty of attacks in K8s were possible because labels had semantic meaning, but the permission model could implicitly granted access to modify labels, even if it was inappropriate. For example, /status subresource allows label updates. Substrate shouldn't repeat this mistake. | | T-06 | High | Attacker gains control of the Substrate API server or an ingress/egress gateway. | Isolate the control plane from the data plane, and isolate data plane ingress from sandboxes. | Don't co-locate ate-apiserver or other control-plane components on the same machines as the untrusted sandboxes. This also supports control-plane / data-plane isolation from a reliability and performance perspective. Consider running any gateway that enables direct interaction with sandboxes on a separate VM from the sandboxes. Consider using a zero-trust architecture where traffic is encrypted and authenticated end-to-end and routed via a mesh. | While the sandbox is trusted as an isolation layer, the residual risk is typically not worth the marginal cost savings of colocating, and isolation is recommend to avoid noisy neighbor problems or other reliability issues that stem from mixing the control-plane and data-plane, especially for large scale deployments of Substrate. | | T-07 | High | Attacker who can create ActorTemplates specifies malicious runtime. | Ensure available runtime can only be configured by administrators. | Consider a mechanism like RuntimeClass to decouple configuration of available runtimes from consumption of available runtimes. | | -| T-08 | High | Attacker who can create ActorTemplates can read or write any storage buckets atelet has access to. | Ensure that bucket access is least-privilege. | Use credentials derived from actor identity to read snapshots. Configure permissions to prevent atelet or nodes from having access to sensitive buckets. Don't support arbitrary URLs in APIs, instead design a standard approach for accessing the correct resource given the ActorTemplate name and compute the mapping internally, subject to scheduling-aware authorization checks. | For example: Attacker creates an ActorTemplate with the runsc URL or golden snapshot URL pointing to an arbitrary bucket in the same project/resource scope as the cluster. If atelet has project-wide access to buckets, this could cause the state to be pulled into the worker pod or malicious actor. Similarly, an attacker could set the snapshots URL to point to an internal infrastructure bucket, causing data to be written to that bucket. Current state for sandbox asset URLs (SandboxConfig): atelet holds no storage credentials. It fetches every asset through the node's object-store plugin, which refuses any URL under a snapshot location before reading it, and returns an object only if its content matches the sha256 the SandboxConfig pins, staging the download where atelet cannot read it until it is verified. Only when the plugin cannot serve a `gs://` URL (no such object, a URL it rejects, no `AssetProvider`, or unreachable) does atelet read it anonymously, which returns only publicly readable objects; a refusal by the plugin is final. A SandboxConfig author therefore cannot use an asset URL to pull another actor's snapshot onto a node, even when assets share a bucket with snapshots, and a modified asset fails the digest check in the plugin and again in atelet. These checks do not protect against a compromised node: root on the node can use the plugin's identity directly, which already reads and writes snapshots. Closing that needs storage-level scoping such as per-actor snapshot credentials: [https://github.com/agent-substrate/substrate/issues/777](https://github.com/agent-substrate/substrate/issues/777) | +| T-08 | High | Attacker who can create ActorTemplates can read or write any storage buckets atelet has access to. | Ensure that bucket access is least-privilege. | Use credentials derived from actor identity to read snapshots. Configure permissions to prevent atelet or nodes from having access to sensitive buckets. Don't support arbitrary URLs in APIs, instead design a standard approach for accessing the correct resource given the ActorTemplate name and compute the mapping internally, subject to scheduling-aware authorization checks. | For example: Attacker creates an ActorTemplate with the runsc URL or golden snapshot URL pointing to an arbitrary bucket in the same project/resource scope as the cluster. If atelet has project-wide access to buckets, this could cause the state to be pulled into the worker pod or malicious actor. Similarly, an attacker could set the snapshots URL to point to an internal infrastructure bucket, causing data to be written to that bucket. Current state for sandbox asset URLs (SandboxConfig): atelet holds no storage credentials. It fetches every asset through the node's object-store plugin, which refuses any URL under a snapshot location before reading it, and returns an object only if its content matches the sha256 the SandboxConfig pins, staging the download where atelet cannot read it until it is verified. Only when the plugin cannot serve a `gs://` URL (no such object, a URL it rejects, no `AssetProvider`, unreachable, or an error it does not recognize) does atelet read it anonymously, which returns only publicly readable objects; any other refusal by the plugin is final. A SandboxConfig author therefore cannot use an asset URL to pull another actor's snapshot onto a node, even when assets share a bucket with snapshots, and a modified asset fails the digest check in the plugin and again in atelet. These checks do not protect against a compromised node: root on the node can use the plugin's identity directly, which already reads and writes snapshots. Closing that needs storage-level scoping such as per-actor snapshot credentials: [https://github.com/agent-substrate/substrate/issues/777](https://github.com/agent-substrate/substrate/issues/777) | | T-09 | Medium | DoS attack against API, gateway, or available cluster resources. | Minimize exposure and ensure APIs and proxies implement appropriately scoped quotas and rate limiting. | Don't co-locate ate-apiserver or other control-plane components on the same machines as the untrusted sandboxes. Don't expose ate-apiserver directly to the internet. Consider an identity-aware WAF (likely up to providers) if external access is required. Use network policy to block direct interaction between untrusted sandboxes and the ate-apiserver. Implement quotas and rate limiting in the API and in proxies. This includes quotas for the number of actors a user can allocate. Use a zero-trust architecture that prevents identity spoofing or intentional misrouting to get around limits. | The gateway calls ate-apiserver to resolve Actor assignments and uses a short-lived cache. A flood of Actor traffic could still create high API load; cache invalidation during Actor rescheduling must avoid stale routing. | | T-10 | Medium | Internal network traffic is intercepted or spoofed | Encrypt all traffic by default | Use mTLS between all system components, including the gateway-to-worker `atunnel` connection. | It may be desirable to rely on cloud providers to transparently encrypt traffic between VMs on their internal network. Worth discussing what makes the most sense. | diff --git a/docs/threats.json b/docs/threats.json index cd1820dbda..4ee01cc49f 100644 --- a/docs/threats.json +++ b/docs/threats.json @@ -61,7 +61,7 @@ "threat": "Attacker who can create ActorTemplates can read or write any storage buckets atelet has access to.", "mitigating_invariants": "Ensure that bucket access is least-privilege.", "suggested_concrete_mitigations": "Use credentials derived from actor identity to read snapshots. Configure permissions to prevent atelet or nodes from having access to sensitive buckets. Don't support arbitrary URLs in APIs, instead design a standard approach for accessing the correct resource given the ActorTemplate name and compute the mapping internally, subject to scheduling-aware authorization checks.", - "notes": "For example: Attacker creates an ActorTemplate with the runsc URL or golden snapshot URL pointing to an arbitrary bucket in the same project/resource scope as the cluster. If atelet has project-wide access to buckets, this could cause the state to be pulled into the worker pod or malicious actor. Similarly, an attacker could set the snapshots URL to point to an internal infrastructure bucket, causing data to be written to that bucket. Current state for sandbox asset URLs (SandboxConfig): atelet holds no storage credentials. It fetches every asset through the node's object-store plugin, which refuses any URL under a snapshot location before reading it, and returns an object only if its content matches the sha256 the SandboxConfig pins, staging the download where atelet cannot read it until it is verified. Only when the plugin cannot serve a `gs://` URL (no such object, a URL it rejects, no `AssetProvider`, or unreachable) does atelet read it anonymously, which returns only publicly readable objects; a refusal by the plugin is final. A SandboxConfig author therefore cannot use an asset URL to pull another actor's snapshot onto a node, even when assets share a bucket with snapshots, and a modified asset fails the digest check in the plugin and again in atelet. These checks do not protect against a compromised node: root on the node can use the plugin's identity directly, which already reads and writes snapshots. Closing that needs storage-level scoping such as per-actor snapshot credentials: [https://github.com/agent-substrate/substrate/issues/777](https://github.com/agent-substrate/substrate/issues/777)" + "notes": "For example: Attacker creates an ActorTemplate with the runsc URL or golden snapshot URL pointing to an arbitrary bucket in the same project/resource scope as the cluster. If atelet has project-wide access to buckets, this could cause the state to be pulled into the worker pod or malicious actor. Similarly, an attacker could set the snapshots URL to point to an internal infrastructure bucket, causing data to be written to that bucket. Current state for sandbox asset URLs (SandboxConfig): atelet holds no storage credentials. It fetches every asset through the node's object-store plugin, which refuses any URL under a snapshot location before reading it, and returns an object only if its content matches the sha256 the SandboxConfig pins, staging the download where atelet cannot read it until it is verified. Only when the plugin cannot serve a `gs://` URL (no such object, a URL it rejects, no `AssetProvider`, unreachable, or an error it does not recognize) does atelet read it anonymously, which returns only publicly readable objects; any other refusal by the plugin is final. A SandboxConfig author therefore cannot use an asset URL to pull another actor's snapshot onto a node, even when assets share a bucket with snapshots, and a modified asset fails the digest check in the plugin and again in atelet. These checks do not protect against a compromised node: root on the node can use the plugin's identity directly, which already reads and writes snapshots. Closing that needs storage-level scoping such as per-actor snapshot credentials: [https://github.com/agent-substrate/substrate/issues/777](https://github.com/agent-substrate/substrate/issues/777)" }, { "threat_id": "T-09", diff --git a/docs/upgrade.md b/docs/upgrade.md index 10bcb912bd..4f9fdda840 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -1,652 +1,317 @@ -# Rolling upgrade runbook - -This runbook upgrades a running Agent Substrate install to a new build -version, one node at a time. No actor loses state. On a cluster with -more than one node, at most one node's worth of capacity is out of -service while the rest of the fleet keeps serving; a single-node cluster -stops fully during step 5. It needs `kubectl`, `kubectl ate`, `go run -./cmd/ate-setup`, `jq`, and `grpcurl`. The numbered steps are the same -on every Kubernetes provider; the provider-specific parts sit in their -own sections, [before](#on-gke) and [after](#after-the-roll-on-gke) -the roll. All of the roll's state lives in cluster objects, so you can -stop at any point and pick up again. [Check your -progress](#check-your-progress) tells you where you left off. - -The order is `ate-controller` first, then the dataplane, then the rest -of the control plane. The controller goes first because it manages -the worker pools the roll creates. The dataplane goes before the rest -of the control plane so that, by the time ate-api-server and atenet -change, every atelet and worker already understands requests from -either version of the control plane. - -## Check your progress - -Come back here after a break, and before you touch anything when -something looks wrong. Three commands say where the roll stands: +# Upgrade runbook + +This runbook upgrades a running Agent Substrate install to a newer release. + +| From → to | Supported | +|---|---| +| A later patch release of the same minor, for example 1.2.1 → 1.2.4 | Yes. Patch releases can be skipped | +| The next minor release, for example 1.2.x → 1.3.y | Yes | +| Skipping a minor release, for example 1.2.x → 1.4.y | No. Upgrade to 1.3 first | +| A new major release, for example 1.x → 2.x | No. Reinstall | +| A minor release before 1.0, for example 0.3.x → 0.4.y | No. Reinstall. Before 1.0, breakage still exists for minor versions | + +If an upgrade goes wrong, you can roll back to the release you upgraded from, and no further. + + + +| Step | What changes | What running actors see | How long | +|---|---|---|---| +| 1 | podcertificate-controller | Nothing | A minute | +| 2 | atelet, one node at a time | Nothing | Seconds to about 6 minutes per node | +| 3 | workers | `SIGTERM`, then a 30-minute window to be suspended before `SIGKILL` | Seconds to hours per pool | +| 4 | ate-api-server, the CRDs and ate-controller, atenet, SandboxConfig, then the rest of the system bundle | API calls, long requests and outbound connections can be cut once; workers may get `SIGTERM` again. See step 4 | A few minutes, or as long as step 3 if workers roll again | + +Rollback is the same list in reverse, from the old release. An actor that crashes along the way goes back to its last snapshot with `kubectl ate revert actor -a `. + +The system upgrade does not change the database engine, the CSI drivers `--setup-csi` installs, the Kubernetes version or the node OS version. + +## Where am I? + +Come back here after a break or a handoff. Each time a Deployment moves to a new image it gets a new ReplicaSet, and the atelet DaemonSet a new ControllerRevision, so their `AGE` tells what has moved. A component has moved when the `AGE` of its ReplicaSet with pods (`DESIRED` above 0), or of atelet's ControllerRevision with the highest `REVISION`, is shorter than the time since the upgrade began. Two ReplicaSets with pods mean a roll is under way. `podcertificate-controller` moves in step 1, `atelet` in step 2, the rest in step 4: ```bash -# One DaemonSet: only the old dataplane is installed, and its label -# value is the $OLD_VERSION you are upgrading from. Two: step 3 is done, -# and the other label value is the version you are upgrading to. -kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version - -# Nodes per version. All on the old one: the per-node roll has not -# started. A mix: it is under way. All on the new one: it is done. -kubectl get nodes -L ate.dev/substrate-version --no-headers \ - | awk '{print $NF}' | sort | uniq -c - -# Worker pools. A clone next to each serving pool: step 4 is done. -kubectl get workerpools -A +kubectl -n podcertificate-controller-system get rs +kubectl -n ate-system get rs,controllerrevision ``` -The commands above do not show whether steps 1, 2 and 6 have run. -Those steps are idempotent, so run them again if you are not sure. +The worker pools move in step 3, and their ReplicaSets read the same way. A pool is done rolling when `UP-TO-DATE` and `READY` equal `WANTED`: + +```bash +kubectl get rs -A -l ate.dev/worker-pool +kubectl get deploy -A -l ate.dev/worker-pool -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,UP-TO-DATE:.status.updatedReplicas,READY:.status.readyReplicas,WANTED:.spec.replicas,PAUSED:.spec.paused' +``` ## Before you start -### Confirm before you begin - -Every item has to hold. Nothing in the roll stops you if one does not. - -- [ ] Every node carries the `ate.dev/substrate-version` label, all - with the same value. That value is `$OLD_VERSION`. - - ```bash - kubectl get nodes -L ate.dev/substrate-version - ``` - -- [ ] The atelet DaemonSet name carries a version suffix. If it does - not, the cluster was installed from an older build and needs a fresh - install instead; its DaemonSet selector cannot be changed in place. - - ```bash - kubectl get ds -n ate-system -l app=atelet - ``` - -- [ ] If you are upgrading across the release that added the ate-system - NetworkPolicies, every namespace whose pods call the Substrate API - directly (not through `kubectl ate`) is labeled - `ate.dev/api-access=true`. The new manifests deny the API to every - other namespace as soon as they are applied. Worker pods are never - admitted, labeled or not. See - [`ate-system-networkpolicies.yaml`](../manifests/ate-install/ate-system-networkpolicies.yaml). - - ```bash - kubectl label namespace ate.dev/api-access=true - ``` - -- [ ] Every serving pool is pinned to `$OLD_VERSION`. An unpinned pool - cannot take part in the roll. - - ```bash - kubectl get workerpools -A \ - -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,PIN:.spec.template.nodeSelector.ate\.dev/substrate-version' - ``` - - Pin a pool whose `PIN` column is `` while no actor is assigned - to its workers, because the edit re-renders the pool's Deployment - (see [the warnings](#three-things-that-break-an-upgrade)): - - ```bash - kubectl -n $NS patch workerpool $OLD_WORKERPOOL --type merge \ - -p "spec: {template: {nodeSelector: {ate.dev/substrate-version: '$OLD_VERSION'}}}" - ``` - -- [ ] Every serving pool is healthy: `READY` equals `DESIRED`. - - ```bash - kubectl get workerpools -A - ``` - -- [ ] Nothing drains worker nodes on its own: node auto-upgrade is off - on every pool that runs workers and none of them is spot or - preemptible, as the - [Create Cluster warning](../tools/setup-gcp/README.md#2-create-cluster) - requires. - -- [ ] If the install uses `--cordon-control-plane` with the bundled - PostgreSQL, at least one node carries - `ate.dev/workloadType=ate-postgres`. Releases before the dedicated - postgres pool ran postgres in the shared `ate-control-plane` pool; see - [Cordoned control plane: postgres pool](#cordoned-control-plane-postgres-pool). - `ate-setup` refuses to apply postgres without one. - - ```bash - kubectl get nodes -l ate.dev/workloadType=ate-postgres - ``` - -- [ ] The installed ate-api-server serves `DrainWorker`. Draining a - worker is one `DrainWorker` RPC, and step 5 calls it with - [grpcurl](https://github.com/fullstorydev/grpcurl). Open access to - the Control API now and keep the port-forward running; step 5 uses - the same `/tmp/ate-ca.pem` and `$TOKEN`. - - ```bash - kubectl -n ate-system port-forward svc/api 8443:443 >/dev/null 2>&1 & - - kubectl get clustertrustbundles -l podcert.ate.dev/canarying=live \ - -o jsonpath='{range .items[?(@.spec.signerName=="servicedns.podcert.ate.dev/identity")]}{.spec.trustBundle}{end}' \ - > /tmp/ate-ca.pem - - TOKEN=$(kubectl -n ate-system create token ate-client \ - --audience=api.ate-system.svc --duration=48h) - - # Must print ateapi.Control.DrainWorker. If it does not, the cluster - # predates the versioned builds and needs a fresh install. - grpcurl -cacert /tmp/ate-ca.pem -authority api.ate-system.svc \ - -H "authorization: Bearer ${TOKEN}" 127.0.0.1:8443 list ateapi.Control \ - | grep DrainWorker - ``` - -### Names used throughout - -| name | what it is | how to get it | -|---|---|---| -| `$OLD_VERSION` | the version label value you are upgrading from | the one atelet DaemonSet's label, see [Check your progress](#check-your-progress) | -| `$NEW_VERSION` | the version label value you are upgrading to | the second DaemonSet's label once step 3 has run | -| `$NS`, `$OLD_WORKERPOOL` | namespace and name of a serving WorkerPool | `kubectl get workerpools -A` | -| `$NEW_WORKERPOOL` | the clone's name | you pick it in step 4, for example `counter-v2` | -| `$NODE` | the node being rolled | picked per iteration in step 5 from `kubectl get nodes` | - -A cluster usually serves more than one WorkerPool, and every serving -pool moves in the same upgrade. Step 4 clones each of them, the -per-node roll in step 5 covers all pools on a node together, and the -retire at the end deletes each old pool. Where the runbook says -`$OLD_WORKERPOOL`, read "each serving pool". - -### Checkout and environment - -Every `go run ./cmd/ate-setup` command below runs from a checkout of -the new release, with the environment and flags the install used: -`PROJECT_ID`, `CLUSTER_NAME` and `CLUSTER_LOCATION` (or `--context`), -and either `KO_DOCKER_REPO` for a build from source or -`--image-repo`/`--image-tag` for prebuilt images. Keep a checkout of -the old release too as rollback runs the same commands from it. - -`ate-setup` takes the version from `$VERSION` when it is set, else -from `git describe` on the checkout for a build from source, or from -`--image-tag` (`ATE_IMAGE_TAG`) for prebuilt images. That value must come out different -from `$OLD_VERSION`, otherwise dataplane upgrade rolls the running atelet in place -instead of adding a second DaemonSet. A tagged checkout differs by -construction; if the install pinned `VERSION`, pin a new value now -and keep it for every command of this upgrade. - -Install the new `kubectl ate` with `go install ./cmd/kubectl-ate`. - -### On GKE - -GKE needs `gcloud` and three more names: - -| name | what it is | how to get it | -|---|---|---| -| `$CLUSTER`, `$ZONE` | the cluster and its location | `gcloud container clusters list` | -| `$NODEPOOL` | the node pool | `gcloud container node-pools list --cluster $CLUSTER --zone $ZONE` | - -Park the autoscaler. During the roll, the old pool's pods that lost -their node sit Pending on purpose: they are the rollback reserve. The -autoscaler reads Pending pods as demand and would add nodes for pods -that must never schedule. Save its config first; you restore it -[after the roll](#after-the-roll-on-gke). +You need git, go and kubectl, and crane for prebuilt images. Agree on a time with actor owners: their running actors get `SIGTERM` in step 3. -```bash -gcloud container node-pools describe $NODEPOOL --cluster $CLUSTER --zone $ZONE \ - --format='json(autoscaling)' # save for after the roll -gcloud container clusters update $CLUSTER --zone $ZONE \ - --node-pool $NODEPOOL --no-enable-autoscaling -``` +Every `ate-setup deploy` run records the settings it was given in `installs/` under the record directory: `ATE_RECORD_DIR` if set, by default `~/.cache/ate-setup` on Linux and `~/Library/Caches/ate-setup` on macOS. It prints the path as `recorded this configuration in `. -If a GKE maintenance window falls inside the roll, add a maintenance -exclusion for it too: a node GKE recreates mid-roll comes back at the -pool's label, which is still `$OLD_VERSION`. +The record's `cluster.substrateVersion` is the installed release. If this machine has no record of the cluster, write one from the settings you installed with ([operator-install.md](operator-install.md) lists them), and set `OLD_RELEASE` to the installed release before you run the block below. -Check that the node pool itself carries -`ate.dev/substrate-version=$OLD_VERSION`, or nodes GKE creates later -arrive unlabeled and nothing schedules to them. If it is missing, stamp -it now with the old value: +If `~/ate-upgrade` is left from an earlier upgrade, delete it first. Fill in the first three lines, then run the rest. `ate-setup` runs from the new checkout. The old checkout is kept for rollback. ```bash -gcloud container node-pools describe $NODEPOOL --cluster $CLUSTER --zone $ZONE \ - --format='get(config.labels)' # copy the value -gcloud container node-pools update $NODEPOOL --cluster $CLUSTER --zone $ZONE \ - --node-labels=,ate.dev/substrate-version=$OLD_VERSION +NEW_RELEASE=v0.4.2 # change me +RECORD=~/.cache/ate-setup/installs/my-cluster.yaml # change me: this cluster's record +kubectl config use-context my-cluster # change me: keep it until the upgrade is done +# export ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING=... # external database only + +OLD_RELEASE=${OLD_RELEASE:-$(sed -n 's/^ *substrateVersion: //p' $RECORD)} +echo "Upgrading '$OLD_RELEASE' -> '$NEW_RELEASE'" # stop if either is empty or not a release tag +mkdir ~/ate-upgrade && cd ~/ate-upgrade +cp $RECORD old.yaml && cp $RECORD new.yaml +sed -i.bak "s/^ tag: .*/ tag: $NEW_RELEASE/" new.yaml # prebuilt images: the new tag +grep 'tag:' old.yaml new.yaml # prebuilt images: new.yaml has the new tag +git clone --branch $OLD_RELEASE https://github.com/agent-substrate/substrate.git old +git clone --branch $NEW_RELEASE https://github.com/agent-substrate/substrate.git new +(cd old && go install ./cmd/kubectl-ate) +cd new ``` -Other providers have their own equivalents: stop whatever adds or -recreates worker nodes during the roll, and make sure a node created -later arrives with the old label. +Run every step from this shell. A record holds no database connection strings: with an external database, uncomment the `export` line, and export `ATE_API_POSTGRES_OWNER_CONNECTION_STRING` too if you set it. -### Cordoned control plane: postgres pool +Keep `old.yaml` as it is, for rollback. With prebuilt images, `new.yaml` now names the new tag. From source, the checkout you run from decides the release. -Skip this section unless the install uses `--cordon-control-plane` with -the bundled PostgreSQL. Under that flag, postgres runs alone in a -one-node pool labeled and tainted -`ate.dev/workloadType=ate-postgres:NoSchedule`, and every other control -plane workload shares the `ate-control-plane` pool. An install from a -release before that split has no postgres pool. The StatefulSet deletes -its running pod before creating the replacement, so without the pool -step 6 would take postgres down with nothing to schedule it on. +> [!CAUTION] +> Do not run `ate-setup setup csi` or `ate-setup delete` during the upgrade, or `ate-setup deploy ate-system` before step 4e. They change components out of upgrade order. -Create the pool before step 6, in the zone of the postgres volume. The -volume is a zonal disk, and a postgres pod on a node in another zone -stays Pending with a volume node affinity conflict. +## Upgrade + +### Step 1. Upgrade podcertificate-controller ```bash -# The zone the postgres volume lives in. -PG_ZONE=$(kubectl get pv \ - "$(kubectl -n ate-system get pvc data-postgres-0 -o jsonpath='{.spec.volumeName}')" \ - -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values[0]}') -echo "$PG_ZONE" - -# A machine type that fits the postgres pod's requests: the size10 -# profile asks for 80 CPUs and 140Gi. -gcloud container node-pools create ate-postgres --cluster $CLUSTER --zone $ZONE \ - --node-locations "$PG_ZONE" --num-nodes 1 --machine-type \ - --node-labels ate.dev/workloadType=ate-postgres \ - --node-taints ate.dev/workloadType=ate-postgres:NoSchedule +go run ./cmd/ate-setup deploy podcertificate-controller --config ~/ate-upgrade/new.yaml ``` -On other providers, create the equivalent: one node in the volume's -zone, carrying the label and the taint. +This deploys the new podcertificate-controller and waits until it publishes its trust bundles. Rerunning is safe. -Step 6 then restarts postgres once onto the new node, which is a short -outage for ate-api-server. The other control plane Deployments roll in -steps 2 and 6 as well. Their old pods still carry the required -anti-affinity of the earlier release, so the new pods land on the -`ate-control-plane` pool's spare node first and stay packed there. Once -`kubectl get pods -n ate-system -o wide` shows no pod older than the -roll, spread them again and shrink the pool, which no longer needs one -node per pod: +- **Done:** the command exits 0. +- **Stuck:** the command fails. [Roll back step 1](#roll-back-step-1). + +### Step 2. Upgrade atelet ```bash -kubectl -n ate-system rollout restart deployment \ - ate-api-server ate-controller atenet-router atenet-egress podcertificate-controller -gcloud container clusters resize $CLUSTER --zone $ZONE \ - --node-pool --num-nodes +go run ./cmd/ate-setup deploy atelet --config ~/ate-upgrade/new.yaml --rollout-timeout 1h ``` -## Three things that break an upgrade +`ate-setup` prints nothing until the roll is done. To watch it, open a second terminal and run: -Each warning comes back at the step where the mistake becomes possible. +```bash +kubectl -n ate-system rollout status ds/atelet +``` -> [!WARNING] -> **Do not delete a node's old worker pods before flipping its -> version label.** Otherwise the old pool reschedules replacements onto -> the same node, and old workers end up next to the new atelet: exactly -> the version skew the roll exists to prevent. (Step 5) +- **Progress:** `Waiting for daemon set "atelet" rollout to finish: 2 out of 5 new pods have been updated...`, then `... 4 of 5 updated pods are available...`. +- **Done:** `daemon set "atelet" successfully rolled out`. +- **Stuck:** the `N out of M` count does not move for more than about 6 minutes. A new atelet pod that cannot start stops the roll and leaves its node without atelet; if you cannot fix it quickly, [roll back step 2](#roll-back-step-2). A pod stuck in ContainerCreating usually means podcertificate-controller is not running. If `deploy atelet` times out, the roll still goes on; keep watching. +- **Actors:** running actors keep running. A node's atelet is down for a few seconds while it restarts. -> [!WARNING] -> **Do not edit a serving worker pool, and do not scale it down.** The -> controller would roll the pool's Deployment straight through live -> actors. A deleted worker pod does go through the eviction path: -> `SIGTERM` is forwarded into the actor's containers and the control -> plane keeps accepting a suspend for 30 minutes, so an actor suspended -> inside that window saves its state and stays resumable. Handling -> `SIGTERM` by exiting cleanly is not enough on its own; the suspend has -> to reach the control plane and finish. An actor still awake when the -> window closes moves to `ACTOR_STATE_CRASHED`: `resume` and `suspend` -> are both refused, and everything since its last snapshot is lost. Call -> `RevertActor` (`kubectl ate revert`) to discard the crashed run and -> return the actor to `ACTOR_STATE_SUSPENDED` at its last external snapshot -> so it can be resumed. Scaling a serving pool down removes pods the same -> way, without suspending the actors on them. (Step 4 clones the pool; it -> never edits it.) +### Step 3. Move each WorkerPool to the new worker image -> [!WARNING] -> **On GKE, do not touch the node pool's label until every node is -> rolled.** A pool label update applies in place to every node in the -> pool, so the whole fleet flips at once, with no drain and no pacing. -> (After the roll) - -## Upgrade +> [!IMPORTANT] +> Do not start step 3 until step 2 is done: new workers need the new atelet. -### 1. Apply the new CRDs +Change the WorkerPool, not its Deployment: `ate-controller` overwrites edits to the Deployment. If your WorkerPools live in Git or are applied by CI, change `workerImage` there as well, or the next sync puts the old image back and rolls every worker again. -From the new release's checkout, apply its CRDs. Nothing running -changes; the new schema is in place for the controller that follows. +Before you patch any pool, list the pools and save the list for rollback. Each pool needs the new image for its sandbox CLASS: ```bash -# in a checkout of the new release -kubectl apply -f manifests/ate-install/generated +kubectl get workerpools -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.sandboxClasses[0].name,IMAGE:.spec.workerImage' | tee ~/ate-upgrade/before-workerpools.txt ``` -### 2. Upgrade ate-controller +If you build from source, build and push both worker images once. This touches nothing in the cluster and ends by printing `ateom-gvisor: ` and `ateom-microvm: `: ```bash -go run ./cmd/ate-setup deploy ate-controller +go run ./cmd/ate-setup publish worker-images --config ~/ate-upgrade/new.yaml | tee ~/ate-upgrade/worker-images.txt ``` -This rolls `ate-controller` (and re-applies the CRDs, which is a -no-op now). +Then move the pools, starting with a small pool of each CLASS that you can afford to disrupt. For each pool: -By convention, changes to how the controller renders worker -pods sit behind WorkerPool fields, so the new controller keeps -rendering the serving pools as they are. A release that breaks that -convention says so in its notes. Expect the pools' Deployments to -roll once here in that case. Every actor is suspended through the -worker eviction path, loses no state, and resumes on demand. If they -roll, wait for `READY` to equal `DESIRED` again on every serving pool -(`kubectl get workerpools -A`) before step 4. +1. Set the pool: -The move of the node state root from `/var/lib/ateom-gvisor` to -`/var/lib/ate` is such a release. A worker that lands on a node whose -atelet still uses the old path reaches it only once step 5 moves that node. -The old directory can be deleted afterwards. + ```bash + NS=my-namespace # change me + WORKERPOOL=my-pool # change me + CLASS=$(kubectl -n $NS get workerpool $WORKERPOOL -o jsonpath='{.spec.sandboxClasses[0].name}') + ``` -### 3. Prepare the new dataplane +2. Set its new image. From source, take the ref after `ateom-$CLASS: ` in `~/ate-upgrade/worker-images.txt`: -The dataplane roll starts with the new atelet DaemonSet, from the -same checkout: + ```bash + NEW_IMAGE=$(sed -n "s/^ateom-$CLASS: //p" ~/ate-upgrade/worker-images.txt) + ``` -```bash -go run ./cmd/ate-setup deploy atelet -``` + With prebuilt images, take `ateom-$CLASS` from your image repo at the new tag, pinned by digest: -It lands next to the old one with zero pods until step 5 flips a -node. + ```bash + NEW_IMAGE=$(sed -n 's/^ repo: //p' ~/ate-upgrade/new.yaml)/ateom-$CLASS:$NEW_RELEASE + NEW_IMAGE=$NEW_IMAGE@$(crane digest $NEW_IMAGE) + ``` -Then read `$NEW_VERSION` off the cluster: +3. Check that `echo "$NS/$WORKERPOOL $CLASS $NEW_IMAGE"` shows `ateom-` plus the pool's CLASS and a full `@sha256:` digest. A bad image still removes the pool's first old worker. +4. Patch the worker pool and wait until ate-controller moves its Deployment to the new image: -```bash -kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version -``` + ```bash + kubectl -n $NS patch workerpool $WORKERPOOL --type merge -p "{\"spec\":{\"workerImage\":\"$NEW_IMAGE\"}}" + kubectl -n $NS wait deployment/$WORKERPOOL --for=jsonpath='{.spec.template.spec.containers[0].image}'="$NEW_IMAGE" --timeout=2m + ``` + +5. Once the first pool of a CLASS has one new worker running, patch the other pools of that CLASS without waiting for the roll to finish. A new worker is running when the newest ReplicaSet of the pool shows READY 1 or more: + + ```bash + kubectl -n $NS get rs -l ate.dev/worker-pool=$WORKERPOOL + ``` + + Then repeat on the other CLASS. + +What to expect: + +- **Progress:** `kubectl -n $NS rollout status deployment/$WORKERPOOL` prints how many new workers are updated and available. Ctrl-C stops only the watch. +- **Done:** Step 3 is done when every pool is done rolling with `kubectl get deploy -A -l ate.dev/worker-pool`. Go to Step 4 once done. +- **Pace:** a WorkerPool replaces 10% of its workers per batch, rounded down, at least one. With spare capacity, every batch has its `SIGTERM` within minutes and the old workers drain in parallel. Without it, each batch waits up to 30 minutes for old workers to exit. +- **Stuck:** rollout status fails with `exceeded its progress deadline` (the roll goes on), or a worker pod stays Terminating for more than 60 minutes, its grace period (a worker normally exits within about 30). Check `kubectl -n $NS get pods -l ate.dev/worker-pool=$WORKERPOOL`. A bad image stops the roll after the first batch: [roll back that pool](#roll-back-step-3). +- **Actors:** each gets `SIGTERM` and 30 minutes to be suspended. Substrate does not suspend them. An actor that is not suspended by then becomes `CRASHED` and loses its work since its last snapshot. [Eviction](api-guide.md#eviction) in the API guide says what an actor needs to keep its state. + +### Step 4. Upgrade the Substrate control plane -Exactly two DaemonSets print. The SUBSTRATE-VERSION that is not `$OLD_VERSION` is -`$NEW_VERSION`, and the new DaemonSet shows 0 `DESIRED` because no node carries -its label yet. `kubectl get nodes -L ate.dev/substrate-version` still -shows every node at `$OLD_VERSION`. If only one DaemonSet prints, the -version did not change (see Checkout and environment) and the command -rolled the running atelet in place. +#### 4a. ate-api-server -Last, build and push the new worker images. The refs print together -at the end. The one for your pool's `sandboxClasses[].name` goes into the clone -in step 4: +The first new API server pod migrates the database schema, and rolling back does not undo that. Back up the database first: with `pg_dump` for the bundled one, or an on-demand backup for Cloud SQL. With an external database, check that `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING` is set in this shell: without it, `deploy apiserver` points the API server at the bundled database. ```bash -go run ./cmd/ate-setup publish worker-images +go run ./cmd/ate-setup deploy apiserver --config ~/ate-upgrade/new.yaml +kubectl -n ate-system rollout status deploy/ate-api-server +kubectl ate get actors -A ``` -If you are using prebuilt images there is nothing to publish. The new worker image -is the release's `ateom-` image under the same repo and -tag as the control plane, pinned by digest: +- **Done:** rollout status prints `successfully rolled out` and `kubectl ate get actors -A` lists your actors. +- **Stuck:** rollout status does not finish, or a new pod keeps restarting. [Roll back step 4](#roll-back-step-4). +- **API clients:** an API call can fail once with a retriable error while an API server pod restarts. Retry it. + +#### 4b. CRDs and ate-controller ```bash -NEW_IMAGE=$IMAGE_REPO/ateom-gvisor:$IMAGE_TAG@$(crane digest $IMAGE_REPO/ateom-gvisor:$IMAGE_TAG) +go run ./cmd/ate-setup deploy ate-controller --config ~/ate-upgrade/new.yaml +kubectl get deploy -A -l ate.dev/worker-pool ``` -### 4. Create the new pool +This applies the new CRDs and ate-controller's RBAC, then deploys the new `ate-controller`. Rerunning is safe. If this release changes the worker pod template, the new ate-controller rolls every pool again, with another round of `SIGTERM`: wait as in step 3. -**Repeat this step for every serving pool the checklist listed**. + -> [!WARNING] -> Clone the old pool. Do not edit it. An edit rolls its Deployment -> straight through the live actors on it (second warning above). +- **Done:** the command exits 0, `kubectl -n ate-system get pods -l app=ate-controller` shows one pod, Running, with RESTARTS 0, and every pool shows READY `n/n` with UP-TO-DATE and AVAILABLE `n`. Check the pools a minute after the new pod starts: a second roll begins only once it reconciles them. +- **Stuck:** the new ate-controller pod keeps restarting. [Roll back step 4](#roll-back-step-4). -Copy the old pool under a new name (`$NEW_WORKERPOOL`, for example -the old name plus `$NEW_VERSION`) and change only `workerImage` and -the version pin. The command below does exactly that; everything -else, including the `metadata.labels` the scheduler matches actors -by, carries over as is. +#### 4c. atenet ```bash -NEW_IMAGE= -# Only read when the old pool predates spec.sandboxClasses. -SANDBOX_CLASS= - -kubectl -n $NS get workerpool $OLD_WORKERPOOL -o json \ - | jq --arg name "$NEW_WORKERPOOL" --arg image "$NEW_IMAGE" --arg version "$NEW_VERSION" \ - --arg class "$SANDBOX_CLASS" ' - {apiVersion, kind, - metadata: {name: $name, namespace: .metadata.namespace, - labels: .metadata.labels}, - spec: (.spec + {workerImage: $image})} - | .spec.template.nodeSelector["ate.dev/substrate-version"] = $version - | .spec.sandboxClasses //= [{name: $class}] - ' \ - | kubectl apply -f - +go run ./cmd/ate-setup deploy atenet --config ~/ate-upgrade/new.yaml +kubectl -n ate-system rollout status deploy/atenet-router +kubectl -n ate-system rollout status deploy/atenet-egress ``` -Do not shrink `spec.template.resources.limits` while cloning. A -worker only accepts an actor whose limits fit under them. +- **Done:** both print `successfully rolled out`. +- **Stuck:** rollout status does not finish, or a new pod keeps restarting. [Roll back step 4](#roll-back-step-4). +- **Actors:** a request to an actor, or an outbound connection, still open about 15 seconds after its `atenet` pod starts draining can be cut once. -Verify: +#### 4d. SandboxConfig ```bash -# $NEW_VERSION, then an image ref that contains @sha256: -kubectl -n $NS get workerpool $NEW_WORKERPOOL \ - -o jsonpath='{.spec.template.nodeSelector.ate\.dev/substrate-version}{"\n"}{.spec.workerImage}{"\n"}' - -# One Deployment per pool; new-pool pods all Pending (no node carries -# the new label yet). -kubectl -n $NS get deploy -l ate.dev/worker-pool -kubectl -n $NS get pods -l ate.dev/worker-pool=$NEW_WORKERPOOL +go run ./cmd/ate-setup deploy sandboxconfig --config ~/ate-upgrade/new.yaml ``` -While both pools serve, placement between them is random, and that -is fine: a snapshot written on either version restores on either -version. The roll converges because step 5 takes old workers out of -service node by node, not because the scheduler prefers the new pool. +This applies the SandboxConfig admission policy and the default SandboxConfig. It does not roll workers. -### 5. Roll each node + -Repeat for every node, one at a time. On a single-node cluster this -step is a full stop: every actor is suspended at once. +- **Done:** the command exits 0. +- **Stuck:** the command fails. [Roll back step 4](#roll-back-step-4). -**a. Drain the node's workers.** Bound actors keep running. Draining -only stops new placements. +#### 4e. The rest of the system bundle ```bash -# A worker's name is its pod's UID; that is what DrainWorker takes. -for w in $(kubectl ate get workers -o json \ - | jq -r --arg node "$NODE" '.workers[] | select(.nodeName == $node) | .metadata.name'); do - grpcurl -cacert /tmp/ate-ca.pem -authority api.ate-system.svc \ - -H "authorization: Bearer ${TOKEN}" \ - -d "{\"worker\": {\"name\": \"${w}\"}}" \ - 127.0.0.1:8443 ateapi.Control/DrainWorker -done +go run ./cmd/ate-setup deploy ate-system --config ~/ate-upgrade/new.yaml --setup-csi=none ``` -**b. See what is still on the node.** Two lists: the node's workers -with the actor each one hosts, and any paused actor whose local -snapshot lives on this node. A paused actor sits on no worker, so it -shows up only in the second list. +This applies what the steps above do not, such as the ate-system NetworkPolicies and the bundled PostgreSQL. The components above are already on the new release and do not roll again. `--setup-csi=none` leaves the CSI drivers as they are. Rerunning is safe. -```bash -kubectl ate get workers -o json | jq -r --arg node "$NODE" ' - ["WORKER", "POD", "ASSIGNED ACTOR"], - (.workers[] | select(.nodeName == $node) - | [.metadata.name, .workerPod, - (.status.assignment.actor | if . then .atespace + "/" + .name else "" end)]) - | @tsv' | column -t -s $'\t' - -kubectl ate get actors -A -o json | jq -r --arg node "$NODE" ' - ["PAUSED_ACTOR", "STATE"], - (.actors[] - | select(.status.assignedNode == $node and (.status.workerAssignment == null)) - | [.metadata.atespace + "/" + .metadata.name, .status.state]) - | @tsv' | column -t -s $'\t' -``` +- **Done:** the command exits 0. +- **Stuck:** the command fails. [Roll back step 4](#roll-back-step-4). +- **API clients:** if the release changes the bundled PostgreSQL, it restarts once, and API calls fail until it is back. -**c. Suspend them at your own pace.** Every actor in either list has to be suspended -before the node moves (`kubectl ate suspend actor -a -`). Suspend releases the worker and uploads the durable -snapshot, so the actor resumes on demand onto any free matching worker -afterwards. Repeat step b until the `ASSIGNED ACTOR` column reads -`` throughout and the paused list is empty. Then rerun the drain -in step a once more right before flipping. +Then install the new `kubectl ate` with `go install ./cmd/kubectl-ate`. Actor owners can now use API fields and commands new in this release, and you can set WorkerPool, SandboxConfig and CSIDriverConfig fields new in this release. -> [!WARNING] -> Do not run e before d. Deleting the old-pool pods while the node -> still carries `$OLD_VERSION` lets the old pool put replacements right -> back on this node, next to the new atelet (first warning above). +The upgrade is done when step 4 is done and [Where am I?](#where-am-i) shows every component and pool moved, every pool done rolling and none paused. Keep `~/ate-upgrade` while you might still roll back, then delete it. -**d. Flip the label.** The old atelet pod leaves on its own and the -new one starts. +## Something's wrong, or I need to roll back -```bash -kubectl label node $NODE ate.dev/substrate-version=$NEW_VERSION --overwrite -``` +Stopping between steps is safe, and so is stopping partway through step 3. A pool you have patched keeps rolling unless you pause it. + +### Pause a pool's roll -Wait for the new `atelet` pod on the node to be Ready before going on. Usually that is seconds; the new `atelet` pod stays Pending only while the old one is still exiting, since it holds the node's host ports until then. +If a pool's new workers look wrong but you are not ready to roll it back, or you want to stop interrupting actors for a while, pause its Deployment. ate-controller leaves this one Deployment change in place. ```bash -# The pod is named after atelet-. -kubectl get pods -n ate-system -l app=atelet --field-selector spec.nodeName=$NODE +kubectl -n $NS rollout pause deployment/$WORKERPOOL +kubectl -n $NS rollout resume deployment/$WORKERPOOL ``` -**e. Delete the node's old-pool worker pods.** Step c emptied them, -but they are still Ready and hold capacity the new pool needs on this -node. Their Deployment cannot reschedule them here anymore. Repeat -per pool if the node hosts several. +Workers already removed still get their 30 minutes. Do not leave a pool paused, and do not start step 4 while one is: a paused pool ignores every later change to its workers. + +### Roll back + +To go back to the old release, roll back every step you began, finished or not, starting with the last. To fix one pool, roll back only that pool. Rollback runs from the old checkout, with `old.yaml`: ```bash -kubectl -n $NS delete pod -l ate.dev/worker-pool=$OLD_WORKERPOOL \ - --field-selector spec.nodeName=$NODE +cd ~/ate-upgrade/old ``` -**f. Confirm the node has moved.** Two checks: +#### Roll back step 4 -```bash -# New-pool workers came up here. -kubectl -n $NS get pods -l ate.dev/worker-pool=$NEW_WORKERPOOL --field-selector spec.nodeName=$NODE +Tell actor owners to stop using API fields new in this release, because the old API server rejects them. Then: -# No old-pool pod is left here. -kubectl -n $NS get pods -l ate.dev/worker-pool=$OLD_WORKERPOOL --field-selector spec.nodeName=$NODE +```bash +go run ./cmd/ate-setup deploy sandboxconfig --config ~/ate-upgrade/old.yaml +go run ./cmd/ate-setup deploy atenet --config ~/ate-upgrade/old.yaml +go run ./cmd/ate-setup deploy ate-controller --config ~/ate-upgrade/old.yaml +go run ./cmd/ate-setup deploy apiserver --config ~/ate-upgrade/old.yaml +go install ./cmd/kubectl-ate ``` -New-pool pods Pending on other nodes are expected until those nodes -move. - -**g. Take the next node.** Start again at step a once `kubectl ate get -workers` shows at least one FREE worker for the suspended actors to -land on. +`deploy ate-controller` restores the old CRDs, RBAC and `ate-controller`. If the release changed the worker pod template, every pool rolls again: wait as in step 3. Fields the new release added to WorkerPools, SandboxConfigs or CSIDriverConfigs are dropped from those objects. Remove them from your YAML files too, or `kubectl apply` rejects the files. -You are done when `kubectl get nodes -L ate.dev/substrate-version` -shows every node at `$NEW_VERSION` (a node that joined mid-roll still -carries `$OLD_VERSION`; apply step 5 to roll it) and every assigned -actor in `kubectl ate get actors -A` sits on a new-pool pod. +The database schema stays new; the old API server drops fields it does not know when it rewrites a record. Step 4e is not undone either: the NetworkPolicies and the bundled PostgreSQL stay as the new release left them. -### 6. Upgrade the rest of the control plane +#### Roll back step 3 -Every actor is now on the new dataplane, running or suspended, and -ate-controller moved in step 2. From the same checkout, move -`ate-api-server` first, then everything else: +Set each patched pool back to its image in `~/ate-upgrade/before-workerpools.txt`: ```bash -go run ./cmd/ate-setup deploy apiserver -go run ./cmd/ate-setup deploy ate-system --credential-provider='' +NS=my-namespace # change me +WORKERPOOL=my-pool # change me +OLD_IMAGE=$(awk -v ns=$NS -v name=$WORKERPOOL '$1 == ns && $2 == name {print $4}' ~/ate-upgrade/before-workerpools.txt) +kubectl -n $NS patch workerpool $WORKERPOOL --type merge -p "{\"spec\":{\"workerImage\":\"$OLD_IMAGE\"}}" ``` -`deploy ate-system` requires the credential provider selection; pass the one -the install was made with, or carry it in `ATE_CREDENTIAL_PROVIDER`. An install -made with `--experimental-egress-credential-injection` and the bundled provider -is `{"name":"k8s.io"}`; one without injection is `{"enabled":false}`. The -selection replaces `--credential-provider-name` and -`--credential-provider-address`: a provider you deploy yourself is -`{"name":"","address":":"}`, with the name bare -(`vault.example.com`, not `ate-secret://vault.example.com`). +If you paused a WorkerPool, resume it now with `kubectl -n $NS rollout resume deployment/$WORKERPOOL`. Then watch: -The second command rolls atenet and converges the rest of the -install, postgres included: on a cordoned install it moves postgres to -its own pool (see [Cordoned control plane: postgres -pool](#cordoned-control-plane-postgres-pool)); it re-resolves and re-applies everything, so it could take a -while. The checklist's port-forward dies when the API server rolls. -Restart it and mint a fresh token if you still need to drain. +```bash +kubectl -n $NS rollout status deployment/$WORKERPOOL +``` -**NOTE**: Until -this step is done the old ate-api-server is still serving, so do not -start using API fields new in this release before upgrade finishes. +Only workers already on the new image are replaced, and their actors get `SIGTERM` again. Wait as in [step 3](#step-3-move-each-workerpool-to-the-new-worker-image). -### After the roll, on GKE +#### Roll back step 2 > [!WARNING] -> Only after every node shows `$NEW_VERSION` in [Check your -> progress](#check-your-progress). Relabeling the node pool applies to -> every node in it at once, with no drain and no pacing. - -Relabel the node pool with `$NEW_VERSION` so nodes created later start -at the new version, then restore the autoscaler from the config you -saved [before the roll](#on-gke): +> Do not roll back step 2 before every pool is back on its old image and `kubectl get pods -A -l ate.dev/worker-pool` shows no Terminating pod. A new worker with an old atelet is not supported. ```bash -# --node-labels REPLACES the pool's full user label set: list the -# current labels first and carry them all over. -gcloud container node-pools describe $NODEPOOL --cluster $CLUSTER --zone $ZONE \ - --format='get(config.labels)' # copy the output - -gcloud container node-pools update $NODEPOOL --cluster $CLUSTER --zone $ZONE \ - --node-labels=,ate.dev/substrate-version=$NEW_VERSION - -# restore the autoscaler config -gcloud container clusters update $CLUSTER --zone $ZONE --node-pool $NODEPOOL \ - --enable-autoscaling --min-nodes --max-nodes +go run ./cmd/ate-setup deploy atelet --config ~/ate-upgrade/old.yaml --rollout-timeout 1h ``` -## Something's wrong, or I need to undo - -First run [Check your progress](#check-your-progress). It tells you -which step you reached, and everything below is keyed on that. - -If a command failed, rerun it: every step is idempotent. The one -exception is drain. There is no undrain; if you drained the wrong -node, check that its workers are empty (suspending as needed), delete -their pods, and let the Deployment's replacements register as fresh -workers. - -To roll back, undo what you did in reverse order: the control plane -first, then the nodes, then ate-controller last. The roll never edits -or deletes the old objects, so the old `WorkerPool` is intact, the pods -that lost their nodes stay Pending, and the old atelet DaemonSet is -still installed. Rolling nodes back is a matter of label flips. Run the -commands from the old release's checkout with the same environment as -the install, `VERSION` included if the install pinned it. - -- Relabeled the GKE node pool: park the autoscaler again as - [before the roll](#on-gke), then move the node pool label back to - `$OLD_VERSION` with the [after the roll](#after-the-roll-on-gke) - command. -- Past step 6: `go run ./cmd/ate-setup deploy apiserver` now, and - `go run ./cmd/ate-setup deploy ate-system` once the nodes are back. -- Past step 5: roll each flipped node back by running step 5 with the - sides swapped: drain, get every actor off the node as in b and c, - flip the label back to `$OLD_VERSION`, and delete the node's - new-pool pods. -- Past step 4: once no actor is assigned to a new-pool worker, delete - each clone: `kubectl -n $NS delete workerpool $NEW_WORKERPOOL`. -- Past step 3: once no node carries `$NEW_VERSION`, delete the new - DaemonSet: `kubectl delete daemonset -n ate-system -l app=atelet,ate.dev/substrate-version=$NEW_VERSION`. -- Past step 2: `go run ./cmd/ate-setup deploy ate-controller`. - -`kubectl get ds -n ate-system -l app=atelet` must not show a third -DaemonSet afterwards. A third means the old checkout produced a -version other than `$OLD_VERSION`: delete it and check `VERSION`. - -Abandoning the upgrade entirely is the whole list, top to bottom. -Retiring the old pool (below) is a separate, deliberate step. As long -as the old objects exist, rollback is one label flip per node. - -## Retire the old pool - -After the new version has soaked, reclaim the reserve. Save the old -pool's spec first: its `workerImage` ref is by digest and stays -pullable, so the saved file is the last-resort way to recreate the -pool. Then check the guards. Deleting the old objects is what ends -the rollback option. +#### Roll back step 1 ```bash -kubectl -n $NS get workerpool $OLD_WORKERPOOL -o yaml > old-pool-backup.yaml - -# Guards: no node still at $OLD_VERSION; no old-pool pod Running (Pending is -# expected); no actor assigned to an old-pool worker. -kubectl get nodes -l ate.dev/substrate-version=$OLD_VERSION -kubectl -n $NS get pods -l ate.dev/worker-pool=$OLD_WORKERPOOL -kubectl ate get workers - -# Retire the old pool (its Deployment and pods go with it) and the -# old atelet DaemonSet. -kubectl -n $NS delete workerpool $OLD_WORKERPOOL -kubectl delete daemonset -n ate-system -l app=atelet,ate.dev/substrate-version=$OLD_VERSION +go run ./cmd/ate-setup deploy podcertificate-controller --config ~/ate-upgrade/old.yaml ``` - -The new pool keeps its name. Names mean nothing to placement, so -`counter-v2` can serve indefinitely, and the next upgrade clones it -to `counter-v3`. diff --git a/internal/actoridjwt/actoridjwt.go b/internal/actoridjwt/actoridjwt.go index 4a40b95974..a0701ba0c4 100644 --- a/internal/actoridjwt/actoridjwt.go +++ b/internal/actoridjwt/actoridjwt.go @@ -20,7 +20,53 @@ import ( "time" ) +// Kind is who a token speaks for. The two kinds carry the same fields but +// under different subjects and claim names, so a relying party can tell an +// actor's own code from a Substrate component acting on the actor's behalf +// and grant them different access. +type Kind int + +const ( + // KindUnspecified is the zero value. ClaimsToWire rejects it, so a Claims + // that forgets to set Kind cannot mint a token of either kind. + KindUnspecified Kind = iota + // KindActor is the actor itself. Subject "actor//", + // claims under "actor.ate.dev". + KindActor + // KindAteomForActor is an ateom, or another Substrate component, acting + // on behalf of the actor. Subject "ateom-for-actor//", + // claims under "ateom-for-actor.ate.dev". + KindAteomForActor +) + +// SubjectPrefix is the first segment of the sub claim for this kind. +func (k Kind) SubjectPrefix() string { + switch k { + case KindActor: + return "actor" + case KindAteomForActor: + return "ateom-for-actor" + } + return "" +} + +// ClaimName is the JWT claim the Substrate fields are stored under for this +// kind. +func (k Kind) ClaimName() string { + switch k { + case KindActor: + return "actor.ate.dev" + case KindAteomForActor: + return "ateom-for-actor.ate.dev" + } + return "" +} + type Claims struct { + // Kind selects the subject prefix and the claim name the Substrate + // fields are stored under. + Kind Kind + // Claims from RFC7519 Issuer string Subject string @@ -50,8 +96,22 @@ type WireClaims struct { IssuedAt float64 `json:"iat,omitempty"` JTI string `json:"jti,omitempty"` - // Claims from ADK's session model. - Substrate WireSubstrateClaims `json:"ate.dev,omitempty"` + // Substrate claims, under the claim name for the token's kind. Exactly + // one is set. + Actor *WireSubstrateClaims `json:"actor.ate.dev,omitempty"` + AteomForActor *WireSubstrateClaims `json:"ateom-for-actor.ate.dev,omitempty"` +} + +// Substrate returns the token's kind and its Substrate claims, or ok false if +// the token carries neither claim or both. +func (w *WireClaims) Substrate() (kind Kind, claims *WireSubstrateClaims, ok bool) { + switch { + case w.Actor != nil && w.AteomForActor == nil: + return KindActor, w.Actor, true + case w.AteomForActor != nil && w.Actor == nil: + return KindAteomForActor, w.AteomForActor, true + } + return 0, nil, false } type WireSubstrateClaims struct { @@ -74,11 +134,19 @@ func ClaimsToWire(claims *Claims) (*WireClaims, error) { NotBefore: float64(claims.NotBefore.Unix()), IssuedAt: float64(claims.IssuedAt.Unix()), JTI: claims.JTI, - Substrate: WireSubstrateClaims{ - Atespace: claims.Substrate.Atespace, - ActorName: claims.Substrate.ActorName, - ActorUID: claims.Substrate.ActorUID, - }, + } + substrate := &WireSubstrateClaims{ + Atespace: claims.Substrate.Atespace, + ActorName: claims.Substrate.ActorName, + ActorUID: claims.Substrate.ActorUID, + } + switch claims.Kind { + case KindActor: + wire.Actor = substrate + case KindAteomForActor: + wire.AteomForActor = substrate + default: + return nil, fmt.Errorf("token kind %d is not one of KindActor or KindAteomForActor", claims.Kind) } return wire, nil diff --git a/internal/actoridjwt/actoridjwt_test.go b/internal/actoridjwt/actoridjwt_test.go new file mode 100644 index 0000000000..9b68bee517 --- /dev/null +++ b/internal/actoridjwt/actoridjwt_test.go @@ -0,0 +1,128 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actoridjwt + +import ( + "encoding/json" + "testing" + "time" +) + +func TestClaimsToWire_ClaimNamePerKind(t *testing.T) { + base := Claims{ + Issuer: "https://idp.ate-system.svc", + Subject: "x", + Audiences: []string{"a"}, + Expiration: time.Unix(1_800_000_000, 0), + Substrate: SubstrateClaims{Atespace: "team-a", ActorName: "my-actor", ActorUID: "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21"}, + } + for _, tc := range []struct { + kind Kind + wantClaim string + otherClaim string + }{ + {KindActor, "actor.ate.dev", "ateom-for-actor.ate.dev"}, + {KindAteomForActor, "ateom-for-actor.ate.dev", "actor.ate.dev"}, + } { + claims := base + claims.Kind = tc.kind + wire, err := ClaimsToWire(&claims) + if err != nil { + t.Fatalf("ClaimsToWire(%v) failed: %v", tc.kind, err) + } + payload, err := json.Marshal(wire) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var raw map[string]json.RawMessage + if err := json.Unmarshal(payload, &raw); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if _, ok := raw[tc.wantClaim]; !ok { + t.Errorf("kind %v: claim %q missing from %s", tc.kind, tc.wantClaim, payload) + } + for _, absent := range []string{tc.otherClaim, "ate.dev"} { + if _, ok := raw[absent]; ok { + t.Errorf("kind %v: unexpected claim %q in %s", tc.kind, absent, payload) + } + } + var sub WireSubstrateClaims + if err := json.Unmarshal(raw[tc.wantClaim], &sub); err != nil { + t.Fatalf("kind %v: decode %q: %v", tc.kind, tc.wantClaim, err) + } + if want := (WireSubstrateClaims{Atespace: "team-a", ActorName: "my-actor", ActorUID: "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21"}); sub != want { + t.Errorf("kind %v: %q = %+v, want %+v", tc.kind, tc.wantClaim, sub, want) + } + } +} + +func TestClaimsToWire_RejectsUnspecifiedKind(t *testing.T) { + for _, kind := range []Kind{KindUnspecified, Kind(99)} { + if _, err := ClaimsToWire(&Claims{Kind: kind, Subject: "x", Audiences: []string{"a"}}); err == nil { + t.Errorf("ClaimsToWire(kind %v) succeeded, want error", kind) + } + } +} + +func TestWireClaims_Substrate(t *testing.T) { + actor := &WireSubstrateClaims{Atespace: "a", ActorName: "n", ActorUID: "u"} + ateom := &WireSubstrateClaims{Atespace: "a2", ActorName: "n2", ActorUID: "u2"} + for _, tc := range []struct { + name string + wire WireClaims + wantOK bool + wantKind Kind + wantClaims *WireSubstrateClaims + }{ + {name: "neither claim", wire: WireClaims{}}, + {name: "both claims", wire: WireClaims{Actor: actor, AteomForActor: ateom}}, + {name: "actor", wire: WireClaims{Actor: actor}, wantOK: true, wantKind: KindActor, wantClaims: actor}, + {name: "ateom-for-actor", wire: WireClaims{AteomForActor: ateom}, wantOK: true, wantKind: KindAteomForActor, wantClaims: ateom}, + } { + t.Run(tc.name, func(t *testing.T) { + kind, claims, ok := tc.wire.Substrate() + if ok != tc.wantOK { + t.Fatalf("ok = %v, want %v", ok, tc.wantOK) + } + if !ok { + return + } + if kind != tc.wantKind { + t.Errorf("kind = %v, want %v", kind, tc.wantKind) + } + if claims != tc.wantClaims { + t.Errorf("claims = %+v, want %+v", claims, tc.wantClaims) + } + }) + } +} + +func TestKind_Names(t *testing.T) { + for _, tc := range []struct { + kind Kind + prefix, claim string + }{ + {KindUnspecified, "", ""}, + {KindActor, "actor", "actor.ate.dev"}, + {KindAteomForActor, "ateom-for-actor", "ateom-for-actor.ate.dev"}, + } { + if got := tc.kind.SubjectPrefix(); got != tc.prefix { + t.Errorf("Kind(%d).SubjectPrefix() = %q, want %q", tc.kind, got, tc.prefix) + } + if got := tc.kind.ClaimName(); got != tc.claim { + t.Errorf("Kind(%d).ClaimName() = %q, want %q", tc.kind, got, tc.claim) + } + } +} diff --git a/internal/ateattr/ateattr.go b/internal/ateattr/ateattr.go index da02cbb244..c9c4bce479 100644 --- a/internal/ateattr/ateattr.go +++ b/internal/ateattr/ateattr.go @@ -27,8 +27,8 @@ import ( "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" - atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/agent-substrate/substrate/pkg/sandboxclass" ) // Dotted ate.* matches the metric-instrument naming (atenet.*, atelet.*), not the @@ -373,12 +373,10 @@ const SandboxClassUnknown = "unknown" // Empty reports as unknown rather than the gvisor default, so a manifest // problem stays visible. func NormalizeSandboxClass(class string) string { - switch atev1alpha1.SandboxClass(class) { - case atev1alpha1.SandboxClassGvisor, atev1alpha1.SandboxClassMicroVM: + if sandboxclass.Valid(class) { return class - default: - return SandboxClassUnknown } + return SandboxClassUnknown } // SandboxClassAttribute sets ate.sandbox.class. No source of the class is diff --git a/internal/ateominit/ateominit.go b/internal/ateominit/ateominit.go new file mode 100644 index 0000000000..2b4cc0a890 --- /dev/null +++ b/internal/ateominit/ateominit.go @@ -0,0 +1,114 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package ateominit runs a minimal init as PID 1 so ateom doesn't have to. +// +// PID 1 inherits every orphan (e.g. the runsc sandbox once `runsc create` +// exits) and must reap them with wait4(-1), which races os/exec for the exit +// status of ateom's own subprocesses. The init re-executes the binary as its +// only child, reaps everything, and exits with the child's status. +package ateominit + +import ( + "fmt" + "os" + "os/signal" + + "golang.org/x/sys/unix" +) + +// forwarded are the signals passed through to ateom. +var forwarded = []os.Signal{ + unix.SIGTERM, unix.SIGINT, unix.SIGHUP, unix.SIGQUIT, unix.SIGUSR1, unix.SIGUSR2, +} + +// RunIfPID1 runs the init and exits if the process is PID 1, and otherwise +// returns. Call it first thing in main. +func RunIfPID1() { + if os.Getpid() != 1 { + return + } + exe, err := os.Executable() + if err != nil { + fmt.Fprintf(os.Stderr, "ateominit: finding the executable: %v\n", err) + os.Exit(1) + } + os.Exit(supervise(exe, os.Args)) +} + +// supervise starts path as a child, reaps until it exits, and returns its exit +// code. +func supervise(path string, argv []string) int { + // A no-op as PID 1, but it lets tests run this as an ordinary process. + if err := unix.Prctl(unix.PR_SET_CHILD_SUBREAPER, 1, 0, 0, 0); err != nil { + fmt.Fprintf(os.Stderr, "ateominit: setting child subreaper: %v\n", err) + return 1 + } + + // Subscribe before starting the child so no exit goes unnoticed. + sigs := make(chan os.Signal, 32) + signal.Notify(sigs, append([]os.Signal{unix.SIGCHLD}, forwarded...)...) + defer signal.Stop(sigs) + + child, err := os.StartProcess(path, argv, &os.ProcAttr{ + Env: os.Environ(), + Files: []*os.File{os.Stdin, os.Stdout, os.Stderr}, + }) + if err != nil { + fmt.Fprintf(os.Stderr, "ateominit: starting %s: %v\n", path, err) + return 1 + } + + for sig := range sigs { + if sig != unix.SIGCHLD { + _ = child.Signal(sig) + } + // Reap after every signal: os/signal drops signals when sigs is full, + // so the SIGCHLD for the child's exit may never arrive. + if status, exited := reap(child.Pid); exited { + return exitCode(status) + } + } + panic("unreachable") +} + +// reap collects every exited child and reports whether pid was among them. +// SIGCHLD coalesces, so one signal may stand for several exits. +func reap(pid int) (unix.WaitStatus, bool) { + var childStatus unix.WaitStatus + childExited := false + for { + var status unix.WaitStatus + got, err := unix.Wait4(-1, &status, unix.WNOHANG, nil) + if err == unix.EINTR { + continue + } + if err != nil || got <= 0 { + return childStatus, childExited + } + if got == pid { + childStatus, childExited = status, true + } + } +} + +// exitCode maps a wait status to a shell-style exit code. +func exitCode(status unix.WaitStatus) int { + if status.Signaled() { + return 128 + int(status.Signal()) + } + return status.ExitStatus() +} diff --git a/internal/ateominit/ateominit_test.go b/internal/ateominit/ateominit_test.go new file mode 100644 index 0000000000..9cca5983f2 --- /dev/null +++ b/internal/ateominit/ateominit_test.go @@ -0,0 +1,76 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package ateominit + +import ( + "os" + "os/signal" + "testing" + "time" + + "golang.org/x/sys/unix" +) + +func TestSuperviseReturnsExitCode(t *testing.T) { + if got := supervise("/bin/sh", []string{"sh", "-c", "exit 3"}); got != 3 { + t.Fatalf("supervise = %d, want 3", got) + } +} + +func TestSuperviseReturnsSignalExitCode(t *testing.T) { + if got := supervise("/bin/sh", []string{"sh", "-c", "kill -KILL $$"}); got != 128+9 { + t.Fatalf("supervise = %d, want %d", got, 128+9) + } +} + +func TestSuperviseReapsOrphans(t *testing.T) { + // The inner sh is orphaned to us and exits while the child still runs. + if got := supervise("/bin/sh", []string{"sh", "-c", "(sh -c 'exit 0' &); sleep 0.5"}); got != 0 { + t.Fatalf("supervise = %d, want 0", got) + } + var status unix.WaitStatus + if pid, err := unix.Wait4(-1, &status, unix.WNOHANG, nil); err != unix.ECHILD { + t.Fatalf("Wait4 = (%d, %v), want ECHILD: a child was left unreaped", pid, err) + } +} + +func TestSuperviseForwardsSignals(t *testing.T) { + // Keep SIGTERM from killing the test before supervise subscribes. + ignore := make(chan os.Signal, 1) + signal.Notify(ignore, unix.SIGTERM) + defer signal.Stop(ignore) + + done := make(chan int) + go func() { done <- supervise("/bin/sh", []string{"sh", "-c", "exec sleep 30"}) }() + // Repeated, since the first may land before supervise subscribes. + tick := time.NewTicker(10 * time.Millisecond) + defer tick.Stop() + timeout := time.After(10 * time.Second) + for { + select { + case got := <-done: + if got != 128+int(unix.SIGTERM) { + t.Fatalf("supervise = %d, want %d", got, 128+int(unix.SIGTERM)) + } + return + case <-tick.C: + _ = unix.Kill(unix.Getpid(), unix.SIGTERM) + case <-timeout: + t.Fatal("SIGTERM was not forwarded to the child") + } + } +} diff --git a/internal/ateomnet/dns/evaluator.go b/internal/ateomnet/dns/evaluator.go new file mode 100644 index 0000000000..e85acf5319 --- /dev/null +++ b/internal/ateomnet/dns/evaluator.go @@ -0,0 +1,80 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package dns + +import ( + "github.com/agent-substrate/substrate/internal/egresspolicy" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" +) + +// evaluator decides whether a DNS hostname is allowed by an EgressPolicy's +// rules. +type evaluator struct { + patterns []egresspolicy.HostnamePattern +} + +// newEvaluator compiles the hostname patterns allowed by policy's rules. A nil +// policy, or one with no matching rules, allows nothing. Invalid patterns are +// ignored so they fail closed without breaking valid patterns alongside them. +func newEvaluator(policy *ateletpb.EgressPolicy) *evaluator { + e := &evaluator{} + var seen map[string]struct{} + for _, rule := range policy.GetRules() { + var dest *ateletpb.EgressDestinations + switch { + case rule.GetHttp() != nil: + dest = rule.GetHttp() + case rule.GetHttps() != nil: + dest = rule.GetHttps() + case rule.GetTlsPassthrough() != nil: + dest = rule.GetTlsPassthrough() + default: + continue + } + for _, raw := range dest.GetHostnames() { + if _, ok := seen[raw]; ok { + continue + } + pattern, err := egresspolicy.ParseHostnamePattern(raw) + if err != nil { + continue + } + if seen == nil { + seen = make(map[string]struct{}) + } + seen[raw] = struct{}{} + e.patterns = append(e.patterns, pattern) + } + } + return e +} + +// allowsHostname reports whether qname is a valid DNS hostname matched by at +// least one rule in the policy. +func (e *evaluator) allowsHostname(qname string) bool { + if e == nil || len(e.patterns) == 0 { + return false + } + host, ok := egresspolicy.NormalizeHostname(qname) + if !ok { + return false + } + for _, p := range e.patterns { + if p.Matches(host) { + return true + } + } + return false +} diff --git a/internal/ateomnet/dns/evaluator_test.go b/internal/ateomnet/dns/evaluator_test.go new file mode 100644 index 0000000000..d1e600ea3c --- /dev/null +++ b/internal/ateomnet/dns/evaluator_test.go @@ -0,0 +1,150 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package dns + +import ( + "testing" + + "github.com/agent-substrate/substrate/internal/proto/ateletpb" +) + +func TestEvaluator(t *testing.T) { + if (*evaluator)(nil).allowsHostname("example.com.") { + t.Error("nil evaluator allowed a hostname") + } + + tests := []struct { + name string + policy *ateletpb.EgressPolicy + allowed []string + blocked []string + }{ + { + name: "nil policy", + policy: nil, + blocked: []string{"example.com.", "api.example.com.", "."}, + }, + { + name: "empty policy", + policy: &ateletpb.EgressPolicy{}, + blocked: []string{"example.com.", "api.example.com.", "."}, + }, + { + name: "rule with no protocol", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{}}}, + blocked: []string{"example.com.", "api.example.com."}, + }, + { + name: "exact host across http, https, and tls_passthrough", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{ + {Http: &ateletpb.EgressDestinations{Hostnames: []string{"http.example.com"}}}, + {Https: &ateletpb.EgressDestinations{Hostnames: []string{"https.example.com"}}}, + {TlsPassthrough: &ateletpb.EgressDestinations{Hostnames: []string{"tls.example.com"}}}, + }}, + allowed: []string{ + "http.example.com.", + "http.example.com", + "HTTP.Example.COM.", + "https.example.com.", + "tls.example.com.", + }, + blocked: []string{ + "example.com.", + "sub.http.example.com.", + "other.example.com.", + "http.example.com..", + ".", + "", + }, + }, + { + name: "leftmost-label wildcard", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"*.example.com"}}, + }}}, + allowed: []string{ + "api.example.com.", + "API.Example.COM.", + "a-b.example.com.", + "1.example.com.", + }, + blocked: []string{ + "example.com.", + "a.b.example.com.", + "notexample.com.", + ".example.com.", + "_dmarc.example.com.", + "-bad.example.com.", + }, + }, + { + name: "star wildcard", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Http: &ateletpb.EgressDestinations{Hostnames: []string{"*"}}, + }}}, + allowed: []string{ + "example.com.", + "api.example.com.", + "a.b.c.example.org.", + "xn--bcher-kva.example.", + }, + blocked: []string{ + "", + ".", + "example.com..", + "192.0.2.1.", + "01.2.3.4.", + "_dmarc.example.com.", + "bücher.example.", + }, + }, + { + name: "invalid patterns are ignored while valid ones still match", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Http: &ateletpb.EgressDestinations{Hostnames: []string{ + "BAD.EXAMPLE.COM", + "*.", + "https://bad.example.com", + "good.example.com", + "*.wild.example.com", + }}, + }}}, + allowed: []string{ + "good.example.com.", + "a.wild.example.com.", + }, + blocked: []string{ + "bad.example.com.", + "wild.example.com.", + }, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + eval := newEvaluator(tc.policy) + for _, qname := range tc.allowed { + if !eval.allowsHostname(qname) { + t.Errorf("allowsHostname(%q) = false, want true", qname) + } + } + for _, qname := range tc.blocked { + if eval.allowsHostname(qname) { + t.Errorf("allowsHostname(%q) = true, want false", qname) + } + } + }) + } +} diff --git a/internal/ateomnet/dns/policy.go b/internal/ateomnet/dns/policy.go new file mode 100644 index 0000000000..39f2e3567b --- /dev/null +++ b/internal/ateomnet/dns/policy.go @@ -0,0 +1,69 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package dns + +import ( + "context" + "log/slog" + "sync" + + "golang.org/x/net/dns/dnsmessage" + + "github.com/agent-substrate/substrate/internal/proto/ateletpb" +) + +// PolicySource supplies the egress policy an actor's DNS queries are checked +// against. +type PolicySource interface { + // EgressPolicy returns the actor's current policy. + // + // Callers must not modify the policy. + EgressPolicy(ctx context.Context) (*ateletpb.EgressPolicy, error) +} + +// egressPolicyCheck returns the queryPolicy that checks each DNS question against +// the actor's egress policy. +func egressPolicyCheck(ctx context.Context, source PolicySource) queryPolicy { + if source == nil { + return func(dnsmessage.Question) bool { return false } + } + var ( + mu sync.Mutex + cached *ateletpb.EgressPolicy + eval *evaluator + ) + return func(q dnsmessage.Question) bool { + policy, err := source.EgressPolicy(ctx) + if err != nil { + slog.DebugContext(ctx, "dns relay refused a query; the actor's egress policy is unavailable", slog.Any("err", err)) + return false + } + mu.Lock() + if policy != cached || eval == nil { + cached = policy + eval = newEvaluator(policy) + } + e := eval + mu.Unlock() + return e.allowsHostname(q.Name.String()) + } +} + +// allowQuery reports whether policy lets the actor ask q. An actor without a +// policy may resolve nothing, as it may reach nothing through the egress +// gateway. +func allowQuery(policy *ateletpb.EgressPolicy, q dnsmessage.Question) bool { + return newEvaluator(policy).allowsHostname(q.Name.String()) +} diff --git a/internal/ateomnet/dns/policy_test.go b/internal/ateomnet/dns/policy_test.go new file mode 100644 index 0000000000..fa1b6f4efa --- /dev/null +++ b/internal/ateomnet/dns/policy_test.go @@ -0,0 +1,399 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package dns + +import ( + "context" + "errors" + "net" + "sync/atomic" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "golang.org/x/net/dns/dnsmessage" + + "github.com/agent-substrate/substrate/internal/ateomnet/dns/protocol" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" +) + +// staticPolicy supplies policy, or fails with err. +type staticPolicy struct { + policy *ateletpb.EgressPolicy + err error +} + +func (p staticPolicy) EgressPolicy(context.Context) (*ateletpb.EgressPolicy, error) { + return p.policy, p.err +} + +// withPolicy supplies an egress policy that allows every hostname. +var withPolicy = staticPolicy{policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Http: &ateletpb.EgressDestinations{Hostnames: []string{"*"}}, +}}}} + +// switchablePolicy supplies whichever policy was stored last. +type switchablePolicy struct { + policy atomic.Pointer[ateletpb.EgressPolicy] +} + +func (p *switchablePolicy) EgressPolicy(context.Context) (*ateletpb.EgressPolicy, error) { + return p.policy.Load(), nil +} + +// blockingPolicy holds each fetch until its context ends or the test releases +// it, as a fetch from an atelet that never answers would. It signals called +// when a fetch starts. +type blockingPolicy struct { + called chan struct{} + release chan struct{} +} + +func (p blockingPolicy) EgressPolicy(ctx context.Context) (*ateletpb.EgressPolicy, error) { + select { + case p.called <- struct{}{}: + default: + } + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-p.release: + return nil, errors.New("released by the test") + } +} + +// namesAsStrings compares DNS names as text rather than as 255-byte arrays. +var namesAsStrings = cmp.Transformer("Name", func(n dnsmessage.Name) string { return n.String() }) + +// checkAnswer fails the test unless answer answers dnsQuery(id) with rcode. +func checkAnswer(t *testing.T, answer []byte, id uint16, rcode dnsmessage.RCode) { + t.Helper() + checkAnswerForQuestion(t, answer, id, exampleQuestion, rcode) +} + +func checkAnswerForQuestion(t *testing.T, answer []byte, id uint16, wantQ dnsmessage.Question, rcode dnsmessage.RCode) { + t.Helper() + var p dnsmessage.Parser + h, err := p.Start(answer) + if err != nil { + t.Fatalf("parsing the relay's answer: %v", err) + } + if h.ID != id || !h.Response || h.RCode != rcode { + t.Errorf("answer header = %+v, want a response to query %#x with %v", h, id, rcode) + } + q, err := p.Question() + if err != nil { + t.Fatalf("parsing the question of the relay's answer: %v", err) + } + if diff := cmp.Diff(wantQ, q, namesAsStrings); diff != "" { + t.Errorf("answer question mismatch (-want +got):\n%s", diff) + } +} + +func dnsQueryForName(t *testing.T, id uint16, name string) ([]byte, dnsmessage.Question) { + t.Helper() + q := dnsmessage.Question{ + Name: dnsmessage.MustNewName(name), + Type: dnsmessage.TypeA, + Class: dnsmessage.ClassINET, + } + raw := buildTestMessage(t, dnsmessage.Header{ + ID: id, + RecursionDesired: true, + }, []dnsmessage.Question{q}) + return raw, q +} + +// newTCPResolver answers each DNS message on its TCP connections with +// respond(query). It returns its address and how many queries it was asked. +func newTCPResolver(t *testing.T, respond func([]byte) []byte) (string, *atomic.Int64) { + t.Helper() + lis, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { lis.Close() }) + var asked atomic.Int64 + go func() { + for { + conn, err := lis.Accept() + if err != nil { + return + } + go func() { + defer conn.Close() + for { + query, err := protocol.ReadTCPFrame(conn) + if err != nil { + return + } + asked.Add(1) + if err := protocol.WriteTCPFrame(conn, respond(query)); err != nil { + return + } + } + }() + } + }() + return lis.Addr().String(), &asked +} + +// dialRelay connects to the relay at addr over network, with a deadline that +// bounds the test's reads and writes. +func dialRelay(t *testing.T, network string, addr net.Addr) net.Conn { + t.Helper() + conn, err := net.Dial(network, addr.String()) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { conn.Close() }) + if err := conn.SetDeadline(time.Now().Add(5 * time.Second)); err != nil { + t.Fatal(err) + } + return conn +} + +// askUDP sends query to the relay and returns its answer. +func askUDP(t *testing.T, conn net.Conn, query []byte) []byte { + t.Helper() + if _, err := conn.Write(query); err != nil { + t.Fatal(err) + } + return readWithin(t, conn) +} + +// askTCP sends query on a TCP connection to the relay and returns the next +// message it sends back. +func askTCP(t *testing.T, conn net.Conn, query []byte) []byte { + t.Helper() + if err := protocol.WriteTCPFrame(conn, query); err != nil { + t.Fatal(err) + } + answer, err := protocol.ReadTCPFrame(conn) + if err != nil { + t.Fatalf("reading the relay's answer: %v", err) + } + return answer +} + +// policySources are the ways an actor can be without a usable egress policy. +var policySources = []struct { + name string + policy PolicySource +}{ + {"no policy", staticPolicy{}}, + {"policy unavailable", staticPolicy{err: errors.New("atelet is unreachable")}}, + {"no policy source", nil}, +} + +// An actor without an egress policy can reach nothing through the egress +// gateway, so it may resolve nothing either. The relay refuses its queries +// without asking a resolver, and does the same when it cannot learn the +// policy. +func TestRelayRefusesUDPQueriesWithoutPolicy(t *testing.T) { + for _, tc := range policySources { + t.Run(tc.name, func(t *testing.T) { + var asked atomic.Int64 + upstream := newFakeResolver(t, func(query []byte) []byte { + asked.Add(1) + return dnsAnswer(query, 0) + }) + relay, err := NewRelayForUpstreams([]string{upstream}) + if err != nil { + t.Fatal(err) + } + _, udpAddr, _ := serveLoopbackWithPolicy(t, relay, tc.policy) + client := dialRelay(t, "udp", udpAddr) + + checkAnswer(t, askUDP(t, client, dnsQuery(0x1111)), 0x1111, dnsmessage.RCodeRefused) + if got := asked.Load(); got != 0 { + t.Errorf("the resolver was asked %d times, want 0", got) + } + }) + } +} + +func TestRelayRefusesTCPQueriesWithoutPolicy(t *testing.T) { + for _, tc := range policySources { + t.Run(tc.name, func(t *testing.T) { + upstream, asked := newTCPResolver(t, func(query []byte) []byte { return dnsAnswer(query, 0) }) + relay, err := NewRelayForUpstreams([]string{upstream}) + if err != nil { + t.Fatal(err) + } + _, _, tcpAddr := serveLoopbackWithPolicy(t, relay, tc.policy) + conn := dialRelay(t, "tcp", tcpAddr) + + // The connection stays open for the actor's next query. + checkAnswer(t, askTCP(t, conn, dnsQuery(0x1111)), 0x1111, dnsmessage.RCodeRefused) + checkAnswer(t, askTCP(t, conn, dnsQuery(0x2222)), 0x2222, dnsmessage.RCodeRefused) + if got := asked.Load(); got != 0 { + t.Errorf("the resolver was asked %d queries, want 0", got) + } + }) + } +} + +// The relay checks each query against the actor's policy as it is then, so a +// policy created or deleted while the actor runs applies to its next query, +// even on a TCP connection it holds open. +func TestRelayFollowsPolicyChanges(t *testing.T) { + answer := func(query []byte) []byte { return dnsAnswer(query, 0) } + for _, tc := range []struct { + network string + upstream func(t *testing.T) string + ask func(*testing.T, net.Conn, []byte) []byte + }{ + { + network: "udp", + upstream: func(t *testing.T) string { return newFakeResolver(t, answer) }, + ask: askUDP, + }, + { + network: "tcp", + upstream: func(t *testing.T) string { + address, _ := newTCPResolver(t, answer) + return address + }, + ask: askTCP, + }, + } { + t.Run(tc.network, func(t *testing.T) { + relay, err := NewRelayForUpstreams([]string{tc.upstream(t)}) + if err != nil { + t.Fatal(err) + } + var policy switchablePolicy + _, udpAddr, tcpAddr := serveLoopbackWithPolicy(t, relay, &policy) + addr := udpAddr + if tc.network == "tcp" { + addr = tcpAddr + } + conn := dialRelay(t, tc.network, addr) + + checkAnswer(t, tc.ask(t, conn, dnsQuery(1)), 1, dnsmessage.RCodeRefused) + policy.policy.Store(&ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Http: &ateletpb.EgressDestinations{Hostnames: []string{"example.com", "*.wild.example"}}, + }}}) + checkAnswer(t, tc.ask(t, conn, dnsQuery(2)), 2, dnsmessage.RCodeSuccess) + wildQuery, wildQ := dnsQueryForName(t, 3, "api.wild.example.") + checkAnswerForQuestion(t, tc.ask(t, conn, wildQuery), 3, wildQ, dnsmessage.RCodeSuccess) + unlistedQuery, unlistedQ := dnsQueryForName(t, 4, "other.example.com.") + checkAnswerForQuestion(t, tc.ask(t, conn, unlistedQuery), 4, unlistedQ, dnsmessage.RCodeRefused) + policy.policy.Store(nil) + checkAnswer(t, tc.ask(t, conn, dnsQuery(5)), 5, dnsmessage.RCodeRefused) + }) + } +} + +// A query waiting on its policy must not hold the relay up when the actor goes +// away: Stop cancels the fetch rather than wait for an atelet that may never +// answer, and returns the query's slot. +func TestStopCancelsAPolicyFetchInProgress(t *testing.T) { + upstream := newFakeResolver(t, func(query []byte) []byte { return dnsAnswer(query, 0) }) + relay, err := NewRelayForUpstreams([]string{upstream}) + if err != nil { + t.Fatal(err) + } + policy := blockingPolicy{called: make(chan struct{}, 1), release: make(chan struct{})} + srv, udpAddr, _ := serveLoopbackWithPolicy(t, relay, policy) + // Runs before the server's own cleanup, so a relay that ignores Stop fails + // the test rather than hang it there. + t.Cleanup(func() { close(policy.release) }) + client := dialRelay(t, "udp", udpAddr) + + if _, err := client.Write(dnsQuery(0x1111)); err != nil { + t.Fatal(err) + } + select { + case <-policy.called: + case <-time.After(5 * time.Second): + t.Fatal("the relay never fetched the actor's policy") + } + + stopCtx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + if err := srv.Stop(stopCtx); err != nil { + t.Fatalf("Stop: %v", err) + } + if got := srv.limiter.inFlight.occupied(); got != 0 { + t.Errorf("%d in-flight slots still held after Stop, want 0", got) + } +} + +func TestAllowQuery(t *testing.T) { + question := func(name string) dnsmessage.Question { + return dnsmessage.Question{ + Name: dnsmessage.MustNewName(name), + Type: dnsmessage.TypeA, + Class: dnsmessage.ClassINET, + } + } + for _, tc := range []struct { + name string + policy *ateletpb.EgressPolicy + q dnsmessage.Question + want bool + }{ + {name: "no policy", policy: nil, q: exampleQuestion, want: false}, + {name: "policy without rules", policy: &ateletpb.EgressPolicy{}, q: exampleQuestion, want: false}, + { + name: "matching exact host", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"example.com"}}, + }}}, + q: exampleQuestion, + want: true, + }, + { + name: "unlisted host", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"example.com"}}, + }}}, + q: question("other.example.com."), + want: false, + }, + { + name: "matching wildcard host", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"*.example.com"}}, + }}}, + q: question("api.example.com."), + want: true, + }, + { + name: "wildcard does not match apex", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"*.example.com"}}, + }}}, + q: exampleQuestion, + want: false, + }, + { + name: "wildcard does not match nested subdomain", + policy: &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"*.example.com"}}, + }}}, + q: question("a.b.example.com."), + want: false, + }, + } { + t.Run(tc.name, func(t *testing.T) { + if got := allowQuery(tc.policy, tc.q); got != tc.want { + t.Errorf("allowQuery = %t, want %t", got, tc.want) + } + }) + } +} diff --git a/internal/ateomnet/dns/readme.md b/internal/ateomnet/dns/readme.md index ac742fe1ef..d7e0d3fba9 100644 --- a/internal/ateomnet/dns/readme.md +++ b/internal/ateomnet/dns/readme.md @@ -7,7 +7,7 @@ Actor DNS relays DNS requests from the actor to the external world. ### Relay - `Relay` -- one per Ateom, holds upstream resolvers, and is used to create the per-Actor `Server`. -- `Relay.Serve()` -- creates a per-Actor `Server` that manages DNS requests for the Actor. Returns a `*Server`. +- `Relay.Serve()` -- creates a per-Actor `Server` that manages DNS requests for the Actor, checked against the Actor's egress policy (see [Egress policy](#egress-policy)). Returns a `*Server`. ### Server @@ -54,7 +54,7 @@ UDP and TCP call up to the DNS protocol layer (`golang.org/x/net/dns/dnsmessage` - **Reply** (synthesized response without upstream round-trip): - `FORMERR` (`RCode = 1`): malformed header/question or `QDCOUNT != 1`. - `NOTIMP` (`RCode = 4`): `OpCode != 0` (non-standard query) or zone transfer `QTYPE` (`AXFR`, `IXFR`). - - `REFUSED` (`RCode = 5`): disallowed by domain or record-type policy. + - `REFUSED` (`RCode = 5`): disallowed by the Actor's egress policy (see [Egress policy](#egress-policy)). - `onUDPRequest(raw []byte, clientAddr net.Addr, upstreams []string)` -> `action`: - Acquires an `inFlight` slot, runs `validateRequest()`, and on `actionForward` records the query in `pendingRequests` with a rewritten `upstreamID`. - `onUDPResponse(raw []byte, from net.Addr)` -> `action`: @@ -64,3 +64,12 @@ UDP and TCP call up to the DNS protocol layer (`golang.org/x/net/dns/dnsmessage` - Runs `validateRequest()` and returns **Drop**, **Reply**, or **Forward** (unmodified frame). - `onTCPResponse(raw []byte)` -> `action`: - Validates `QR == 1` and parses header + question, returning **Drop** or **Deliver**. + +#### Egress policy + +`policy.go` and `evaluator.go`: DNS bypasses the egress gateway, so the relay enforces the Actor's egress policy itself. + +- `PolicySource` supplies the Actor's `EgressPolicy`. The ateom passes atunnel's per-Actor source, which caches the policy for a few seconds so a policy created, updated, or deleted while the Actor runs takes effect without a restart, and keeps serving the stale cached entry if `atelet` is temporarily unreachable. +- `evaluator` (`evaluator.go`) compiles the hostname and wildcard patterns from an `EgressPolicy`'s `http`, `https`, and `tls_passthrough` rules and evaluates normalized `QNAME`s via `allowsHostname(qname)`. +- `egressPolicyCheck` adapts the source to the `dnsHandler`'s `queryPolicy` hook, caching the compiled `*evaluator` for the current `*ateletpb.EgressPolicy` pointer and consulting it for every query. Fetching uses the `Server`'s context, so `Stop()` cancels a fetch in progress. +- `allowQuery` evaluates a single `dnsmessage.Question` against a policy using `evaluator`. diff --git a/internal/ateomnet/dns/relay.go b/internal/ateomnet/dns/relay.go index e7f421342e..e37b146928 100644 --- a/internal/ateomnet/dns/relay.go +++ b/internal/ateomnet/dns/relay.go @@ -32,9 +32,8 @@ const ( ) // Relay validates and forwards UDP and TCP DNS queries to the worker pod's -// resolvers. It listens in the sandbox's gateway namespace and dials from the -// worker's. DNS bypasses the egress tunnel and is not checked against egress -// policy. +// resolvers. It listens in the sandbox's gateway netns and dials from the +// worker's netns. type Relay struct { upstreams []string @@ -98,6 +97,6 @@ func NewRelayForUpstreams(upstreams []string) (*Relay, error) { // serveOn serves DNS on netC's sockets, which the caller has already bound, // and dials upstreams with netC's dialer. -func (r *Relay) serveOn(ctx context.Context, netC *netConn) (*Server, error) { - return newServer(ctx, &serverConfig{upstreams: r.upstreams}, netC) +func (r *Relay) serveOn(ctx context.Context, netC *netConn, policy PolicySource) (*Server, error) { + return newServer(ctx, &serverConfig{upstreams: r.upstreams, policy: policy}, netC) } diff --git a/internal/ateomnet/dns/relay_linux.go b/internal/ateomnet/dns/relay_linux.go index 53f0185b3f..7aa6c121a8 100644 --- a/internal/ateomnet/dns/relay_linux.go +++ b/internal/ateomnet/dns/relay_linux.go @@ -26,7 +26,7 @@ import ( ) // Serve serves UDP and TCP DNS in the sandbox's local gateway namespace. -func (r *Relay) Serve(ctx context.Context, ns netns.Handle) (*Server, error) { +func (r *Relay) Serve(ctx context.Context, ns netns.Handle, policy PolicySource) (*Server, error) { // Bind the wildcard because the microVM tap's gateway address is added later. address := net.JoinHostPort("0.0.0.0", strconv.Itoa(dnsPort)) @@ -59,5 +59,5 @@ func (r *Relay) Serve(ctx context.Context, ns netns.Handle) (*Server, error) { } netC.egressUDP = egressUDP - return r.serveOn(ctx, &netC) + return r.serveOn(ctx, &netC, policy) } diff --git a/internal/ateomnet/dns/relay_test.go b/internal/ateomnet/dns/relay_test.go index 104622c033..0c1e85283f 100644 --- a/internal/ateomnet/dns/relay_test.go +++ b/internal/ateomnet/dns/relay_test.go @@ -183,8 +183,16 @@ func newSilentResolver(t *testing.T) (address string, asked <-chan struct{}) { return pc.LocalAddr().String(), seen } -// serveLoopback runs the relay on loopback sockets and stops it on cleanup. +// serveLoopback runs the relay on loopback sockets for an actor with an +// egress policy, and stops it on cleanup. func serveLoopback(t *testing.T, relay *Relay) (*Server, net.Addr, net.Addr) { + t.Helper() + return serveLoopbackWithPolicy(t, relay, withPolicy) +} + +// serveLoopbackWithPolicy is serveLoopback for an actor whose egress policy +// policy supplies. +func serveLoopbackWithPolicy(t *testing.T, relay *Relay, policy PolicySource) (*Server, net.Addr, net.Addr) { t.Helper() pc, err := net.ListenPacket("udp", "127.0.0.1:0") if err != nil { @@ -199,7 +207,7 @@ func serveLoopback(t *testing.T, relay *Relay) (*Server, net.Addr, net.Addr) { dialer: *relay.dialer, udp: pc, tcpListener: lis, - }) + }, policy) if err != nil { pc.Close() t.Fatal(err) diff --git a/internal/ateomnet/dns/serve_linux_test.go b/internal/ateomnet/dns/serve_linux_test.go index d7ad48aa3a..3943070a6c 100644 --- a/internal/ateomnet/dns/serve_linux_test.go +++ b/internal/ateomnet/dns/serve_linux_test.go @@ -41,7 +41,7 @@ func TestClosingSandboxDNSStopsServing(t *testing.T) { if err != nil { t.Fatal(err) } - srv, err := relay.Serve(context.Background(), ns) + srv, err := relay.Serve(context.Background(), ns, nil) if err != nil { t.Fatal(err) } diff --git a/internal/ateomnet/dns/server.go b/internal/ateomnet/dns/server.go index 23987fbb54..b629c3c7d6 100644 --- a/internal/ateomnet/dns/server.go +++ b/internal/ateomnet/dns/server.go @@ -26,6 +26,9 @@ import ( type serverConfig struct { upstreams []string + // policy supplies the actor's egress policy, which each query is checked + // against. Nil refuses every query. + policy PolicySource } type netConn struct { @@ -84,7 +87,8 @@ func newServer( } s.pendingRequests = newPendingRequests(s.limiter) - s.dns = newDNSHandler(s.pendingRequests, s.limiter, nil) + // The check fetches with serveCtx, so that Stop cancels one in progress. + s.dns = newDNSHandler(s.pendingRequests, s.limiter, egressPolicyCheck(serveCtx, config.policy)) s.udp = newUDPHandler(netC.udp, netC.egressUDP, udpAddrs, s.dns, s.pendingRequests) s.tcp = newTCPHandler(config.upstreams, netC.tcpListener, netC.dialer, s.dns, s.limiter) diff --git a/internal/ateomnet/sandbox.go b/internal/ateomnet/sandbox.go index 03f5453699..a0b59acae2 100644 --- a/internal/ateomnet/sandbox.go +++ b/internal/ateomnet/sandbox.go @@ -69,6 +69,10 @@ type SandboxNetworkConfig struct { // EgressPort is where atunnel serves this actor. Every TCP connection the // actor makes is redirected to it, whatever port it was aimed at. EgressPort uint16 + + // DNSPolicy supplies the egress policy the actor's DNS queries are checked + // against. Nil refuses every query. + DNSPolicy dns.PolicySource } // SetupSandboxNetwork creates isolated networking with fixed sandbox addresses. @@ -410,6 +414,7 @@ type SandboxSession struct { // ServeSandbox builds a sandbox's network and serves egress and DNS from its // gateway namespace. A nil server leaves that unserved, which fails closed. +// DNS queries are checked against the policy cfg.DNSPolicy supplies. func ServeSandbox(ctx context.Context, cfg SandboxNetworkConfig, egress egressServer, resolver *dns.Relay) (_ *SandboxSession, retErr error) { network, err := SetupSandboxNetwork(ctx, cfg) if err != nil { @@ -423,7 +428,7 @@ func ServeSandbox(ctx context.Context, cfg SandboxNetworkConfig, egress egressSe }() if resolver != nil { - dnsServer, err := resolver.Serve(ctx, network.GatewayNetNS) + dnsServer, err := resolver.Serve(ctx, network.GatewayNetNS, cfg.DNSPolicy) if err != nil { return nil, err } diff --git a/internal/ateomstats/usage.go b/internal/ateomstats/usage.go index b3801fd961..20646f0243 100644 --- a/internal/ateomstats/usage.go +++ b/internal/ateomstats/usage.go @@ -75,7 +75,7 @@ func UsageAttrs(pool Pool, kind string, s *ateompb.WorkloadStatsSample) []slog.A }), slog.String(string(ateattr.WorkerPoolNamespaceKey), pool.Namespace), slog.String(string(ateattr.WorkerPoolNameKey), pool.Name), - slog.String(string(ateattr.SandboxClassKey), SandboxClassLabel(s.GetSandboxClass())), + slog.String(string(ateattr.SandboxClassKey), ateattr.NormalizeSandboxClass(s.GetSandboxClass())), slog.String(string(ateattr.StatsSourceKey), StatsSourceLabel(s.GetSource())), slog.String(string(ateattr.StatsKindKey), kind), slog.Int64(string(ateattr.ActorEpochKey), s.GetEpochUnixNano()), @@ -94,18 +94,6 @@ func UsageAttrs(pool Pool, kind string, s *ateompb.WorkloadStatsSample) []slog.A return attrs } -// SandboxClassLabel maps the wire enum to the ate.sandbox.class values. -func SandboxClassLabel(c ateompb.SandboxClass) string { - switch c { - case ateompb.SandboxClass_SANDBOX_CLASS_GVISOR: - return "gvisor" - case ateompb.SandboxClass_SANDBOX_CLASS_MICROVM: - return "microvm" - default: - return ateattr.SandboxClassUnknown - } -} - // StatsSourceLabel maps the wire enum to the ate.stats.source values. func StatsSourceLabel(s ateompb.StatsSource) string { switch s { diff --git a/internal/ateomstats/usage_test.go b/internal/ateomstats/usage_test.go index a6efc61d7c..397e63476d 100644 --- a/internal/ateomstats/usage_test.go +++ b/internal/ateomstats/usage_test.go @@ -33,7 +33,7 @@ var testPool = Pool{Namespace: "team-a", Name: "default"} func measuredSample() *ateompb.WorkloadStatsSample { return &ateompb.WorkloadStatsSample{ Atespace: "ns", ActorName: "a", ActorUid: "uid-1", ActorTemplateAtespace: "ns", ActorTemplateName: "t", - SandboxClass: ateompb.SandboxClass_SANDBOX_CLASS_GVISOR, + SandboxClass: "gvisor", Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: 40 << 20, MemoryPeakBytes: 48 << 20, @@ -111,6 +111,24 @@ func TestUsageAttrsPendingHasNoMeasurements(t *testing.T) { } } +func TestUsageAttrsNormalizesSandboxClass(t *testing.T) { + t.Parallel() + for class, want := range map[string]string{ + "microvm": "microvm", + "gvisor": "gvisor", + "gVisor": ateattr.SandboxClassUnknown, + "": ateattr.SandboxClassUnknown, + "kata": ateattr.SandboxClassUnknown, + } { + s := measuredSample() + s.SandboxClass = class + got := attrMap(UsageAttrs(testPool, ateattr.StatsKindPeriodic, s)) + if v := got[string(ateattr.SandboxClassKey)].String(); v != want { + t.Errorf("sandbox class %q: %s = %q, want %q", class, ateattr.SandboxClassKey, v, want) + } + } +} + func TestUsageAttrsOmitsUnreportedPeak(t *testing.T) { t.Parallel() s := measuredSample() @@ -205,17 +223,8 @@ func TestStartSamplerSurvivesAPanic(t *testing.T) { }) } -func TestLabels(t *testing.T) { +func TestStatsSourceLabel(t *testing.T) { t.Parallel() - for c, want := range map[ateompb.SandboxClass]string{ - ateompb.SandboxClass_SANDBOX_CLASS_GVISOR: "gvisor", - ateompb.SandboxClass_SANDBOX_CLASS_MICROVM: "microvm", - ateompb.SandboxClass_SANDBOX_CLASS_UNSPECIFIED: ateattr.SandboxClassUnknown, - } { - if got := SandboxClassLabel(c); got != want { - t.Errorf("SandboxClassLabel(%v) = %q, want %q", c, got, want) - } - } for src, want := range map[ateompb.StatsSource]string{ ateompb.StatsSource_STATS_SOURCE_CGROUP: ateattr.StatsSourceCgroup, ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT: ateattr.StatsSourceGuestAgent, diff --git a/internal/ateomtunnel/ateomtunnel.go b/internal/ateomtunnel/ateomtunnel.go index c434d65c3f..a80671a620 100644 --- a/internal/ateomtunnel/ateomtunnel.go +++ b/internal/ateomtunnel/ateomtunnel.go @@ -21,6 +21,7 @@ package ateomtunnel import ( "context" + "crypto/tls" "errors" "fmt" "log/slog" @@ -28,8 +29,10 @@ import ( "net/url" "time" + "github.com/agent-substrate/substrate/internal/ateletdial" "github.com/agent-substrate/substrate/internal/ateomnet/dns" "github.com/agent-substrate/substrate/internal/atunnel" + "github.com/agent-substrate/substrate/internal/egresspolicy" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/proto/ateompb" @@ -66,6 +69,8 @@ type Config struct { EgressListenAddress string // EgressTrustBundle verifies the remote egress gateway's serving cert. EgressTrustBundle string + // DNSEgressPolicy selects the DNS egress policy enforcement mode. + DNSEgressPolicy string } // RegisterFlags registers the atunnel flags on fs and returns the Config they @@ -82,6 +87,7 @@ func RegisterFlags(fs FlagSet) *Config { fs.StringVar(&c.BrokerIdentity, "atunnel-broker-identity", installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace), "SPIFFE identity the node-local atelet must present on the credential broker connection. Override when atelet runs outside the default namespace.") fs.StringVar(&c.EgressListenAddress, "atunnel-egress-listen-address", "0.0.0.0:15001", "Address for transparently intercepted actor egress TCP") fs.StringVar(&c.EgressTrustBundle, "atunnel-egress-trust-bundle", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "Service DNS trust bundle for the remote egress gateway") + fs.StringVar(&c.DNSEgressPolicy, "dns-egress-policy", "", "DNS egress policy enforcement mode (must be "+egresspolicy.DNSEgressPolicyEnforceInAtunnel+")") return c } @@ -98,12 +104,17 @@ type Tunnel struct { DNSRelay *dns.Relay cfg Config + // ateletTLS is used to authenticate to atelet to fetch egress policies. + ateletTLS *tls.Config } // Start builds the DNS relay and starts serving actor ingress, which proxies // to upstream inside each sandbox. Egress binds only in sandbox namespaces, so // nothing listens for it here. func Start(ctx context.Context, cfg Config, upstream string) (*Tunnel, error) { + if err := egresspolicy.ValidateDNSEgressPolicy(cfg.DNSEgressPolicy); err != nil { + return nil, err + } upstreamURL, err := url.Parse(upstream) if err != nil { return nil, fmt.Errorf("while parsing atunnel upstream: %w", err) @@ -113,6 +124,10 @@ func Start(ctx context.Context, cfg Config, upstream string) (*Tunnel, error) { if err != nil { return nil, fmt.Errorf("configuring DNS relay: %w", err) } + ateletTLS, err := ateletdial.TLSConfig(cfg.CredentialBundle, cfg.TrustBundle, cfg.BrokerIdentity) + if err != nil { + return nil, fmt.Errorf("configuring the atelet connection: %w", err) + } // Ingress ingress, err := atunnel.NewServer(atunnel.Config{ @@ -146,7 +161,8 @@ func Start(ctx context.Context, cfg Config, upstream string) (*Tunnel, error) { Egress: egress, EgressPort: egressPort, DNSRelay: dnsRelay, - cfg: cfg}, + cfg: cfg, + ateletTLS: ateletTLS}, nil } diff --git a/internal/ateomtunnel/ateomtunnel_test.go b/internal/ateomtunnel/ateomtunnel_test.go index 7f9954a31f..a25221fae7 100644 --- a/internal/ateomtunnel/ateomtunnel_test.go +++ b/internal/ateomtunnel/ateomtunnel_test.go @@ -33,8 +33,10 @@ import ( "testing" "time" + "github.com/agent-substrate/substrate/internal/egresspolicy" "github.com/agent-substrate/substrate/internal/proto/ateompb" "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/internal/substratex509" "github.com/spf13/pflag" ) @@ -48,6 +50,7 @@ func TestRegisterFlags(t *testing.T) { "--atunnel-broker-identity=broker", "--atunnel-egress-listen-address=0.0.0.0:3", "--atunnel-egress-trust-bundle=egress-trust", + "--dns-egress-policy=enforce-in-atunnel", } want := Config{ ListenAddress: ":1", @@ -58,6 +61,7 @@ func TestRegisterFlags(t *testing.T) { BrokerIdentity: "broker", EgressListenAddress: "0.0.0.0:3", EgressTrustBundle: "egress-trust", + DNSEgressPolicy: egresspolicy.DNSEgressPolicyEnforceInAtunnel, } std := flag.NewFlagSet("std", flag.ContinueOnError) @@ -79,6 +83,9 @@ func TestRegisterFlags(t *testing.T) { if tc.cfg.CredentialBundle == "" || tc.cfg.TrustBundle == "" || tc.cfg.ClientIdentity == "" || tc.cfg.BrokerIdentity == "" || tc.cfg.EgressTrustBundle == "" { t.Errorf("unset default: %+v", *tc.cfg) } + if tc.cfg.DNSEgressPolicy != "" { + t.Errorf("DNSEgressPolicy default = %q, want empty", tc.cfg.DNSEgressPolicy) + } if err := tc.parse(args); err != nil { t.Fatal(err) } @@ -91,6 +98,21 @@ func TestRegisterFlags(t *testing.T) { // testConfig returns a Config over fresh credentials and free loopback ports. func testConfig(t *testing.T) Config { + t.Helper() + cfg, _ := testConfigAndCA(t) + return cfg +} + +// testCA is the self-signed worker certificate in a test Config, which is also +// the Config's trust bundle. +type testCA struct { + cert *x509.Certificate + key *ecdsa.PrivateKey +} + +// testConfigAndCA is testConfig, also returning the certificate that both +// authenticates the worker and is the trust bundle. +func testConfigAndCA(t *testing.T) (Config, testCA) { t.Helper() dir := t.TempDir() @@ -109,10 +131,21 @@ func testConfig(t *testing.T) Config { ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth}, BasicConstraintsValid: true, } + // Worker certificates name the worker's Pod, which atelet connections need. + if err := substratex509.AddPodIdentityToCertificate(&substratex509.PodIdentity{ + Namespace: "workers", ServiceAccountName: "default", ServiceAccountUID: "sa-uid", + PodName: "worker", PodUID: "pod-uid", NodeName: testNodeName, NodeUID: testNodeUID, + }, template); err != nil { + t.Fatal(err) + } der, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key) if err != nil { t.Fatal(err) } + cert, err := x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } keyDER, err := x509.MarshalPKCS8PrivateKey(key) if err != nil { t.Fatal(err) @@ -145,9 +178,16 @@ func testConfig(t *testing.T) Config { BrokerIdentity: "spiffe://cluster.local/ns/ate-system/sa/atelet", EgressListenAddress: "0.0.0.0:15001", EgressTrustBundle: trust, - } + DNSEgressPolicy: egresspolicy.DNSEgressPolicyEnforceInAtunnel, + }, testCA{cert: cert, key: key} } +// The node the test worker runs on. +const ( + testNodeName = "node" + testNodeUID = "node-uid" +) + // freeAddress returns a loopback address nothing is listening on. func freeAddress(t *testing.T) string { t.Helper() @@ -200,6 +240,14 @@ func TestStartErrors(t *testing.T) { edit func(t *testing.T, cfg *Config) wantErr bool }{ + "missing dns egress policy": { + edit: func(_ *testing.T, cfg *Config) { cfg.DNSEgressPolicy = "" }, + wantErr: true, + }, + "invalid dns egress policy": { + edit: func(_ *testing.T, cfg *Config) { cfg.DNSEgressPolicy = "invalid" }, + wantErr: true, + }, "no resolvers": { edit: func(t *testing.T, _ *Config) { empty := filepath.Join(t.TempDir(), "resolv.conf") @@ -214,6 +262,10 @@ func TestStartErrors(t *testing.T) { edit: func(t *testing.T, cfg *Config) { cfg.CredentialBundle = filepath.Join(t.TempDir(), "absent.pem") }, wantErr: true, }, + "no atelet identity": { + edit: func(_ *testing.T, cfg *Config) { cfg.BrokerIdentity = "" }, + wantErr: true, + }, "ingress address in use": { edit: func(_ *testing.T, cfg *Config) { cfg.ListenAddress = inUse.Addr().String() }, wantErr: true, diff --git a/internal/ateomtunnel/egresspolicy.go b/internal/ateomtunnel/egresspolicy.go new file mode 100644 index 0000000000..9f41847e9e --- /dev/null +++ b/internal/ateomtunnel/egresspolicy.go @@ -0,0 +1,195 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package ateomtunnel + +import ( + "context" + "fmt" + "log/slog" + "sync" + "time" + + "golang.org/x/sync/singleflight" + + "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/ateletdial" + "github.com/agent-substrate/substrate/internal/ateomnet/dns" + "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "github.com/agent-substrate/substrate/internal/resources" +) + +const ( + // egressPolicyTTL is how long an actor's egress policy is reused before it + // is fetched again. + // + // TODO: tune once the cost of fetching is known at scale. + egressPolicyTTL = 5 * time.Second + + // egressPolicyFetchTimeout bounds one fetch. + egressPolicyFetchTimeout = 2 * time.Second + + // egressPolicyStaleWarnInterval is the minimum interval between warning + // logs while a stale cached policy is used because atelet is unreachable. + egressPolicyStaleWarnInterval = 5 * time.Minute +) + +// actorEgressPolicy supplies an actor's egress policy to its DNS relay. +type actorEgressPolicy struct { + fetch func(context.Context) (*ateletpb.EgressPolicy, error) + ttl time.Duration + now func() time.Time + // log identifies the actor on the source's records. + log *slog.Logger + + // flight shares one fetch among the queries waiting on it to avoid duplicate + // requests for the same information. + flight singleflight.Group + + mu sync.Mutex + fetched bool + policy *ateletpb.EgressPolicy + expires time.Time + // failing is set while fetches fail, so that the first failure in a run is + // logged at error level and subsequent stale-cache warnings are rate-limited. + failing bool + nextStaleWarn time.Time +} + +// ActorEgressPolicy returns the source of actor's egress policy for its DNS +// relay. +func (t *Tunnel) ActorEgressPolicy(actor resources.ActorAttribution) dns.PolicySource { + return &actorEgressPolicy{ + fetch: func(ctx context.Context) (*ateletpb.EgressPolicy, error) { + conn, err := ateletdial.Dial(nodepath.AteomSupportSocket, t.ateletTLS) + if err != nil { + return nil, err + } + defer conn.Close() + return getEgressPolicy(ctx, ateletpb.NewAteomSupportClient(conn), actor.Ref) + }, + ttl: egressPolicyTTL, + now: time.Now, + log: slog.New(slog.Default().Handler().WithAttrs(ateattr.ActorLogAttrs(actor))), + } +} + +// EgressPolicy returns the actor's egress policy, or nil if it has none. +// +// Callers must not modify the policy as it may be shared in response to concurrent calls +// to EgressPolicy. +func (p *actorEgressPolicy) EgressPolicy(ctx context.Context) (*ateletpb.EgressPolicy, error) { + if policy, ok := p.cached(); ok { + return policy, nil + } + results := p.flight.DoChan("", func() (any, error) { + // A fetch may have finished since this query found the policy stale. + if policy, ok := p.cached(); ok { + return policy, nil + } + // Not canceled with the query that started the fetch: the others + // waiting on it still need its result. + fetchCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), egressPolicyFetchTimeout) + defer cancel() + policy, err := p.fetch(fetchCtx) + return p.record(ctx, policy, err) + }) + select { + case res := <-results: + if res.Err != nil { + return nil, res.Err + } + policy, _ := res.Val.(*ateletpb.EgressPolicy) + return policy, nil + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +// cached returns the policy last fetched, unless it has expired. +func (p *actorEgressPolicy) cached() (*ateletpb.EgressPolicy, bool) { + p.mu.Lock() + defer p.mu.Unlock() + if !p.fetched || !p.now().Before(p.expires) { + return nil, false + } + return p.policy, true +} + +// record updates the cache with a fetch's result and returns the policy and +// error EgressPolicy should report. When a refresh fails after a policy has +// already been fetched, record keeps and returns the stale entry (fail static), +// logging an error on the first failure and a warning every +// egressPolicyStaleWarnInterval while stale entries are used. +func (p *actorEgressPolicy) record(ctx context.Context, policy *ateletpb.EgressPolicy, err error) (*ateletpb.EgressPolicy, error) { + p.mu.Lock() + defer p.mu.Unlock() + + now := p.now() + if err != nil { + if !p.fetched { + level := slog.LevelDebug + if !p.failing { + level = slog.LevelError + } + p.failing = true + p.log.LogAttrs(ctx, level, "Could not fetch the actor's egress policy; refusing its DNS queries", slog.Any("err", err)) + return nil, err + } + + level := slog.LevelDebug + switch { + case !p.failing: + level = slog.LevelError + p.failing = true + p.nextStaleWarn = now.Add(egressPolicyStaleWarnInterval) + case !now.Before(p.nextStaleWarn): + level = slog.LevelWarn + p.nextStaleWarn = now.Add(egressPolicyStaleWarnInterval) + } + p.log.LogAttrs(ctx, level, "Could not refresh the actor's egress policy; using stale cached policy", slog.Any("err", err)) + return p.policy, nil + } + + if p.failing || !p.fetched || (policy == nil) != (p.policy == nil) { + if policy == nil { + p.log.InfoContext(ctx, "Actor has no egress policy; blocking all DNS queries") + } else { + p.log.InfoContext(ctx, "Actor has an egress policy") + } + } + + p.failing = false + p.nextStaleWarn = time.Time{} + p.fetched = true + p.policy = policy + p.expires = now.Add(p.ttl) + return policy, nil +} + +// getEgressPolicy asks atelet for actor's egress policy, returning nil if the +// actor has none. +func getEgressPolicy(ctx context.Context, client ateletpb.AteomSupportClient, actor resources.ActorRef) (*ateletpb.EgressPolicy, error) { + resp, err := client.GetActorEgressPolicy(ctx, &ateletpb.GetActorEgressPolicyRequest{ + ActorAtespace: actor.Atespace, + ActorName: actor.Name, + }) + if err != nil { + return nil, fmt.Errorf("while fetching the actor's egress policy: %w", err) + } + return resp.GetEgressPolicy(), nil +} diff --git a/internal/ateomtunnel/egresspolicy_test.go b/internal/ateomtunnel/egresspolicy_test.go new file mode 100644 index 0000000000..587f993d6d --- /dev/null +++ b/internal/ateomtunnel/egresspolicy_test.go @@ -0,0 +1,536 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package ateomtunnel + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "log/slog" + "math/big" + "net" + "net/url" + "os" + "path/filepath" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials" + "google.golang.org/grpc/status" + "google.golang.org/protobuf/testing/protocmp" + + "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/internal/substratex509" +) + +var testActor = resources.ActorAttribution{Ref: resources.ActorRef{Atespace: "space", Name: "actor"}, UID: "uid"} + +// fakeClock moves only when advanced. +type fakeClock struct { + mu sync.Mutex + t time.Time +} + +func (c *fakeClock) now() time.Time { + c.mu.Lock() + defer c.mu.Unlock() + return c.t +} + +func (c *fakeClock) advance(d time.Duration) { + c.mu.Lock() + defer c.mu.Unlock() + c.t = c.t.Add(d) +} + +// fetchResult is what one fetch returns. +type fetchResult struct { + policy *ateletpb.EgressPolicy + err error +} + +// scriptedFetch returns results in order, one per fetch, and counts them. +type scriptedFetch struct { + results []fetchResult + fetches atomic.Int32 +} + +func (f *scriptedFetch) fetch(context.Context) (*ateletpb.EgressPolicy, error) { + r := f.results[f.fetches.Add(1)-1] + return r.policy, r.err +} + +// newTestSource returns a source over fetch whose clock moves only when the +// test advances it. +func newTestSource(fetch func(context.Context) (*ateletpb.EgressPolicy, error)) (*actorEgressPolicy, *fakeClock) { + clock := &fakeClock{t: time.Unix(1_000_000, 0)} + return &actorEgressPolicy{ + fetch: fetch, + ttl: egressPolicyTTL, + now: clock.now, + log: slog.New(slog.DiscardHandler), + }, clock +} + +func TestActorEgressPolicyReusesAFetchForItsTTL(t *testing.T) { + policy := &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{Http: &ateletpb.EgressDestinations{Hostnames: []string{"example.com"}}}}} + script := &scriptedFetch{results: []fetchResult{{policy: policy}, {policy: policy}}} + source, clock := newTestSource(script.fetch) + + for _, step := range []struct { + advance time.Duration + wantFetches int32 + }{ + {0, 1}, + {egressPolicyTTL - time.Nanosecond, 1}, + {time.Nanosecond, 2}, + } { + clock.advance(step.advance) + got, err := source.EgressPolicy(context.Background()) + if err != nil || got != policy { + t.Fatalf("EgressPolicy = %v, %v; want %v", got, err, policy) + } + if n := script.fetches.Load(); n != step.wantFetches { + t.Errorf("%d fetches after %v more, want %d", n, step.advance, step.wantFetches) + } + } +} + +// Having no policy is the answer that refuses an actor's DNS, and it is reused +// like any other. +func TestActorEgressPolicyReusesNoPolicy(t *testing.T) { + script := &scriptedFetch{results: []fetchResult{{}}} + source, _ := newTestSource(script.fetch) + + for range 2 { + got, err := source.EgressPolicy(context.Background()) + if err != nil || got != nil { + t.Fatalf("EgressPolicy = %v, %v; want nil, nil", got, err) + } + } + if n := script.fetches.Load(); n != 1 { + t.Errorf("%d fetches, want 1", n) + } +} + +// A failed initial fetch (before any policy is cached) is not reused: the next +// query fetches again rather than being refused for the rest of the TTL. +func TestActorEgressPolicyRetriesAfterAFailure(t *testing.T) { + errUnavailable := errors.New("atelet is unavailable") + policy := &ateletpb.EgressPolicy{} + script := &scriptedFetch{results: []fetchResult{{err: errUnavailable}, {policy: policy}}} + source, _ := newTestSource(script.fetch) + + if _, err := source.EgressPolicy(context.Background()); !errors.Is(err, errUnavailable) { + t.Fatalf("EgressPolicy error = %v, want %v", err, errUnavailable) + } + got, err := source.EgressPolicy(context.Background()) + if err != nil || got != policy { + t.Fatalf("EgressPolicy after the failure = %v, %v; want %v", got, err, policy) + } +} + +// Once a policy (or no-policy) has been cached, a failed refresh keeps serving +// the stale cache entry (fail static) until atelet responds again. +func TestActorEgressPolicyFailsStaticOnRefreshFailure(t *testing.T) { + errUnavailable := errors.New("atelet is unavailable") + policy1 := &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{Http: &ateletpb.EgressDestinations{Hostnames: []string{"one.example.com"}}}}} + policy2 := &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{Http: &ateletpb.EgressDestinations{Hostnames: []string{"two.example.com"}}}}} + script := &scriptedFetch{results: []fetchResult{ + {policy: policy1}, + {err: errUnavailable}, + {err: errUnavailable}, + {policy: policy2}, + {policy: nil}, + {err: errUnavailable}, + }} + source, clock := newTestSource(script.fetch) + + for i, want := range []*ateletpb.EgressPolicy{policy1, policy1, policy1, policy2, nil, nil} { + got, err := source.EgressPolicy(context.Background()) + if err != nil || got != want { + t.Fatalf("step %d: EgressPolicy = %v, %v; want %v, nil", i, got, err, want) + } + clock.advance(egressPolicyTTL) + } +} + +// The queries an actor sends together wait on one fetch rather than each +// asking atelet. +func TestActorEgressPolicySharesOneFetch(t *testing.T) { + policy := &ateletpb.EgressPolicy{} + started := make(chan struct{}) + release := make(chan struct{}) + var fetches atomic.Int32 + source, _ := newTestSource(func(context.Context) (*ateletpb.EgressPolicy, error) { + if fetches.Add(1) == 1 { + close(started) + } + <-release + return policy, nil + }) + + const queries = 8 + errs := make(chan error, queries) + for range queries { + go func() { + got, err := source.EgressPolicy(context.Background()) + if err == nil && got != policy { + err = fmt.Errorf("EgressPolicy = %v, want %v", got, policy) + } + errs <- err + }() + } + <-started + close(release) + for range queries { + if err := <-errs; err != nil { + t.Error(err) + } + } + if n := fetches.Load(); n != 1 { + t.Errorf("%d fetches for %d queries at once, want 1", n, queries) + } +} + +// A query stops waiting when its context ends, but the fetch it started goes on +// for the other queries, bounded by its own deadline. +func TestActorEgressPolicyQueryStopsWaitingWhenCanceled(t *testing.T) { + policy := &ateletpb.EgressPolicy{} + started := make(chan struct{}) + release := make(chan struct{}) + fetchCtxErr := make(chan error, 1) + var fetches atomic.Int32 + source, _ := newTestSource(func(ctx context.Context) (*ateletpb.EgressPolicy, error) { + fetches.Add(1) + close(started) + <-release + if _, ok := ctx.Deadline(); !ok { + fetchCtxErr <- errors.New("the fetch has no deadline") + } else { + fetchCtxErr <- ctx.Err() + } + return policy, nil + }) + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { + _, err := source.EgressPolicy(ctx) + done <- err + }() + <-started + cancel() + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Errorf("EgressPolicy error = %v, want %v", err, context.Canceled) + } + case <-time.After(5 * time.Second): + t.Fatal("the query kept waiting after its context was canceled") + } + + close(release) + if err := <-fetchCtxErr; err != nil { + t.Errorf("the fetch's context: %v; want it to outlive the query", err) + } + got, err := source.EgressPolicy(context.Background()) + if err != nil || got != policy { + t.Fatalf("EgressPolicy = %v, %v; want %v", got, err, policy) + } + if n := fetches.Load(); n != 1 { + t.Errorf("%d fetches, want 1", n) + } +} + +// recordingHandler keeps the level and message of each record. +type recordingHandler struct { + mu sync.Mutex + records []string +} + +func (h *recordingHandler) Enabled(context.Context, slog.Level) bool { return true } + +func (h *recordingHandler) Handle(_ context.Context, r slog.Record) error { + h.mu.Lock() + defer h.mu.Unlock() + h.records = append(h.records, r.Level.String()+" "+r.Message) + return nil +} + +func (h *recordingHandler) WithAttrs([]slog.Attr) slog.Handler { return h } +func (h *recordingHandler) WithGroup(string) slog.Handler { return h } + +// The source logs when the actor gains or loses its policy, an error on the +// first failure in a run, and a warning every 5 minutes while a stale cached +// policy is used. +func TestActorEgressPolicyLogsChanges(t *testing.T) { + errUnavailable := errors.New("atelet is unavailable") + policy := &ateletpb.EgressPolicy{} + steps := []struct { + advance time.Duration + result fetchResult + }{ + // Initial fetch fails twice before any entry is cached. + {0, fetchResult{err: errUnavailable}}, + {0, fetchResult{err: errUnavailable}}, + // First successful fetch caches "no policy". + {0, fetchResult{}}, + {egressPolicyTTL, fetchResult{}}, + // Refresh fails: first failure logs ERROR, 1ns before 5m logs DEBUG, + // at 5m logs WARN, within the next 5m logs DEBUG, at 10m logs WARN. + {egressPolicyTTL, fetchResult{err: errUnavailable}}, + {egressPolicyStaleWarnInterval - time.Nanosecond, fetchResult{err: errUnavailable}}, + {time.Nanosecond, fetchResult{err: errUnavailable}}, + {egressPolicyTTL, fetchResult{err: errUnavailable}}, + {egressPolicyStaleWarnInterval - egressPolicyTTL, fetchResult{err: errUnavailable}}, + // Recovery logs INFO and resets failure state. + {egressPolicyTTL, fetchResult{policy: policy}}, + {egressPolicyTTL, fetchResult{policy: policy}}, + // A new outage logs ERROR again on its first failure. + {egressPolicyTTL, fetchResult{err: errUnavailable}}, + {egressPolicyTTL, fetchResult{}}, + } + results := make([]fetchResult, len(steps)) + for i, s := range steps { + results[i] = s.result + } + script := &scriptedFetch{results: results} + source, clock := newTestSource(script.fetch) + var logged recordingHandler + source.log = slog.New(&logged) + + for _, s := range steps { + clock.advance(s.advance) + _, _ = source.EgressPolicy(context.Background()) + } + want := []string{ + "ERROR Could not fetch the actor's egress policy; refusing its DNS queries", + "DEBUG Could not fetch the actor's egress policy; refusing its DNS queries", + "INFO Actor has no egress policy; blocking all DNS queries", + "ERROR Could not refresh the actor's egress policy; using stale cached policy", + "DEBUG Could not refresh the actor's egress policy; using stale cached policy", + "WARN Could not refresh the actor's egress policy; using stale cached policy", + "DEBUG Could not refresh the actor's egress policy; using stale cached policy", + "WARN Could not refresh the actor's egress policy; using stale cached policy", + "INFO Actor has an egress policy", + "ERROR Could not refresh the actor's egress policy; using stale cached policy", + "INFO Actor has no egress policy; blocking all DNS queries", + } + logged.mu.Lock() + defer logged.mu.Unlock() + if diff := cmp.Diff(want, logged.records); diff != "" { + t.Errorf("records mismatch (-want +got):\n%s", diff) + } +} + +// fakeSupportClient answers egress policy requests with resp or err. +type fakeSupportClient struct { + ateletpb.AteomSupportClient + resp *ateletpb.GetActorEgressPolicyResponse + err error + got *ateletpb.GetActorEgressPolicyRequest +} + +func (c *fakeSupportClient) GetActorEgressPolicy(_ context.Context, req *ateletpb.GetActorEgressPolicyRequest, _ ...grpc.CallOption) (*ateletpb.GetActorEgressPolicyResponse, error) { + c.got = req + return c.resp, c.err +} + +func TestGetEgressPolicy(t *testing.T) { + policy := &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"example.com"}}, + }}} + errUnavailable := status.Error(codes.Unavailable, "the control plane is unavailable") + + for _, tc := range []struct { + name string + resp *ateletpb.GetActorEgressPolicyResponse + err error + want *ateletpb.EgressPolicy + wantErr bool + }{ + {name: "policy", resp: &ateletpb.GetActorEgressPolicyResponse{EgressPolicy: policy}, want: policy}, + {name: "no policy", resp: &ateletpb.GetActorEgressPolicyResponse{}}, + {name: "atelet error", err: errUnavailable, wantErr: true}, + } { + t.Run(tc.name, func(t *testing.T) { + client := &fakeSupportClient{resp: tc.resp, err: tc.err} + got, err := getEgressPolicy(context.Background(), client, testActor.Ref) + if (err != nil) != tc.wantErr { + t.Fatalf("getEgressPolicy error = %v, wantErr %t", err, tc.wantErr) + } + if tc.err != nil && !errors.Is(err, tc.err) { + t.Errorf("getEgressPolicy error = %v, want it to wrap %v", err, tc.err) + } + if (got == nil) != (tc.want == nil) { + t.Fatalf("getEgressPolicy = %v, want %v", got, tc.want) + } + if diff := cmp.Diff(tc.want, got, protocmp.Transform()); diff != "" { + t.Errorf("policy mismatch (-want +got):\n%s", diff) + } + wantReq := &ateletpb.GetActorEgressPolicyRequest{ActorAtespace: "space", ActorName: "actor"} + if diff := cmp.Diff(wantReq, client.got, protocmp.Transform()); diff != "" { + t.Errorf("request mismatch (-want +got):\n%s", diff) + } + }) + } +} + +// fakeAtelet serves an egress policy response as the node's atelet would. +type fakeAtelet struct { + ateletpb.UnimplementedAteomSupportServer + + mu sync.Mutex + resp *ateletpb.GetActorEgressPolicyResponse + got []*ateletpb.GetActorEgressPolicyRequest +} + +func (a *fakeAtelet) GetActorEgressPolicy(_ context.Context, req *ateletpb.GetActorEgressPolicyRequest) (*ateletpb.GetActorEgressPolicyResponse, error) { + a.mu.Lock() + defer a.mu.Unlock() + a.got = append(a.got, req) + return a.resp, nil +} + +func (a *fakeAtelet) respond(resp *ateletpb.GetActorEgressPolicyResponse) { + a.mu.Lock() + defer a.mu.Unlock() + a.resp = resp +} + +// serveAtelet serves a fake atelet on the AteomSupport socket, presenting a +// certificate issued by ca to the atelet identity ateletID on the test node. +func serveAtelet(t *testing.T, ca testCA, ateletID string) *fakeAtelet { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + id, err := url.Parse(ateletID) + if err != nil { + t.Fatal(err) + } + template := &x509.Certificate{ + SerialNumber: big.NewInt(2), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + URIs: []*url.URL{id}, + } + if err := substratex509.AddPodIdentityToCertificate(&substratex509.PodIdentity{ + Namespace: "ate-system", ServiceAccountName: "atelet", ServiceAccountUID: "atelet-sa-uid", + PodName: "atelet", PodUID: "atelet-uid", NodeName: testNodeName, NodeUID: testNodeUID, + }, template); err != nil { + t.Fatal(err) + } + der, err := x509.CreateCertificate(rand.Reader, template, ca.cert, &key.PublicKey, ca.key) + if err != nil { + t.Fatal(err) + } + clientCAs := x509.NewCertPool() + clientCAs.AddCert(ca.cert) + + // t.TempDir() embeds the test name, which can overrun the unix socket + // path limit, so the socket gets its own short directory. + socketDir, err := os.MkdirTemp("", "ateomtunnel") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.RemoveAll(socketDir) }) + socket := filepath.Join(socketDir, "support.sock") + old := nodepath.AteomSupportSocket + nodepath.AteomSupportSocket = socket + t.Cleanup(func() { nodepath.AteomSupportSocket = old }) + + lis, err := net.Listen("unix", socket) + if err != nil { + t.Fatal(err) + } + server := grpc.NewServer(grpc.Creds(credentials.NewTLS(&tls.Config{ + MinVersion: tls.VersionTLS13, + Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key}}, + ClientAuth: tls.RequireAndVerifyClientCert, + ClientCAs: clientCAs, + }))) + atelet := &fakeAtelet{} + ateletpb.RegisterAteomSupportServer(server, atelet) + go func() { _ = server.Serve(lis) }() + t.Cleanup(server.Stop) + return atelet +} + +// The tunnel fetches an actor's policy from the atelet on its node, over the +// worker's own credentials. +func TestTunnelFetchesActorEgressPolicyFromAtelet(t *testing.T) { + cfg, ca := testConfigAndCA(t) + tunnel := startTunnel(t, cfg) + atelet := serveAtelet(t, ca, cfg.BrokerIdentity) + + policy := &ateletpb.EgressPolicy{Rules: []*ateletpb.EgressRule{{ + Https: &ateletpb.EgressDestinations{Hostnames: []string{"example.com"}}, + }}} + for _, tc := range []struct { + name string + resp *ateletpb.GetActorEgressPolicyResponse + want *ateletpb.EgressPolicy + }{ + {name: "policy", resp: &ateletpb.GetActorEgressPolicyResponse{EgressPolicy: policy}, want: policy}, + // An empty policy encodes to no bytes, and must still arrive as a policy. + {name: "empty policy", resp: &ateletpb.GetActorEgressPolicyResponse{EgressPolicy: &ateletpb.EgressPolicy{}}, want: &ateletpb.EgressPolicy{}}, + {name: "no policy", resp: &ateletpb.GetActorEgressPolicyResponse{}}, + } { + t.Run(tc.name, func(t *testing.T) { + atelet.respond(tc.resp) + got, err := tunnel.ActorEgressPolicy(testActor).EgressPolicy(context.Background()) + if err != nil { + t.Fatal(err) + } + if (got == nil) != (tc.want == nil) { + t.Fatalf("EgressPolicy = %v, want %v", got, tc.want) + } + if diff := cmp.Diff(tc.want, got, protocmp.Transform()); diff != "" { + t.Errorf("policy mismatch (-want +got):\n%s", diff) + } + }) + } + + atelet.mu.Lock() + defer atelet.mu.Unlock() + if len(atelet.got) != 3 { + t.Fatalf("atelet was asked %d times, want 3", len(atelet.got)) + } + want := &ateletpb.GetActorEgressPolicyRequest{ActorAtespace: "space", ActorName: "actor"} + for _, got := range atelet.got { + if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { + t.Errorf("request mismatch (-want +got):\n%s", diff) + } + } +} diff --git a/internal/atunnel/egress.go b/internal/atunnel/egress.go index 634404d13b..c63678ad56 100644 --- a/internal/atunnel/egress.go +++ b/internal/atunnel/egress.go @@ -48,6 +48,8 @@ type OriginalDestination func(net.Conn) (string, error) // is assigned to its worker. type Egress struct { originalDestination OriginalDestination + // tunnelOpenTimeout bounds the wait for the gateway's CONNECT answer. + tunnelOpenTimeout time.Duration mu sync.Mutex // Keyed by actor UID, supplied by the namespace-specific listener. @@ -71,8 +73,13 @@ func NewEgress(originalDestination OriginalDestination) (*Egress, error) { if originalDestination == nil { return nil, fmt.Errorf("atunnel: original destination resolver is required") } + // The gateway answers within its 5s authorization timeout; a longer wait + // means it has queued the CONNECT behind its connection cap, and closing + // the connection is what frees that queue slot. + const tunnelOpenTimeout = 10 * time.Second return &Egress{ originalDestination: originalDestination, + tunnelOpenTimeout: tunnelOpenTimeout, active: map[string]*egressActivation{}, }, nil } @@ -298,7 +305,9 @@ func (e *Egress) handle(downstream net.Conn, active *egressActivation) { slog.WarnContext(active.ctx, "atunnel failed to resolve original egress destination", slog.Any("err", err)) return } - upstream, err := active.dialer.DialContext(active.ctx, destination) + dialCtx, cancelDial := context.WithTimeout(active.ctx, e.tunnelOpenTimeout) + upstream, err := active.dialer.DialContext(dialCtx, destination) + cancelDial() if err != nil { slog.WarnContext(active.ctx, "atunnel failed to open egress tunnel", slog.String("destination", destination), slog.Any("err", err)) return diff --git a/internal/atunnel/egress_test.go b/internal/atunnel/egress_test.go index 62fd936106..ebc724d66f 100644 --- a/internal/atunnel/egress_test.go +++ b/internal/atunnel/egress_test.go @@ -480,6 +480,49 @@ func TestEgressRejectsInactiveConnection(t *testing.T) { } } +// A gateway at its connection cap takes the CONNECT and queues it without +// answering. The open fails at the deadline instead of waiting for as long as +// the actor is active, the actor's connection closes, and the gateway sees the +// stream reset, which is what takes the CONNECT out of its queue. +func TestEgressBoundsHowLongATunnelWaitsToOpen(t *testing.T) { + ca := newTestCA(t) + // The handler's context ends when the client resets the stream. + withdrawn := make(chan struct{}, 1) + gatewayAddress, _ := serveTestH2ConnectGateway(t, ca, http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + <-r.Context().Done() + withdrawn <- struct{}{} + })) + client := newTestClient(t, ca, WithDialer(dialFixedAddress(gatewayAddress))) + + egress, err := NewEgress(func(net.Conn) (string, error) { return "192.0.2.10:443", nil }) + if err != nil { + t.Fatal(err) + } + egress.tunnelOpenTimeout = 100 * time.Millisecond + source := fakeActorCertificateSource{expiresAt: time.Now().Add(time.Hour)} + if err := egress.Activate(testActorUID, client, source, time.Now().Add(time.Hour)); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = egress.Deactivate(context.Background(), testActorUID) }) + + actor, proxy := net.Pipe() + defer actor.Close() + // Set before handle runs: once the proxy side closes, the pipe refuses + // new deadlines. + if err := actor.SetReadDeadline(time.Now().Add(5 * time.Second)); err != nil { + t.Fatal(err) + } + egress.mu.Lock() + active := egress.active[testActorUID] + egress.mu.Unlock() + egress.handle(proxy, active) + + receiveWithin(t, withdrawn, "the gateway to see the CONNECT withdrawn") + if _, err := actor.Read(make([]byte, 1)); !errors.Is(err, io.EOF) { + t.Errorf("reading the actor's connection = %v, want it closed (EOF) once the tunnel failed to open", err) + } +} + type egressDialerFunc func(context.Context, string) (net.Conn, error) func (f egressDialerFunc) DialContext(ctx context.Context, destination string) (net.Conn, error) { diff --git a/internal/childreap/childreap.go b/internal/childreap/childreap.go deleted file mode 100644 index 0a50aedcb8..0000000000 --- a/internal/childreap/childreap.go +++ /dev/null @@ -1,271 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// Package childreap reaps orphaned children without racing subprocess waits. -// Reaping waits for tracked subprocesses to finish because wait4(-1) can -// consume an exit status expected by os/exec. -//go:build unix - -package childreap - -import ( - "context" - "log/slog" - "os" - "os/exec" - "os/signal" - "sync" - "time" - - "golang.org/x/sys/unix" -) - -// MaxDefer is how long reaping waits before blocking new subprocesses. -const MaxDefer = 60 * time.Second - -// MaxDrain is how long a forced drain waits before abandoning the reap. -const MaxDrain = 30 * time.Second - -// Reaper collects orphaned children. The zero value is not usable; call New. -type Reaper struct { - mu sync.Mutex - // cond is broadcast when inFlight reaches zero or reaping ends. - cond *sync.Cond - // inFlight counts tracked subprocesses. - inFlight int - // reaping blocks new subprocesses while wait4 runs. - reaping bool - // draining blocks new subprocesses until inFlight reaches zero. - draining bool - // abandoned records that a reap gave up with children left unreaped. - abandoned bool - // retry re-runs an abandoned reap once the subprocesses that held it off - // are gone. SIGCHLD fires on a transition, so a child that died before the - // abandoned round will never announce itself again. - retry chan struct{} -} - -func New() *Reaper { - r := &Reaper{retry: make(chan struct{}, 1)} - r.cond = sync.NewCond(&r.mu) - return r -} - -// Enter prevents reaping until the returned function is called. -// -// defer reaper.Enter()() -// -// Do not use Enter for long-lived subprocesses; they prevent reaping. -func (r *Reaper) Enter() func() { - r.enter() - return r.leave -} - -// RunCommand runs cmd while excluding child reaping. -func (r *Reaper) RunCommand(cmd *exec.Cmd) error { - defer r.Enter()() - return cmd.Run() -} - -// CombinedOutput is RunCommand for callers that need cmd.CombinedOutput. -func (r *Reaper) CombinedOutput(cmd *exec.Cmd) ([]byte, error) { - defer r.Enter()() - return cmd.CombinedOutput() -} - -func (r *Reaper) enter() { - r.mu.Lock() - defer r.mu.Unlock() - for r.reaping || r.draining { - r.cond.Wait() - } - r.inFlight++ -} - -func (r *Reaper) leave() { - r.mu.Lock() - defer r.mu.Unlock() - r.inFlight-- - if r.inFlight < 0 { - // The exclusion is now meaningless: a later reap would see a quiet - // process that is not quiet and take a subprocess's exit status. - panic("childreap: leave called more times than Enter") - } - if r.inFlight == 0 { - if r.abandoned { - r.abandoned = false - select { - case r.retry <- struct{}{}: - default: - } - } - r.cond.Broadcast() - } -} - -// Run reaps until ctx ends. Call it once in its own goroutine. -func (r *Reaper) Run(ctx context.Context) { - sigs := make(chan os.Signal, 1) - signal.Notify(sigs, unix.SIGCHLD) - defer signal.Stop(sigs) - - for { - select { - case <-ctx.Done(): - return - case <-sigs: - case <-r.retry: - } - // A burst of SIGCHLD needs only one pass over wait4. - drain(sigs) - r.reapOnce(ctx) - } -} - -// drain empties ch without blocking. -func drain(ch <-chan os.Signal) { - for { - select { - case <-ch: - default: - return - } - } -} - -// reapOnce waits for a quiet moment, then collects everything waiting. -func (r *Reaper) reapOnce(ctx context.Context) { - if !r.acquire(ctx) { - return - } - defer r.release() - for { - var status unix.WaitStatus - pid, err := unix.Wait4(-1, &status, unix.WNOHANG, nil) - switch err { - case nil: - if pid <= 0 { - // Children exist but none have exited. Another SIGCHLD will - // bring us back. - return - } - case unix.EINTR: - continue - case unix.ECHILD: - return - default: - slog.WarnContext(ctx, "Reaping children failed", slog.Any("err", err)) - return - } - } -} - -// acquire waits for in-flight subprocesses, blocking new ones after MaxDefer. -// It returns false if ctx ends or the drain exceeds MaxDrain. -func (r *Reaper) acquire(ctx context.Context) bool { - if ctx.Err() != nil { - return false - } - // Nothing to wait for is the ordinary case, and it needs no deadlines. - if r.tryAcquire() { - return true - } - return r.awaitAcquire(ctx) -} - -// tryAcquire takes the reap if no subprocess is running, and reports whether -// it did. -func (r *Reaper) tryAcquire() bool { - r.mu.Lock() - defer r.mu.Unlock() - if r.inFlight > 0 || r.reaping { - return false - } - r.draining = false - r.reaping = true - return true -} - -// awaitAcquire is acquire once something is in the way, so it is the only path -// that pays for the deadlines. -func (r *Reaper) awaitAcquire(ctx context.Context) bool { - // sync.Cond has no timed wait, so each deadline is a timer that broadcasts - // and the loop reads the clock itself. - startedAt := time.Now() - drainAt := startedAt.Add(MaxDefer) - giveUpAt := drainAt.Add(MaxDrain) - for _, at := range []time.Time{drainAt, giveUpAt} { - timer := time.AfterFunc(time.Until(at), r.wake) - defer timer.Stop() - } - - stop := context.AfterFunc(ctx, r.wake) - defer stop() - - r.mu.Lock() - defer r.mu.Unlock() - for { - if ctx.Err() != nil { - r.stopDrainingLocked() - return false - } - // Another reap is in progress (it drained on our behalf); let it. - if r.reaping { - r.cond.Wait() - continue - } - now := time.Now() - if r.inFlight == 0 { - if !now.Before(drainAt) { - slog.WarnContext(ctx, "Reaped children only after holding off subprocesses", - slog.Duration("waited", now.Sub(startedAt))) - } - r.draining = false - r.reaping = true - return true - } - if !now.Before(giveUpAt) { - // Reaping now could consume the tracked subprocess's exit status. - slog.ErrorContext(ctx, "Gave up reaping: a subprocess has held off the reaper too long", - slog.Duration("waited", now.Sub(startedAt)), slog.Int("inFlight", r.inFlight)) - r.abandoned = true - r.stopDrainingLocked() - return false - } - if !now.Before(drainAt) { - r.draining = true - } - r.cond.Wait() - } -} - -// wake only signals acquire; timer callbacks may run after acquire returns. -func (r *Reaper) wake() { - r.mu.Lock() - defer r.mu.Unlock() - r.cond.Broadcast() -} - -// stopDrainingLocked ends an abandoned drain. Callers hold r.mu. -func (r *Reaper) stopDrainingLocked() { - r.draining = false - r.cond.Broadcast() -} - -func (r *Reaper) release() { - r.mu.Lock() - defer r.mu.Unlock() - r.reaping = false - r.cond.Broadcast() -} diff --git a/internal/childreap/childreap_test.go b/internal/childreap/childreap_test.go deleted file mode 100644 index 7fa6b26e21..0000000000 --- a/internal/childreap/childreap_test.go +++ /dev/null @@ -1,307 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -//go:build unix - -package childreap - -import ( - "context" - "os/exec" - "slices" - "sync" - "testing" - "testing/synctest" - "time" -) - -// Short subprocesses must stay short while a long one is always in flight -// beside them. This needs continuous arrival with mixed durations, not just N -// concurrent commands. -func TestShortSubprocessesAreNotPacedByLongOnes(t *testing.T) { - const ( - shortRunners = 8 - short = 20 * time.Millisecond - long = 400 * time.Millisecond - testFor = 3 * time.Second - ) - - r := New() - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go r.Run(ctx) - - run, stop := context.WithTimeout(ctx, testFor) - defer stop() - var wg sync.WaitGroup - - // Keep SIGCHLD pending throughout the test. - for range 4 { - wg.Go(func() { - for run.Err() == nil { - cmd := exec.Command("true") - if cmd.Start() != nil { - return - } - _ = cmd.Wait() - } - }) - } - - // Keep one long subprocess in flight. - wg.Go(func() { - for run.Err() == nil { - _ = runUnder(r, exec.Command("/bin/sleep", "0.4")) - } - }) - - var mu sync.Mutex - var samples []time.Duration - for range shortRunners { - wg.Go(func() { - for run.Err() == nil { - started := time.Now() - err := runUnder(r, exec.Command("/bin/sleep", "0.02")) - elapsed := time.Since(started) - mu.Lock() - if err != nil { - t.Errorf("short subprocess: %v", err) - } - samples = append(samples, elapsed) - mu.Unlock() - } - }) - } - wg.Wait() - - if len(samples) < 50 { - t.Fatalf("only %d short subprocesses completed in %s; too few to judge", len(samples), testFor) - } - slices.Sort(samples) - p90 := samples[len(samples)*90/100] - - // Distinguish normal overhead from pacing by the long subprocess. - if limit := 8 * short; p90 > limit { - t.Errorf("p90 of %d short subprocesses was %s, want under %s; they are being paced by the long one (median %s, max %s)", - len(samples), p90.Round(time.Millisecond), limit, - samples[len(samples)/2].Round(time.Millisecond), samples[len(samples)-1].Round(time.Millisecond)) - } - t.Logf("%d short subprocesses: p50=%s p90=%s max=%s", len(samples), - samples[len(samples)/2].Round(time.Millisecond), p90.Round(time.Millisecond), - samples[len(samples)-1].Round(time.Millisecond)) -} - -func runUnder(r *Reaper, cmd *exec.Cmd) error { - return r.RunCommand(cmd) -} - -func TestReapCollectsOrphans(t *testing.T) { - r := New() - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go r.Run(ctx) - - // Leave the child for the reaper. - cmd := exec.Command("true") - if err := cmd.Start(); err != nil { - t.Fatalf("start: %v", err) - } - pid := cmd.Process.Pid - - deadline := time.Now().Add(10 * time.Second) - for { - // An unreaped zombie still responds to signal 0. - if err := cmd.Process.Signal(nil); err != nil { - return // reaped - } - if time.Now().After(deadline) { - t.Fatalf("child %d was never reaped", pid) - } - time.Sleep(10 * time.Millisecond) - } -} - -// TestExecIsExcludedWhileReaping verifies that wait4 cannot race Enter. -func TestExecIsExcludedWhileReaping(t *testing.T) { - synctest.Test(t, func(t *testing.T) { - r := New() - - // Enter the reaper's critical section directly. - if !r.acquire(context.Background()) { - t.Fatal("acquire failed on an idle reaper") - } - - entered := make(chan struct{}) - go func() { - defer r.Enter()() - close(entered) - }() - synctest.Wait() - - select { - case <-entered: - t.Fatal("Enter returned while a reap was in progress") - default: - } - - r.release() - <-entered - }) -} - -func TestMaxDeferDrainsInFlightSubprocesses(t *testing.T) { - synctest.Test(t, func(t *testing.T) { - r := New() - leave := r.Enter() - - acquired := make(chan bool) - go func() { - acquired <- r.acquire(t.Context()) - }() - synctest.Wait() - time.Sleep(MaxDefer) - synctest.Wait() - - r.mu.Lock() - draining := r.draining - r.mu.Unlock() - if !draining { - t.Fatal("reaper never began draining") - } - - entered := make(chan func(), 1) - go func() { - entered <- r.Enter() - }() - synctest.Wait() - select { - case <-entered: - t.Fatal("new subprocess entered while the reaper was draining") - default: - } - - leave() - if ok := <-acquired; !ok { - t.Fatal("reaper did not acquire after in-flight subprocess exited") - } - r.release() - - leaveEntered := <-entered - leaveEntered() - }) -} - -// An abandoned drain must not block later subprocesses. -func TestAnEntryThatNeverLeavesDoesNotBlockLaterCommands(t *testing.T) { - const horizon = 10 * time.Minute - - synctest.Test(t, func(t *testing.T) { - r := New() - - defer r.Enter()() - - gaveUp := make(chan bool, 1) - go func() { gaveUp <- r.acquire(t.Context()) }() - - // The reaper must not consume the tracked child's status. - synctest.Wait() - select { - case got := <-gaveUp: - t.Fatalf("acquire returned %v immediately, want it to wait for the entry", got) - default: - } - - time.Sleep(horizon) - synctest.Wait() - select { - case got := <-gaveUp: - if got { - t.Fatal("acquire reported it reaped, want it to have given the round up") - } - default: - t.Fatalf("acquire still waiting after %s; the held entry blocks reaping forever", horizon) - } - - done := make(chan struct{}) - go func() { - defer close(done) - defer r.Enter()() - }() - synctest.Wait() - select { - case <-done: - default: - t.Fatal("a later Enter is still blocked by the abandoned drain") - } - }) -} - -// A timer callback after acquire returns must not alter reaper state. -func TestALateDeadlineCallbackLeavesNothingBehind(t *testing.T) { - r := New() - - if !r.acquire(t.Context()) { - t.Fatal("acquire failed on an idle reaper") - } - r.release() - - r.wake() - - done := make(chan struct{}) - go func() { - defer close(done) - defer r.Enter()() - }() - select { - case <-done: - case <-time.After(10 * time.Second): - t.Fatal("a later Enter is blocked: a deadline callback left reaper state behind") - } -} - -// An abandoned round leaves children unreaped, and SIGCHLD fires on a -// transition, so nothing will announce them again. Releasing the entry that -// held the reaper off has to be what brings it back. -func TestAnAbandonedRoundRetriesOnceTheEntryLeaves(t *testing.T) { - const horizon = 10 * time.Minute - - synctest.Test(t, func(t *testing.T) { - r := New() - - leave := r.Enter() - - gaveUp := make(chan bool, 1) - go func() { gaveUp <- r.acquire(t.Context()) }() - - time.Sleep(horizon) - synctest.Wait() - if got := <-gaveUp; got { - t.Fatal("acquire reported it reaped, want it to have given the round up") - } - - select { - case <-r.retry: - t.Fatal("the reaper was re-armed while the entry was still held") - default: - } - - leave() - synctest.Wait() - select { - case <-r.retry: - default: - t.Fatal("leaving the entry did not re-arm the reaper; the orphans it gave up on stay zombies") - } - }) -} diff --git a/internal/e2e/fixture.go b/internal/e2e/fixture.go index 43ecf5ef56..e3cf5167d4 100644 --- a/internal/e2e/fixture.go +++ b/internal/e2e/fixture.go @@ -42,7 +42,7 @@ func substrateTemplateSubstitutions(bucket, name string, trustBundles []string) // gvisor-default is the cluster-wide SandboxConfig // manifests/ate-install ships; config_name is required, so the // fixtures name it explicitly. - "${TEMPLATE_SANDBOX_CONFIG}": "sandboxConfig:\n sandboxClass: SANDBOX_CLASS_GVISOR\n configName: gvisor-default", + "${TEMPLATE_SANDBOX_CONFIG}": "sandboxConfig:\n sandboxClass: gvisor\n configName: gvisor-default", "${TEMPLATE_RESOURCES}": "", // Off unless the caller opts in; see WithTrustBundle. "${TEMPLATE_TRUST_BUNDLE}": "", @@ -63,7 +63,7 @@ func substrateTemplateSubstitutions(bucket, name string, trustBundles []string) inline["${FIXTURE_SUFFIX}"] = "-" + SandboxClassMicroVM + "-" + name // The cluster-wide SandboxConfig hack/install-microvm-deps.sh installs; // a missing or stale one fails loudly at template creation. - blocks["${TEMPLATE_SANDBOX_CONFIG}"] = "sandboxConfig:\n sandboxClass: SANDBOX_CLASS_MICROVM\n configName: microvm" + blocks["${TEMPLATE_SANDBOX_CONFIG}"] = "sandboxConfig:\n sandboxClass: microvm\n configName: microvm" // Only for fixtures that declare no limits of their own. Without them the // guest boots at ateom's default size (2GiB), and several of those // do not fit beside the demo pools on CI's single kind node. These size diff --git a/internal/e2e/sandbox_test.go b/internal/e2e/sandbox_test.go index 357c96b893..c035a88039 100644 --- a/internal/e2e/sandbox_test.go +++ b/internal/e2e/sandbox_test.go @@ -142,8 +142,8 @@ func TestRenderSubstrateFixtures_GVisor(t *testing.T) { } for _, tmpl := range templates { name := tmpl.GetMetadata().GetName() - if got := tmpl.GetSandboxConfig().GetSandboxClass(); got != ateapipb.SandboxClass_SANDBOX_CLASS_GVISOR { - t.Errorf("template %s sandboxClass = %v, want GVISOR", name, got) + if got := tmpl.GetSandboxConfig().GetSandboxClass(); got != "gvisor" { + t.Errorf("template %s sandboxClass = %q, want gvisor", name, got) } // The templates name the cluster-wide gvisor-default // SandboxConfig explicitly: config_name is required. @@ -191,8 +191,8 @@ func TestRenderSubstrateFixtures_MicroVM(t *testing.T) { } for _, tmpl := range templates { name := tmpl.GetMetadata().GetName() - if got := tmpl.GetSandboxConfig().GetSandboxClass(); got != ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM { - t.Errorf("template %s sandboxClass = %v, want MICROVM — it must match the pool's or no worker is eligible", name, got) + if got := tmpl.GetSandboxConfig().GetSandboxClass(); got != SandboxClassMicroVM { + t.Errorf("template %s sandboxClass = %q, want microvm — it must match the pool's or no worker is eligible", name, got) } // Named explicitly (see fixture.go), so a missing or stale // microvm install fails loudly. diff --git a/internal/e2e/suites/credentials/credentials_test.go b/internal/e2e/suites/credentials/credentials_test.go index 730fae0eb8..52892bcdd2 100644 --- a/internal/e2e/suites/credentials/credentials_test.go +++ b/internal/e2e/suites/credentials/credentials_test.go @@ -68,7 +68,8 @@ func TestKubernetesCredentialInjection(t *testing.T) { }, VolumeMounts: []corev1.VolumeMount{{Name: "token", MountPath: "/run/token", ReadOnly: true}}, }) - host := "credential-origin." + namespace + ".svc" + // Match the DNS query's fully qualified name in the egress policy. + host := "credential-origin." + namespace + ".svc.cluster.local" router, err := e2e.NewRouterClient(ctx) require.NoError(t, err) t.Cleanup(router.Close) diff --git a/internal/e2e/suites/demo/demo_test.go b/internal/e2e/suites/demo/demo_test.go index b8c1fc6157..f45cd2d741 100644 --- a/internal/e2e/suites/demo/demo_test.go +++ b/internal/e2e/suites/demo/demo_test.go @@ -29,10 +29,14 @@ import ( "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/internal/atenet" "github.com/agent-substrate/substrate/internal/e2e" + "github.com/agent-substrate/substrate/internal/hardware" "github.com/agent-substrate/substrate/internal/resources" + atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/google/go-cmp/cmp" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" + "google.golang.org/protobuf/testing/protocmp" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/client-go/tools/portforward" @@ -106,6 +110,21 @@ func TestActorSnapshotLifecycle(t *testing.T) { t.Fatalf("failed to initialize ActorTemplate: %v", err) } + goldenStamp := goldenSnapshotRuntime(t, ctx, clients, at) + wantClass := string(atev1alpha1.SandboxClassGvisor) + if e2e.IsMicroVM() { + wantClass = string(atev1alpha1.SandboxClassMicroVM) + } + if got := goldenStamp.GetSandboxClass(); got != wantClass { + t.Errorf("golden snapshot sandbox_class = %q, want %q", got, wantClass) + } + if got := goldenStamp.GetVersion().GetSchemaVersion(); got != hardware.SchemaVersionV1 { + t.Errorf("golden snapshot schema_version = %q, want %q", got, hardware.SchemaVersionV1) + } + if runtimeAttribute(goldenStamp, hardware.AttrArchitecture) == "" { + t.Errorf("golden snapshot stamp %v has no %q attribute", goldenStamp, hardware.AttrArchitecture) + } + sourceName := "snapshot-source-" + nsObj.Name cloneName := "snapshot-clone-" + nsObj.Name for _, name := range []string{sourceName, cloneName} { @@ -117,16 +136,32 @@ func TestActorSnapshotLifecycle(t *testing.T) { }) } - if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ + sourceCreated, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: demoAtespace, Name: sourceName}, ActorTemplate: e2e.TemplateRef(at), - }}); err != nil { + }}) + if err != nil { t.Fatalf("failed to create source Actor: %v", err) } - if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: sourceName}}); err != nil { + if diff := cmp.Diff(goldenStamp, durableSnapshot(sourceCreated.GetStatus()).GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Fatalf("golden snapshot stamp mismatch (-golden +actor):\n%s", diff) + } + sourceResumed, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: sourceName}}) + if err != nil { t.Fatalf("failed to resume source Actor: %v", err) } waitForActorState(ctx, t, clients, sourceName, ateapipb.ActorState_ACTOR_STATE_RUNNING) + assignment := sourceResumed.GetActor().GetStatus().GetWorkerAssignment() + worker, err := clients.SubstrateAPI.GetWorker(ctx, &ateapipb.GetWorkerRequest{Worker: assignment.GetWorker()}) + if err != nil { + t.Fatalf("failed to get assigned Worker %s: %v", assignment.GetWorker().GetName(), err) + } + if diff := cmp.Diff(worker.GetStatus().GetDefaultRuntime(), assignment.GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Fatalf("worker_assignment.sandbox_runtime mismatch (-worker default_runtime +assignment):\n%s", diff) + } + if !hardware.Matches(assignment.GetSandboxRuntime(), goldenStamp) { + t.Fatalf("assigned runtime %v cannot restore the golden stamp %v", assignment.GetSandboxRuntime(), goldenStamp) + } response, err := callActor(t, resources.ActorRef{Atespace: demoAtespace, Name: sourceName}) if err != nil { t.Fatalf("failed to call source Actor: %v", err) @@ -147,6 +182,10 @@ func TestActorSnapshotLifecycle(t *testing.T) { if snapshotURI == "" { t.Fatal("suspended Actor has no external snapshot") } + snapshotStamp := durableSnapshot(suspended.GetActor().GetStatus()).GetSandboxRuntime() + if diff := cmp.Diff(assignment.GetSandboxRuntime(), snapshotStamp, protocmp.Transform()); diff != "" { + t.Fatalf("suspended snapshot stamp mismatch (-assignment +snapshot):\n%s", diff) + } // The tag is given its own copy of the snapshot the suspend just wrote, so // it survives the Actor being suspended again or deleted. tagToUpdate, err := clients.SubstrateAPI.CreateTag(ctx, &ateapipb.CreateTagRequest{ @@ -165,6 +204,9 @@ func TestActorSnapshotLifecycle(t *testing.T) { if _, err := resources.ParseSnapshotURI(tagSnapshotURI(tagToUpdate)); err != nil { t.Errorf("ParseSnapshotURI(%q): %v", tagSnapshotURI(tagToUpdate), err) } + if diff := cmp.Diff(snapshotStamp, tagToUpdate.GetStatus().GetSnapshot().GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Fatalf("tag snapshot stamp mismatch (-actor snapshot +tag):\n%s", diff) + } listed, err := clients.SubstrateAPI.ListTags(ctx, &ateapipb.ListTagsRequest{Atespace: demoAtespace}) if err != nil { t.Fatalf("failed to list Tags: %v", err) @@ -187,18 +229,26 @@ func TestActorSnapshotLifecycle(t *testing.T) { t.Fatalf("failed to publish Tag: %v", err) } - if _, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{ + cloned, err := clients.SubstrateAPI.CreateActor(ctx, &ateapipb.CreateActorRequest{ Actor: &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: demoAtespace, Name: cloneName}, ActorTemplate: e2e.TemplateRef(at), SourceTag: tagRef, }, - }); err != nil { + }) + if err != nil { t.Fatalf("failed to create Actor from Tag: %v", err) } - if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: cloneName}}); err != nil { + if diff := cmp.Diff(snapshotStamp, durableSnapshot(cloned.GetStatus()).GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Fatalf("cloned actor snapshot stamp mismatch (-tag +clone):\n%s", diff) + } + cloneResumed, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: cloneName}}) + if err != nil { t.Fatalf("failed to resume cloned Actor: %v", err) } + if diff := cmp.Diff(snapshotStamp, cloneResumed.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("cloned actor assignment stamp mismatch (-tag +assignment):\n%s", diff) + } waitForActorState(ctx, t, clients, cloneName, ateapipb.ActorState_ACTOR_STATE_RUNNING) response, err = callActor(t, resources.ActorRef{Atespace: demoAtespace, Name: cloneName}) if err != nil { @@ -433,7 +483,7 @@ func TestDeleteActorAnyStateWithExternalVolume(t *testing.T) { } waitForActorState(ctx, t, clients, actorName, ateapipb.ActorState_ACTOR_STATE_RUNNING) - // Verify volume exists in actor (status should be CREATED) + // Verify volume exists in actor (it should be provisioned) actor, err := clients.SubstrateAPI.GetActor(ctx, &ateapipb.GetActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorName}, }) @@ -444,8 +494,8 @@ func TestDeleteActorAnyStateWithExternalVolume(t *testing.T) { t.Fatalf("expected actor to have volumes, got 0") } for _, vol := range actor.GetStatus().GetExternalVolumes() { - if vol.Status != ateapipb.ExternalVolume_STATUS_CREATED { - t.Fatalf("expected volume %q to be CREATED, got %s", vol.Name, vol.Status) + if vol.StorageVolumeId == "" { + t.Fatalf("expected volume %q to be provisioned, got no storage volume ID", vol.Name) } } @@ -632,12 +682,17 @@ func runActorLifecycleTestCase(t *testing.T, prefix string, createTemplate func( // Resuming the actor t.Logf("Resuming Actor %q...", actorID) - if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{ + resumed1, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID}, - }); err != nil { + }) + if err != nil { t.Fatalf("failed to resume Actor: %v", err) } waitForActorState(ctx, t, clients, actorID, ateapipb.ActorState_ACTOR_STATE_RUNNING) + assignedRuntime := resumed1.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime() + if assignedRuntime == nil { + t.Fatalf("resumed Actor %q has nil WorkerAssignment.SandboxRuntime", actorID) + } resp, err := callActor(t, resources.ActorRef{Atespace: demoAtespace, Name: actorID}) if err != nil { @@ -655,21 +710,30 @@ func runActorLifecycleTestCase(t *testing.T, prefix string, createTemplate func( // Pausing the actor // t.Logf("Pausing Actor %q...", actorID) - if _, err := clients.SubstrateAPI.PauseActor(ctx, &ateapipb.PauseActorRequest{ + paused, err := clients.SubstrateAPI.PauseActor(ctx, &ateapipb.PauseActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID}, - }); err != nil { + }) + if err != nil { t.Fatalf("failed to pause Actor: %v", err) } waitForActorPaused(ctx, t, clients, actorID) + if diff := cmp.Diff(assignedRuntime, localSnapshot(paused.GetActor().GetStatus()).GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("local pause snapshot stamp mismatch (-assignment +local):\n%s", diff) + } // Resuming the actor t.Logf("Resuming Actor %q again...", actorID) - if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{ + resumed2, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapipb.ResumeActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID}, - }); err != nil { + }) + if err != nil { t.Fatalf("failed to resume Actor again: %v", err) } waitForActorState(ctx, t, clients, actorID, ateapipb.ActorState_ACTOR_STATE_RUNNING) + assignedRuntime2 := resumed2.GetActor().GetStatus().GetWorkerAssignment().GetSandboxRuntime() + if diff := cmp.Diff(assignedRuntime, assignedRuntime2, protocmp.Transform()); diff != "" { + t.Errorf("resume-from-paused assignment stamp mismatch (-first +second):\n%s", diff) + } resp, err = callActor(t, resources.ActorRef{Atespace: demoAtespace, Name: actorID}) if err != nil { @@ -696,12 +760,16 @@ func runActorLifecycleTestCase(t *testing.T, prefix string, createTemplate func( waitForActorPaused(ctx, t, clients, actorID) } t.Logf("Suspending Actor %q...", actorID) - if _, err := clients.SubstrateAPI.SuspendActor(ctx, &ateapipb.SuspendActorRequest{ + suspended, err := clients.SubstrateAPI.SuspendActor(ctx, &ateapipb.SuspendActorRequest{ Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID}, - }); err != nil { + }) + if err != nil { t.Fatalf("failed to suspend Actor: %v", err) } waitForActorState(ctx, t, clients, actorID, ateapipb.ActorState_ACTOR_STATE_SUSPENDED) + if diff := cmp.Diff(assignedRuntime2, durableSnapshot(suspended.GetActor().GetStatus()).GetSandboxRuntime(), protocmp.Transform()); diff != "" { + t.Errorf("durable snapshot stamp after suspend mismatch (-assignment +snapshot):\n%s", diff) + } if tc.suspendWhilePaused { // The suspend must end the node pinning: a suspended actor's snapshot @@ -783,13 +851,50 @@ func durableSnapshotURI(status *ateapipb.ActorStatus) string { return durableSnapshotStorage(status).GetObject().GetSnapshotUri() } -func hasLocalSnapshot(status *ateapipb.ActorStatus) bool { +func localSnapshot(status *ateapipb.ActorStatus) *ateapipb.Snapshot { + var best *ateapipb.Snapshot for _, snap := range status.GetSnapshots() { if snap.GetLocality() != "" && snap.GetUuid() != "" { - return true + if best == nil || snap.GetGeneration() > best.GetGeneration() { + best = snap + } + } + } + return best +} + +func hasLocalSnapshot(status *ateapipb.ActorStatus) bool { + return localSnapshot(status) != nil +} + +func goldenSnapshotRuntime(t *testing.T, ctx context.Context, clients *e2e.Clients, at *ateapipb.ActorTemplate) *ateapipb.SandboxRuntime { + t.Helper() + current, err := clients.SubstrateAPI.GetActorTemplate(ctx, &ateapipb.GetActorTemplateRequest{ActorTemplate: e2e.TemplateRef(at)}) + if err != nil { + t.Fatalf("GetActorTemplate: %v", err) + } + goldenTag := current.GetStatus().GetGoldenSnapshotStatus().GetGoldenTag() + if goldenTag.GetName() == "" { + t.Fatalf("ActorTemplate %s has no golden tag", e2e.TemplateRef(at)) + } + tag, err := clients.SubstrateAPI.GetTag(ctx, &ateapipb.GetTagRequest{Tag: goldenTag}) + if err != nil { + t.Fatalf("GetTag(golden %s/%s): %v", goldenTag.GetAtespace(), goldenTag.GetName(), err) + } + stamp := tag.GetStatus().GetSnapshot().GetSandboxRuntime() + if stamp == nil { + t.Fatalf("golden snapshot %v carries no sandbox_runtime", tag.GetStatus().GetSnapshot()) + } + return stamp +} + +func runtimeAttribute(rt *ateapipb.SandboxRuntime, key string) string { + for _, attr := range rt.GetVersion().GetAttributes() { + if attr.GetKey() == key { + return attr.GetValue() } } - return false + return "" } func tagSnapshotURI(tag *ateapipb.Tag) string { diff --git a/internal/e2e/suites/egresscredinject/egresscredinject_test.go b/internal/e2e/suites/egresscredinject/egresscredinject_test.go index 16f8ce5bd4..f7ea9fec93 100644 --- a/internal/e2e/suites/egresscredinject/egresscredinject_test.go +++ b/internal/e2e/suites/egresscredinject/egresscredinject_test.go @@ -324,8 +324,8 @@ func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Client // Exercise each credential outcome on both HTTP and HTTPS: the echo // origin's credential resolves, and each fail-closed host's cannot. rules := []*ateapipb.EgressRule{ - e2e.EgressInjectHeader("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost), - e2e.EgressInjectHeaderHTTP("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost), + e2e.EgressInjectHeader("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost, echoHost+".cluster.local"), + e2e.EgressInjectHeaderHTTP("Authorization", "Bearer ", e2e.CredentialInjectionURI, echoHost, echoHost+".cluster.local"), } for _, credential := range []struct { host string diff --git a/internal/e2e/suites/egressmitm/egressmitm_test.go b/internal/e2e/suites/egressmitm/egressmitm_test.go index 315d93b0db..672563d3cc 100644 --- a/internal/e2e/suites/egressmitm/egressmitm_test.go +++ b/internal/e2e/suites/egressmitm/egressmitm_test.go @@ -360,7 +360,7 @@ func createAndResumeActor(t *testing.T, ctx context.Context, clients *e2e.Client // passthrough origin's fully qualified alias is allowed on 8443 only, // while the probe dials 443. e2e.EnsureEgressPolicy(t, ctx, clients, ref, - e2e.EgressAllowHTTP(mitm.SvcName()), + e2e.EgressAllowHTTP(mitm.SvcName(), mitm.SvcName()+".cluster.local"), e2e.EgressAllowHTTPS(mitm.SvcName()), e2e.EgressAllowPassthrough(passthrough.SvcName()), e2e.EgressAllowPassthroughOnPorts([]int32{8443}, passthrough.SvcName()+".cluster.local"), diff --git a/internal/e2e/suites/networking/egresspolicy_test.go b/internal/e2e/suites/networking/egresspolicy_test.go index 410d5a8755..86d6ec6e98 100644 --- a/internal/e2e/suites/networking/egresspolicy_test.go +++ b/internal/e2e/suites/networking/egresspolicy_test.go @@ -114,8 +114,13 @@ func TestActorEgressHTTPSByHostname(t *testing.T) { passthroughSpec.Namespace = mitm.Namespace passthrough := e2e.DeployServerPod(t, ctx, passthroughSpec) + // Include an HTTP rule for the .cluster.local FQDNs so cluster DNS search-path + // resolution succeeds while port 443 TLS rules still match only SvcName(). _, actorName, _ := createAndResumeActorWithEgress(t, ctx, "egress-sni", e2e.EgressFixture(), - e2e.EgressAllowHTTPS(mitm.SvcName()), e2e.EgressAllowPassthrough(passthrough.SvcName())) + e2e.EgressAllowHTTP(mitm.SvcName()+".cluster.local", passthrough.SvcName()+".cluster.local"), + e2e.EgressAllowHTTPS(mitm.SvcName()), + e2e.EgressAllowPassthrough(passthrough.SvcName()), + ) router := mustRouterClient(t, ctx) defer router.Close() actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} @@ -178,7 +183,7 @@ func TestActorEgressHTTPSByHostname(t *testing.T) { func TestActorEgressPassthroughPolicyRefresh(t *testing.T) { ctx := t.Context() dataplane := e2e.CurrentAtenetDataplane() - _, address, rootCA := prepareProtocolOrigin(t, ctx, e2e.ServerPod{ + origin, address, rootCA := prepareProtocolOrigin(t, ctx, e2e.ServerPod{ Name: "passthrough-origin", ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", Args: []string{"http"}, @@ -215,7 +220,12 @@ func TestActorEgressPassthroughPolicyRefresh(t *testing.T) { // Allow connection to be fully established before changing the policy. assertEgressGatewayConnect(t, ctx, since, actorAtespace, actorName, "443") - e2e.EnsureEgressPolicy(t, ctx, e2e.GetClients(), actorRef.ToObjectRef(), e2e.EgressAllowPassthrough("foo.bar.baz")) + // Keep DNS allowed through an HTTP rule on port 80 so the second HTTPS + // request reaches the gateway and tests revocation of TLS passthrough. + e2e.EnsureEgressPolicy(t, ctx, e2e.GetClients(), actorRef.ToObjectRef(), + e2e.EgressAllowHTTP(origin.SvcName()+".cluster.local"), + e2e.EgressAllowPassthrough("foo.bar.baz"), + ) // Wait for the policy cached in ext_proc server to expire, so it will fetch the new one. // ext_proc cache freshness interval is 10 seconds. time.Sleep(11 * time.Second) @@ -251,8 +261,8 @@ func TestActorEgressPendingPlaintextHeadersDoNotBlockMITM(t *testing.T) { actorAtespace, actorName, _ := createAndResumeActorWithEgress( t, ctx, "egress-slow-headers", e2e.EgressFixture(), - e2e.EgressAllowHTTPS(mitm.SvcName()), - e2e.EgressAllowHTTP(plaintext.SvcName()), + e2e.EgressAllowHTTPS(mitm.SvcName(), mitm.SvcName()+".cluster.local"), + e2e.EgressAllowHTTP(plaintext.SvcName(), plaintext.SvcName()+".cluster.local"), ) router := mustRouterClient(t, ctx) defer router.Close() @@ -294,6 +304,117 @@ func TestActorEgressPendingPlaintextHeadersDoNotBlockMITM(t *testing.T) { } } +// TestActorDNSRequiresPolicy: the sandbox's DNS relay refuses the queries of an +// actor with no EgressPolicy or for a host outside its policy, and answers them +// once a rule in the policy matches the host, without a restart. +func TestActorDNSRequiresPolicy(t *testing.T) { + ctx := context.Background() + origin := egressHTTPTarget() + target := e2e.DeployServerPod(t, ctx, origin) + name := fmt.Sprintf("%s.%s.svc.cluster.local", origin.Name, target.Namespace) + url := "http://" + name + "/healthz" + + _, actorName, _ := createAndResumeActorWithEgress(t, ctx, "dns-nopolicy", e2e.EgressFixture()) + router := mustRouterClient(t, ctx) + defer router.Close() + actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} + waitForActorRoute(t, ctx, router, actorRef) + + status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, url, notTransient) + if status != http.StatusBadGateway || !isRefusedLookup(string(body), name) { + t.Fatalf("fetch of %s by an actor with no policy returned HTTP %d, want its lookup refused; body: %s", url, status, body) + } + t.Logf("lookup was refused as expected: %s", body) + + // The relay fetches the policy again within seconds, so the retries below + // see it. A wildcard rule covering the target namespace's Services allows + // name while still refusing an unlisted host. + wildcard := fmt.Sprintf("*.%s.svc.cluster.local", target.Namespace) + e2e.EnsureEgressPolicy(t, ctx, e2e.GetClients(), actorRef.ToObjectRef(), e2e.EgressAllowHTTP(wildcard)) + status, body = fetchThroughEgressActorUntil(t, ctx, router, actorRef, url, reached) + if status != http.StatusOK { + t.Fatalf("fetch of %s after the actor was given a policy returned HTTP %d, want 200; body: %s", url, status, body) + } + + const unlisted = "unlisted.example.com" + status, body = fetchThroughEgressActorUntil(t, ctx, router, actorRef, "http://"+unlisted+"/healthz", notTransient) + if status != http.StatusBadGateway || !isRefusedLookup(string(body), unlisted) { + t.Fatalf("fetch of unlisted host %s returned HTTP %d, want its lookup refused; body: %s", unlisted, status, body) + } +} + +// isRefusedLookup reports whether body is the egress demo's report of a lookup +// of name that the resolver refused. Go's resolver reports a REFUSED answer as +// "server misbehaving". +func isRefusedLookup(body, name string) bool { + return strings.Contains(body, "lookup "+name) && strings.Contains(body, "server misbehaving") +} + +// hostnamePolicyActor creates an actor whose policy allows HTTPS to +// example.com, TLS passthrough to example.edu, and cleartext HTTP to +// example.org (so example.org resolves in DNS while still being denied over +// HTTPS on 443), and waits until it is routable. +func hostnamePolicyActor(t *testing.T, ctx context.Context) (*e2e.RouterClient, resources.ActorRef) { + t.Helper() + _, actorName, _ := createAndResumeActorWithEgress(t, ctx, "egress-sni", e2e.EgressFixture(), + e2e.EgressAllowHTTPS("example.com"), + e2e.EgressAllowPassthrough("example.edu"), + e2e.EgressAllowHTTP("example.org"), + ) + router := mustRouterClient(t, ctx) + t.Cleanup(func() { router.Close() }) + actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} + waitForActorRoute(t, ctx, router, actorRef) + return router, actorRef +} + +func isMitmCert(body string) bool { + // example.* domains use certs signed by Google Trust Services + // If this substring is not present it means dataplane used a minted cert + return !strings.Contains(body, "O=Google Trust Services") +} + +// TestActorEgressHTTPSByHostnameMITM: the gateway terminates the TLS and decides +// each request by name: example.com 200, example.org 403. +func TestActorEgressHTTPSByHostnameMITM(t *testing.T) { + ctx := context.Background() + dataplane := e2e.CurrentAtenetDataplane() + router, actorRef := hostnamePolicyActor(t, ctx) + + status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.com/", reached) + if status != http.StatusOK { + t.Fatalf("fetch of the allowed host returned HTTP %d, want 200; body: %s", status, body) + } + if !isMitmCert(string(body)) { + t.Fatalf("request did not use minted cert; body: %s", body) + } + status, body = fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.org/", notTransient) + if !dataplane.IsEgressPolicyDenied(status, string(body)) { + t.Fatalf("fetch of a host outside the policy returned HTTP %d, want an egress-policy denial; body: %s", status, body) + } + t.Logf("denied on the decrypted request: %s", body) +} + +// TestActorEgressHTTPSByHostnamePassthrough: the gateway acts as TCP proxy fetching +// allowed SNI. +func TestActorEgressHTTPSByHostnamePassthrough(t *testing.T) { + ctx := context.Background() + dataplane := e2e.CurrentAtenetDataplane() + router, actorRef := hostnamePolicyActor(t, ctx) + + status, body := fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.edu/", reached) + if status != http.StatusOK { + t.Fatalf("fetch of the allowed host returned HTTP %d, want 200; body: %s", status, body) + } + if isMitmCert(string(body)) { + t.Fatalf("request used minted cert; body: %s", body) + } + status, body = fetchThroughEgressActorUntil(t, ctx, router, actorRef, "https://example.org/", notTransient) + if !dataplane.IsEgressPolicyDenied(status, string(body)) { + t.Fatalf("fetch of a host outside the policy returned HTTP %d, want an egress-policy denial; body: %s", status, body) + } +} + // fetchThroughEgressActorUntil is fetchThroughEgressActor with the caller // deciding which answer is final. func fetchThroughEgressActorUntil(t *testing.T, ctx context.Context, router *e2e.RouterClient, actorRef resources.ActorRef, url string, done func(status int, body []byte) bool) (int, []byte) { diff --git a/internal/e2e/suites/networking/grpcingress_test.go b/internal/e2e/suites/networking/grpcingress_test.go index e919cb49d2..2cd02e8522 100644 --- a/internal/e2e/suites/networking/grpcingress_test.go +++ b/internal/e2e/suites/networking/grpcingress_test.go @@ -109,15 +109,15 @@ func TestIngressProtocolDowngrade(t *testing.T) { } body, _ := io.ReadAll(resp.Body) resp.Body.Close() - // The counter cannot speak h2c. Through atunnel's HTTPS listener Envoy - // returns HTTP 502; through a CONNECT tunnel the failure is gRPC's - // Unavailable status. + // The counter cannot speak h2c. Both dataplanes reach atunnel through + // a CONNECT tunnel, where the failure is gRPC's Unavailable status. An + // HTTP 502 means Envoy went through atunnel's HTTPS listener instead. grpcStatus := resp.Header.Get("Grpc-Status") if grpcStatus == "" { grpcStatus = resp.Trailer.Get("Grpc-Status") } - if resp.StatusCode != http.StatusBadGateway && !(resp.StatusCode == http.StatusOK && grpcStatus == "14") { - t.Fatalf("gRPC-shaped POST = %d, grpc-status %q (body %q), want HTTP 502 or gRPC Unavailable: gRPC must not be silently downgraded to HTTP/1.1", resp.StatusCode, grpcStatus, body) + if resp.StatusCode != http.StatusOK || grpcStatus != "14" { + t.Fatalf("gRPC-shaped POST = %d, grpc-status %q (body %q), want gRPC Unavailable: gRPC must not be silently downgraded to HTTP/1.1", resp.StatusCode, grpcStatus, body) } }) } diff --git a/internal/e2e/suites/networking/networking_test.go b/internal/e2e/suites/networking/networking_test.go index 0291dfe5cd..088631868a 100644 --- a/internal/e2e/suites/networking/networking_test.go +++ b/internal/e2e/suites/networking/networking_test.go @@ -130,9 +130,7 @@ func TestActorEgress(t *testing.T) { assertEgressGatewayConnect(t, ctx, since, actorAtespace, actorName, strconv.Itoa(origin.Port)) - // The Actor resolves the name itself -- DNS leaves over the UDP masquerade, - // not the tunnel, and atunnel forwards the resolved address, never the - // name -- so dialing by name covers a leg the by-address fetch above skips. + // Test resolution by DNS. t.Run("by DNS name", func(t *testing.T) { url := fmt.Sprintf("http://%s.%s.svc.cluster.local/healthz", origin.Name, target.Namespace) status, body := fetchThroughEgressActor(t, ctx, router, actorRef, url) diff --git a/internal/egresspolicy/egresspolicy.go b/internal/egresspolicy/egresspolicy.go index ba8d9a76fd..3655d15261 100644 --- a/internal/egresspolicy/egresspolicy.go +++ b/internal/egresspolicy/egresspolicy.go @@ -389,13 +389,24 @@ func NormalizeAuthority(authority string) (Destination, error) { return Destination{IP: addr.Unmap(), Port: port}, nil } - name := lowerASCII(strings.TrimSuffix(host, ".")) - if !isHostname(name) { + name, ok := NormalizeHostname(host) + if !ok { return Destination{}, fmt.Errorf("authority %q is neither a DNS hostname nor an IP literal", authority) } return Destination{Hostname: name, Port: port}, nil } +// NormalizeHostname turns raw into the canonical form HostnamePattern.Matches +// expects: ASCII lowercased, one trailing dot removed, and checked as a +// DNS-1123 subdomain whose last label is not all digits. +func NormalizeHostname(raw string) (string, bool) { + name := lowerASCII(strings.TrimSuffix(raw, ".")) + if !isHostname(name) { + return "", false + } + return name, true +} + // isHostname reports whether name is a lowercase DNS-1123 subdomain whose last // label is not all digits. RFC 1123 section 2.1 requires that, and it is what // keeps a dotted-decimal address from passing as a name, including spellings @@ -429,3 +440,19 @@ func lowerASCII(s string) string { } return string(b) } + +// DNSEgressPolicyEnforceInAtunnel is the --dns-egress-policy mode that +// enforces an Actor's EgressPolicy on DNS queries in the ateom DNS relay. +const DNSEgressPolicyEnforceInAtunnel = "enforce-in-atunnel" + +// ValidateDNSEgressPolicy checks the --dns-egress-policy flag value. +func ValidateDNSEgressPolicy(mode string) error { + switch mode { + case "": + return errors.New("--dns-egress-policy is required") + case DNSEgressPolicyEnforceInAtunnel: + return nil + default: + return fmt.Errorf("unsupported --dns-egress-policy %q (must be %q)", mode, DNSEgressPolicyEnforceInAtunnel) + } +} diff --git a/internal/egresspolicy/egresspolicy_test.go b/internal/egresspolicy/egresspolicy_test.go index c26994db72..9488d50e99 100644 --- a/internal/egresspolicy/egresspolicy_test.go +++ b/internal/egresspolicy/egresspolicy_test.go @@ -186,6 +186,34 @@ func TestNormalizeAuthority(t *testing.T) { } } +func TestNormalizeHostname(t *testing.T) { + tests := []struct { + raw string + want string + wantOK bool + }{ + {raw: "example.com", want: "example.com", wantOK: true}, + {raw: "example.com.", want: "example.com", wantOK: true}, + {raw: "API.Example.COM.", want: "api.example.com", wantOK: true}, + {raw: "xn--bcher-kva.example.", want: "xn--bcher-kva.example", wantOK: true}, + {raw: "", wantOK: false}, + {raw: ".", wantOK: false}, + {raw: "example.com..", wantOK: false}, + {raw: "192.0.2.1", wantOK: false}, + {raw: "192.0.2.1.", wantOK: false}, + {raw: "01.2.3.4.", wantOK: false}, + {raw: "_dmarc.example.com.", wantOK: false}, + {raw: "bücher.example.", wantOK: false}, + {raw: "*.example.com", wantOK: false}, + } + for _, tc := range tests { + got, ok := NormalizeHostname(tc.raw) + if ok != tc.wantOK || got != tc.want { + t.Errorf("NormalizeHostname(%q) = (%q, %v), want (%q, %v)", tc.raw, got, ok, tc.want, tc.wantOK) + } + } +} + func TestCompileReportsAndDropsInvalidEntries(t *testing.T) { compiled, errs := Compile(policy( httpRule("good.example.com", "BAD.example.com"), @@ -497,3 +525,20 @@ func TestSNIRules(t *testing.T) { }) } } + +func TestValidateDNSEgressPolicy(t *testing.T) { + tests := []struct { + mode string + wantErr bool + }{ + {mode: DNSEgressPolicyEnforceInAtunnel, wantErr: false}, + {mode: "", wantErr: true}, + {mode: "unknown", wantErr: true}, + } + for _, tc := range tests { + err := ValidateDNSEgressPolicy(tc.mode) + if (err != nil) != tc.wantErr { + t.Errorf("ValidateDNSEgressPolicy(%q) error = %v, wantErr = %v", tc.mode, err, tc.wantErr) + } + } +} diff --git a/internal/hardware/hardware.go b/internal/hardware/hardware.go index 99930e506e..827f59dd62 100644 --- a/internal/hardware/hardware.go +++ b/internal/hardware/hardware.go @@ -70,9 +70,12 @@ func Matches(worker, snap *ateapipb.SandboxRuntime) bool { } // MatchesCompat reports whether worker can restore a snapshot stamped with -// snap: same non-empty schema version, and every attribute on snap present on -// worker with an equal value. Keys only the worker has are ignored; a new -// schema version is how older snapshots are excluded. +// snap: the same non-empty schema version, and every attribute on snap +// present on worker with an equal value. Keys only the worker has are +// ignored, so a worker can start reporting a new key under the same schema +// version without invalidating the snapshots taken before it; removing or +// recomputing a key is a new schema version, under which older snapshots no +// longer match. func MatchesCompat(worker, snap *ateapipb.VersionedSandboxCompat) bool { if worker == nil || snap == nil { return false @@ -101,3 +104,43 @@ func MatchesCompat(worker, snap *ateapipb.VersionedSandboxCompat) bool { } return true } + +// MatchingRuntime returns the SandboxRuntime from status that can restore a +// snapshot stamped with snap (checking DefaultRuntime first, then +// RestorableRuntimes in order), or nil if none matches. When snap is nil, +// DefaultRuntime is returned. +func MatchingRuntime(status *ateapipb.WorkerStatus, snap *ateapipb.SandboxRuntime) *ateapipb.SandboxRuntime { + if snap == nil { + return status.GetDefaultRuntime() + } + if def := status.GetDefaultRuntime(); Matches(def, snap) { + return def + } + for _, rt := range status.GetRestorableRuntimes() { + if Matches(rt, snap) { + return rt + } + } + return nil +} + +// WorkerMatches reports whether a worker with status can restore a snapshot +// stamped with snap. A nil snapshot SandboxRuntime imposes no constraint. +func WorkerMatches(status *ateapipb.WorkerStatus, snap *ateapipb.SandboxRuntime) bool { + if snap == nil { + return true + } + return MatchingRuntime(status, snap) != nil +} + +// RuntimeFor returns the runtime a worker with status runs an actor under +// whose snapshot is stamped with snap: the matching runtime when it can +// restore the snapshot, else the DefaultRuntime the guest boots fresh under. +// This is what an assignment records and what the actor's next checkpoint +// is stamped with. +func RuntimeFor(status *ateapipb.WorkerStatus, snap *ateapipb.SandboxRuntime) *ateapipb.SandboxRuntime { + if rt := MatchingRuntime(status, snap); rt != nil { + return rt + } + return status.GetDefaultRuntime() +} diff --git a/internal/hardware/hardware_test.go b/internal/hardware/hardware_test.go index 5f0da993b3..652313c7ef 100644 --- a/internal/hardware/hardware_test.go +++ b/internal/hardware/hardware_test.go @@ -80,7 +80,7 @@ func TestMatches(t *testing.T) { {"different attribute value fails", arm64GVisor, amd64GVisor, false}, {"different sandbox class fails", amd64MicroVM, amd64GVisor, false}, {"different schema version fails", amd64GVisorV2, amd64GVisor, false}, - {"attribute the snapshot predates is not checked", amd64GVisorExtraAttr, amd64GVisor, true}, + {"attribute only the worker has is ignored", amd64GVisorExtraAttr, amd64GVisor, true}, {"attribute missing on worker fails", amd64GVisor, amd64GVisorExtraAttr, false}, {"nil worker fails when snapshot is stamped", nil, amd64GVisor, false}, {"nil version fails", &ateapipb.SandboxRuntime{SandboxClass: "gvisor"}, amd64GVisor, false}, @@ -101,6 +101,7 @@ func TestMatchesCompat(t *testing.T) { amd64 := &ateapipb.AttributeEntry{Key: AttrArchitecture, Value: "amd64"} arm64 := &ateapipb.AttributeEntry{Key: AttrArchitecture, Value: "arm64"} cpu := &ateapipb.AttributeEntry{Key: "cpu_features", Value: "GenuineIntel"} + hash := &ateapipb.AttributeEntry{Key: "gvisor_asset_hash", Value: "abc"} tests := []struct { name string @@ -110,6 +111,7 @@ func TestMatchesCompat(t *testing.T) { {"equal", compat("v1", amd64), compat("v1", amd64), true}, {"key only the worker has is ignored", compat("v1", amd64, cpu), compat("v1", amd64), true}, {"key only the snapshot has fails", compat("v1", amd64), compat("v1", amd64, cpu), false}, + {"key only the snapshot has fails even at equal count", compat("v1", amd64, cpu), compat("v1", amd64, hash), false}, {"value differs", compat("v1", arm64), compat("v1", amd64), false}, {"schema version differs", compat("v2", amd64), compat("v1", amd64), false}, {"both schema versions empty", compat("", amd64), compat("", amd64), false}, @@ -128,3 +130,87 @@ func TestMatchesCompat(t *testing.T) { }) } } + +func TestMatchingRuntimeAndWorkerMatches(t *testing.T) { + rt := func(class, arch, hash string) *ateapipb.SandboxRuntime { + attrs := []*ateapipb.AttributeEntry{{Key: AttrArchitecture, Value: arch}} + if hash != "" { + attrs = append(attrs, &ateapipb.AttributeEntry{Key: "gvisor_asset_hash", Value: hash}) + } + return &ateapipb.SandboxRuntime{ + SandboxClass: class, + Version: &ateapipb.VersionedSandboxCompat{ + SchemaVersion: SchemaVersionV1, + Attributes: attrs, + }, + } + } + + defRT := rt("gvisor", "amd64", "v2-hash") + restorable1 := rt("gvisor", "amd64", "v1-hash") + restorable2 := rt("gvisor", "amd64", "v0-hash") + unmatchedRT := rt("gvisor", "arm64", "v2-hash") + + status := &ateapipb.WorkerStatus{ + DefaultRuntime: defRT, + RestorableRuntimes: []*ateapipb.SandboxRuntime{restorable1, restorable2}, + } + + t.Run("nil snap matches any worker status and returns DefaultRuntime", func(t *testing.T) { + if !WorkerMatches(nil, nil) { + t.Errorf("WorkerMatches(nil, nil) = false, want true") + } + if !WorkerMatches(&ateapipb.WorkerStatus{}, nil) { + t.Errorf("WorkerMatches(empty, nil) = false, want true") + } + if got := MatchingRuntime(status, nil); got != defRT { + t.Errorf("MatchingRuntime(status, nil) = %v, want DefaultRuntime", got) + } + }) + + t.Run("matches DefaultRuntime", func(t *testing.T) { + if !WorkerMatches(status, defRT) { + t.Errorf("WorkerMatches(status, defRT) = false, want true") + } + if got := MatchingRuntime(status, defRT); got != defRT { + t.Errorf("MatchingRuntime(status, defRT) = %v, want defRT", got) + } + }) + + t.Run("matches RestorableRuntimes entry", func(t *testing.T) { + if !WorkerMatches(status, restorable2) { + t.Errorf("WorkerMatches(status, restorable2) = false, want true") + } + if got := MatchingRuntime(status, restorable2); got != restorable2 { + t.Errorf("MatchingRuntime(status, restorable2) = %v, want restorable2", got) + } + }) + + t.Run("returns nil and false when neither DefaultRuntime nor RestorableRuntimes match", func(t *testing.T) { + if WorkerMatches(status, unmatchedRT) { + t.Errorf("WorkerMatches(status, unmatchedRT) = true, want false") + } + if got := MatchingRuntime(status, unmatchedRT); got != nil { + t.Errorf("MatchingRuntime(status, unmatchedRT) = %v, want nil", got) + } + if WorkerMatches(nil, defRT) { + t.Errorf("WorkerMatches(nil, defRT) = true, want false") + } + }) + + t.Run("RuntimeFor is the match, else the default a cold boot runs on", func(t *testing.T) { + if got := RuntimeFor(status, restorable2); got != restorable2 { + t.Errorf("RuntimeFor(status, restorable2) = %v, want restorable2", got) + } + if got := RuntimeFor(status, unmatchedRT); got != defRT { + t.Errorf("RuntimeFor(status, unmatchedRT) = %v, want DefaultRuntime", got) + } + if got := RuntimeFor(status, nil); got != defRT { + t.Errorf("RuntimeFor(status, nil) = %v, want DefaultRuntime", got) + } + // A worker that reported no runtimes runs the actor under none. + if got := RuntimeFor(&ateapipb.WorkerStatus{}, unmatchedRT); got != nil { + t.Errorf("RuntimeFor(empty, unmatchedRT) = %v, want nil", got) + } + }) +} diff --git a/internal/localjwtauthority/localjwtauthority_test.go b/internal/localjwtauthority/localjwtauthority_test.go index 4b3e8e5243..bc7f538a2f 100644 --- a/internal/localjwtauthority/localjwtauthority_test.go +++ b/internal/localjwtauthority/localjwtauthority_test.go @@ -166,7 +166,7 @@ func TestSignJWTHeader(t *testing.T) { ActiveForSigning: "key-1", } - jwt, err := pool.SignJWT(&actoridjwt.Claims{Subject: "actor/a/b", Audiences: []string{"aud"}}) + jwt, err := pool.SignJWT(&actoridjwt.Claims{Kind: actoridjwt.KindActor, Subject: "actor/a/b", Audiences: []string{"aud"}}) if err != nil { t.Fatalf("Unexpected error signing JWT: %v", err) } @@ -234,7 +234,7 @@ func TestPoolRotation(t *testing.T) { if diff := cmp.Diff([]string{"next"}, authorityIDs(pool)); diff != "" { t.Errorf("authorities (-want +got):\n%s", diff) } - jwt, err := pool.SignJWT(&actoridjwt.Claims{Subject: "actor/a/b", Audiences: []string{"aud"}}) + jwt, err := pool.SignJWT(&actoridjwt.Claims{Kind: actoridjwt.KindActor, Subject: "actor/a/b", Audiences: []string{"aud"}}) if err != nil { t.Fatalf("SignJWT: %v", err) } @@ -345,7 +345,7 @@ func TestGenerateAuthority(t *testing.T) { if err != nil { t.Fatal(err) } - jwt, err := loaded.SignJWT(&actoridjwt.Claims{Subject: "actor/a/b", Audiences: []string{"aud"}}) + jwt, err := loaded.SignJWT(&actoridjwt.Claims{Kind: actoridjwt.KindActor, Subject: "actor/a/b", Audiences: []string{"aud"}}) if err != nil { t.Fatal(err) } diff --git a/internal/objectstoreplugin/status.go b/internal/objectstoreplugin/status.go index aa9008eb3b..472229fa60 100644 --- a/internal/objectstoreplugin/status.go +++ b/internal/objectstoreplugin/status.go @@ -17,6 +17,8 @@ package objectstoreplugin import ( "context" "errors" + "fmt" + "log/slog" "github.com/agent-substrate/substrate/internal/objectstore" "github.com/agent-substrate/substrate/pkg/objectstorage" @@ -37,5 +39,29 @@ func toStatus(err error) error { case errors.Is(err, context.DeadlineExceeded): return status.Error(codes.DeadlineExceeded, err.Error()) } + // The caller sees only Internal, and atelet deliberately drops a plugin's + // status text, so this log line is the one record of why the plugin failed. + slog.Warn("Object-store plugin failed with an unrecognized error; returning Internal", internalErrorAttrs(err)...) return status.Error(codes.Internal, err.Error()) } + +// internalErrorAttrs describes err for the log without its text, which may +// quote a URI, header or credential: only what the storage backend describes +// of the service's response (see objectstorage.DescribeServiceError), and the +// Go type of err's innermost error. +func internalErrorAttrs(err error) []any { + var attrs []any + if desc, ok := objectstorage.DescribeServiceError(err); ok { + attrs = append(attrs, slog.Int("http_status", desc.HTTPStatus)) + if desc.Code != "" { + attrs = append(attrs, slog.String("service_code", desc.Code)) + } + if desc.Message != "" { + attrs = append(attrs, slog.String("message", desc.Message)) + } + } + for next := errors.Unwrap(err); next != nil; next = errors.Unwrap(err) { + err = next + } + return append(attrs, slog.String("error_type", fmt.Sprintf("%T", err))) +} diff --git a/internal/objectstoreplugin/status_test.go b/internal/objectstoreplugin/status_test.go new file mode 100644 index 0000000000..6cf45397c7 --- /dev/null +++ b/internal/objectstoreplugin/status_test.go @@ -0,0 +1,85 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package objectstoreplugin + +import ( + "bytes" + "context" + "errors" + "fmt" + "log/slog" + "strings" + "testing" + + "github.com/agent-substrate/substrate/pkg/objectstorage" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// responseError is a storage service failure that reports its HTTP status, +// as a backend SDK's response error does. +type responseError struct { + code int + text string +} + +func (e *responseError) Error() string { return e.text } +func (e *responseError) HTTPStatusCode() int { return e.code } + +// TestToStatusLogsUnrecognizedErrors checks that an error toStatus does not +// recognize, which callers see only as Internal, is logged at WARN with its +// HTTP status and Go type but not its text, and that recognized errors are not +// logged. +func TestToStatusLogsUnrecognizedErrors(t *testing.T) { + const secret = "SECRET-SIG" + for _, tc := range []struct { + name string + err error + code codes.Code + wantLogs []string // empty: nothing is logged + }{ + {"storage response error", fmt.Errorf("while reading gs://b/o?sig=%s: %w", secret, &responseError{412, "precondition failed for gs://b/o " + secret}), codes.Internal, + []string{`"level":"WARN"`, `"http_status":412`, `"error_type":"*objectstoreplugin.responseError"`}}, + {"other error", fmt.Errorf("while reading gs://b/o?sig=%s: %w", secret, errors.New("boom")), codes.Internal, + []string{`"level":"WARN"`, `"error_type":"*errors.errorString"`}}, + {"not found", fmt.Errorf("while reading: %w", objectstorage.ErrObjectNotFound), codes.NotFound, nil}, + {"canceled", context.Canceled, codes.Canceled, nil}, + } { + t.Run(tc.name, func(t *testing.T) { + var buf bytes.Buffer + orig := slog.Default() + t.Cleanup(func() { slog.SetDefault(orig) }) + slog.SetDefault(slog.New(slog.NewJSONHandler(&buf, nil))) + + if got := status.Code(toStatus(tc.err)); got != tc.code { + t.Errorf("toStatus(%v) code = %s, want %s", tc.err, got, tc.code) + } + logs := buf.String() + if len(tc.wantLogs) == 0 && logs != "" { + t.Errorf("logged a recognized error:\n%s", logs) + } + for _, want := range tc.wantLogs { + if !strings.Contains(logs, want) { + t.Errorf("logs missing %s:\n%s", want, logs) + } + } + for _, leak := range []string{secret, "gs://b/o", "boom", "precondition failed"} { + if strings.Contains(logs, leak) { + t.Errorf("logs carry %q:\n%s", leak, logs) + } + } + }) + } +} diff --git a/internal/oidcdiscovery/discovery.go b/internal/oidcdiscovery/discovery.go index 8ffe3ec411..27e9a2b175 100644 --- a/internal/oidcdiscovery/discovery.go +++ b/internal/oidcdiscovery/discovery.go @@ -58,6 +58,6 @@ func DiscoveryDocument(issuer string, algorithms []string) ([]byte, error) { ResponseTypesSupported: []string{"id_token"}, SubjectTypesSupported: []string{"public"}, IDTokenSigningAlgValuesSupported: algs, - ClaimsSupported: []string{"iss", "sub", "aud", "exp", "nbf", "iat", "jti", "ate.dev"}, + ClaimsSupported: []string{"iss", "sub", "aud", "exp", "nbf", "iat", "jti", "actor.ate.dev", "ateom-for-actor.ate.dev"}, }) } diff --git a/internal/oidcdiscovery/discovery_test.go b/internal/oidcdiscovery/discovery_test.go index 31dfaf55e7..e7c54d70d9 100644 --- a/internal/oidcdiscovery/discovery_test.go +++ b/internal/oidcdiscovery/discovery_test.go @@ -32,7 +32,7 @@ func TestDiscoveryDocument(t *testing.T) { `"response_types_supported":["id_token"],` + `"subject_types_supported":["public"],` + `"id_token_signing_alg_values_supported":["RS256"],` + - `"claims_supported":["iss","sub","aud","exp","nbf","iat","jti","ate.dev"]}` + `"claims_supported":["iss","sub","aud","exp","nbf","iat","jti","actor.ate.dev","ateom-for-actor.ate.dev"]}` if diff := cmp.Diff(want, string(got)); diff != "" { t.Errorf("DiscoveryDocument mismatch (-want +got):\n%s", diff) } diff --git a/internal/proto/ateletpb/atelet.pb.go b/internal/proto/ateletpb/atelet.pb.go index 16b2fc7103..4ebb7b9b23 100644 --- a/internal/proto/ateletpb/atelet.pb.go +++ b/internal/proto/ateletpb/atelet.pb.go @@ -298,6 +298,7 @@ type SandboxRuntime struct { // // +k8s:required // +k8s:maxLength=63 + // +k8s:customValidation # gvisor or microvm SandboxClass string `protobuf:"bytes,1,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // Primary compatibility version stamped onto an actor's snapshot when a // checkpoint is taken on this runtime. Snapshots matching `version` @@ -808,6 +809,375 @@ func (*RequestActorSuspendResponse) Descriptor() ([]byte, []int) { return file_atelet_proto_rawDescGZIP(), []int{9} } +type GetActorEgressPolicyRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The actor the calling worker is hosting. atelet reads its policy from the + // control plane by name. + // + // +k8s:required + // +k8s:format=k8s-short-name + ActorAtespace string `protobuf:"bytes,1,opt,name=actor_atespace,json=actorAtespace,proto3" json:"actor_atespace,omitempty"` + // +k8s:required + // +k8s:format=k8s-short-name + ActorName string `protobuf:"bytes,2,opt,name=actor_name,json=actorName,proto3" json:"actor_name,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *GetActorEgressPolicyRequest) Reset() { + *x = GetActorEgressPolicyRequest{} + mi := &file_atelet_proto_msgTypes[10] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *GetActorEgressPolicyRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetActorEgressPolicyRequest) ProtoMessage() {} + +func (x *GetActorEgressPolicyRequest) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[10] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetActorEgressPolicyRequest.ProtoReflect.Descriptor instead. +func (*GetActorEgressPolicyRequest) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{10} +} + +func (x *GetActorEgressPolicyRequest) GetActorAtespace() string { + if x != nil { + return x.ActorAtespace + } + return "" +} + +func (x *GetActorEgressPolicyRequest) GetActorName() string { + if x != nil { + return x.ActorName + } + return "" +} + +type GetActorEgressPolicyResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The actor's policy, copied from the control plane's. Unset if the actor + // has no policy, which denies all of its egress, DNS included. + EgressPolicy *EgressPolicy `protobuf:"bytes,1,opt,name=egress_policy,json=egressPolicy,proto3" json:"egress_policy,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *GetActorEgressPolicyResponse) Reset() { + *x = GetActorEgressPolicyResponse{} + mi := &file_atelet_proto_msgTypes[11] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *GetActorEgressPolicyResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetActorEgressPolicyResponse) ProtoMessage() {} + +func (x *GetActorEgressPolicyResponse) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[11] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetActorEgressPolicyResponse.ProtoReflect.Descriptor instead. +func (*GetActorEgressPolicyResponse) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{11} +} + +func (x *GetActorEgressPolicyResponse) GetEgressPolicy() *EgressPolicy { + if x != nil { + return x.EgressPolicy + } + return nil +} + +// EgressPolicy is the part of the control plane's EgressPolicy that a worker +// enforces itself: which destinations each rule allows. The resource metadata +// and the rules' effects are left out; only the egress gateway applies them, +// so the credentials they refer to never reach the worker. +type EgressPolicy struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Rules are an unordered set of allows, as in the control plane's policy. + Rules []*EgressRule `protobuf:"bytes,1,rep,name=rules,proto3" json:"rules,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *EgressPolicy) Reset() { + *x = EgressPolicy{} + mi := &file_atelet_proto_msgTypes[12] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *EgressPolicy) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*EgressPolicy) ProtoMessage() {} + +func (x *EgressPolicy) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[12] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use EgressPolicy.ProtoReflect.Descriptor instead. +func (*EgressPolicy) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{12} +} + +func (x *EgressPolicy) GetRules() []*EgressRule { + if x != nil { + return x.Rules + } + return nil +} + +// EgressRule mirrors the control plane's EgressRule: exactly one field is set, +// naming the rule's protocol. A rule with none set has a protocol this copy +// does not carry yet, and allows nothing. +type EgressRule struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Cleartext HTTP to the named hosts. + Http *EgressDestinations `protobuf:"bytes,1,opt,name=http,proto3" json:"http,omitempty"` + // HTTPS to the named hosts, intercepted by the egress gateway. + Https *EgressDestinations `protobuf:"bytes,2,opt,name=https,proto3" json:"https,omitempty"` + // TLS to the named servers, forwarded without decryption. + TlsPassthrough *EgressDestinations `protobuf:"bytes,3,opt,name=tls_passthrough,json=tlsPassthrough,proto3" json:"tls_passthrough,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *EgressRule) Reset() { + *x = EgressRule{} + mi := &file_atelet_proto_msgTypes[13] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *EgressRule) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*EgressRule) ProtoMessage() {} + +func (x *EgressRule) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[13] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use EgressRule.ProtoReflect.Descriptor instead. +func (*EgressRule) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{13} +} + +func (x *EgressRule) GetHttp() *EgressDestinations { + if x != nil { + return x.Http + } + return nil +} + +func (x *EgressRule) GetHttps() *EgressDestinations { + if x != nil { + return x.Https + } + return nil +} + +func (x *EgressRule) GetTlsPassthrough() *EgressDestinations { + if x != nil { + return x.TlsPassthrough + } + return nil +} + +// EgressDestinations is the hostnames and ports a rule allows, copied from the +// control plane's HTTPRule, HTTPSRule, or TLSPassthroughRule. +type EgressDestinations struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Hostnames as the control plane's rule has them, wildcards included. + Hostnames []string `protobuf:"bytes,1,rep,name=hostnames,proto3" json:"hostnames,omitempty"` + // The ports the rule allows, with the control plane's defaults applied. + Ports *EgressPorts `protobuf:"bytes,2,opt,name=ports,proto3" json:"ports,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *EgressDestinations) Reset() { + *x = EgressDestinations{} + mi := &file_atelet_proto_msgTypes[14] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *EgressDestinations) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*EgressDestinations) ProtoMessage() {} + +func (x *EgressDestinations) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[14] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use EgressDestinations.ProtoReflect.Descriptor instead. +func (*EgressDestinations) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{14} +} + +func (x *EgressDestinations) GetHostnames() []string { + if x != nil { + return x.Hostnames + } + return nil +} + +func (x *EgressDestinations) GetPorts() *EgressPorts { + if x != nil { + return x.Ports + } + return nil +} + +// EgressPorts mirrors the control plane's Ports: either all ports, or the +// listed numbers. +type EgressPorts struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Set if the rule allows every port. + All *EgressAllPorts `protobuf:"bytes,1,opt,name=all,proto3" json:"all,omitempty"` + // The port numbers the rule allows, when all is unset. + Numbers []int32 `protobuf:"varint,2,rep,packed,name=numbers,proto3" json:"numbers,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *EgressPorts) Reset() { + *x = EgressPorts{} + mi := &file_atelet_proto_msgTypes[15] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *EgressPorts) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*EgressPorts) ProtoMessage() {} + +func (x *EgressPorts) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[15] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use EgressPorts.ProtoReflect.Descriptor instead. +func (*EgressPorts) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{15} +} + +func (x *EgressPorts) GetAll() *EgressAllPorts { + if x != nil { + return x.All + } + return nil +} + +func (x *EgressPorts) GetNumbers() []int32 { + if x != nil { + return x.Numbers + } + return nil +} + +// EgressAllPorts marks a rule as allowing every port. +type EgressAllPorts struct { + state protoimpl.MessageState `protogen:"open.v1"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *EgressAllPorts) Reset() { + *x = EgressAllPorts{} + mi := &file_atelet_proto_msgTypes[16] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *EgressAllPorts) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*EgressAllPorts) ProtoMessage() {} + +func (x *EgressAllPorts) ProtoReflect() protoreflect.Message { + mi := &file_atelet_proto_msgTypes[16] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use EgressAllPorts.ProtoReflect.Descriptor instead. +func (*EgressAllPorts) Descriptor() ([]byte, []int) { + return file_atelet_proto_rawDescGZIP(), []int{16} +} + type MintActorCertificateRequest struct { state protoimpl.MessageState `protogen:"open.v1"` // The actor for which the certificate should be issued. @@ -836,7 +1206,7 @@ type MintActorCertificateRequest struct { func (x *MintActorCertificateRequest) Reset() { *x = MintActorCertificateRequest{} - mi := &file_atelet_proto_msgTypes[10] + mi := &file_atelet_proto_msgTypes[17] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -848,7 +1218,7 @@ func (x *MintActorCertificateRequest) String() string { func (*MintActorCertificateRequest) ProtoMessage() {} func (x *MintActorCertificateRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[10] + mi := &file_atelet_proto_msgTypes[17] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -861,7 +1231,7 @@ func (x *MintActorCertificateRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use MintActorCertificateRequest.ProtoReflect.Descriptor instead. func (*MintActorCertificateRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{10} + return file_atelet_proto_rawDescGZIP(), []int{17} } func (x *MintActorCertificateRequest) GetActorAtespace() string { @@ -902,7 +1272,7 @@ type MintActorCertificateResponse struct { func (x *MintActorCertificateResponse) Reset() { *x = MintActorCertificateResponse{} - mi := &file_atelet_proto_msgTypes[11] + mi := &file_atelet_proto_msgTypes[18] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -914,7 +1284,7 @@ func (x *MintActorCertificateResponse) String() string { func (*MintActorCertificateResponse) ProtoMessage() {} func (x *MintActorCertificateResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[11] + mi := &file_atelet_proto_msgTypes[18] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -927,7 +1297,7 @@ func (x *MintActorCertificateResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use MintActorCertificateResponse.ProtoReflect.Descriptor instead. func (*MintActorCertificateResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{11} + return file_atelet_proto_rawDescGZIP(), []int{18} } func (x *MintActorCertificateResponse) GetActorCertificates() [][]byte { @@ -973,7 +1343,7 @@ type TerminateRequest struct { func (x *TerminateRequest) Reset() { *x = TerminateRequest{} - mi := &file_atelet_proto_msgTypes[12] + mi := &file_atelet_proto_msgTypes[19] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -985,7 +1355,7 @@ func (x *TerminateRequest) String() string { func (*TerminateRequest) ProtoMessage() {} func (x *TerminateRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[12] + mi := &file_atelet_proto_msgTypes[19] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -998,7 +1368,7 @@ func (x *TerminateRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use TerminateRequest.ProtoReflect.Descriptor instead. func (*TerminateRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{12} + return file_atelet_proto_rawDescGZIP(), []int{19} } func (x *TerminateRequest) GetWorkerPodUid() string { @@ -1058,7 +1428,7 @@ type TerminateResponse struct { func (x *TerminateResponse) Reset() { *x = TerminateResponse{} - mi := &file_atelet_proto_msgTypes[13] + mi := &file_atelet_proto_msgTypes[20] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1070,7 +1440,7 @@ func (x *TerminateResponse) String() string { func (*TerminateResponse) ProtoMessage() {} func (x *TerminateResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[13] + mi := &file_atelet_proto_msgTypes[20] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1083,7 +1453,7 @@ func (x *TerminateResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use TerminateResponse.ProtoReflect.Descriptor instead. func (*TerminateResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{13} + return file_atelet_proto_rawDescGZIP(), []int{20} } type RunRequest struct { @@ -1139,7 +1509,7 @@ type RunRequest struct { func (x *RunRequest) Reset() { *x = RunRequest{} - mi := &file_atelet_proto_msgTypes[14] + mi := &file_atelet_proto_msgTypes[21] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1151,7 +1521,7 @@ func (x *RunRequest) String() string { func (*RunRequest) ProtoMessage() {} func (x *RunRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[14] + mi := &file_atelet_proto_msgTypes[21] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1164,7 +1534,7 @@ func (x *RunRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RunRequest.ProtoReflect.Descriptor instead. func (*RunRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{14} + return file_atelet_proto_rawDescGZIP(), []int{21} } func (x *RunRequest) GetWorkerPodUid() string { @@ -1260,7 +1630,7 @@ type EgressGateway struct { func (x *EgressGateway) Reset() { *x = EgressGateway{} - mi := &file_atelet_proto_msgTypes[15] + mi := &file_atelet_proto_msgTypes[22] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1272,7 +1642,7 @@ func (x *EgressGateway) String() string { func (*EgressGateway) ProtoMessage() {} func (x *EgressGateway) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[15] + mi := &file_atelet_proto_msgTypes[22] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1285,7 +1655,7 @@ func (x *EgressGateway) ProtoReflect() protoreflect.Message { // Deprecated: Use EgressGateway.ProtoReflect.Descriptor instead. func (*EgressGateway) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{15} + return file_atelet_proto_rawDescGZIP(), []int{22} } func (x *EgressGateway) GetAddress() string { @@ -1320,7 +1690,7 @@ type AssetFile struct { func (x *AssetFile) Reset() { *x = AssetFile{} - mi := &file_atelet_proto_msgTypes[16] + mi := &file_atelet_proto_msgTypes[23] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1332,7 +1702,7 @@ func (x *AssetFile) String() string { func (*AssetFile) ProtoMessage() {} func (x *AssetFile) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[16] + mi := &file_atelet_proto_msgTypes[23] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1345,7 +1715,7 @@ func (x *AssetFile) ProtoReflect() protoreflect.Message { // Deprecated: Use AssetFile.ProtoReflect.Descriptor instead. func (*AssetFile) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{16} + return file_atelet_proto_rawDescGZIP(), []int{23} } func (x *AssetFile) GetUrl() string { @@ -1376,7 +1746,7 @@ type ArchAssets struct { func (x *ArchAssets) Reset() { *x = ArchAssets{} - mi := &file_atelet_proto_msgTypes[17] + mi := &file_atelet_proto_msgTypes[24] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1388,7 +1758,7 @@ func (x *ArchAssets) String() string { func (*ArchAssets) ProtoMessage() {} func (x *ArchAssets) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[17] + mi := &file_atelet_proto_msgTypes[24] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1401,7 +1771,7 @@ func (x *ArchAssets) ProtoReflect() protoreflect.Message { // Deprecated: Use ArchAssets.ProtoReflect.Descriptor instead. func (*ArchAssets) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{17} + return file_atelet_proto_rawDescGZIP(), []int{24} } func (x *ArchAssets) GetFiles() map[string]*AssetFile { @@ -1418,9 +1788,6 @@ func (x *ArchAssets) GetFiles() map[string]*AssetFile { // the release tarball; legacy "runsc", a bare binary, is still accepted). type SandboxAssets struct { state protoimpl.MessageState `protogen:"open.v1"` - // TODO: send this as an enum, like ateapi's and ateom's SandboxClass, so - // the tags can bound it and the custom validation goes away. - // // +k8s:required // +k8s:maxLength=63 # guardrail; the class names are a few characters // +k8s:customValidation # gvisor or microvm @@ -1445,7 +1812,7 @@ type SandboxAssets struct { func (x *SandboxAssets) Reset() { *x = SandboxAssets{} - mi := &file_atelet_proto_msgTypes[18] + mi := &file_atelet_proto_msgTypes[25] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1457,7 +1824,7 @@ func (x *SandboxAssets) String() string { func (*SandboxAssets) ProtoMessage() {} func (x *SandboxAssets) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[18] + mi := &file_atelet_proto_msgTypes[25] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1470,7 +1837,7 @@ func (x *SandboxAssets) ProtoReflect() protoreflect.Message { // Deprecated: Use SandboxAssets.ProtoReflect.Descriptor instead. func (*SandboxAssets) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{18} + return file_atelet_proto_rawDescGZIP(), []int{25} } func (x *SandboxAssets) GetSandboxClass() string { @@ -1515,7 +1882,7 @@ type WorkloadSpec struct { func (x *WorkloadSpec) Reset() { *x = WorkloadSpec{} - mi := &file_atelet_proto_msgTypes[19] + mi := &file_atelet_proto_msgTypes[26] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1527,7 +1894,7 @@ func (x *WorkloadSpec) String() string { func (*WorkloadSpec) ProtoMessage() {} func (x *WorkloadSpec) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[19] + mi := &file_atelet_proto_msgTypes[26] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1540,7 +1907,7 @@ func (x *WorkloadSpec) ProtoReflect() protoreflect.Message { // Deprecated: Use WorkloadSpec.ProtoReflect.Descriptor instead. func (*WorkloadSpec) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{19} + return file_atelet_proto_rawDescGZIP(), []int{26} } func (x *WorkloadSpec) GetContainers() []*Container { @@ -1565,7 +1932,7 @@ type DurableDirVolume struct { func (x *DurableDirVolume) Reset() { *x = DurableDirVolume{} - mi := &file_atelet_proto_msgTypes[20] + mi := &file_atelet_proto_msgTypes[27] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1577,7 +1944,7 @@ func (x *DurableDirVolume) String() string { func (*DurableDirVolume) ProtoMessage() {} func (x *DurableDirVolume) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[20] + mi := &file_atelet_proto_msgTypes[27] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1590,7 +1957,7 @@ func (x *DurableDirVolume) ProtoReflect() protoreflect.Message { // Deprecated: Use DurableDirVolume.ProtoReflect.Descriptor instead. func (*DurableDirVolume) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{20} + return file_atelet_proto_rawDescGZIP(), []int{27} } type ExternalVolumeSource struct { @@ -1624,7 +1991,7 @@ type ExternalVolumeSource struct { func (x *ExternalVolumeSource) Reset() { *x = ExternalVolumeSource{} - mi := &file_atelet_proto_msgTypes[21] + mi := &file_atelet_proto_msgTypes[28] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1636,7 +2003,7 @@ func (x *ExternalVolumeSource) String() string { func (*ExternalVolumeSource) ProtoMessage() {} func (x *ExternalVolumeSource) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[21] + mi := &file_atelet_proto_msgTypes[28] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1649,7 +2016,7 @@ func (x *ExternalVolumeSource) ProtoReflect() protoreflect.Message { // Deprecated: Use ExternalVolumeSource.ProtoReflect.Descriptor instead. func (*ExternalVolumeSource) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{21} + return file_atelet_proto_rawDescGZIP(), []int{28} } func (x *ExternalVolumeSource) GetStorageVolumeId() string { @@ -1694,7 +2061,7 @@ type ImageVolumeSource struct { func (x *ImageVolumeSource) Reset() { *x = ImageVolumeSource{} - mi := &file_atelet_proto_msgTypes[22] + mi := &file_atelet_proto_msgTypes[29] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1706,7 +2073,7 @@ func (x *ImageVolumeSource) String() string { func (*ImageVolumeSource) ProtoMessage() {} func (x *ImageVolumeSource) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[22] + mi := &file_atelet_proto_msgTypes[29] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1719,7 +2086,7 @@ func (x *ImageVolumeSource) ProtoReflect() protoreflect.Message { // Deprecated: Use ImageVolumeSource.ProtoReflect.Descriptor instead. func (*ImageVolumeSource) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{22} + return file_atelet_proto_rawDescGZIP(), []int{29} } func (x *ImageVolumeSource) GetReference() string { @@ -1751,7 +2118,7 @@ type ActorMetadataItem struct { func (x *ActorMetadataItem) Reset() { *x = ActorMetadataItem{} - mi := &file_atelet_proto_msgTypes[23] + mi := &file_atelet_proto_msgTypes[30] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1763,7 +2130,7 @@ func (x *ActorMetadataItem) String() string { func (*ActorMetadataItem) ProtoMessage() {} func (x *ActorMetadataItem) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[23] + mi := &file_atelet_proto_msgTypes[30] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1776,7 +2143,7 @@ func (x *ActorMetadataItem) ProtoReflect() protoreflect.Message { // Deprecated: Use ActorMetadataItem.ProtoReflect.Descriptor instead. func (*ActorMetadataItem) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{23} + return file_atelet_proto_rawDescGZIP(), []int{30} } func (x *ActorMetadataItem) GetField() ActorMetadataField { @@ -1812,7 +2179,7 @@ type ActorMetadataDataSource struct { func (x *ActorMetadataDataSource) Reset() { *x = ActorMetadataDataSource{} - mi := &file_atelet_proto_msgTypes[24] + mi := &file_atelet_proto_msgTypes[31] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1824,7 +2191,7 @@ func (x *ActorMetadataDataSource) String() string { func (*ActorMetadataDataSource) ProtoMessage() {} func (x *ActorMetadataDataSource) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[24] + mi := &file_atelet_proto_msgTypes[31] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1837,7 +2204,7 @@ func (x *ActorMetadataDataSource) ProtoReflect() protoreflect.Message { // Deprecated: Use ActorMetadataDataSource.ProtoReflect.Descriptor instead. func (*ActorMetadataDataSource) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{24} + return file_atelet_proto_rawDescGZIP(), []int{31} } func (x *ActorMetadataDataSource) GetItems() []*ActorMetadataItem { @@ -1871,7 +2238,7 @@ type TrustBundleDataSource struct { func (x *TrustBundleDataSource) Reset() { *x = TrustBundleDataSource{} - mi := &file_atelet_proto_msgTypes[25] + mi := &file_atelet_proto_msgTypes[32] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1883,7 +2250,7 @@ func (x *TrustBundleDataSource) String() string { func (*TrustBundleDataSource) ProtoMessage() {} func (x *TrustBundleDataSource) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[25] + mi := &file_atelet_proto_msgTypes[32] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1896,7 +2263,7 @@ func (x *TrustBundleDataSource) ProtoReflect() protoreflect.Message { // Deprecated: Use TrustBundleDataSource.ProtoReflect.Descriptor instead. func (*TrustBundleDataSource) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{25} + return file_atelet_proto_rawDescGZIP(), []int{32} } func (x *TrustBundleDataSource) GetPath() string { @@ -1931,7 +2298,7 @@ type SystemInfoDataSource struct { func (x *SystemInfoDataSource) Reset() { *x = SystemInfoDataSource{} - mi := &file_atelet_proto_msgTypes[26] + mi := &file_atelet_proto_msgTypes[33] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1943,7 +2310,7 @@ func (x *SystemInfoDataSource) String() string { func (*SystemInfoDataSource) ProtoMessage() {} func (x *SystemInfoDataSource) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[26] + mi := &file_atelet_proto_msgTypes[33] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1956,7 +2323,7 @@ func (x *SystemInfoDataSource) ProtoReflect() protoreflect.Message { // Deprecated: Use SystemInfoDataSource.ProtoReflect.Descriptor instead. func (*SystemInfoDataSource) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{26} + return file_atelet_proto_rawDescGZIP(), []int{33} } func (x *SystemInfoDataSource) GetActorMetadata() *ActorMetadataDataSource { @@ -1989,7 +2356,7 @@ type SystemInfoVolume struct { func (x *SystemInfoVolume) Reset() { *x = SystemInfoVolume{} - mi := &file_atelet_proto_msgTypes[27] + mi := &file_atelet_proto_msgTypes[34] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2001,7 +2368,7 @@ func (x *SystemInfoVolume) String() string { func (*SystemInfoVolume) ProtoMessage() {} func (x *SystemInfoVolume) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[27] + mi := &file_atelet_proto_msgTypes[34] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2014,7 +2381,7 @@ func (x *SystemInfoVolume) ProtoReflect() protoreflect.Message { // Deprecated: Use SystemInfoVolume.ProtoReflect.Descriptor instead. func (*SystemInfoVolume) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{27} + return file_atelet_proto_rawDescGZIP(), []int{34} } func (x *SystemInfoVolume) GetDataSources() []*SystemInfoDataSource { @@ -2050,7 +2417,7 @@ type Volume struct { func (x *Volume) Reset() { *x = Volume{} - mi := &file_atelet_proto_msgTypes[28] + mi := &file_atelet_proto_msgTypes[35] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2062,7 +2429,7 @@ func (x *Volume) String() string { func (*Volume) ProtoMessage() {} func (x *Volume) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[28] + mi := &file_atelet_proto_msgTypes[35] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2075,7 +2442,7 @@ func (x *Volume) ProtoReflect() protoreflect.Message { // Deprecated: Use Volume.ProtoReflect.Descriptor instead. func (*Volume) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{28} + return file_atelet_proto_rawDescGZIP(), []int{35} } func (x *Volume) GetName() string { @@ -2132,7 +2499,7 @@ type VolumeMount struct { func (x *VolumeMount) Reset() { *x = VolumeMount{} - mi := &file_atelet_proto_msgTypes[29] + mi := &file_atelet_proto_msgTypes[36] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2144,7 +2511,7 @@ func (x *VolumeMount) String() string { func (*VolumeMount) ProtoMessage() {} func (x *VolumeMount) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[29] + mi := &file_atelet_proto_msgTypes[36] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2157,7 +2524,7 @@ func (x *VolumeMount) ProtoReflect() protoreflect.Message { // Deprecated: Use VolumeMount.ProtoReflect.Descriptor instead. func (*VolumeMount) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{29} + return file_atelet_proto_rawDescGZIP(), []int{36} } func (x *VolumeMount) GetName() string { @@ -2222,7 +2589,7 @@ type Container struct { func (x *Container) Reset() { *x = Container{} - mi := &file_atelet_proto_msgTypes[30] + mi := &file_atelet_proto_msgTypes[37] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2234,7 +2601,7 @@ func (x *Container) String() string { func (*Container) ProtoMessage() {} func (x *Container) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[30] + mi := &file_atelet_proto_msgTypes[37] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2247,7 +2614,7 @@ func (x *Container) ProtoReflect() protoreflect.Message { // Deprecated: Use Container.ProtoReflect.Descriptor instead. func (*Container) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{30} + return file_atelet_proto_rawDescGZIP(), []int{37} } func (x *Container) GetName() string { @@ -2324,7 +2691,7 @@ type SecurityContext struct { func (x *SecurityContext) Reset() { *x = SecurityContext{} - mi := &file_atelet_proto_msgTypes[31] + mi := &file_atelet_proto_msgTypes[38] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2336,7 +2703,7 @@ func (x *SecurityContext) String() string { func (*SecurityContext) ProtoMessage() {} func (x *SecurityContext) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[31] + mi := &file_atelet_proto_msgTypes[38] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2349,7 +2716,7 @@ func (x *SecurityContext) ProtoReflect() protoreflect.Message { // Deprecated: Use SecurityContext.ProtoReflect.Descriptor instead. func (*SecurityContext) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{31} + return file_atelet_proto_rawDescGZIP(), []int{38} } func (x *SecurityContext) GetCapabilities() *Capabilities { @@ -2379,7 +2746,7 @@ type Capabilities struct { func (x *Capabilities) Reset() { *x = Capabilities{} - mi := &file_atelet_proto_msgTypes[32] + mi := &file_atelet_proto_msgTypes[39] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2391,7 +2758,7 @@ func (x *Capabilities) String() string { func (*Capabilities) ProtoMessage() {} func (x *Capabilities) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[32] + mi := &file_atelet_proto_msgTypes[39] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2404,7 +2771,7 @@ func (x *Capabilities) ProtoReflect() protoreflect.Message { // Deprecated: Use Capabilities.ProtoReflect.Descriptor instead. func (*Capabilities) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{32} + return file_atelet_proto_rawDescGZIP(), []int{39} } func (x *Capabilities) GetAdd() []string { @@ -2442,7 +2809,7 @@ type ResourceLimits struct { func (x *ResourceLimits) Reset() { *x = ResourceLimits{} - mi := &file_atelet_proto_msgTypes[33] + mi := &file_atelet_proto_msgTypes[40] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2454,7 +2821,7 @@ func (x *ResourceLimits) String() string { func (*ResourceLimits) ProtoMessage() {} func (x *ResourceLimits) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[33] + mi := &file_atelet_proto_msgTypes[40] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2467,7 +2834,7 @@ func (x *ResourceLimits) ProtoReflect() protoreflect.Message { // Deprecated: Use ResourceLimits.ProtoReflect.Descriptor instead. func (*ResourceLimits) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{33} + return file_atelet_proto_rawDescGZIP(), []int{40} } func (x *ResourceLimits) GetMemoryBytes() int64 { @@ -2501,7 +2868,7 @@ type EnvEntry struct { func (x *EnvEntry) Reset() { *x = EnvEntry{} - mi := &file_atelet_proto_msgTypes[34] + mi := &file_atelet_proto_msgTypes[41] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2513,7 +2880,7 @@ func (x *EnvEntry) String() string { func (*EnvEntry) ProtoMessage() {} func (x *EnvEntry) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[34] + mi := &file_atelet_proto_msgTypes[41] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2526,7 +2893,7 @@ func (x *EnvEntry) ProtoReflect() protoreflect.Message { // Deprecated: Use EnvEntry.ProtoReflect.Descriptor instead. func (*EnvEntry) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{34} + return file_atelet_proto_rawDescGZIP(), []int{41} } func (x *EnvEntry) GetName() string { @@ -2561,7 +2928,7 @@ type WakeupProbe struct { func (x *WakeupProbe) Reset() { *x = WakeupProbe{} - mi := &file_atelet_proto_msgTypes[35] + mi := &file_atelet_proto_msgTypes[42] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2573,7 +2940,7 @@ func (x *WakeupProbe) String() string { func (*WakeupProbe) ProtoMessage() {} func (x *WakeupProbe) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[35] + mi := &file_atelet_proto_msgTypes[42] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2586,7 +2953,7 @@ func (x *WakeupProbe) ProtoReflect() protoreflect.Message { // Deprecated: Use WakeupProbe.ProtoReflect.Descriptor instead. func (*WakeupProbe) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{35} + return file_atelet_proto_rawDescGZIP(), []int{42} } func (x *WakeupProbe) GetHttpGet() *HttpGetAction { @@ -2624,7 +2991,7 @@ type HttpGetAction struct { func (x *HttpGetAction) Reset() { *x = HttpGetAction{} - mi := &file_atelet_proto_msgTypes[36] + mi := &file_atelet_proto_msgTypes[43] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2636,7 +3003,7 @@ func (x *HttpGetAction) String() string { func (*HttpGetAction) ProtoMessage() {} func (x *HttpGetAction) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[36] + mi := &file_atelet_proto_msgTypes[43] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2649,7 +3016,7 @@ func (x *HttpGetAction) ProtoReflect() protoreflect.Message { // Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead. func (*HttpGetAction) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{36} + return file_atelet_proto_rawDescGZIP(), []int{43} } func (x *HttpGetAction) GetPath() string { @@ -2674,7 +3041,7 @@ type RunResponse struct { func (x *RunResponse) Reset() { *x = RunResponse{} - mi := &file_atelet_proto_msgTypes[37] + mi := &file_atelet_proto_msgTypes[44] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2686,7 +3053,7 @@ func (x *RunResponse) String() string { func (*RunResponse) ProtoMessage() {} func (x *RunResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[37] + mi := &file_atelet_proto_msgTypes[44] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2699,7 +3066,7 @@ func (x *RunResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RunResponse.ProtoReflect.Descriptor instead. func (*RunResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{37} + return file_atelet_proto_rawDescGZIP(), []int{44} } type ObjectStorage struct { @@ -2717,7 +3084,7 @@ type ObjectStorage struct { func (x *ObjectStorage) Reset() { *x = ObjectStorage{} - mi := &file_atelet_proto_msgTypes[38] + mi := &file_atelet_proto_msgTypes[45] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2729,7 +3096,7 @@ func (x *ObjectStorage) String() string { func (*ObjectStorage) ProtoMessage() {} func (x *ObjectStorage) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[38] + mi := &file_atelet_proto_msgTypes[45] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2742,7 +3109,7 @@ func (x *ObjectStorage) ProtoReflect() protoreflect.Message { // Deprecated: Use ObjectStorage.ProtoReflect.Descriptor instead. func (*ObjectStorage) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{38} + return file_atelet_proto_rawDescGZIP(), []int{45} } func (x *ObjectStorage) GetSnapshotUri() string { @@ -2792,8 +3159,7 @@ type Snapshot struct { // usable local copy. // // TODO: Temporarily, this is optional while we haven't implemented deferred - // uploads. Restore also uses its presence to choose between the local copy - // and object storage; see RestoreRequest.snapshot. + // uploads. // // +k8s:optional Object *ObjectStorage `protobuf:"bytes,4,opt,name=object,proto3" json:"object,omitempty"` @@ -2803,7 +3169,7 @@ type Snapshot struct { func (x *Snapshot) Reset() { *x = Snapshot{} - mi := &file_atelet_proto_msgTypes[39] + mi := &file_atelet_proto_msgTypes[46] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2815,7 +3181,7 @@ func (x *Snapshot) String() string { func (*Snapshot) ProtoMessage() {} func (x *Snapshot) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[39] + mi := &file_atelet_proto_msgTypes[46] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2828,7 +3194,7 @@ func (x *Snapshot) ProtoReflect() protoreflect.Message { // Deprecated: Use Snapshot.ProtoReflect.Descriptor instead. func (*Snapshot) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{39} + return file_atelet_proto_rawDescGZIP(), []int{46} } func (x *Snapshot) GetSnapshotUuid() string { @@ -2875,7 +3241,7 @@ type SnapshotReport struct { func (x *SnapshotReport) Reset() { *x = SnapshotReport{} - mi := &file_atelet_proto_msgTypes[40] + mi := &file_atelet_proto_msgTypes[47] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2887,7 +3253,7 @@ func (x *SnapshotReport) String() string { func (*SnapshotReport) ProtoMessage() {} func (x *SnapshotReport) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[40] + mi := &file_atelet_proto_msgTypes[47] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2900,7 +3266,7 @@ func (x *SnapshotReport) ProtoReflect() protoreflect.Message { // Deprecated: Use SnapshotReport.ProtoReflect.Descriptor instead. func (*SnapshotReport) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{40} + return file_atelet_proto_rawDescGZIP(), []int{47} } func (x *SnapshotReport) GetSnapshot() *Snapshot { @@ -2955,7 +3321,7 @@ type ActorSnapshotState struct { func (x *ActorSnapshotState) Reset() { *x = ActorSnapshotState{} - mi := &file_atelet_proto_msgTypes[41] + mi := &file_atelet_proto_msgTypes[48] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -2967,7 +3333,7 @@ func (x *ActorSnapshotState) String() string { func (*ActorSnapshotState) ProtoMessage() {} func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[41] + mi := &file_atelet_proto_msgTypes[48] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -2980,7 +3346,7 @@ func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message { // Deprecated: Use ActorSnapshotState.ProtoReflect.Descriptor instead. func (*ActorSnapshotState) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{41} + return file_atelet_proto_rawDescGZIP(), []int{48} } func (x *ActorSnapshotState) GetLatestLocal() *SnapshotReport { @@ -3024,7 +3390,7 @@ type UploadConfig struct { func (x *UploadConfig) Reset() { *x = UploadConfig{} - mi := &file_atelet_proto_msgTypes[42] + mi := &file_atelet_proto_msgTypes[49] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3036,7 +3402,7 @@ func (x *UploadConfig) String() string { func (*UploadConfig) ProtoMessage() {} func (x *UploadConfig) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[42] + mi := &file_atelet_proto_msgTypes[49] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3049,7 +3415,7 @@ func (x *UploadConfig) ProtoReflect() protoreflect.Message { // Deprecated: Use UploadConfig.ProtoReflect.Descriptor instead. func (*UploadConfig) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{42} + return file_atelet_proto_rawDescGZIP(), []int{49} } func (x *UploadConfig) GetMode() UploadMode { @@ -3112,7 +3478,7 @@ type CheckpointRequest struct { func (x *CheckpointRequest) Reset() { *x = CheckpointRequest{} - mi := &file_atelet_proto_msgTypes[43] + mi := &file_atelet_proto_msgTypes[50] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3124,7 +3490,7 @@ func (x *CheckpointRequest) String() string { func (*CheckpointRequest) ProtoMessage() {} func (x *CheckpointRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[43] + mi := &file_atelet_proto_msgTypes[50] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3137,7 +3503,7 @@ func (x *CheckpointRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointRequest.ProtoReflect.Descriptor instead. func (*CheckpointRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{43} + return file_atelet_proto_rawDescGZIP(), []int{50} } func (x *CheckpointRequest) GetWorkerPodUid() string { @@ -3219,7 +3585,7 @@ type CheckpointResponse struct { func (x *CheckpointResponse) Reset() { *x = CheckpointResponse{} - mi := &file_atelet_proto_msgTypes[44] + mi := &file_atelet_proto_msgTypes[51] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3231,7 +3597,7 @@ func (x *CheckpointResponse) String() string { func (*CheckpointResponse) ProtoMessage() {} func (x *CheckpointResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[44] + mi := &file_atelet_proto_msgTypes[51] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3244,7 +3610,7 @@ func (x *CheckpointResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointResponse.ProtoReflect.Descriptor instead. func (*CheckpointResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{44} + return file_atelet_proto_rawDescGZIP(), []int{51} } func (x *CheckpointResponse) GetState() *ActorSnapshotState { @@ -3286,9 +3652,9 @@ type RestoreRequest struct { // detaching the actor, it waits for the detach to finish and then reads // `snapshot.object`. Fails if neither copy exists: that generation is lost. // - // TODO: Temporarily, whether `snapshot.object` is set chooses the copy: - // atelet reads only the local copy when it is unset, and only - // `snapshot.object` when it is set, with no fallback either way. + // TODO: Temporarily, the control plane does not set `snapshot.object` when + // it restores a paused actor, so atelet has nothing to fall back to if that + // actor's local copy is missing or unusable. // // +k8s:required Snapshot *Snapshot `protobuf:"bytes,8,opt,name=snapshot,proto3" json:"snapshot,omitempty"` @@ -3319,7 +3685,7 @@ type RestoreRequest struct { func (x *RestoreRequest) Reset() { *x = RestoreRequest{} - mi := &file_atelet_proto_msgTypes[45] + mi := &file_atelet_proto_msgTypes[52] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3331,7 +3697,7 @@ func (x *RestoreRequest) String() string { func (*RestoreRequest) ProtoMessage() {} func (x *RestoreRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[45] + mi := &file_atelet_proto_msgTypes[52] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3344,7 +3710,7 @@ func (x *RestoreRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreRequest.ProtoReflect.Descriptor instead. func (*RestoreRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{45} + return file_atelet_proto_rawDescGZIP(), []int{52} } func (x *RestoreRequest) GetWorkerPodUid() string { @@ -3439,7 +3805,7 @@ type RestoreResponse struct { func (x *RestoreResponse) Reset() { *x = RestoreResponse{} - mi := &file_atelet_proto_msgTypes[46] + mi := &file_atelet_proto_msgTypes[53] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3451,7 +3817,7 @@ func (x *RestoreResponse) String() string { func (*RestoreResponse) ProtoMessage() {} func (x *RestoreResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[46] + mi := &file_atelet_proto_msgTypes[53] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3464,7 +3830,7 @@ func (x *RestoreResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreResponse.ProtoReflect.Descriptor instead. func (*RestoreResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{46} + return file_atelet_proto_rawDescGZIP(), []int{53} } type PromoteSnapshotRequest struct { @@ -3501,7 +3867,7 @@ type PromoteSnapshotRequest struct { func (x *PromoteSnapshotRequest) Reset() { *x = PromoteSnapshotRequest{} - mi := &file_atelet_proto_msgTypes[47] + mi := &file_atelet_proto_msgTypes[54] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3513,7 +3879,7 @@ func (x *PromoteSnapshotRequest) String() string { func (*PromoteSnapshotRequest) ProtoMessage() {} func (x *PromoteSnapshotRequest) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[47] + mi := &file_atelet_proto_msgTypes[54] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3526,7 +3892,7 @@ func (x *PromoteSnapshotRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use PromoteSnapshotRequest.ProtoReflect.Descriptor instead. func (*PromoteSnapshotRequest) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{47} + return file_atelet_proto_rawDescGZIP(), []int{54} } func (x *PromoteSnapshotRequest) GetAtespace() string { @@ -3585,7 +3951,7 @@ type PromoteSnapshotResponse struct { func (x *PromoteSnapshotResponse) Reset() { *x = PromoteSnapshotResponse{} - mi := &file_atelet_proto_msgTypes[48] + mi := &file_atelet_proto_msgTypes[55] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -3597,7 +3963,7 @@ func (x *PromoteSnapshotResponse) String() string { func (*PromoteSnapshotResponse) ProtoMessage() {} func (x *PromoteSnapshotResponse) ProtoReflect() protoreflect.Message { - mi := &file_atelet_proto_msgTypes[48] + mi := &file_atelet_proto_msgTypes[55] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -3610,7 +3976,7 @@ func (x *PromoteSnapshotResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use PromoteSnapshotResponse.ProtoReflect.Descriptor instead. func (*PromoteSnapshotResponse) Descriptor() ([]byte, []int) { - return file_atelet_proto_rawDescGZIP(), []int{48} + return file_atelet_proto_rawDescGZIP(), []int{55} } func (x *PromoteSnapshotResponse) GetState() *ActorSnapshotState { @@ -3654,7 +4020,27 @@ const file_atelet_proto_rawDesc = "" + "\n" + "actor_name\x18\x02 \x01(\tR\tactorName\x12\x1b\n" + "\tactor_uid\x18\x03 \x01(\tR\bactorUid\"\x1d\n" + - "\x1bRequestActorSuspendResponse\"\xc0\x01\n" + + "\x1bRequestActorSuspendResponse\"c\n" + + "\x1bGetActorEgressPolicyRequest\x12%\n" + + "\x0eactor_atespace\x18\x01 \x01(\tR\ractorAtespace\x12\x1d\n" + + "\n" + + "actor_name\x18\x02 \x01(\tR\tactorName\"Y\n" + + "\x1cGetActorEgressPolicyResponse\x129\n" + + "\regress_policy\x18\x01 \x01(\v2\x14.atelet.EgressPolicyR\fegressPolicy\"8\n" + + "\fEgressPolicy\x12(\n" + + "\x05rules\x18\x01 \x03(\v2\x12.atelet.EgressRuleR\x05rules\"\xb3\x01\n" + + "\n" + + "EgressRule\x12.\n" + + "\x04http\x18\x01 \x01(\v2\x1a.atelet.EgressDestinationsR\x04http\x120\n" + + "\x05https\x18\x02 \x01(\v2\x1a.atelet.EgressDestinationsR\x05https\x12C\n" + + "\x0ftls_passthrough\x18\x03 \x01(\v2\x1a.atelet.EgressDestinationsR\x0etlsPassthrough\"]\n" + + "\x12EgressDestinations\x12\x1c\n" + + "\thostnames\x18\x01 \x03(\tR\thostnames\x12)\n" + + "\x05ports\x18\x02 \x01(\v2\x13.atelet.EgressPortsR\x05ports\"Q\n" + + "\vEgressPorts\x12(\n" + + "\x03all\x18\x01 \x01(\v2\x16.atelet.EgressAllPortsR\x03all\x12\x18\n" + + "\anumbers\x18\x02 \x03(\x05R\anumbers\"\x10\n" + + "\x0eEgressAllPorts\"\xc0\x01\n" + "\x1bMintActorCertificateRequest\x12%\n" + "\x0eactor_atespace\x18\x03 \x01(\tR\ractorAtespace\x12\x1d\n" + "\n" + @@ -3856,11 +4242,12 @@ const file_atelet_proto_rawDesc = "" + "UploadMode\x12\x1b\n" + "\x17UPLOAD_MODE_UNSPECIFIED\x10\x00\x12\x18\n" + "\x14UPLOAD_MODE_DEFERRED\x10\x01\x12\x19\n" + - "\x15UPLOAD_MODE_IMMEDIATE\x10\x022\xa8\x02\n" + + "\x15UPLOAD_MODE_IMMEDIATE\x10\x022\x8d\x03\n" + "\fAteomSupport\x12c\n" + "\x14MintActorCertificate\x12#.atelet.MintActorCertificateRequest\x1a$.atelet.MintActorCertificateResponse\"\x00\x12Q\n" + "\x0eRegisterWorker\x12\x1d.atelet.RegisterWorkerRequest\x1a\x1e.atelet.RegisterWorkerResponse\"\x00\x12`\n" + - "\x13RequestActorSuspend\x12\".atelet.RequestActorSuspendRequest\x1a#.atelet.RequestActorSuspendResponse\"\x002\xde\x02\n" + + "\x13RequestActorSuspend\x12\".atelet.RequestActorSuspendRequest\x1a#.atelet.RequestActorSuspendResponse\"\x00\x12c\n" + + "\x14GetActorEgressPolicy\x12#.atelet.GetActorEgressPolicyRequest\x1a$.atelet.GetActorEgressPolicyResponse\"\x002\xde\x02\n" + "\vAteomHerder\x120\n" + "\x03Run\x12\x12.atelet.RunRequest\x1a\x13.atelet.RunResponse\"\x00\x12E\n" + "\n" + @@ -3882,7 +4269,7 @@ func file_atelet_proto_rawDescGZIP() []byte { } var file_atelet_proto_enumTypes = make([]protoimpl.EnumInfo, 3) -var file_atelet_proto_msgTypes = make([]protoimpl.MessageInfo, 53) +var file_atelet_proto_msgTypes = make([]protoimpl.MessageInfo, 60) var file_atelet_proto_goTypes = []any{ (ActorMetadataField)(0), // 0: atelet.ActorMetadataField (SnapshotFidelity)(0), // 1: atelet.SnapshotFidelity @@ -3897,50 +4284,57 @@ var file_atelet_proto_goTypes = []any{ (*RegisterWorkerResponse)(nil), // 10: atelet.RegisterWorkerResponse (*RequestActorSuspendRequest)(nil), // 11: atelet.RequestActorSuspendRequest (*RequestActorSuspendResponse)(nil), // 12: atelet.RequestActorSuspendResponse - (*MintActorCertificateRequest)(nil), // 13: atelet.MintActorCertificateRequest - (*MintActorCertificateResponse)(nil), // 14: atelet.MintActorCertificateResponse - (*TerminateRequest)(nil), // 15: atelet.TerminateRequest - (*TerminateResponse)(nil), // 16: atelet.TerminateResponse - (*RunRequest)(nil), // 17: atelet.RunRequest - (*EgressGateway)(nil), // 18: atelet.EgressGateway - (*AssetFile)(nil), // 19: atelet.AssetFile - (*ArchAssets)(nil), // 20: atelet.ArchAssets - (*SandboxAssets)(nil), // 21: atelet.SandboxAssets - (*WorkloadSpec)(nil), // 22: atelet.WorkloadSpec - (*DurableDirVolume)(nil), // 23: atelet.DurableDirVolume - (*ExternalVolumeSource)(nil), // 24: atelet.ExternalVolumeSource - (*ImageVolumeSource)(nil), // 25: atelet.ImageVolumeSource - (*ActorMetadataItem)(nil), // 26: atelet.ActorMetadataItem - (*ActorMetadataDataSource)(nil), // 27: atelet.ActorMetadataDataSource - (*TrustBundleDataSource)(nil), // 28: atelet.TrustBundleDataSource - (*SystemInfoDataSource)(nil), // 29: atelet.SystemInfoDataSource - (*SystemInfoVolume)(nil), // 30: atelet.SystemInfoVolume - (*Volume)(nil), // 31: atelet.Volume - (*VolumeMount)(nil), // 32: atelet.VolumeMount - (*Container)(nil), // 33: atelet.Container - (*SecurityContext)(nil), // 34: atelet.SecurityContext - (*Capabilities)(nil), // 35: atelet.Capabilities - (*ResourceLimits)(nil), // 36: atelet.ResourceLimits - (*EnvEntry)(nil), // 37: atelet.EnvEntry - (*WakeupProbe)(nil), // 38: atelet.WakeupProbe - (*HttpGetAction)(nil), // 39: atelet.HttpGetAction - (*RunResponse)(nil), // 40: atelet.RunResponse - (*ObjectStorage)(nil), // 41: atelet.ObjectStorage - (*Snapshot)(nil), // 42: atelet.Snapshot - (*SnapshotReport)(nil), // 43: atelet.SnapshotReport - (*ActorSnapshotState)(nil), // 44: atelet.ActorSnapshotState - (*UploadConfig)(nil), // 45: atelet.UploadConfig - (*CheckpointRequest)(nil), // 46: atelet.CheckpointRequest - (*CheckpointResponse)(nil), // 47: atelet.CheckpointResponse - (*RestoreRequest)(nil), // 48: atelet.RestoreRequest - (*RestoreResponse)(nil), // 49: atelet.RestoreResponse - (*PromoteSnapshotRequest)(nil), // 50: atelet.PromoteSnapshotRequest - (*PromoteSnapshotResponse)(nil), // 51: atelet.PromoteSnapshotResponse - nil, // 52: atelet.ArchAssets.FilesEntry - nil, // 53: atelet.SandboxAssets.AssetsEntry - nil, // 54: atelet.ExternalVolumeSource.VolumeContextEntry - nil, // 55: atelet.ExternalVolumeSource.PublishContextEntry - (*timestamppb.Timestamp)(nil), // 56: google.protobuf.Timestamp + (*GetActorEgressPolicyRequest)(nil), // 13: atelet.GetActorEgressPolicyRequest + (*GetActorEgressPolicyResponse)(nil), // 14: atelet.GetActorEgressPolicyResponse + (*EgressPolicy)(nil), // 15: atelet.EgressPolicy + (*EgressRule)(nil), // 16: atelet.EgressRule + (*EgressDestinations)(nil), // 17: atelet.EgressDestinations + (*EgressPorts)(nil), // 18: atelet.EgressPorts + (*EgressAllPorts)(nil), // 19: atelet.EgressAllPorts + (*MintActorCertificateRequest)(nil), // 20: atelet.MintActorCertificateRequest + (*MintActorCertificateResponse)(nil), // 21: atelet.MintActorCertificateResponse + (*TerminateRequest)(nil), // 22: atelet.TerminateRequest + (*TerminateResponse)(nil), // 23: atelet.TerminateResponse + (*RunRequest)(nil), // 24: atelet.RunRequest + (*EgressGateway)(nil), // 25: atelet.EgressGateway + (*AssetFile)(nil), // 26: atelet.AssetFile + (*ArchAssets)(nil), // 27: atelet.ArchAssets + (*SandboxAssets)(nil), // 28: atelet.SandboxAssets + (*WorkloadSpec)(nil), // 29: atelet.WorkloadSpec + (*DurableDirVolume)(nil), // 30: atelet.DurableDirVolume + (*ExternalVolumeSource)(nil), // 31: atelet.ExternalVolumeSource + (*ImageVolumeSource)(nil), // 32: atelet.ImageVolumeSource + (*ActorMetadataItem)(nil), // 33: atelet.ActorMetadataItem + (*ActorMetadataDataSource)(nil), // 34: atelet.ActorMetadataDataSource + (*TrustBundleDataSource)(nil), // 35: atelet.TrustBundleDataSource + (*SystemInfoDataSource)(nil), // 36: atelet.SystemInfoDataSource + (*SystemInfoVolume)(nil), // 37: atelet.SystemInfoVolume + (*Volume)(nil), // 38: atelet.Volume + (*VolumeMount)(nil), // 39: atelet.VolumeMount + (*Container)(nil), // 40: atelet.Container + (*SecurityContext)(nil), // 41: atelet.SecurityContext + (*Capabilities)(nil), // 42: atelet.Capabilities + (*ResourceLimits)(nil), // 43: atelet.ResourceLimits + (*EnvEntry)(nil), // 44: atelet.EnvEntry + (*WakeupProbe)(nil), // 45: atelet.WakeupProbe + (*HttpGetAction)(nil), // 46: atelet.HttpGetAction + (*RunResponse)(nil), // 47: atelet.RunResponse + (*ObjectStorage)(nil), // 48: atelet.ObjectStorage + (*Snapshot)(nil), // 49: atelet.Snapshot + (*SnapshotReport)(nil), // 50: atelet.SnapshotReport + (*ActorSnapshotState)(nil), // 51: atelet.ActorSnapshotState + (*UploadConfig)(nil), // 52: atelet.UploadConfig + (*CheckpointRequest)(nil), // 53: atelet.CheckpointRequest + (*CheckpointResponse)(nil), // 54: atelet.CheckpointResponse + (*RestoreRequest)(nil), // 55: atelet.RestoreRequest + (*RestoreResponse)(nil), // 56: atelet.RestoreResponse + (*PromoteSnapshotRequest)(nil), // 57: atelet.PromoteSnapshotRequest + (*PromoteSnapshotResponse)(nil), // 58: atelet.PromoteSnapshotResponse + nil, // 59: atelet.ArchAssets.FilesEntry + nil, // 60: atelet.SandboxAssets.AssetsEntry + nil, // 61: atelet.ExternalVolumeSource.VolumeContextEntry + nil, // 62: atelet.ExternalVolumeSource.PublishContextEntry + (*timestamppb.Timestamp)(nil), // 63: google.protobuf.Timestamp } var file_atelet_proto_depIdxs = []int32{ 7, // 0: atelet.RegisterWorkerRequest.capacity:type_name -> atelet.WorkerResources @@ -3950,75 +4344,84 @@ var file_atelet_proto_depIdxs = []int32{ 6, // 4: atelet.VersionedSandboxCompat.attributes:type_name -> atelet.AttributeEntry 8, // 5: atelet.WorkerResources.resources:type_name -> atelet.Resources 9, // 6: atelet.Resources.limits:type_name -> atelet.Limits - 22, // 7: atelet.TerminateRequest.spec:type_name -> atelet.WorkloadSpec - 22, // 8: atelet.RunRequest.spec:type_name -> atelet.WorkloadSpec - 21, // 9: atelet.RunRequest.sandbox_assets:type_name -> atelet.SandboxAssets - 18, // 10: atelet.RunRequest.egress_gateway:type_name -> atelet.EgressGateway - 52, // 11: atelet.ArchAssets.files:type_name -> atelet.ArchAssets.FilesEntry - 53, // 12: atelet.SandboxAssets.assets:type_name -> atelet.SandboxAssets.AssetsEntry - 33, // 13: atelet.WorkloadSpec.containers:type_name -> atelet.Container - 31, // 14: atelet.WorkloadSpec.volumes:type_name -> atelet.Volume - 54, // 15: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry - 55, // 16: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry - 0, // 17: atelet.ActorMetadataItem.field:type_name -> atelet.ActorMetadataField - 26, // 18: atelet.ActorMetadataDataSource.items:type_name -> atelet.ActorMetadataItem - 27, // 19: atelet.SystemInfoDataSource.actor_metadata:type_name -> atelet.ActorMetadataDataSource - 28, // 20: atelet.SystemInfoDataSource.trust_bundle:type_name -> atelet.TrustBundleDataSource - 29, // 21: atelet.SystemInfoVolume.data_sources:type_name -> atelet.SystemInfoDataSource - 23, // 22: atelet.Volume.durable_dir:type_name -> atelet.DurableDirVolume - 24, // 23: atelet.Volume.external:type_name -> atelet.ExternalVolumeSource - 30, // 24: atelet.Volume.system_info:type_name -> atelet.SystemInfoVolume - 25, // 25: atelet.Volume.image:type_name -> atelet.ImageVolumeSource - 37, // 26: atelet.Container.env:type_name -> atelet.EnvEntry - 38, // 27: atelet.Container.wakeup_probe:type_name -> atelet.WakeupProbe - 32, // 28: atelet.Container.volume_mounts:type_name -> atelet.VolumeMount - 34, // 29: atelet.Container.security_context:type_name -> atelet.SecurityContext - 36, // 30: atelet.Container.resources:type_name -> atelet.ResourceLimits - 35, // 31: atelet.SecurityContext.capabilities:type_name -> atelet.Capabilities - 39, // 32: atelet.WakeupProbe.http_get:type_name -> atelet.HttpGetAction - 1, // 33: atelet.Snapshot.fidelity:type_name -> atelet.SnapshotFidelity - 41, // 34: atelet.Snapshot.object:type_name -> atelet.ObjectStorage - 42, // 35: atelet.SnapshotReport.snapshot:type_name -> atelet.Snapshot - 56, // 36: atelet.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp - 56, // 37: atelet.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp - 43, // 38: atelet.ActorSnapshotState.latest_local:type_name -> atelet.SnapshotReport - 43, // 39: atelet.ActorSnapshotState.uploading:type_name -> atelet.SnapshotReport - 43, // 40: atelet.ActorSnapshotState.latest_durable:type_name -> atelet.SnapshotReport - 56, // 41: atelet.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp - 2, // 42: atelet.UploadConfig.mode:type_name -> atelet.UploadMode - 22, // 43: atelet.CheckpointRequest.spec:type_name -> atelet.WorkloadSpec - 42, // 44: atelet.CheckpointRequest.snapshot:type_name -> atelet.Snapshot - 45, // 45: atelet.CheckpointRequest.upload_config:type_name -> atelet.UploadConfig - 44, // 46: atelet.CheckpointResponse.state:type_name -> atelet.ActorSnapshotState - 22, // 47: atelet.RestoreRequest.spec:type_name -> atelet.WorkloadSpec - 42, // 48: atelet.RestoreRequest.snapshot:type_name -> atelet.Snapshot - 18, // 49: atelet.RestoreRequest.egress_gateway:type_name -> atelet.EgressGateway - 21, // 50: atelet.RestoreRequest.sandbox_assets:type_name -> atelet.SandboxAssets - 42, // 51: atelet.PromoteSnapshotRequest.snapshot:type_name -> atelet.Snapshot - 44, // 52: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState - 19, // 53: atelet.ArchAssets.FilesEntry.value:type_name -> atelet.AssetFile - 20, // 54: atelet.SandboxAssets.AssetsEntry.value:type_name -> atelet.ArchAssets - 13, // 55: atelet.AteomSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest - 3, // 56: atelet.AteomSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest - 11, // 57: atelet.AteomSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest - 17, // 58: atelet.AteomHerder.Run:input_type -> atelet.RunRequest - 46, // 59: atelet.AteomHerder.Checkpoint:input_type -> atelet.CheckpointRequest - 48, // 60: atelet.AteomHerder.Restore:input_type -> atelet.RestoreRequest - 50, // 61: atelet.AteomHerder.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest - 15, // 62: atelet.AteomHerder.Terminate:input_type -> atelet.TerminateRequest - 14, // 63: atelet.AteomSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse - 10, // 64: atelet.AteomSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse - 12, // 65: atelet.AteomSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse - 40, // 66: atelet.AteomHerder.Run:output_type -> atelet.RunResponse - 47, // 67: atelet.AteomHerder.Checkpoint:output_type -> atelet.CheckpointResponse - 49, // 68: atelet.AteomHerder.Restore:output_type -> atelet.RestoreResponse - 51, // 69: atelet.AteomHerder.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse - 16, // 70: atelet.AteomHerder.Terminate:output_type -> atelet.TerminateResponse - 63, // [63:71] is the sub-list for method output_type - 55, // [55:63] is the sub-list for method input_type - 55, // [55:55] is the sub-list for extension type_name - 55, // [55:55] is the sub-list for extension extendee - 0, // [0:55] is the sub-list for field type_name + 15, // 7: atelet.GetActorEgressPolicyResponse.egress_policy:type_name -> atelet.EgressPolicy + 16, // 8: atelet.EgressPolicy.rules:type_name -> atelet.EgressRule + 17, // 9: atelet.EgressRule.http:type_name -> atelet.EgressDestinations + 17, // 10: atelet.EgressRule.https:type_name -> atelet.EgressDestinations + 17, // 11: atelet.EgressRule.tls_passthrough:type_name -> atelet.EgressDestinations + 18, // 12: atelet.EgressDestinations.ports:type_name -> atelet.EgressPorts + 19, // 13: atelet.EgressPorts.all:type_name -> atelet.EgressAllPorts + 29, // 14: atelet.TerminateRequest.spec:type_name -> atelet.WorkloadSpec + 29, // 15: atelet.RunRequest.spec:type_name -> atelet.WorkloadSpec + 28, // 16: atelet.RunRequest.sandbox_assets:type_name -> atelet.SandboxAssets + 25, // 17: atelet.RunRequest.egress_gateway:type_name -> atelet.EgressGateway + 59, // 18: atelet.ArchAssets.files:type_name -> atelet.ArchAssets.FilesEntry + 60, // 19: atelet.SandboxAssets.assets:type_name -> atelet.SandboxAssets.AssetsEntry + 40, // 20: atelet.WorkloadSpec.containers:type_name -> atelet.Container + 38, // 21: atelet.WorkloadSpec.volumes:type_name -> atelet.Volume + 61, // 22: atelet.ExternalVolumeSource.volume_context:type_name -> atelet.ExternalVolumeSource.VolumeContextEntry + 62, // 23: atelet.ExternalVolumeSource.publish_context:type_name -> atelet.ExternalVolumeSource.PublishContextEntry + 0, // 24: atelet.ActorMetadataItem.field:type_name -> atelet.ActorMetadataField + 33, // 25: atelet.ActorMetadataDataSource.items:type_name -> atelet.ActorMetadataItem + 34, // 26: atelet.SystemInfoDataSource.actor_metadata:type_name -> atelet.ActorMetadataDataSource + 35, // 27: atelet.SystemInfoDataSource.trust_bundle:type_name -> atelet.TrustBundleDataSource + 36, // 28: atelet.SystemInfoVolume.data_sources:type_name -> atelet.SystemInfoDataSource + 30, // 29: atelet.Volume.durable_dir:type_name -> atelet.DurableDirVolume + 31, // 30: atelet.Volume.external:type_name -> atelet.ExternalVolumeSource + 37, // 31: atelet.Volume.system_info:type_name -> atelet.SystemInfoVolume + 32, // 32: atelet.Volume.image:type_name -> atelet.ImageVolumeSource + 44, // 33: atelet.Container.env:type_name -> atelet.EnvEntry + 45, // 34: atelet.Container.wakeup_probe:type_name -> atelet.WakeupProbe + 39, // 35: atelet.Container.volume_mounts:type_name -> atelet.VolumeMount + 41, // 36: atelet.Container.security_context:type_name -> atelet.SecurityContext + 43, // 37: atelet.Container.resources:type_name -> atelet.ResourceLimits + 42, // 38: atelet.SecurityContext.capabilities:type_name -> atelet.Capabilities + 46, // 39: atelet.WakeupProbe.http_get:type_name -> atelet.HttpGetAction + 1, // 40: atelet.Snapshot.fidelity:type_name -> atelet.SnapshotFidelity + 48, // 41: atelet.Snapshot.object:type_name -> atelet.ObjectStorage + 49, // 42: atelet.SnapshotReport.snapshot:type_name -> atelet.Snapshot + 63, // 43: atelet.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp + 63, // 44: atelet.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp + 50, // 45: atelet.ActorSnapshotState.latest_local:type_name -> atelet.SnapshotReport + 50, // 46: atelet.ActorSnapshotState.uploading:type_name -> atelet.SnapshotReport + 50, // 47: atelet.ActorSnapshotState.latest_durable:type_name -> atelet.SnapshotReport + 63, // 48: atelet.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp + 2, // 49: atelet.UploadConfig.mode:type_name -> atelet.UploadMode + 29, // 50: atelet.CheckpointRequest.spec:type_name -> atelet.WorkloadSpec + 49, // 51: atelet.CheckpointRequest.snapshot:type_name -> atelet.Snapshot + 52, // 52: atelet.CheckpointRequest.upload_config:type_name -> atelet.UploadConfig + 51, // 53: atelet.CheckpointResponse.state:type_name -> atelet.ActorSnapshotState + 29, // 54: atelet.RestoreRequest.spec:type_name -> atelet.WorkloadSpec + 49, // 55: atelet.RestoreRequest.snapshot:type_name -> atelet.Snapshot + 25, // 56: atelet.RestoreRequest.egress_gateway:type_name -> atelet.EgressGateway + 28, // 57: atelet.RestoreRequest.sandbox_assets:type_name -> atelet.SandboxAssets + 49, // 58: atelet.PromoteSnapshotRequest.snapshot:type_name -> atelet.Snapshot + 51, // 59: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState + 26, // 60: atelet.ArchAssets.FilesEntry.value:type_name -> atelet.AssetFile + 27, // 61: atelet.SandboxAssets.AssetsEntry.value:type_name -> atelet.ArchAssets + 20, // 62: atelet.AteomSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest + 3, // 63: atelet.AteomSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest + 11, // 64: atelet.AteomSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest + 13, // 65: atelet.AteomSupport.GetActorEgressPolicy:input_type -> atelet.GetActorEgressPolicyRequest + 24, // 66: atelet.AteomHerder.Run:input_type -> atelet.RunRequest + 53, // 67: atelet.AteomHerder.Checkpoint:input_type -> atelet.CheckpointRequest + 55, // 68: atelet.AteomHerder.Restore:input_type -> atelet.RestoreRequest + 57, // 69: atelet.AteomHerder.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest + 22, // 70: atelet.AteomHerder.Terminate:input_type -> atelet.TerminateRequest + 21, // 71: atelet.AteomSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse + 10, // 72: atelet.AteomSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse + 12, // 73: atelet.AteomSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse + 14, // 74: atelet.AteomSupport.GetActorEgressPolicy:output_type -> atelet.GetActorEgressPolicyResponse + 47, // 75: atelet.AteomHerder.Run:output_type -> atelet.RunResponse + 54, // 76: atelet.AteomHerder.Checkpoint:output_type -> atelet.CheckpointResponse + 56, // 77: atelet.AteomHerder.Restore:output_type -> atelet.RestoreResponse + 58, // 78: atelet.AteomHerder.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse + 23, // 79: atelet.AteomHerder.Terminate:output_type -> atelet.TerminateResponse + 71, // [71:80] is the sub-list for method output_type + 62, // [62:71] is the sub-list for method input_type + 62, // [62:62] is the sub-list for extension type_name + 62, // [62:62] is the sub-list for extension extendee + 0, // [0:62] is the sub-list for field type_name } func init() { file_atelet_proto_init() } @@ -4026,15 +4429,15 @@ func file_atelet_proto_init() { if File_atelet_proto != nil { return } - file_atelet_proto_msgTypes[14].OneofWrappers = []any{} - file_atelet_proto_msgTypes[45].OneofWrappers = []any{} + file_atelet_proto_msgTypes[21].OneofWrappers = []any{} + file_atelet_proto_msgTypes[52].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_atelet_proto_rawDesc), len(file_atelet_proto_rawDesc)), NumEnums: 3, - NumMessages: 53, + NumMessages: 60, NumExtensions: 0, NumServices: 2, }, diff --git a/internal/proto/ateletpb/atelet.proto b/internal/proto/ateletpb/atelet.proto index aff782894d..06ce0fff84 100644 --- a/internal/proto/ateletpb/atelet.proto +++ b/internal/proto/ateletpb/atelet.proto @@ -45,6 +45,12 @@ service AteomSupport { // would need: the control plane answers this by driving a suspend back // through atelet into the calling ateom. rpc RequestActorSuspend(RequestActorSuspendRequest) returns (RequestActorSuspendResponse) {} + + // GetActorEgressPolicy returns the egress policy of the actor this + // worker hosts, which atelet reads from the control plane's + // Control.GetActorEgressPolicy. An error is the control plane's, passed + // through as it was given. + rpc GetActorEgressPolicy(GetActorEgressPolicyRequest) returns (GetActorEgressPolicyResponse) {} } message RegisterWorkerRequest { @@ -78,6 +84,7 @@ message SandboxRuntime { // // +k8s:required // +k8s:maxLength=63 + // +k8s:customValidation # gvisor or microvm string sandbox_class = 1; // Primary compatibility version stamped onto an actor's snapshot when a @@ -204,6 +211,71 @@ message RequestActorSuspendRequest { message RequestActorSuspendResponse { } +message GetActorEgressPolicyRequest { + // The actor the calling worker is hosting. atelet reads its policy from the + // control plane by name. + // + // +k8s:required + // +k8s:format=k8s-short-name + string actor_atespace = 1; + + // +k8s:required + // +k8s:format=k8s-short-name + string actor_name = 2; +} + +message GetActorEgressPolicyResponse { + // The actor's policy, copied from the control plane's. Unset if the actor + // has no policy, which denies all of its egress, DNS included. + EgressPolicy egress_policy = 1; +} + +// EgressPolicy is the part of the control plane's EgressPolicy that a worker +// enforces itself: which destinations each rule allows. The resource metadata +// and the rules' effects are left out; only the egress gateway applies them, +// so the credentials they refer to never reach the worker. +message EgressPolicy { + // Rules are an unordered set of allows, as in the control plane's policy. + repeated EgressRule rules = 1; +} + +// EgressRule mirrors the control plane's EgressRule: exactly one field is set, +// naming the rule's protocol. A rule with none set has a protocol this copy +// does not carry yet, and allows nothing. +message EgressRule { + // Cleartext HTTP to the named hosts. + EgressDestinations http = 1; + + // HTTPS to the named hosts, intercepted by the egress gateway. + EgressDestinations https = 2; + + // TLS to the named servers, forwarded without decryption. + EgressDestinations tls_passthrough = 3; +} + +// EgressDestinations is the hostnames and ports a rule allows, copied from the +// control plane's HTTPRule, HTTPSRule, or TLSPassthroughRule. +message EgressDestinations { + // Hostnames as the control plane's rule has them, wildcards included. + repeated string hostnames = 1; + + // The ports the rule allows, with the control plane's defaults applied. + EgressPorts ports = 2; +} + +// EgressPorts mirrors the control plane's Ports: either all ports, or the +// listed numbers. +message EgressPorts { + // Set if the rule allows every port. + EgressAllPorts all = 1; + + // The port numbers the rule allows, when all is unset. + repeated int32 numbers = 2; +} + +// EgressAllPorts marks a rule as allowing every port. +message EgressAllPorts {} + message MintActorCertificateRequest { // The actor for which the certificate should be issued. // @@ -420,9 +492,6 @@ message ArchAssets { // atelet's backend code interprets the asset names (gVisor expects "gvisor", // the release tarball; legacy "runsc", a bare binary, is still accepted). message SandboxAssets { - // TODO: send this as an enum, like ateapi's and ateom's SandboxClass, so - // the tags can bound it and the custom validation goes away. - // // +k8s:required // +k8s:maxLength=63 # guardrail; the class names are a few characters // +k8s:customValidation # gvisor or microvm @@ -837,8 +906,7 @@ message Snapshot { // usable local copy. // // TODO: Temporarily, this is optional while we haven't implemented deferred - // uploads. Restore also uses its presence to choose between the local copy - // and object storage; see RestoreRequest.snapshot. + // uploads. // // +k8s:optional ObjectStorage object = 4; @@ -1022,9 +1090,9 @@ message RestoreRequest { // detaching the actor, it waits for the detach to finish and then reads // `snapshot.object`. Fails if neither copy exists: that generation is lost. // - // TODO: Temporarily, whether `snapshot.object` is set chooses the copy: - // atelet reads only the local copy when it is unset, and only - // `snapshot.object` when it is set, with no fallback either way. + // TODO: Temporarily, the control plane does not set `snapshot.object` when + // it restores a paused actor, so atelet has nothing to fall back to if that + // actor's local copy is missing or unusable. // // +k8s:required Snapshot snapshot = 8; diff --git a/internal/proto/ateletpb/atelet_grpc.pb.go b/internal/proto/ateletpb/atelet_grpc.pb.go index ce18960786..38896e9e80 100644 --- a/internal/proto/ateletpb/atelet_grpc.pb.go +++ b/internal/proto/ateletpb/atelet_grpc.pb.go @@ -36,6 +36,7 @@ const ( AteomSupport_MintActorCertificate_FullMethodName = "/atelet.AteomSupport/MintActorCertificate" AteomSupport_RegisterWorker_FullMethodName = "/atelet.AteomSupport/RegisterWorker" AteomSupport_RequestActorSuspend_FullMethodName = "/atelet.AteomSupport/RequestActorSuspend" + AteomSupport_GetActorEgressPolicy_FullMethodName = "/atelet.AteomSupport/GetActorEgressPolicy" ) // AteomSupportClient is the client API for AteomSupport service. @@ -65,6 +66,11 @@ type AteomSupportClient interface { // would need: the control plane answers this by driving a suspend back // through atelet into the calling ateom. RequestActorSuspend(ctx context.Context, in *RequestActorSuspendRequest, opts ...grpc.CallOption) (*RequestActorSuspendResponse, error) + // GetActorEgressPolicy returns the egress policy of the actor this + // worker hosts, which atelet reads from the control plane's + // Control.GetActorEgressPolicy. An error is the control plane's, passed + // through as it was given. + GetActorEgressPolicy(ctx context.Context, in *GetActorEgressPolicyRequest, opts ...grpc.CallOption) (*GetActorEgressPolicyResponse, error) } type ateomSupportClient struct { @@ -105,6 +111,16 @@ func (c *ateomSupportClient) RequestActorSuspend(ctx context.Context, in *Reques return out, nil } +func (c *ateomSupportClient) GetActorEgressPolicy(ctx context.Context, in *GetActorEgressPolicyRequest, opts ...grpc.CallOption) (*GetActorEgressPolicyResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(GetActorEgressPolicyResponse) + err := c.cc.Invoke(ctx, AteomSupport_GetActorEgressPolicy_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + // AteomSupportServer is the server API for AteomSupport service. // All implementations must embed UnimplementedAteomSupportServer // for forward compatibility. @@ -132,6 +148,11 @@ type AteomSupportServer interface { // would need: the control plane answers this by driving a suspend back // through atelet into the calling ateom. RequestActorSuspend(context.Context, *RequestActorSuspendRequest) (*RequestActorSuspendResponse, error) + // GetActorEgressPolicy returns the egress policy of the actor this + // worker hosts, which atelet reads from the control plane's + // Control.GetActorEgressPolicy. An error is the control plane's, passed + // through as it was given. + GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*GetActorEgressPolicyResponse, error) mustEmbedUnimplementedAteomSupportServer() } @@ -151,6 +172,9 @@ func (UnimplementedAteomSupportServer) RegisterWorker(context.Context, *Register func (UnimplementedAteomSupportServer) RequestActorSuspend(context.Context, *RequestActorSuspendRequest) (*RequestActorSuspendResponse, error) { return nil, status.Error(codes.Unimplemented, "method RequestActorSuspend not implemented") } +func (UnimplementedAteomSupportServer) GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*GetActorEgressPolicyResponse, error) { + return nil, status.Error(codes.Unimplemented, "method GetActorEgressPolicy not implemented") +} func (UnimplementedAteomSupportServer) mustEmbedUnimplementedAteomSupportServer() {} func (UnimplementedAteomSupportServer) testEmbeddedByValue() {} @@ -226,6 +250,24 @@ func _AteomSupport_RequestActorSuspend_Handler(srv interface{}, ctx context.Cont return interceptor(ctx, in, info, handler) } +func _AteomSupport_GetActorEgressPolicy_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(GetActorEgressPolicyRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(AteomSupportServer).GetActorEgressPolicy(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: AteomSupport_GetActorEgressPolicy_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(AteomSupportServer).GetActorEgressPolicy(ctx, req.(*GetActorEgressPolicyRequest)) + } + return interceptor(ctx, in, info, handler) +} + // AteomSupport_ServiceDesc is the grpc.ServiceDesc for AteomSupport service. // It's only intended for direct use with grpc.RegisterService, // and not to be introspected or modified (even as a copy) @@ -245,6 +287,10 @@ var AteomSupport_ServiceDesc = grpc.ServiceDesc{ MethodName: "RequestActorSuspend", Handler: _AteomSupport_RequestActorSuspend_Handler, }, + { + MethodName: "GetActorEgressPolicy", + Handler: _AteomSupport_GetActorEgressPolicy_Handler, + }, }, Streams: []grpc.StreamDesc{}, Metadata: "atelet.proto", diff --git a/internal/proto/ateompb/ateom.pb.go b/internal/proto/ateompb/ateom.pb.go index bfadf9f340..03eb7137d9 100644 --- a/internal/proto/ateompb/ateom.pb.go +++ b/internal/proto/ateompb/ateom.pb.go @@ -97,60 +97,6 @@ func (SnapshotFidelity) EnumDescriptor() ([]byte, []int) { return file_ateom_proto_rawDescGZIP(), []int{0} } -// SandboxClass is the sandbox runtime family that produced a sample. Mirrors -// the ate.dev/v1alpha1 SandboxClass the WorkerPool and ActorTemplate are -// configured with; an ateom binary only ever reports its own. -type SandboxClass int32 - -const ( - SandboxClass_SANDBOX_CLASS_UNSPECIFIED SandboxClass = 0 - // gVisor / runsc (cmd/ateom-gvisor). - SandboxClass_SANDBOX_CLASS_GVISOR SandboxClass = 1 - // Micro-VM (cmd/ateom-microvm). - SandboxClass_SANDBOX_CLASS_MICROVM SandboxClass = 2 -) - -// Enum value maps for SandboxClass. -var ( - SandboxClass_name = map[int32]string{ - 0: "SANDBOX_CLASS_UNSPECIFIED", - 1: "SANDBOX_CLASS_GVISOR", - 2: "SANDBOX_CLASS_MICROVM", - } - SandboxClass_value = map[string]int32{ - "SANDBOX_CLASS_UNSPECIFIED": 0, - "SANDBOX_CLASS_GVISOR": 1, - "SANDBOX_CLASS_MICROVM": 2, - } -) - -func (x SandboxClass) Enum() *SandboxClass { - p := new(SandboxClass) - *p = x - return p -} - -func (x SandboxClass) String() string { - return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) -} - -func (SandboxClass) Descriptor() protoreflect.EnumDescriptor { - return file_ateom_proto_enumTypes[1].Descriptor() -} - -func (SandboxClass) Type() protoreflect.EnumType { - return &file_ateom_proto_enumTypes[1] -} - -func (x SandboxClass) Number() protoreflect.EnumNumber { - return protoreflect.EnumNumber(x) -} - -// Deprecated: Use SandboxClass.Descriptor instead. -func (SandboxClass) EnumDescriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{1} -} - // StatsSource is how a sample's measurements were obtained. The two are not // measured the same way and should not be compared against each other as though // they were. @@ -195,11 +141,11 @@ func (x StatsSource) String() string { } func (StatsSource) Descriptor() protoreflect.EnumDescriptor { - return file_ateom_proto_enumTypes[2].Descriptor() + return file_ateom_proto_enumTypes[1].Descriptor() } func (StatsSource) Type() protoreflect.EnumType { - return &file_ateom_proto_enumTypes[2] + return &file_ateom_proto_enumTypes[1] } func (x StatsSource) Number() protoreflect.EnumNumber { @@ -208,7 +154,7 @@ func (x StatsSource) Number() protoreflect.EnumNumber { // Deprecated: Use StatsSource.Descriptor instead. func (StatsSource) EnumDescriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{2} + return file_ateom_proto_rawDescGZIP(), []int{1} } // ActorDirs is the set of per-actor directories atelet prepares on the volume @@ -1748,8 +1694,10 @@ type WorkloadStatsSample struct { ActorUid string `protobuf:"bytes,3,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` ActorTemplateAtespace string `protobuf:"bytes,4,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` ActorTemplateName string `protobuf:"bytes,5,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` - // The sandbox runtime family that produced this sample. - SandboxClass SandboxClass `protobuf:"varint,6,opt,name=sandbox_class,json=sandboxClass,proto3,enum=ateom.SandboxClass" json:"sandbox_class,omitempty"` + // The sandbox runtime family that produced this sample, in the + // ate.dev/v1alpha1 SandboxClass form: "gvisor" (cmd/ateom-gvisor) or + // "microvm" (cmd/ateom-microvm). An ateom binary only ever reports its own. + SandboxClass string `protobuf:"bytes,6,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // How the measurements below were obtained. Source StatsSource `protobuf:"varint,7,opt,name=source,proto3,enum=ateom.StatsSource" json:"source,omitempty"` // Measurements. All four are zero when source is STATS_SOURCE_UNSPECIFIED, @@ -1846,11 +1794,11 @@ func (x *WorkloadStatsSample) GetActorTemplateName() string { return "" } -func (x *WorkloadStatsSample) GetSandboxClass() SandboxClass { +func (x *WorkloadStatsSample) GetSandboxClass() string { if x != nil { return x.SandboxClass } - return SandboxClass_SANDBOX_CLASS_UNSPECIFIED + return "" } func (x *WorkloadStatsSample) GetSource() StatsSource { @@ -2181,15 +2129,15 @@ const file_ateom_proto_rawDesc = "" + "\x0f_egress_gateway\"\x19\n" + "\x17RestoreWorkloadResponse\"6\n" + "\x17GetWorkloadStatsRequest\x12\x1b\n" + - "\tactor_uid\x18\x01 \x01(\tR\bactorUid\"\xd3\x04\n" + + "\tactor_uid\x18\x01 \x01(\tR\bactorUid\"\xbe\x04\n" + "\x13WorkloadStatsSample\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + "actor_name\x18\x02 \x01(\tR\tactorName\x12\x1b\n" + "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x126\n" + "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + - "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x128\n" + - "\rsandbox_class\x18\x06 \x01(\x0e2\x13.ateom.SandboxClassR\fsandboxClass\x12*\n" + + "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12#\n" + + "\rsandbox_class\x18\x06 \x01(\tR\fsandboxClass\x12*\n" + "\x06source\x18\a \x01(\x0e2\x12.ateom.StatsSourceR\x06source\x120\n" + "\x14memory_current_bytes\x18\b \x01(\x04R\x12memoryCurrentBytes\x12*\n" + "\x11memory_peak_bytes\x18\t \x01(\x04R\x0fmemoryPeakBytes\x127\n" + @@ -2208,10 +2156,6 @@ const file_ateom_proto_rawDesc = "" + "\x19SNAPSHOT_FIDELITY_VOLUMES\x10\x01\x12\x1c\n" + "\x18SNAPSHOT_FIDELITY_ROOTFS\x10\x02\x12\x1c\n" + "\x18SNAPSHOT_FIDELITY_MEMORY\x10\x03*b\n" + - "\fSandboxClass\x12\x1d\n" + - "\x19SANDBOX_CLASS_UNSPECIFIED\x10\x00\x12\x18\n" + - "\x14SANDBOX_CLASS_GVISOR\x10\x01\x12\x19\n" + - "\x15SANDBOX_CLASS_MICROVM\x10\x02*b\n" + "\vStatsSource\x12\x1c\n" + "\x18STATS_SOURCE_UNSPECIFIED\x10\x00\x12\x17\n" + "\x13STATS_SOURCE_CGROUP\x10\x01\x12\x1c\n" + @@ -2236,87 +2180,85 @@ func file_ateom_proto_rawDescGZIP() []byte { return file_ateom_proto_rawDescData } -var file_ateom_proto_enumTypes = make([]protoimpl.EnumInfo, 3) +var file_ateom_proto_enumTypes = make([]protoimpl.EnumInfo, 2) var file_ateom_proto_msgTypes = make([]protoimpl.MessageInfo, 28) var file_ateom_proto_goTypes = []any{ (SnapshotFidelity)(0), // 0: ateom.SnapshotFidelity - (SandboxClass)(0), // 1: ateom.SandboxClass - (StatsSource)(0), // 2: ateom.StatsSource - (*ActorDirs)(nil), // 3: ateom.ActorDirs - (*TerminateWorkloadRequest)(nil), // 4: ateom.TerminateWorkloadRequest - (*TerminateWorkloadResponse)(nil), // 5: ateom.TerminateWorkloadResponse - (*RunWorkloadRequest)(nil), // 6: ateom.RunWorkloadRequest - (*EgressGateway)(nil), // 7: ateom.EgressGateway - (*WorkloadSpec)(nil), // 8: ateom.WorkloadSpec - (*Container)(nil), // 9: ateom.Container - (*ContainerSpec)(nil), // 10: ateom.ContainerSpec - (*ResourceLimits)(nil), // 11: ateom.ResourceLimits - (*VolumeMount)(nil), // 12: ateom.VolumeMount - (*DurableDirVolumeMount)(nil), // 13: ateom.DurableDirVolumeMount - (*SystemInfoVolumeMount)(nil), // 14: ateom.SystemInfoVolumeMount - (*ImageVolumeMount)(nil), // 15: ateom.ImageVolumeMount - (*WakeupProbe)(nil), // 16: ateom.WakeupProbe - (*HttpGetAction)(nil), // 17: ateom.HttpGetAction - (*RunWorkloadResponse)(nil), // 18: ateom.RunWorkloadResponse - (*CheckpointWorkloadRequest)(nil), // 19: ateom.CheckpointWorkloadRequest - (*CheckpointWorkloadResponse)(nil), // 20: ateom.CheckpointWorkloadResponse - (*RestoreWorkloadRequest)(nil), // 21: ateom.RestoreWorkloadRequest - (*RestoreWorkloadResponse)(nil), // 22: ateom.RestoreWorkloadResponse - (*GetWorkloadStatsRequest)(nil), // 23: ateom.GetWorkloadStatsRequest - (*WorkloadStatsSample)(nil), // 24: ateom.WorkloadStatsSample - (*GetWorkloadStatsResponse)(nil), // 25: ateom.GetWorkloadStatsResponse - (*GetActiveWorkloadStatsRequest)(nil), // 26: ateom.GetActiveWorkloadStatsRequest - (*GetActiveWorkloadStatsResponse)(nil), // 27: ateom.GetActiveWorkloadStatsResponse - nil, // 28: ateom.RunWorkloadRequest.RuntimeAssetPathsEntry - nil, // 29: ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntry - nil, // 30: ateom.RestoreWorkloadRequest.RuntimeAssetPathsEntry + (StatsSource)(0), // 1: ateom.StatsSource + (*ActorDirs)(nil), // 2: ateom.ActorDirs + (*TerminateWorkloadRequest)(nil), // 3: ateom.TerminateWorkloadRequest + (*TerminateWorkloadResponse)(nil), // 4: ateom.TerminateWorkloadResponse + (*RunWorkloadRequest)(nil), // 5: ateom.RunWorkloadRequest + (*EgressGateway)(nil), // 6: ateom.EgressGateway + (*WorkloadSpec)(nil), // 7: ateom.WorkloadSpec + (*Container)(nil), // 8: ateom.Container + (*ContainerSpec)(nil), // 9: ateom.ContainerSpec + (*ResourceLimits)(nil), // 10: ateom.ResourceLimits + (*VolumeMount)(nil), // 11: ateom.VolumeMount + (*DurableDirVolumeMount)(nil), // 12: ateom.DurableDirVolumeMount + (*SystemInfoVolumeMount)(nil), // 13: ateom.SystemInfoVolumeMount + (*ImageVolumeMount)(nil), // 14: ateom.ImageVolumeMount + (*WakeupProbe)(nil), // 15: ateom.WakeupProbe + (*HttpGetAction)(nil), // 16: ateom.HttpGetAction + (*RunWorkloadResponse)(nil), // 17: ateom.RunWorkloadResponse + (*CheckpointWorkloadRequest)(nil), // 18: ateom.CheckpointWorkloadRequest + (*CheckpointWorkloadResponse)(nil), // 19: ateom.CheckpointWorkloadResponse + (*RestoreWorkloadRequest)(nil), // 20: ateom.RestoreWorkloadRequest + (*RestoreWorkloadResponse)(nil), // 21: ateom.RestoreWorkloadResponse + (*GetWorkloadStatsRequest)(nil), // 22: ateom.GetWorkloadStatsRequest + (*WorkloadStatsSample)(nil), // 23: ateom.WorkloadStatsSample + (*GetWorkloadStatsResponse)(nil), // 24: ateom.GetWorkloadStatsResponse + (*GetActiveWorkloadStatsRequest)(nil), // 25: ateom.GetActiveWorkloadStatsRequest + (*GetActiveWorkloadStatsResponse)(nil), // 26: ateom.GetActiveWorkloadStatsResponse + nil, // 27: ateom.RunWorkloadRequest.RuntimeAssetPathsEntry + nil, // 28: ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntry + nil, // 29: ateom.RestoreWorkloadRequest.RuntimeAssetPathsEntry } var file_ateom_proto_depIdxs = []int32{ - 8, // 0: ateom.TerminateWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 3, // 1: ateom.TerminateWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 8, // 2: ateom.RunWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 28, // 3: ateom.RunWorkloadRequest.runtime_asset_paths:type_name -> ateom.RunWorkloadRequest.RuntimeAssetPathsEntry - 7, // 4: ateom.RunWorkloadRequest.egress_gateway:type_name -> ateom.EgressGateway - 3, // 5: ateom.RunWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 9, // 6: ateom.WorkloadSpec.containers:type_name -> ateom.Container - 16, // 7: ateom.Container.wakeup_probe:type_name -> ateom.WakeupProbe - 13, // 8: ateom.Container.durable_dir_volume_mounts:type_name -> ateom.DurableDirVolumeMount - 12, // 9: ateom.Container.csi_volume_mounts:type_name -> ateom.VolumeMount - 14, // 10: ateom.Container.system_info_volume_mounts:type_name -> ateom.SystemInfoVolumeMount - 15, // 11: ateom.Container.image_volume_mounts:type_name -> ateom.ImageVolumeMount - 10, // 12: ateom.Container.container_spec:type_name -> ateom.ContainerSpec - 11, // 13: ateom.ContainerSpec.resources:type_name -> ateom.ResourceLimits - 17, // 14: ateom.WakeupProbe.http_get:type_name -> ateom.HttpGetAction - 8, // 15: ateom.CheckpointWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 29, // 16: ateom.CheckpointWorkloadRequest.runtime_asset_paths:type_name -> ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntry + 7, // 0: ateom.TerminateWorkloadRequest.spec:type_name -> ateom.WorkloadSpec + 2, // 1: ateom.TerminateWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs + 7, // 2: ateom.RunWorkloadRequest.spec:type_name -> ateom.WorkloadSpec + 27, // 3: ateom.RunWorkloadRequest.runtime_asset_paths:type_name -> ateom.RunWorkloadRequest.RuntimeAssetPathsEntry + 6, // 4: ateom.RunWorkloadRequest.egress_gateway:type_name -> ateom.EgressGateway + 2, // 5: ateom.RunWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs + 8, // 6: ateom.WorkloadSpec.containers:type_name -> ateom.Container + 15, // 7: ateom.Container.wakeup_probe:type_name -> ateom.WakeupProbe + 12, // 8: ateom.Container.durable_dir_volume_mounts:type_name -> ateom.DurableDirVolumeMount + 11, // 9: ateom.Container.csi_volume_mounts:type_name -> ateom.VolumeMount + 13, // 10: ateom.Container.system_info_volume_mounts:type_name -> ateom.SystemInfoVolumeMount + 14, // 11: ateom.Container.image_volume_mounts:type_name -> ateom.ImageVolumeMount + 9, // 12: ateom.Container.container_spec:type_name -> ateom.ContainerSpec + 10, // 13: ateom.ContainerSpec.resources:type_name -> ateom.ResourceLimits + 16, // 14: ateom.WakeupProbe.http_get:type_name -> ateom.HttpGetAction + 7, // 15: ateom.CheckpointWorkloadRequest.spec:type_name -> ateom.WorkloadSpec + 28, // 16: ateom.CheckpointWorkloadRequest.runtime_asset_paths:type_name -> ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntry 0, // 17: ateom.CheckpointWorkloadRequest.fidelity:type_name -> ateom.SnapshotFidelity - 3, // 18: ateom.CheckpointWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 8, // 19: ateom.RestoreWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 30, // 20: ateom.RestoreWorkloadRequest.runtime_asset_paths:type_name -> ateom.RestoreWorkloadRequest.RuntimeAssetPathsEntry + 2, // 18: ateom.CheckpointWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs + 7, // 19: ateom.RestoreWorkloadRequest.spec:type_name -> ateom.WorkloadSpec + 29, // 20: ateom.RestoreWorkloadRequest.runtime_asset_paths:type_name -> ateom.RestoreWorkloadRequest.RuntimeAssetPathsEntry 0, // 21: ateom.RestoreWorkloadRequest.fidelity:type_name -> ateom.SnapshotFidelity - 7, // 22: ateom.RestoreWorkloadRequest.egress_gateway:type_name -> ateom.EgressGateway - 3, // 23: ateom.RestoreWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 1, // 24: ateom.WorkloadStatsSample.sandbox_class:type_name -> ateom.SandboxClass - 2, // 25: ateom.WorkloadStatsSample.source:type_name -> ateom.StatsSource - 24, // 26: ateom.GetWorkloadStatsResponse.sample:type_name -> ateom.WorkloadStatsSample - 24, // 27: ateom.GetActiveWorkloadStatsResponse.samples:type_name -> ateom.WorkloadStatsSample - 6, // 28: ateom.Ateom.RunWorkload:input_type -> ateom.RunWorkloadRequest - 19, // 29: ateom.Ateom.CheckpointWorkload:input_type -> ateom.CheckpointWorkloadRequest - 21, // 30: ateom.Ateom.RestoreWorkload:input_type -> ateom.RestoreWorkloadRequest - 23, // 31: ateom.Ateom.GetWorkloadStats:input_type -> ateom.GetWorkloadStatsRequest - 26, // 32: ateom.Ateom.GetActiveWorkloadStats:input_type -> ateom.GetActiveWorkloadStatsRequest - 4, // 33: ateom.Ateom.TerminateWorkload:input_type -> ateom.TerminateWorkloadRequest - 18, // 34: ateom.Ateom.RunWorkload:output_type -> ateom.RunWorkloadResponse - 20, // 35: ateom.Ateom.CheckpointWorkload:output_type -> ateom.CheckpointWorkloadResponse - 22, // 36: ateom.Ateom.RestoreWorkload:output_type -> ateom.RestoreWorkloadResponse - 25, // 37: ateom.Ateom.GetWorkloadStats:output_type -> ateom.GetWorkloadStatsResponse - 27, // 38: ateom.Ateom.GetActiveWorkloadStats:output_type -> ateom.GetActiveWorkloadStatsResponse - 5, // 39: ateom.Ateom.TerminateWorkload:output_type -> ateom.TerminateWorkloadResponse - 34, // [34:40] is the sub-list for method output_type - 28, // [28:34] is the sub-list for method input_type - 28, // [28:28] is the sub-list for extension type_name - 28, // [28:28] is the sub-list for extension extendee - 0, // [0:28] is the sub-list for field type_name + 6, // 22: ateom.RestoreWorkloadRequest.egress_gateway:type_name -> ateom.EgressGateway + 2, // 23: ateom.RestoreWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs + 1, // 24: ateom.WorkloadStatsSample.source:type_name -> ateom.StatsSource + 23, // 25: ateom.GetWorkloadStatsResponse.sample:type_name -> ateom.WorkloadStatsSample + 23, // 26: ateom.GetActiveWorkloadStatsResponse.samples:type_name -> ateom.WorkloadStatsSample + 5, // 27: ateom.Ateom.RunWorkload:input_type -> ateom.RunWorkloadRequest + 18, // 28: ateom.Ateom.CheckpointWorkload:input_type -> ateom.CheckpointWorkloadRequest + 20, // 29: ateom.Ateom.RestoreWorkload:input_type -> ateom.RestoreWorkloadRequest + 22, // 30: ateom.Ateom.GetWorkloadStats:input_type -> ateom.GetWorkloadStatsRequest + 25, // 31: ateom.Ateom.GetActiveWorkloadStats:input_type -> ateom.GetActiveWorkloadStatsRequest + 3, // 32: ateom.Ateom.TerminateWorkload:input_type -> ateom.TerminateWorkloadRequest + 17, // 33: ateom.Ateom.RunWorkload:output_type -> ateom.RunWorkloadResponse + 19, // 34: ateom.Ateom.CheckpointWorkload:output_type -> ateom.CheckpointWorkloadResponse + 21, // 35: ateom.Ateom.RestoreWorkload:output_type -> ateom.RestoreWorkloadResponse + 24, // 36: ateom.Ateom.GetWorkloadStats:output_type -> ateom.GetWorkloadStatsResponse + 26, // 37: ateom.Ateom.GetActiveWorkloadStats:output_type -> ateom.GetActiveWorkloadStatsResponse + 4, // 38: ateom.Ateom.TerminateWorkload:output_type -> ateom.TerminateWorkloadResponse + 33, // [33:39] is the sub-list for method output_type + 27, // [27:33] is the sub-list for method input_type + 27, // [27:27] is the sub-list for extension type_name + 27, // [27:27] is the sub-list for extension extendee + 0, // [0:27] is the sub-list for field type_name } func init() { file_ateom_proto_init() } @@ -2331,7 +2273,7 @@ func file_ateom_proto_init() { File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_ateom_proto_rawDesc), len(file_ateom_proto_rawDesc)), - NumEnums: 3, + NumEnums: 2, NumMessages: 28, NumExtensions: 0, NumServices: 1, diff --git a/internal/proto/ateompb/ateom.proto b/internal/proto/ateompb/ateom.proto index 22002d1df0..3a253cca13 100644 --- a/internal/proto/ateompb/ateom.proto +++ b/internal/proto/ateompb/ateom.proto @@ -394,17 +394,6 @@ message GetWorkloadStatsRequest { string actor_uid = 1; } -// SandboxClass is the sandbox runtime family that produced a sample. Mirrors -// the ate.dev/v1alpha1 SandboxClass the WorkerPool and ActorTemplate are -// configured with; an ateom binary only ever reports its own. -enum SandboxClass { - SANDBOX_CLASS_UNSPECIFIED = 0; - // gVisor / runsc (cmd/ateom-gvisor). - SANDBOX_CLASS_GVISOR = 1; - // Micro-VM (cmd/ateom-microvm). - SANDBOX_CLASS_MICROVM = 2; -} - // StatsSource is how a sample's measurements were obtained. The two are not // measured the same way and should not be compared against each other as though // they were. @@ -440,8 +429,10 @@ message WorkloadStatsSample { string actor_template_atespace = 4; string actor_template_name = 5; - // The sandbox runtime family that produced this sample. - SandboxClass sandbox_class = 6; + // The sandbox runtime family that produced this sample, in the + // ate.dev/v1alpha1 SandboxClass form: "gvisor" (cmd/ateom-gvisor) or + // "microvm" (cmd/ateom-microvm). An ateom binary only ever reports its own. + string sandbox_class = 6; // How the measurements below were obtained. StatsSource source = 7; diff --git a/internal/protoredact/protoredact_test.go b/internal/protoredact/protoredact_test.go index e8a4f59816..ada4fd8151 100644 --- a/internal/protoredact/protoredact_test.go +++ b/internal/protoredact/protoredact_test.go @@ -417,6 +417,7 @@ func TestDebugRedactFieldsArePinned(t *testing.T) { want := map[string]bool{ "ateapi.EnvVar.value": true, "ateapi.MintActorJwtResponse.actor_jwt": true, + "ateapi.MintAteomActorJwtResponse.actor_jwt": true, "atelet.AssetFile.url": true, "atelet.EnvEntry.value": true, "ateom.ContainerSpec.env": true, diff --git a/internal/resources/validate.go b/internal/resources/validate.go index 050a3303a7..ec9cb78d1e 100644 --- a/internal/resources/validate.go +++ b/internal/resources/validate.go @@ -30,6 +30,7 @@ import ( "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/proto/ateompb" "github.com/agent-substrate/substrate/internal/volumepath" + "github.com/agent-substrate/substrate/pkg/sandboxclass" "github.com/distribution/reference" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" @@ -144,6 +145,15 @@ func ValidateSnapshotFidelity(fidelity ateompb.SnapshotFidelity, supported []ate return field.ErrorList{field.NotSupported(fldPath, fidelity, names)} } +// ValidateSandboxClass requires one of the sandboxclass values. An empty value +// is left to the required tag. +func ValidateSandboxClass(fldPath *field.Path, value string) field.ErrorList { + if value == "" || sandboxclass.Valid(value) { + return nil + } + return field.ErrorList{field.NotSupported(fldPath, value, sandboxclass.All())} +} + func validateAbsDir(dir string, fldPath *field.Path) field.ErrorList { if dir == "" { return field.ErrorList{field.Required(fldPath, "")} diff --git a/internal/resources/validate_test.go b/internal/resources/validate_test.go index aacdadda4b..d923207ae6 100644 --- a/internal/resources/validate_test.go +++ b/internal/resources/validate_test.go @@ -261,6 +261,26 @@ func TestValidateUUID(t *testing.T) { } } +func TestValidateSandboxClass(t *testing.T) { + path := field.NewPath("sandbox_class") + tests := []struct { + name string + class string + want field.ErrorList + }{ + {name: "gvisor", class: "gvisor"}, + {name: "microvm", class: "microvm"}, + {name: "empty left to tags", class: ""}, + {name: "unknown class", class: "kata", want: field.ErrorList{field.NotSupported[string](path, nil, nil)}}, + {name: "case-sensitive", class: "GVISOR", want: field.ErrorList{field.NotSupported[string](path, nil, nil)}}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + field.ErrorMatcher{}.ByType().ByField().Test(t, tt.want, ValidateSandboxClass(path, tt.class)) + }) + } +} + func TestValidateLimit(t *testing.T) { path := field.NewPath("limits").Index(0) quantityPath := path.Child("quantity") diff --git a/internal/serverboot/serverboot.go b/internal/serverboot/serverboot.go index 652a44ce62..bdd0c61c24 100644 --- a/internal/serverboot/serverboot.go +++ b/internal/serverboot/serverboot.go @@ -27,6 +27,8 @@ import ( "log/slog" "net/http" "os" + "slices" + "sync" "sync/atomic" "time" @@ -36,12 +38,14 @@ import ( promclient "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" prombridge "go.opentelemetry.io/contrib/bridges/prometheus" + "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" "go.opentelemetry.io/otel/exporters/prometheus" "go.opentelemetry.io/otel/propagation" + "go.opentelemetry.io/otel/sdk/instrumentation" sdkmetric "go.opentelemetry.io/otel/sdk/metric" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" @@ -327,7 +331,7 @@ func newMeterProvider(ctx context.Context, serviceName string, push, relayCapabl if err != nil { return nil, fmt.Errorf("create metric resource: %w", err) } - opts := []sdkmetric.Option{sdkmetric.WithResource(res)} + opts := []sdkmetric.Option{sdkmetric.WithResource(res), sdkmetric.WithView(dropRPCServerAddress)} if push { expOpts := []otlpmetricgrpc.Option{ // GKE managed metrics doesn't support validating the TLS certs of the collector. @@ -356,6 +360,16 @@ func newMeterProvider(ctx context.Context, serviceName string, push, relayCapabl return mp, nil } +// dropRPCServerAddress removes server.address from the otelgrpc instruments. +// otelgrpc reads it from the remote address of the client connection: the +// socket path atelet dials holds the ateom pod UID, and a TCP address is the IP +// of one server pod. Either way the key has one value per pod. server.port stays, +// because it has one value per service. +var dropRPCServerAddress = sdkmetric.NewView( + sdkmetric.Instrument{Scope: instrumentation.Scope{Name: otelgrpc.ScopeName}}, + sdkmetric.Stream{AttributeFilter: attribute.NewDenyKeysFilter(semconv.ServerAddressKey)}, +) + // Fatal logs msg + err and exits with status 1. For startup-time // fail-fast where there's no recovery path. func Fatal(ctx context.Context, msg string, err error, additionalArgs ...any) { @@ -375,19 +389,72 @@ func ShutdownProvider(name string, shutdown func(context.Context) error) { } } +// checkTimeout bounds the dependency checks of one /readyz request. It sits +// under the kubelet's default 1s probe timeout so a hung dependency gets a 503 +// rather than a timed-out probe. +const checkTimeout = 500 * time.Millisecond + // Readiness is a predicate for process readiness. The zero value // reports ready. Calling MarkNotReady flips it permanently to not ready, // which causes /readyz to start returning "Service Unavailable" (503). +// Checks added with AddCheck make /readyz return 503 only while they fail. type Readiness struct { notReady atomic.Bool + + mu sync.RWMutex + checks []*readinessCheck +} + +type readinessCheck struct { + name string + fn func(context.Context) error + failing atomic.Bool } // MarkNotReady makes /readyz return 503 from now on. func (r *Readiness) MarkNotReady() { r.notReady.Store(true) } -// Ready reports whether /readyz returns 200. +// Ready reports whether MarkNotReady has not been called. It does not run the +// checks added with AddCheck. func (r *Readiness) Ready() bool { return !r.notReady.Load() } +// AddCheck registers a dependency check that /readyz runs on every request. +// While it returns an error, or does not return within checkTimeout, /readyz +// returns 503, and it returns 200 again once the check passes. Liveness is +// unaffected. Check errors are logged when a check starts and stops failing, +// and never sent to the client. +func (r *Readiness) AddCheck(name string, check func(context.Context) error) { + r.mu.Lock() + defer r.mu.Unlock() + r.checks = append(r.checks, &readinessCheck{name: name, fn: check}) +} + +// notReadyReason returns why /readyz should answer 503, or "" if it should +// answer 200. Draining wins over the checks, which are not run. +func (r *Readiness) notReadyReason(ctx context.Context) string { + if !r.Ready() { + return "draining" + } + r.mu.RLock() + checks := slices.Clone(r.checks) + r.mu.RUnlock() + + ctx, cancel := context.WithTimeout(ctx, checkTimeout) + defer cancel() + for _, c := range checks { + if err := c.fn(ctx); err != nil { + if !c.failing.Swap(true) { + slog.WarnContext(ctx, "Readiness check failing", slog.String("check", c.name), slog.Any("err", err)) + } + return c.name + " check failed" + } + if c.failing.Swap(false) { + slog.InfoContext(ctx, "Readiness check recovered", slog.String("check", c.name)) + } + } + return "" +} + // httpServerReadHeaderTimeout bounds how long a client can take to send the // request headers, so a slow client cannot hold a connection open forever. const httpServerReadHeaderTimeout = 10 * time.Second @@ -407,12 +474,13 @@ func StartMetricsServer(ctx context.Context, addr string) { } // StartHealthServer runs an HTTP server exposing /readyz and /healthz for the -// kubelet probes. /readyz returns 200 while readiness is Ready and 503 after -// MarkNotReady. /healthz always returns 200, so liveness keeps succeeding -// while a draining server fails readiness. Blocks until the server stops; -// designed to be `go`-launched. A serve failure exits the process: a pod whose -// probes can never succeed never turns Ready, so dying loudly lets the kubelet -// restart it instead of leaving a pod that looks alive but never gets work. +// kubelet probes. /readyz returns 200 while readiness is Ready and every check +// passes, and 503 after MarkNotReady or while a check fails. /healthz always +// returns 200, so liveness keeps succeeding while a draining server fails +// readiness. Blocks until the server stops; designed to be `go`-launched. A +// serve failure exits the process: a pod whose probes can never succeed never +// turns Ready, so dying loudly lets the kubelet restart it instead of leaving a +// pod that looks alive but never gets work. func StartHealthServer(ctx context.Context, addr string, readiness *Readiness) { slog.InfoContext(ctx, "Starting health HTTP server", slog.String("addr", addr)) if err := listenAndServe(addr, healthMux(readiness)); err != nil { @@ -455,9 +523,9 @@ func healthMux(readiness *Readiness) *http.ServeMux { } func readinessHandler(readiness *Readiness) http.Handler { - return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - if !readiness.Ready() { - http.Error(w, "draining", http.StatusServiceUnavailable) + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if reason := readiness.notReadyReason(r.Context()); reason != "" { + http.Error(w, reason, http.StatusServiceUnavailable) return } w.WriteHeader(http.StatusOK) diff --git a/internal/serverboot/serverboot_test.go b/internal/serverboot/serverboot_test.go index 1ee6ab44c5..c17e796d23 100644 --- a/internal/serverboot/serverboot_test.go +++ b/internal/serverboot/serverboot_test.go @@ -17,6 +17,7 @@ package serverboot import ( "bytes" "context" + "errors" "log/slog" "net" "net/http" @@ -29,6 +30,8 @@ import ( "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" + "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" + "go.opentelemetry.io/otel/attribute" sdkmetric "go.opentelemetry.io/otel/sdk/metric" "go.opentelemetry.io/otel/sdk/metric/metricdata" "go.opentelemetry.io/otel/sdk/resource" @@ -37,6 +40,8 @@ import ( coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1" "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/health" + healthpb "google.golang.org/grpc/health/grpc_health_v1" "github.com/agent-substrate/substrate/internal/ateattr" ) @@ -177,6 +182,63 @@ func TestMeterProviderRelayAttribute(t *testing.T) { } } +// A real call through the otelgrpc client handler, so the test covers the +// scope name and the key that otelgrpc uses, not only the View. +func TestMeterProviderDropsRPCServerAddress(t *testing.T) { + reader := sdkmetric.NewManualReader() + mp, err := newMeterProvider(context.Background(), "atelet", false, false, nil, nil, reader) + if err != nil { + t.Fatalf("newMeterProvider: %v", err) + } + t.Cleanup(func() { mp.Shutdown(context.Background()) }) + + lis, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + srv := grpc.NewServer() + healthpb.RegisterHealthServer(srv, health.NewServer()) + go srv.Serve(lis) + t.Cleanup(srv.Stop) + + conn, err := grpc.NewClient(lis.Addr().String(), + grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelgrpc.WithMeterProvider(mp))), + ) + if err != nil { + t.Fatalf("grpc.NewClient: %v", err) + } + t.Cleanup(func() { conn.Close() }) + if _, err := healthpb.NewHealthClient(conn).Check(context.Background(), &healthpb.HealthCheckRequest{}); err != nil { + t.Fatalf("Check: %v", err) + } + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + var points []metricdata.HistogramDataPoint[float64] + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name == "rpc.client.call.duration" { + points = append(points, m.Data.(metricdata.Histogram[float64]).DataPoints...) + } + } + } + if len(points) != 1 { + t.Fatalf("rpc.client.call.duration has %d data points, want 1", len(points)) + } + attrs := points[0].Attributes + if v, ok := attrs.Value("server.address"); ok { + t.Errorf("server.address = %q, want absent", v.String()) + } + for _, key := range []attribute.Key{"server.port", "rpc.method"} { + if _, ok := attrs.Value(key); !ok { + t.Errorf("%s is absent, want present", key) + } + } +} + func TestReadyzDrainsWhileHealthzStaysUp(t *testing.T) { readiness := &Readiness{} mux := healthMux(readiness) @@ -205,6 +267,108 @@ func TestReadyzStaticWithZeroValueReadiness(t *testing.T) { } } +func TestReadyzFollowsCheck(t *testing.T) { + var down atomic.Bool + readiness := &Readiness{} + readiness.AddCheck("store", func(context.Context) error { + if down.Load() { + return errors.New("failed to connect to host=db.internal user=ate") + } + return nil + }) + mux := healthMux(readiness) + + if got := getCode(t, mux, "/readyz"); got != http.StatusOK { + t.Errorf("/readyz with a passing check = %d, want %d", got, http.StatusOK) + } + + down.Store(true) + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/readyz", nil)) + if rec.Code != http.StatusServiceUnavailable { + t.Errorf("/readyz with a failing check = %d, want %d", rec.Code, http.StatusServiceUnavailable) + } + if !strings.Contains(rec.Body.String(), "store") || strings.Contains(rec.Body.String(), "db.internal") { + t.Errorf("/readyz body = %q, want it to name the check and not leak the error", rec.Body.String()) + } + if got := getCode(t, mux, "/healthz"); got != http.StatusOK { + t.Errorf("/healthz with a failing check = %d, want %d (liveness must not follow a dependency)", got, http.StatusOK) + } + + down.Store(false) + if got := getCode(t, mux, "/readyz"); got != http.StatusOK { + t.Errorf("/readyz after the check recovers = %d, want %d", got, http.StatusOK) + } +} + +func TestReadyzCheckTimeout(t *testing.T) { + readiness := &Readiness{} + readiness.AddCheck("store", func(ctx context.Context) error { + <-ctx.Done() + return ctx.Err() + }) + mux := healthMux(readiness) + + start := time.Now() + if got := getCode(t, mux, "/readyz"); got != http.StatusServiceUnavailable { + t.Errorf("/readyz with a hung check = %d, want %d", got, http.StatusServiceUnavailable) + } + if elapsed := time.Since(start); elapsed > time.Second { + t.Errorf("/readyz took %v with a hung check, want under the 1s kubelet probe timeout", elapsed) + } +} + +func TestReadyzDrainSkipsChecks(t *testing.T) { + var calls atomic.Int32 + readiness := &Readiness{} + readiness.AddCheck("store", func(context.Context) error { + calls.Add(1) + return nil + }) + mux := healthMux(readiness) + + readiness.MarkNotReady() + if got := getCode(t, mux, "/readyz"); got != http.StatusServiceUnavailable { + t.Errorf("/readyz during drain = %d, want %d", got, http.StatusServiceUnavailable) + } + if got := calls.Load(); got != 0 { + t.Errorf("check ran %d times during drain, want 0", got) + } +} + +func TestReadyzLogsCheckTransitionsOnce(t *testing.T) { + var buf bytes.Buffer + prev := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil))) + t.Cleanup(func() { slog.SetDefault(prev) }) + + var down atomic.Bool + readiness := &Readiness{} + readiness.AddCheck("store", func(context.Context) error { + if down.Load() { + return errors.New("connection refused") + } + return nil + }) + mux := healthMux(readiness) + + down.Store(true) + for range 3 { + getCode(t, mux, "/readyz") + } + down.Store(false) + for range 3 { + getCode(t, mux, "/readyz") + } + + if got := strings.Count(buf.String(), "Readiness check failing"); got != 1 { + t.Errorf("logged failure %d times over 3 failing requests, want 1:\n%s", got, buf.String()) + } + if got := strings.Count(buf.String(), "Readiness check recovered"); got != 1 { + t.Errorf("logged recovery %d times over 3 passing requests, want 1:\n%s", got, buf.String()) + } +} + // The probes and /metrics live on separate servers, so turning one off can // never take the other with it. func TestMetricsAndHealthServeSeparateEndpoints(t *testing.T) { diff --git a/internal/volume/csi/plugin.go b/internal/volume/csi/plugin.go index 739f6904b4..326dc8617f 100644 --- a/internal/volume/csi/plugin.go +++ b/internal/volume/csi/plugin.go @@ -462,6 +462,10 @@ func resolveTLSConfig(cfg *v1alpha1.CSIDriverConfig, paths tlsPaths) (*tls.Confi return nil, fmt.Errorf("only pod identity TLS is supported in this configuration") } + if tlsCfg.ServerName == "" { + return nil, fmt.Errorf("tls.serverName is required when TLS is enabled") + } + caCache := newCAPoolCache(paths.caCert) // Verify CA pool exists, is readable, and populate the initial cache. diff --git a/internal/volume/csi/tls_test.go b/internal/volume/csi/tls_test.go index 5a56aac2c6..0bf4d6f2e7 100644 --- a/internal/volume/csi/tls_test.go +++ b/internal/volume/csi/tls_test.go @@ -28,6 +28,7 @@ import ( "os" "path/filepath" "slices" + "strings" "testing" "time" @@ -298,6 +299,18 @@ func TestResolveTLSConfigRejectsManualCerts(t *testing.T) { } } +func TestResolveTLSConfigRejectsEmptyServerName(t *testing.T) { + t.Parallel() + cfg := driverConfig("", &v1alpha1.CSIDriverTLSConfig{Enabled: true, UsePodIdentity: true}) + _, err := resolveTLSConfig(cfg, defaultTLSPaths) + if err == nil { + t.Fatal("resolveTLSConfig accepted empty serverName when TLS is enabled") + } + if want := "tls.serverName is required when TLS is enabled"; !strings.Contains(err.Error(), want) { + t.Fatalf("resolveTLSConfig error = %v, want substring %q", err, want) + } +} + func TestResolveTLSConfigFields(t *testing.T) { t.Parallel() const serverName = "my-service.default.svc" diff --git a/manifests/ate-install/ate-api-server.yaml b/manifests/ate-install/ate-api-server.yaml index 77e3a4843d..260b5a3363 100644 --- a/manifests/ate-install/ate-api-server.yaml +++ b/manifests/ate-install/ate-api-server.yaml @@ -246,7 +246,9 @@ spec: failureThreshold: 24 # /healthz stays 200 while a terminating pod drains; /readyz # turns 503, so liveness and readiness diverge correctly during - # shutdown. + # shutdown. /readyz also turns 503 while the store is unreachable, + # pinged over its own connection so a busy request pool cannot fail + # it, and /healthz stays 200: the pool reconnects without a restart. livenessProbe: httpGet: path: /healthz diff --git a/manifests/ate-install/ate-controller.yaml b/manifests/ate-install/ate-controller.yaml index 9220cb8d24..0f0f1df6a2 100644 --- a/manifests/ate-install/ate-controller.yaml +++ b/manifests/ate-install/ate-controller.yaml @@ -105,6 +105,7 @@ spec: args: - --ateapi-ca-file=/run/servicedns-ca/trust-bundle.pem - --ateapi-client-cert=/run/podidentity.podcert.ate.dev/credential-bundle.pem + - --dns-egress-policy=enforce-in-atunnel env: - name: POD_NAME valueFrom: diff --git a/manifests/ate-install/atenet-router.yaml b/manifests/ate-install/atenet-router.yaml index f45791688d..3310864aac 100644 --- a/manifests/ate-install/atenet-router.yaml +++ b/manifests/ate-install/atenet-router.yaml @@ -359,9 +359,9 @@ spec: - name: "servicedns" mountPath: "/run/servicedns.podcert.ate.dev" # podidentity credential + trust bundles for actor-facing (upstream) - # mTLS. Envoy's ORIGINAL_DST actor cluster presents this credential - # bundle as its client cert when dialing the actor's atunnel ingress - # server on :443, and validates that server against the trust bundle. + # mTLS. Envoy presents this credential bundle as its client cert when + # opening CONNECT tunnels to the worker's atunnel on :8443, and + # validates that server against the trust bundle. # The issued SPIFFE identity is # spiffe://cluster.local/ns/ate-system/sa/atenet-router, which atunnel's # --atunnel-client-identity allows. diff --git a/manifests/ate-install/generated/ate.dev_csidriverconfigs.yaml b/manifests/ate-install/generated/ate.dev_csidriverconfigs.yaml index 31bdc43f8d..edf4c95431 100644 --- a/manifests/ate-install/generated/ate.dev_csidriverconfigs.yaml +++ b/manifests/ate-install/generated/ate.dev_csidriverconfigs.yaml @@ -89,7 +89,9 @@ spec: description: Enabled controls whether TLS is used. type: boolean serverName: - description: ServerName override for TLS verification. + description: |- + ServerName specifies the expected server name (SAN) for TLS certificate verification. + Required when TLS is enabled. type: string usePodIdentity: description: UsePodIdentity indicates whether to reuse Substrate's @@ -102,6 +104,9 @@ spec: - message: tls.usePodIdentity must be true when tls.enabled is true; manual certificates are not yet supported rule: '!self.enabled || (has(self.usePodIdentity) && self.usePodIdentity)' + - message: tls.serverName is required when tls.enabled is true + rule: '!self.enabled || (has(self.serverName) && size(self.serverName) + > 0)' required: - controllerEndpoint - driverName diff --git a/manifests/ate-install/postgres-size10/postgres-config-patch.yaml b/manifests/ate-install/postgres-size10/postgres-config-patch.yaml index 8e6bba44d6..020494b6bc 100644 --- a/manifests/ate-install/postgres-size10/postgres-config-patch.yaml +++ b/manifests/ate-install/postgres-size10/postgres-config-patch.yaml @@ -46,9 +46,9 @@ data: wal_buffers = 64MB huge_pages = try - # 2 ate-api-server replicas at pool_max_conns=64 + 3-conn watch pool each - # = 134 steady-state; 300 leaves room to double replicas and for psql / - # monitoring / migrations. + # 2 ate-api-server replicas at pool_max_conns=64 + 3-conn watch pool + + # 1-conn readiness pool each = 136 steady-state; 300 leaves room to double + # replicas and for psql / monitoring / migrations. max_connections = 300 # 1 GB max_wal_size (the compiled default) forces a checkpoint every few diff --git a/pkg/api/v1alpha1/csidriverconfig_types.go b/pkg/api/v1alpha1/csidriverconfig_types.go index 02d095245f..e790237669 100644 --- a/pkg/api/v1alpha1/csidriverconfig_types.go +++ b/pkg/api/v1alpha1/csidriverconfig_types.go @@ -51,6 +51,7 @@ type CSIDriverConfigSpec struct { // CSIDriverTLSConfig holds TLS and mTLS configuration for CSI driver connections. // +kubebuilder:validation:XValidation:rule="!self.enabled || (has(self.usePodIdentity) && self.usePodIdentity)",message="tls.usePodIdentity must be true when tls.enabled is true; manual certificates are not yet supported" +// +kubebuilder:validation:XValidation:rule="!self.enabled || (has(self.serverName) && size(self.serverName) > 0)",message="tls.serverName is required when tls.enabled is true" type CSIDriverTLSConfig struct { // Enabled controls whether TLS is used. // +required @@ -61,7 +62,8 @@ type CSIDriverTLSConfig struct { // +optional UsePodIdentity bool `json:"usePodIdentity,omitempty"` - // ServerName override for TLS verification. + // ServerName specifies the expected server name (SAN) for TLS certificate verification. + // Required when TLS is enabled. // +optional ServerName string `json:"serverName,omitempty"` } diff --git a/pkg/api/v1alpha1/sandboxconfig_types.go b/pkg/api/v1alpha1/sandboxconfig_types.go index 131f9cb50c..5659e4c0f8 100644 --- a/pkg/api/v1alpha1/sandboxconfig_types.go +++ b/pkg/api/v1alpha1/sandboxconfig_types.go @@ -16,6 +16,8 @@ package v1alpha1 import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + "github.com/agent-substrate/substrate/pkg/sandboxclass" ) // SandboxClass selects the sandbox runtime family. It is shared by WorkerPool @@ -24,10 +26,10 @@ type SandboxClass string const ( // SandboxClassGvisor is the gVisor/runsc runtime (cmd/ateom-gvisor). Default. - SandboxClassGvisor SandboxClass = "gvisor" + SandboxClassGvisor SandboxClass = sandboxclass.GVisor // SandboxClassMicroVM is the micro-VM runtime (cmd/ateom-microvm); needs // /dev/kvm and vhost devices. - SandboxClassMicroVM SandboxClass = "microvm" + SandboxClassMicroVM SandboxClass = sandboxclass.MicroVM ) // AssetFile is one content-addressed file that atelet fetches for a sandbox diff --git a/pkg/objectstorage/gcs.go b/pkg/objectstorage/gcs.go index e0328708c4..1f6de99360 100644 --- a/pkg/objectstorage/gcs.go +++ b/pkg/objectstorage/gcs.go @@ -17,6 +17,7 @@ package objectstorage import ( "bytes" "context" + "encoding/xml" "errors" "fmt" "io" @@ -25,9 +26,36 @@ import ( "cloud.google.com/go/storage" "github.com/googleapis/gax-go/v2" + "google.golang.org/api/googleapi" "google.golang.org/api/option" ) +// describeGCSError describes a GCS API error in err's chain for +// DescribeServiceError. Reads use the XML API, whose error body carries a +// generic Code and Message and puts the caller and resource in Details, which +// is not read. JSON API errors quote the caller and resource in their message, +// so only their reason is kept. +func describeGCSError(err error) (ServiceError, bool) { + apiErr, ok := errors.AsType[*googleapi.Error](err) + if !ok { + return ServiceError{}, false + } + desc := ServiceError{HTTPStatus: apiErr.Code} + if len(apiErr.Errors) > 0 { + desc.Code = apiErr.Errors[0].Reason + return desc, true + } + var body struct { + XMLName xml.Name `xml:"Error"` + Code string `xml:"Code"` + Message string `xml:"Message"` + } + if xml.Unmarshal([]byte(apiErr.Body), &body) == nil { + desc.Code, desc.Message = body.Code, body.Message + } + return desc, true +} + type gcsClient struct { client *storage.Client // opts are how client was built, so the pool below is built the same way. diff --git a/pkg/objectstorage/gcs_errors_test.go b/pkg/objectstorage/gcs_errors_test.go new file mode 100644 index 0000000000..4e540833c3 --- /dev/null +++ b/pkg/objectstorage/gcs_errors_test.go @@ -0,0 +1,95 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package objectstorage + +import ( + "errors" + "fmt" + "strings" + "testing" + + "google.golang.org/api/googleapi" +) + +// TestDescribeGCSError checks what a GCS API error is described as: an XML API +// error's status, Code and Message but never its Details, and a JSON API +// error's status and reason but never its message, which names the caller and +// resource. +func TestDescribeGCSError(t *testing.T) { + const secret = "caller@project.iam.gserviceaccount.com does not have storage.objects.get access to gvisor/releases/runsc" + for _, tc := range []struct { + name string + err error + want ServiceError + }{ + { + // What GCS returned to the plugin's Workload Identity token for + // gs://gvisor (#2613). + name: "XML API error", + err: &googleapi.Error{ + Code: 412, + Body: "PreconditionFailedThe operation requires that Uniform Bucket Level Access be enabled.
" + secret + "
", + }, + want: ServiceError{HTTPStatus: 412, Code: "PreconditionFailed", Message: "The operation requires that Uniform Bucket Level Access be enabled."}, + }, + { + name: "XML API error, wrapped", + err: fmt.Errorf("while opening gs://gvisor/releases/runsc: %w", &googleapi.Error{ + Code: 403, + Body: "AccessDeniedAccess denied.
" + secret + "
", + }), + want: ServiceError{HTTPStatus: 403, Code: "AccessDenied", Message: "Access denied."}, + }, + { + name: "JSON API error", + err: &googleapi.Error{ + Code: 403, + Message: secret, + Errors: []googleapi.ErrorItem{{Reason: "forbidden", Message: secret}}, + Body: `{"error":{"code":403,"message":"` + secret + `"}}`, + }, + want: ServiceError{HTTPStatus: 403, Code: "forbidden"}, + }, + { + name: "body that is not an XML error", + err: &googleapi.Error{Code: 502, Body: "Bad Gateway " + secret + ""}, + want: ServiceError{HTTPStatus: 502}, + }, + { + name: "empty body", + err: &googleapi.Error{Code: 500}, + want: ServiceError{HTTPStatus: 500}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + got, ok := DescribeServiceError(tc.err) + if !ok { + t.Fatalf("DescribeServiceError(%v) found no service error", tc.err) + } + if got != tc.want { + t.Errorf("DescribeServiceError = %+v, want %+v", got, tc.want) + } + if strings.Contains(got.Code+got.Message, "caller@") { + t.Errorf("description %+v names the caller or resource", got) + } + }) + } + + t.Run("not a GCS error", func(t *testing.T) { + if got, ok := describeGCSError(errors.New("boom")); ok { + t.Errorf("describeGCSError(plain error) = %+v, true; want false", got) + } + }) +} diff --git a/pkg/objectstorage/objects.go b/pkg/objectstorage/objects.go index 7c37fbbfc0..6e697747cb 100644 --- a/pkg/objectstorage/objects.go +++ b/pkg/objectstorage/objects.go @@ -38,6 +38,35 @@ var tracer = otel.Tracer("objectstorage") // storage backend, as opposed to a transient fetch failure. var ErrObjectNotFound = errors.New("object not found in storage backend") +// ServiceError describes a storage service's failed response in terms its +// backend has judged safe to log: never a URI, header or credential. +type ServiceError struct { + // HTTPStatus is the status of the service's response. + HTTPStatus int + // Code is the service's error code, such as "PreconditionFailed", if the + // backend reports one. + Code string + // Message is the service's own description, if the backend reports one + // it knows names no caller, resource or credential. + Message string +} + +// DescribeServiceError describes the storage service response that err +// carries, if any. Each backend decides what of its errors is safe to +// describe; an error in err's chain from any other source with an +// HTTPStatusCode() int method, as the S3 SDK's response errors have, is +// described by its status alone. +func DescribeServiceError(err error) (ServiceError, bool) { + if desc, ok := describeGCSError(err); ok { + return desc, true + } + var withStatus interface{ HTTPStatusCode() int } + if errors.As(err, &withStatus) { + return ServiceError{HTTPStatus: withStatus.HTTPStatusCode()}, true + } + return ServiceError{}, false +} + type ObjectStorage interface { GetObject(ctx context.Context, bucket, object string) (io.ReadCloser, error) PutObject(ctx context.Context, bucket, object string, reader io.Reader) error diff --git a/pkg/proto/ateapipb/ateapi.pb.go b/pkg/proto/ateapipb/ateapi.pb.go index 8b92f27931..74b2b36834 100644 --- a/pkg/proto/ateapipb/ateapi.pb.go +++ b/pkg/proto/ateapipb/ateapi.pb.go @@ -23,6 +23,7 @@ package ateapipb import ( protoreflect "google.golang.org/protobuf/reflect/protoreflect" protoimpl "google.golang.org/protobuf/runtime/protoimpl" + descriptorpb "google.golang.org/protobuf/types/descriptorpb" timestamppb "google.golang.org/protobuf/types/known/timestamppb" reflect "reflect" sync "sync" @@ -318,57 +319,6 @@ func (ActorState) EnumDescriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{4} } -// SandboxClass selects the sandbox runtime family. Snapshots are not portable -// across classes. -type SandboxClass int32 - -const ( - SandboxClass_SANDBOX_CLASS_UNSPECIFIED SandboxClass = 0 - SandboxClass_SANDBOX_CLASS_GVISOR SandboxClass = 1 - SandboxClass_SANDBOX_CLASS_MICROVM SandboxClass = 2 // Keep this in sync with SandboxConfig.sandbox_class's maximum. -) - -// Enum value maps for SandboxClass. -var ( - SandboxClass_name = map[int32]string{ - 0: "SANDBOX_CLASS_UNSPECIFIED", - 1: "SANDBOX_CLASS_GVISOR", - 2: "SANDBOX_CLASS_MICROVM", - } - SandboxClass_value = map[string]int32{ - "SANDBOX_CLASS_UNSPECIFIED": 0, - "SANDBOX_CLASS_GVISOR": 1, - "SANDBOX_CLASS_MICROVM": 2, - } -) - -func (x SandboxClass) Enum() *SandboxClass { - p := new(SandboxClass) - *p = x - return p -} - -func (x SandboxClass) String() string { - return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) -} - -func (SandboxClass) Descriptor() protoreflect.EnumDescriptor { - return file_ateapi_proto_enumTypes[5].Descriptor() -} - -func (SandboxClass) Type() protoreflect.EnumType { - return &file_ateapi_proto_enumTypes[5] -} - -func (x SandboxClass) Number() protoreflect.EnumNumber { - return protoreflect.EnumNumber(x) -} - -// Deprecated: Use SandboxClass.Descriptor instead. -func (SandboxClass) EnumDescriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{5} -} - // GoldenSnapshotMode selects whether a template builds a golden snapshot. type GoldenSnapshotMode int32 @@ -403,11 +353,11 @@ func (x GoldenSnapshotMode) String() string { } func (GoldenSnapshotMode) Descriptor() protoreflect.EnumDescriptor { - return file_ateapi_proto_enumTypes[6].Descriptor() + return file_ateapi_proto_enumTypes[5].Descriptor() } func (GoldenSnapshotMode) Type() protoreflect.EnumType { - return &file_ateapi_proto_enumTypes[6] + return &file_ateapi_proto_enumTypes[5] } func (x GoldenSnapshotMode) Number() protoreflect.EnumNumber { @@ -416,7 +366,7 @@ func (x GoldenSnapshotMode) Number() protoreflect.EnumNumber { // Deprecated: Use GoldenSnapshotMode.Descriptor instead. func (GoldenSnapshotMode) EnumDescriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{6} + return file_ateapi_proto_rawDescGZIP(), []int{5} } // ActorMetadataField selects one identity field of the actor. @@ -456,11 +406,11 @@ func (x ActorMetadataField) String() string { } func (ActorMetadataField) Descriptor() protoreflect.EnumDescriptor { - return file_ateapi_proto_enumTypes[7].Descriptor() + return file_ateapi_proto_enumTypes[6].Descriptor() } func (ActorMetadataField) Type() protoreflect.EnumType { - return &file_ateapi_proto_enumTypes[7] + return &file_ateapi_proto_enumTypes[6] } func (x ActorMetadataField) Number() protoreflect.EnumNumber { @@ -469,7 +419,7 @@ func (x ActorMetadataField) Number() protoreflect.EnumNumber { // Deprecated: Use ActorMetadataField.Descriptor instead. func (ActorMetadataField) EnumDescriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{7} + return file_ateapi_proto_rawDescGZIP(), []int{6} } type WorkerState int32 @@ -478,8 +428,12 @@ const ( WorkerState_WORKER_STATE_UNSPECIFIED WorkerState = 0 // Ready; schedulable. WorkerState_WORKER_STATE_ACTIVE WorkerState = 1 - // Pod terminating. Not schedulable. - WorkerState_WORKER_STATE_DRAINING WorkerState = 2 // Keep this in sync with WorkerStatus.state's maximum. + // Pod terminating. Not schedulable. Final: a draining Worker never becomes + // active or unavailable again. + WorkerState_WORKER_STATE_DRAINING WorkerState = 2 + // Registered but not Ready, or its ateom restarting. Not schedulable. Every + // Worker starts here until MarkWorkerAvailable. + WorkerState_WORKER_STATE_UNAVAILABLE WorkerState = 3 // Keep this in sync with WorkerStatus.state's maximum. ) // Enum value maps for WorkerState. @@ -488,11 +442,13 @@ var ( 0: "WORKER_STATE_UNSPECIFIED", 1: "WORKER_STATE_ACTIVE", 2: "WORKER_STATE_DRAINING", + 3: "WORKER_STATE_UNAVAILABLE", } WorkerState_value = map[string]int32{ "WORKER_STATE_UNSPECIFIED": 0, "WORKER_STATE_ACTIVE": 1, "WORKER_STATE_DRAINING": 2, + "WORKER_STATE_UNAVAILABLE": 3, } ) @@ -507,11 +463,11 @@ func (x WorkerState) String() string { } func (WorkerState) Descriptor() protoreflect.EnumDescriptor { - return file_ateapi_proto_enumTypes[8].Descriptor() + return file_ateapi_proto_enumTypes[7].Descriptor() } func (WorkerState) Type() protoreflect.EnumType { - return &file_ateapi_proto_enumTypes[8] + return &file_ateapi_proto_enumTypes[7] } func (x WorkerState) Number() protoreflect.EnumNumber { @@ -520,62 +476,60 @@ func (x WorkerState) Number() protoreflect.EnumNumber { // Deprecated: Use WorkerState.Descriptor instead. func (WorkerState) EnumDescriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{8} + return file_ateapi_proto_rawDescGZIP(), []int{7} } -type ExternalVolume_Status int32 +// Scope is whether a top-level resource belongs to an atespace. +type Scope int32 const ( - ExternalVolume_STATUS_UNSPECIFIED ExternalVolume_Status = 0 - // Volume creation pending in the storage system. - ExternalVolume_STATUS_PENDING ExternalVolume_Status = 1 - // Volume successfully created in the storage system. - ExternalVolume_STATUS_CREATED ExternalVolume_Status = 2 - // Volume being deleted from the storage system. - ExternalVolume_STATUS_DELETING ExternalVolume_Status = 3 // Keep this in sync with ExternalVolume.status's maximum. + // The scope is not set. + Scope_SCOPE_UNSPECIFIED Scope = 0 + // The resource belongs to an atespace, identified by (atespace, name). + Scope_SCOPE_ATESPACE Scope = 1 + // The resource belongs to no atespace, identified by its name alone. + Scope_SCOPE_GLOBAL Scope = 2 ) -// Enum value maps for ExternalVolume_Status. +// Enum value maps for Scope. var ( - ExternalVolume_Status_name = map[int32]string{ - 0: "STATUS_UNSPECIFIED", - 1: "STATUS_PENDING", - 2: "STATUS_CREATED", - 3: "STATUS_DELETING", - } - ExternalVolume_Status_value = map[string]int32{ - "STATUS_UNSPECIFIED": 0, - "STATUS_PENDING": 1, - "STATUS_CREATED": 2, - "STATUS_DELETING": 3, + Scope_name = map[int32]string{ + 0: "SCOPE_UNSPECIFIED", + 1: "SCOPE_ATESPACE", + 2: "SCOPE_GLOBAL", + } + Scope_value = map[string]int32{ + "SCOPE_UNSPECIFIED": 0, + "SCOPE_ATESPACE": 1, + "SCOPE_GLOBAL": 2, } ) -func (x ExternalVolume_Status) Enum() *ExternalVolume_Status { - p := new(ExternalVolume_Status) +func (x Scope) Enum() *Scope { + p := new(Scope) *p = x return p } -func (x ExternalVolume_Status) String() string { +func (x Scope) String() string { return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) } -func (ExternalVolume_Status) Descriptor() protoreflect.EnumDescriptor { - return file_ateapi_proto_enumTypes[9].Descriptor() +func (Scope) Descriptor() protoreflect.EnumDescriptor { + return file_ateapi_proto_enumTypes[8].Descriptor() } -func (ExternalVolume_Status) Type() protoreflect.EnumType { - return &file_ateapi_proto_enumTypes[9] +func (Scope) Type() protoreflect.EnumType { + return &file_ateapi_proto_enumTypes[8] } -func (x ExternalVolume_Status) Number() protoreflect.EnumNumber { +func (x Scope) Number() protoreflect.EnumNumber { return protoreflect.EnumNumber(x) } -// Deprecated: Use ExternalVolume_Status.Descriptor instead. -func (ExternalVolume_Status) EnumDescriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{5, 0} +// Deprecated: Use Scope.Descriptor instead. +func (Scope) EnumDescriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{8} } // ObjectSnapshot addresses a snapshot held in object storage. @@ -735,8 +689,13 @@ type Snapshot struct { // // +k8s:optional DurableSnapshot *SnapshotStorage `protobuf:"bytes,7,opt,name=durable_snapshot,json=durableSnapshot,proto3" json:"durable_snapshot,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // sandbox_runtime is the runtime the actor ran under when the snapshot was + // taken. + // + // +k8s:optional + SandboxRuntime *SandboxRuntime `protobuf:"bytes,8,opt,name=sandbox_runtime,json=sandboxRuntime,proto3" json:"sandbox_runtime,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *Snapshot) Reset() { @@ -818,6 +777,13 @@ func (x *Snapshot) GetDurableSnapshot() *SnapshotStorage { return nil } +func (x *Snapshot) GetSandboxRuntime() *SandboxRuntime { + if x != nil { + return x.SandboxRuntime + } + return nil +} + // Selector matches worker pools by label. // Only equality-based matching is supported. type Selector struct { @@ -1037,10 +1003,6 @@ type ExternalVolume struct { // +k8s:update=NoUnset # set-once, like name above // +k8s:customValidation VolumeType string `protobuf:"bytes,3,opt,name=volume_type,json=volumeType,proto3" json:"volume_type,omitempty"` - // +k8s:optional - // +k8s:minimum=1 - // +k8s:maximum=3 # keep this in sync with the Status enum - Status ExternalVolume_Status `protobuf:"varint,4,opt,name=status,proto3,enum=ateapi.ExternalVolume_Status" json:"status,omitempty"` // volume_context contains metadata returned by the CSI driver during volume // provisioning, needed by the node plugin for mounting (e.g. attachment // info). Keys and values are the driver's own, so the map is only bounded, @@ -1104,13 +1066,6 @@ func (x *ExternalVolume) GetVolumeType() string { return "" } -func (x *ExternalVolume) GetStatus() ExternalVolume_Status { - if x != nil { - return x.Status - } - return ExternalVolume_STATUS_UNSPECIFIED -} - func (x *ExternalVolume) GetVolumeContext() map[string]string { if x != nil { return x.VolumeContext @@ -2216,9 +2171,14 @@ type WorkerAssignment struct { // // +k8s:optional // +k8s:minimum=0 - WorkerEpoch int64 `protobuf:"varint,8,opt,name=worker_epoch,json=workerEpoch,proto3" json:"worker_epoch,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + WorkerEpoch int64 `protobuf:"varint,8,opt,name=worker_epoch,json=workerEpoch,proto3" json:"worker_epoch,omitempty"` + // sandbox_runtime is the sandbox runtime and compatibility version the Actor + // was started or restored under on this Worker. + // + // +k8s:optional + SandboxRuntime *SandboxRuntime `protobuf:"bytes,9,opt,name=sandbox_runtime,json=sandboxRuntime,proto3" json:"sandbox_runtime,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *WorkerAssignment) Reset() { @@ -2307,6 +2267,13 @@ func (x *WorkerAssignment) GetWorkerEpoch() int64 { return 0 } +func (x *WorkerAssignment) GetSandboxRuntime() *SandboxRuntime { + if x != nil { + return x.SandboxRuntime + } + return nil +} + // TagStatus is the server-owned state of a Tag. // The tag is fixed to one external snapshot once its creation completes, so // the rest is immutable provenance. @@ -2939,12 +2906,13 @@ func (x *ActorTemplateStatus) GetGoldenSnapshotStatus() *GoldenSnapshotStatus { // SandboxConfig selects the sandbox runtime for an ActorTemplate. type SandboxConfig struct { state protoimpl.MessageState `protogen:"open.v1"` - // sandbox_class selects the sandbox runtime family. + // sandbox_class selects the sandbox runtime family: "gvisor" or "microvm". + // Snapshots are not portable across classes. // // +k8s:required - // +k8s:minimum=1 - // +k8s:maximum=2 # keep this in sync with the SandboxClass enum - SandboxClass SandboxClass `protobuf:"varint,1,opt,name=sandbox_class,json=sandboxClass,proto3,enum=ateapi.SandboxClass" json:"sandbox_class,omitempty"` + // +k8s:maxLength=63 + // +k8s:customValidation # must be "gvisor" or "microvm" + SandboxClass string `protobuf:"bytes,1,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // config_name names the cluster-scoped SandboxConfig Kubernetes object // supplying the sandbox binaries. Required; must match sandbox_class. // @@ -2985,11 +2953,11 @@ func (*SandboxConfig) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{29} } -func (x *SandboxConfig) GetSandboxClass() SandboxClass { +func (x *SandboxConfig) GetSandboxClass() string { if x != nil { return x.SandboxClass } - return SandboxClass_SANDBOX_CLASS_UNSPECIFIED + return "" } func (x *SandboxConfig) GetConfigName() string { @@ -3017,7 +2985,7 @@ type SnapshotConfig struct { // captured or is too expensive to capture at the time, for example memory // under host pressure. A process resumed in place on its worker keeps its // memory regardless of this setting. ROOTFS requires - // sandbox_config.sandbox_class SANDBOX_CLASS_MICROVM. Defaults to MEMORY + // sandbox_config.sandbox_class "microvm". Defaults to MEMORY // when unset. // // +k8s:required @@ -5741,8 +5709,13 @@ type MintActorJwtResponse struct { // * exp, nbf, iat: Numeric Unix timestamps. nbf is five minutes before iat // to allow for clock skew. // * jti: A random token ID. - // * `ate.dev`: A JSON object with the actor's `atespace`, `actorName`, and - // `actorUID`. + // * `actor.ate.dev`: A JSON object with the actor's `atespace`, + // `actorName`, and `actorUID`. + // + // The subject and claim name say the token speaks for the actor itself. + // A token minted for a Substrate component acting on the actor's behalf + // (WorkerService.MintAteomActorJwt) uses `ateom-for-actor` in both places + // instead, so a relying party can grant the two different access. ActorJwt string `protobuf:"bytes,1,opt,name=actor_jwt,json=actorJwt,proto3" json:"actor_jwt,omitempty"` // expire_time is when actor_jwt expires. Equal to its exp claim. ExpireTime *timestamppb.Timestamp `protobuf:"bytes,2,opt,name=expire_time,json=expireTime,proto3" json:"expire_time,omitempty"` @@ -6858,6 +6831,196 @@ func (x *DrainWorkerResponse) GetWorker() *Worker { return nil } +// MarkWorkerAvailableRequest names the Worker to mark ACTIVE. +type MarkWorkerAvailableRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The Worker to mark ACTIVE. + // + // +k8s:required + // +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix + Worker *ObjectRef `protobuf:"bytes,1,opt,name=worker,proto3" json:"worker,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *MarkWorkerAvailableRequest) Reset() { + *x = MarkWorkerAvailableRequest{} + mi := &file_ateapi_proto_msgTypes[96] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *MarkWorkerAvailableRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MarkWorkerAvailableRequest) ProtoMessage() {} + +func (x *MarkWorkerAvailableRequest) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[96] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MarkWorkerAvailableRequest.ProtoReflect.Descriptor instead. +func (*MarkWorkerAvailableRequest) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{96} +} + +func (x *MarkWorkerAvailableRequest) GetWorker() *ObjectRef { + if x != nil { + return x.Worker + } + return nil +} + +// MarkWorkerAvailableResponse holds the Worker marked ACTIVE. +type MarkWorkerAvailableResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The Worker marked ACTIVE. + Worker *Worker `protobuf:"bytes,1,opt,name=worker,proto3" json:"worker,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *MarkWorkerAvailableResponse) Reset() { + *x = MarkWorkerAvailableResponse{} + mi := &file_ateapi_proto_msgTypes[97] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *MarkWorkerAvailableResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MarkWorkerAvailableResponse) ProtoMessage() {} + +func (x *MarkWorkerAvailableResponse) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[97] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MarkWorkerAvailableResponse.ProtoReflect.Descriptor instead. +func (*MarkWorkerAvailableResponse) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{97} +} + +func (x *MarkWorkerAvailableResponse) GetWorker() *Worker { + if x != nil { + return x.Worker + } + return nil +} + +// MarkWorkerUnavailableRequest names the Worker to mark UNAVAILABLE. +type MarkWorkerUnavailableRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The Worker to mark UNAVAILABLE. + // + // +k8s:required + // +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix + Worker *ObjectRef `protobuf:"bytes,1,opt,name=worker,proto3" json:"worker,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *MarkWorkerUnavailableRequest) Reset() { + *x = MarkWorkerUnavailableRequest{} + mi := &file_ateapi_proto_msgTypes[98] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *MarkWorkerUnavailableRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MarkWorkerUnavailableRequest) ProtoMessage() {} + +func (x *MarkWorkerUnavailableRequest) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[98] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MarkWorkerUnavailableRequest.ProtoReflect.Descriptor instead. +func (*MarkWorkerUnavailableRequest) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{98} +} + +func (x *MarkWorkerUnavailableRequest) GetWorker() *ObjectRef { + if x != nil { + return x.Worker + } + return nil +} + +// MarkWorkerUnavailableResponse holds the Worker marked UNAVAILABLE. +type MarkWorkerUnavailableResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The Worker marked UNAVAILABLE. + Worker *Worker `protobuf:"bytes,1,opt,name=worker,proto3" json:"worker,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *MarkWorkerUnavailableResponse) Reset() { + *x = MarkWorkerUnavailableResponse{} + mi := &file_ateapi_proto_msgTypes[99] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *MarkWorkerUnavailableResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MarkWorkerUnavailableResponse) ProtoMessage() {} + +func (x *MarkWorkerUnavailableResponse) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[99] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MarkWorkerUnavailableResponse.ProtoReflect.Descriptor instead. +func (*MarkWorkerUnavailableResponse) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{99} +} + +func (x *MarkWorkerUnavailableResponse) GetWorker() *Worker { + if x != nil { + return x.Worker + } + return nil +} + // List actors with pagination. Provides "soft" guarantees: actors may be // missed or duplicated if the system state changes during pagination. type ListActorsRequest struct { @@ -6877,7 +7040,7 @@ type ListActorsRequest struct { func (x *ListActorsRequest) Reset() { *x = ListActorsRequest{} - mi := &file_ateapi_proto_msgTypes[96] + mi := &file_ateapi_proto_msgTypes[100] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -6889,7 +7052,7 @@ func (x *ListActorsRequest) String() string { func (*ListActorsRequest) ProtoMessage() {} func (x *ListActorsRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[96] + mi := &file_ateapi_proto_msgTypes[100] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -6902,7 +7065,7 @@ func (x *ListActorsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ListActorsRequest.ProtoReflect.Descriptor instead. func (*ListActorsRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{96} + return file_ateapi_proto_rawDescGZIP(), []int{100} } func (x *ListActorsRequest) GetAtespace() string { @@ -6931,7 +7094,7 @@ type ListActorsResponse struct { func (x *ListActorsResponse) Reset() { *x = ListActorsResponse{} - mi := &file_ateapi_proto_msgTypes[97] + mi := &file_ateapi_proto_msgTypes[101] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -6943,7 +7106,7 @@ func (x *ListActorsResponse) String() string { func (*ListActorsResponse) ProtoMessage() {} func (x *ListActorsResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[97] + mi := &file_ateapi_proto_msgTypes[101] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -6956,7 +7119,7 @@ func (x *ListActorsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ListActorsResponse.ProtoReflect.Descriptor instead. func (*ListActorsResponse) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{97} + return file_ateapi_proto_rawDescGZIP(), []int{101} } func (x *ListActorsResponse) GetActors() []*Actor { @@ -7026,6 +7189,7 @@ type Worker struct { // // +k8s:optional // +k8s:maxLength=63 + // +k8s:customValidation # gvisor or microvm // +k8s:immutable SandboxClass string `protobuf:"bytes,8,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // labels mirror the WorkerPool object's Kubernetes labels, which selectors @@ -7052,8 +7216,9 @@ type Worker struct { // +k8s:customValidation Epoch int64 `protobuf:"varint,12,opt,name=epoch,proto3" json:"epoch,omitempty"` // Output-only server-managed state. Absent from Create/Update request - // payloads; whatever a request carries here is ignored. DrainWorker is the - // only way a client moves state, and the assignments are the scheduler's. + // payloads; whatever a request carries here is ignored. A client moves + // state only through DrainWorker, MarkWorkerAvailable and + // MarkWorkerUnavailable, and the assignments are the scheduler's. // // +k8s:optional Status *WorkerStatus `protobuf:"bytes,11,opt,name=status,proto3" json:"status,omitempty"` @@ -7063,7 +7228,7 @@ type Worker struct { func (x *Worker) Reset() { *x = Worker{} - mi := &file_ateapi_proto_msgTypes[98] + mi := &file_ateapi_proto_msgTypes[102] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7075,7 +7240,7 @@ func (x *Worker) String() string { func (*Worker) ProtoMessage() {} func (x *Worker) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[98] + mi := &file_ateapi_proto_msgTypes[102] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7088,7 +7253,7 @@ func (x *Worker) ProtoReflect() protoreflect.Message { // Deprecated: Use Worker.ProtoReflect.Descriptor instead. func (*Worker) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{98} + return file_ateapi_proto_rawDescGZIP(), []int{102} } func (x *Worker) GetMetadata() *ResourceMetadata { @@ -7172,7 +7337,7 @@ type WorkerStatus struct { state protoimpl.MessageState `protogen:"open.v1"` // +k8s:required // +k8s:minimum=1 - // +k8s:maximum=2 # keep this in sync with the WorkerState enum + // +k8s:maximum=3 # keep this in sync with the WorkerState enum State WorkerState `protobuf:"varint,1,opt,name=state,proto3,enum=ateapi.WorkerState" json:"state,omitempty"` // The total resource capacity of the Worker. // @@ -7217,7 +7382,7 @@ type WorkerStatus struct { func (x *WorkerStatus) Reset() { *x = WorkerStatus{} - mi := &file_ateapi_proto_msgTypes[99] + mi := &file_ateapi_proto_msgTypes[103] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7229,7 +7394,7 @@ func (x *WorkerStatus) String() string { func (*WorkerStatus) ProtoMessage() {} func (x *WorkerStatus) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[99] + mi := &file_ateapi_proto_msgTypes[103] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7242,7 +7407,7 @@ func (x *WorkerStatus) ProtoReflect() protoreflect.Message { // Deprecated: Use WorkerStatus.ProtoReflect.Descriptor instead. func (*WorkerStatus) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{99} + return file_ateapi_proto_rawDescGZIP(), []int{103} } func (x *WorkerStatus) GetState() WorkerState { @@ -7295,6 +7460,7 @@ type SandboxRuntime struct { // // +k8s:required // +k8s:maxLength=63 + // +k8s:customValidation # gvisor or microvm SandboxClass string `protobuf:"bytes,1,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // Primary compatibility version stamped onto an actor's snapshot when a // checkpoint is taken on this runtime. Snapshots matching `version` @@ -7309,7 +7475,7 @@ type SandboxRuntime struct { func (x *SandboxRuntime) Reset() { *x = SandboxRuntime{} - mi := &file_ateapi_proto_msgTypes[100] + mi := &file_ateapi_proto_msgTypes[104] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7321,7 +7487,7 @@ func (x *SandboxRuntime) String() string { func (*SandboxRuntime) ProtoMessage() {} func (x *SandboxRuntime) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[100] + mi := &file_ateapi_proto_msgTypes[104] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7334,7 +7500,7 @@ func (x *SandboxRuntime) ProtoReflect() protoreflect.Message { // Deprecated: Use SandboxRuntime.ProtoReflect.Descriptor instead. func (*SandboxRuntime) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{100} + return file_ateapi_proto_rawDescGZIP(), []int{104} } func (x *SandboxRuntime) GetSandboxClass() string { @@ -7366,8 +7532,10 @@ type VersionedSandboxCompat struct { // features after masking (e.g. "architecture", "cpu_features"). // Matched against a snapshot under the same schema_version: every attribute // on the snapshot must be present on the worker with an equal value; keys - // only the worker has are ignored. Bump schema_version when a key's meaning - // changes or older snapshots must no longer match. + // only the worker has are ignored, so a worker can gain keys under one + // schema_version without invalidating older snapshots. Removing or + // recomputing a key is a new schema_version, under which older snapshots + // no longer match. // Order as sent does not matter: ate-api-server sorts attributes by key // before recording them. // @@ -7382,7 +7550,7 @@ type VersionedSandboxCompat struct { func (x *VersionedSandboxCompat) Reset() { *x = VersionedSandboxCompat{} - mi := &file_ateapi_proto_msgTypes[101] + mi := &file_ateapi_proto_msgTypes[105] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7394,7 +7562,7 @@ func (x *VersionedSandboxCompat) String() string { func (*VersionedSandboxCompat) ProtoMessage() {} func (x *VersionedSandboxCompat) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[101] + mi := &file_ateapi_proto_msgTypes[105] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7407,7 +7575,7 @@ func (x *VersionedSandboxCompat) ProtoReflect() protoreflect.Message { // Deprecated: Use VersionedSandboxCompat.ProtoReflect.Descriptor instead. func (*VersionedSandboxCompat) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{101} + return file_ateapi_proto_rawDescGZIP(), []int{105} } func (x *VersionedSandboxCompat) GetSchemaVersion() string { @@ -7443,7 +7611,7 @@ type AttributeEntry struct { func (x *AttributeEntry) Reset() { *x = AttributeEntry{} - mi := &file_ateapi_proto_msgTypes[102] + mi := &file_ateapi_proto_msgTypes[106] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7455,7 +7623,7 @@ func (x *AttributeEntry) String() string { func (*AttributeEntry) ProtoMessage() {} func (x *AttributeEntry) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[102] + mi := &file_ateapi_proto_msgTypes[106] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7468,7 +7636,7 @@ func (x *AttributeEntry) ProtoReflect() protoreflect.Message { // Deprecated: Use AttributeEntry.ProtoReflect.Descriptor instead. func (*AttributeEntry) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{102} + return file_ateapi_proto_rawDescGZIP(), []int{106} } func (x *AttributeEntry) GetKey() string { @@ -7507,7 +7675,7 @@ type WorkerResources struct { func (x *WorkerResources) Reset() { *x = WorkerResources{} - mi := &file_ateapi_proto_msgTypes[103] + mi := &file_ateapi_proto_msgTypes[107] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7519,7 +7687,7 @@ func (x *WorkerResources) String() string { func (*WorkerResources) ProtoMessage() {} func (x *WorkerResources) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[103] + mi := &file_ateapi_proto_msgTypes[107] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7532,7 +7700,7 @@ func (x *WorkerResources) ProtoReflect() protoreflect.Message { // Deprecated: Use WorkerResources.ProtoReflect.Descriptor instead. func (*WorkerResources) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{103} + return file_ateapi_proto_rawDescGZIP(), []int{107} } func (x *WorkerResources) GetResources() *Resources { @@ -7585,7 +7753,7 @@ type ActorAssignment struct { func (x *ActorAssignment) Reset() { *x = ActorAssignment{} - mi := &file_ateapi_proto_msgTypes[104] + mi := &file_ateapi_proto_msgTypes[108] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7597,7 +7765,7 @@ func (x *ActorAssignment) String() string { func (*ActorAssignment) ProtoMessage() {} func (x *ActorAssignment) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[104] + mi := &file_ateapi_proto_msgTypes[108] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7610,7 +7778,7 @@ func (x *ActorAssignment) ProtoReflect() protoreflect.Message { // Deprecated: Use ActorAssignment.ProtoReflect.Descriptor instead. func (*ActorAssignment) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{104} + return file_ateapi_proto_rawDescGZIP(), []int{108} } func (x *ActorAssignment) GetMetadata() *ResourceMetadata { @@ -7681,7 +7849,7 @@ type RegisterWorkerRequest struct { func (x *RegisterWorkerRequest) Reset() { *x = RegisterWorkerRequest{} - mi := &file_ateapi_proto_msgTypes[105] + mi := &file_ateapi_proto_msgTypes[109] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7693,7 +7861,7 @@ func (x *RegisterWorkerRequest) String() string { func (*RegisterWorkerRequest) ProtoMessage() {} func (x *RegisterWorkerRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[105] + mi := &file_ateapi_proto_msgTypes[109] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7706,7 +7874,7 @@ func (x *RegisterWorkerRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RegisterWorkerRequest.ProtoReflect.Descriptor instead. func (*RegisterWorkerRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{105} + return file_ateapi_proto_rawDescGZIP(), []int{109} } func (x *RegisterWorkerRequest) GetWorker() *ObjectRef { @@ -7747,7 +7915,7 @@ type RegisterWorkerResponse struct { func (x *RegisterWorkerResponse) Reset() { *x = RegisterWorkerResponse{} - mi := &file_ateapi_proto_msgTypes[106] + mi := &file_ateapi_proto_msgTypes[110] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7759,7 +7927,7 @@ func (x *RegisterWorkerResponse) String() string { func (*RegisterWorkerResponse) ProtoMessage() {} func (x *RegisterWorkerResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[106] + mi := &file_ateapi_proto_msgTypes[110] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7772,7 +7940,7 @@ func (x *RegisterWorkerResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RegisterWorkerResponse.ProtoReflect.Descriptor instead. func (*RegisterWorkerResponse) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{106} + return file_ateapi_proto_rawDescGZIP(), []int{110} } func (x *RegisterWorkerResponse) GetWorker() *Worker { @@ -7811,7 +7979,7 @@ type MintAteomActorCertificateRequest struct { func (x *MintAteomActorCertificateRequest) Reset() { *x = MintAteomActorCertificateRequest{} - mi := &file_ateapi_proto_msgTypes[107] + mi := &file_ateapi_proto_msgTypes[111] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7823,7 +7991,7 @@ func (x *MintAteomActorCertificateRequest) String() string { func (*MintAteomActorCertificateRequest) ProtoMessage() {} func (x *MintAteomActorCertificateRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[107] + mi := &file_ateapi_proto_msgTypes[111] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7836,7 +8004,7 @@ func (x *MintAteomActorCertificateRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use MintAteomActorCertificateRequest.ProtoReflect.Descriptor instead. func (*MintAteomActorCertificateRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{107} + return file_ateapi_proto_rawDescGZIP(), []int{111} } func (x *MintAteomActorCertificateRequest) GetActor() *ObjectRef { @@ -7873,7 +8041,7 @@ type MintAteomActorCertificateResponse struct { func (x *MintAteomActorCertificateResponse) Reset() { *x = MintAteomActorCertificateResponse{} - mi := &file_ateapi_proto_msgTypes[108] + mi := &file_ateapi_proto_msgTypes[112] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7885,7 +8053,7 @@ func (x *MintAteomActorCertificateResponse) String() string { func (*MintAteomActorCertificateResponse) ProtoMessage() {} func (x *MintAteomActorCertificateResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[108] + mi := &file_ateapi_proto_msgTypes[112] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7898,7 +8066,7 @@ func (x *MintAteomActorCertificateResponse) ProtoReflect() protoreflect.Message // Deprecated: Use MintAteomActorCertificateResponse.ProtoReflect.Descriptor instead. func (*MintAteomActorCertificateResponse) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{108} + return file_ateapi_proto_rawDescGZIP(), []int{112} } func (x *MintAteomActorCertificateResponse) GetActorCertificates() [][]byte { @@ -7908,6 +8076,150 @@ func (x *MintAteomActorCertificateResponse) GetActorCertificates() [][]byte { return nil } +// Request message for MintAteomActorJwt. +type MintAteomActorJwtRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The actor for which the JWT should be issued. + // + // Must be a valid actor that currently exists according to the actor store. + // + // +k8s:required + // +k8s:subfield(atespace)=+k8s:required + Actor *ObjectRef `protobuf:"bytes,1,opt,name=actor,proto3" json:"actor,omitempty"` + // The UID of the actor --- used to guard against deletion and recreation of + // an actor with the same name. + // + // +k8s:required + // +k8s:format=k8s-uuid + ActorUid string `protobuf:"bytes,2,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` + // The audiences the minted JWT is bound to. Tokens are only issued with + // audience bindings, so at least one is required. + // + // +k8s:required + // +k8s:maxItems=16 # guardrail; tokens realistically bind a handful of audiences + // +k8s:listType=set + // +k8s:eachVal=+k8s:minLength=1 + // +k8s:eachVal=+k8s:maxLength=512 # audiences are caller-defined URIs; bound only + Audiences []string `protobuf:"bytes,3,rep,name=audiences,proto3" json:"audiences,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *MintAteomActorJwtRequest) Reset() { + *x = MintAteomActorJwtRequest{} + mi := &file_ateapi_proto_msgTypes[113] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *MintAteomActorJwtRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MintAteomActorJwtRequest) ProtoMessage() {} + +func (x *MintAteomActorJwtRequest) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[113] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MintAteomActorJwtRequest.ProtoReflect.Descriptor instead. +func (*MintAteomActorJwtRequest) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{113} +} + +func (x *MintAteomActorJwtRequest) GetActor() *ObjectRef { + if x != nil { + return x.Actor + } + return nil +} + +func (x *MintAteomActorJwtRequest) GetActorUid() string { + if x != nil { + return x.ActorUid + } + return "" +} + +func (x *MintAteomActorJwtRequest) GetAudiences() []string { + if x != nil { + return x.Audiences + } + return nil +} + +// Response message for MintAteomActorJwt. +type MintAteomActorJwtResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The signed JWT. Its claims are those described on + // MintActorJwtResponse.actor_jwt, except that it speaks for an ateom + // acting on behalf of the actor rather than for the actor itself: + // + // * sub: `ateom-for-actor//`. + // * `ateom-for-actor.ate.dev`: A JSON object with the actor's `atespace`, + // `actorName`, and `actorUID`. + // + // A relying party that trusts actor JWTs must not accept this token as the + // actor, and vice versa. + ActorJwt string `protobuf:"bytes,1,opt,name=actor_jwt,json=actorJwt,proto3" json:"actor_jwt,omitempty"` + // expire_time is when actor_jwt expires. Equal to its exp claim. + ExpireTime *timestamppb.Timestamp `protobuf:"bytes,2,opt,name=expire_time,json=expireTime,proto3" json:"expire_time,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *MintAteomActorJwtResponse) Reset() { + *x = MintAteomActorJwtResponse{} + mi := &file_ateapi_proto_msgTypes[114] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *MintAteomActorJwtResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MintAteomActorJwtResponse) ProtoMessage() {} + +func (x *MintAteomActorJwtResponse) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[114] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MintAteomActorJwtResponse.ProtoReflect.Descriptor instead. +func (*MintAteomActorJwtResponse) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{114} +} + +func (x *MintAteomActorJwtResponse) GetActorJwt() string { + if x != nil { + return x.ActorJwt + } + return "" +} + +func (x *MintAteomActorJwtResponse) GetExpireTime() *timestamppb.Timestamp { + if x != nil { + return x.ExpireTime + } + return nil +} + type RequestActorSuspendRequest struct { state protoimpl.MessageState `protogen:"open.v1"` // The Worker making the request. atespace is always empty; Workers are @@ -7936,7 +8248,7 @@ type RequestActorSuspendRequest struct { func (x *RequestActorSuspendRequest) Reset() { *x = RequestActorSuspendRequest{} - mi := &file_ateapi_proto_msgTypes[109] + mi := &file_ateapi_proto_msgTypes[115] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -7948,7 +8260,7 @@ func (x *RequestActorSuspendRequest) String() string { func (*RequestActorSuspendRequest) ProtoMessage() {} func (x *RequestActorSuspendRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[109] + mi := &file_ateapi_proto_msgTypes[115] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -7961,7 +8273,7 @@ func (x *RequestActorSuspendRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RequestActorSuspendRequest.ProtoReflect.Descriptor instead. func (*RequestActorSuspendRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{109} + return file_ateapi_proto_rawDescGZIP(), []int{115} } func (x *RequestActorSuspendRequest) GetWorker() *ObjectRef { @@ -7995,7 +8307,7 @@ type RequestActorSuspendResponse struct { func (x *RequestActorSuspendResponse) Reset() { *x = RequestActorSuspendResponse{} - mi := &file_ateapi_proto_msgTypes[110] + mi := &file_ateapi_proto_msgTypes[116] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8007,7 +8319,7 @@ func (x *RequestActorSuspendResponse) String() string { func (*RequestActorSuspendResponse) ProtoMessage() {} func (x *RequestActorSuspendResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[110] + mi := &file_ateapi_proto_msgTypes[116] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8020,7 +8332,7 @@ func (x *RequestActorSuspendResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RequestActorSuspendResponse.ProtoReflect.Descriptor instead. func (*RequestActorSuspendResponse) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{110} + return file_ateapi_proto_rawDescGZIP(), []int{116} } func (x *RequestActorSuspendResponse) GetActor() *Actor { @@ -8051,7 +8363,7 @@ type ReportActorSnapshotStateRequest struct { func (x *ReportActorSnapshotStateRequest) Reset() { *x = ReportActorSnapshotStateRequest{} - mi := &file_ateapi_proto_msgTypes[111] + mi := &file_ateapi_proto_msgTypes[117] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8063,7 +8375,7 @@ func (x *ReportActorSnapshotStateRequest) String() string { func (*ReportActorSnapshotStateRequest) ProtoMessage() {} func (x *ReportActorSnapshotStateRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[111] + mi := &file_ateapi_proto_msgTypes[117] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8076,7 +8388,7 @@ func (x *ReportActorSnapshotStateRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ReportActorSnapshotStateRequest.ProtoReflect.Descriptor instead. func (*ReportActorSnapshotStateRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{111} + return file_ateapi_proto_rawDescGZIP(), []int{117} } func (x *ReportActorSnapshotStateRequest) GetActor() *ObjectRef { @@ -8108,7 +8420,7 @@ type ReportActorSnapshotStateResponse struct { func (x *ReportActorSnapshotStateResponse) Reset() { *x = ReportActorSnapshotStateResponse{} - mi := &file_ateapi_proto_msgTypes[112] + mi := &file_ateapi_proto_msgTypes[118] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8120,7 +8432,7 @@ func (x *ReportActorSnapshotStateResponse) String() string { func (*ReportActorSnapshotStateResponse) ProtoMessage() {} func (x *ReportActorSnapshotStateResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[112] + mi := &file_ateapi_proto_msgTypes[118] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8133,7 +8445,7 @@ func (x *ReportActorSnapshotStateResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ReportActorSnapshotStateResponse.ProtoReflect.Descriptor instead. func (*ReportActorSnapshotStateResponse) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{112} + return file_ateapi_proto_rawDescGZIP(), []int{118} } // ActorSnapshotState is what atelet holds for one Actor on its node. It @@ -8167,7 +8479,7 @@ type ActorSnapshotState struct { func (x *ActorSnapshotState) Reset() { *x = ActorSnapshotState{} - mi := &file_ateapi_proto_msgTypes[113] + mi := &file_ateapi_proto_msgTypes[119] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8179,7 +8491,7 @@ func (x *ActorSnapshotState) String() string { func (*ActorSnapshotState) ProtoMessage() {} func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[113] + mi := &file_ateapi_proto_msgTypes[119] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8192,7 +8504,7 @@ func (x *ActorSnapshotState) ProtoReflect() protoreflect.Message { // Deprecated: Use ActorSnapshotState.ProtoReflect.Descriptor instead. func (*ActorSnapshotState) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{113} + return file_ateapi_proto_rawDescGZIP(), []int{119} } func (x *ActorSnapshotState) GetLatestLocal() *SnapshotReport { @@ -8264,7 +8576,7 @@ type SnapshotReport struct { func (x *SnapshotReport) Reset() { *x = SnapshotReport{} - mi := &file_ateapi_proto_msgTypes[114] + mi := &file_ateapi_proto_msgTypes[120] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8276,7 +8588,7 @@ func (x *SnapshotReport) String() string { func (*SnapshotReport) ProtoMessage() {} func (x *SnapshotReport) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[114] + mi := &file_ateapi_proto_msgTypes[120] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8289,7 +8601,7 @@ func (x *SnapshotReport) ProtoReflect() protoreflect.Message { // Deprecated: Use SnapshotReport.ProtoReflect.Descriptor instead. func (*SnapshotReport) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{114} + return file_ateapi_proto_rawDescGZIP(), []int{120} } func (x *SnapshotReport) GetUuid() string { @@ -8359,7 +8671,7 @@ type AccessPolicy struct { func (x *AccessPolicy) Reset() { *x = AccessPolicy{} - mi := &file_ateapi_proto_msgTypes[115] + mi := &file_ateapi_proto_msgTypes[121] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8371,7 +8683,7 @@ func (x *AccessPolicy) String() string { func (*AccessPolicy) ProtoMessage() {} func (x *AccessPolicy) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[115] + mi := &file_ateapi_proto_msgTypes[121] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8384,7 +8696,7 @@ func (x *AccessPolicy) ProtoReflect() protoreflect.Message { // Deprecated: Use AccessPolicy.ProtoReflect.Descriptor instead. func (*AccessPolicy) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{115} + return file_ateapi_proto_rawDescGZIP(), []int{121} } func (x *AccessPolicy) GetMetadata() *ResourceMetadata { @@ -8426,7 +8738,7 @@ type Binding struct { func (x *Binding) Reset() { *x = Binding{} - mi := &file_ateapi_proto_msgTypes[116] + mi := &file_ateapi_proto_msgTypes[122] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8438,7 +8750,7 @@ func (x *Binding) String() string { func (*Binding) ProtoMessage() {} func (x *Binding) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[116] + mi := &file_ateapi_proto_msgTypes[122] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8451,7 +8763,7 @@ func (x *Binding) ProtoReflect() protoreflect.Message { // Deprecated: Use Binding.ProtoReflect.Descriptor instead. func (*Binding) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{116} + return file_ateapi_proto_rawDescGZIP(), []int{122} } func (x *Binding) GetRole() string { @@ -8483,7 +8795,7 @@ type GetGlobalAccessPolicyRequest struct { func (x *GetGlobalAccessPolicyRequest) Reset() { *x = GetGlobalAccessPolicyRequest{} - mi := &file_ateapi_proto_msgTypes[117] + mi := &file_ateapi_proto_msgTypes[123] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8495,7 +8807,7 @@ func (x *GetGlobalAccessPolicyRequest) String() string { func (*GetGlobalAccessPolicyRequest) ProtoMessage() {} func (x *GetGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[117] + mi := &file_ateapi_proto_msgTypes[123] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8508,7 +8820,7 @@ func (x *GetGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetGlobalAccessPolicyRequest.ProtoReflect.Descriptor instead. func (*GetGlobalAccessPolicyRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{117} + return file_ateapi_proto_rawDescGZIP(), []int{123} } func (x *GetGlobalAccessPolicyRequest) GetName() string { @@ -8532,7 +8844,7 @@ type CreateGlobalAccessPolicyRequest struct { func (x *CreateGlobalAccessPolicyRequest) Reset() { *x = CreateGlobalAccessPolicyRequest{} - mi := &file_ateapi_proto_msgTypes[118] + mi := &file_ateapi_proto_msgTypes[124] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8544,7 +8856,7 @@ func (x *CreateGlobalAccessPolicyRequest) String() string { func (*CreateGlobalAccessPolicyRequest) ProtoMessage() {} func (x *CreateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[118] + mi := &file_ateapi_proto_msgTypes[124] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8557,7 +8869,7 @@ func (x *CreateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use CreateGlobalAccessPolicyRequest.ProtoReflect.Descriptor instead. func (*CreateGlobalAccessPolicyRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{118} + return file_ateapi_proto_rawDescGZIP(), []int{124} } func (x *CreateGlobalAccessPolicyRequest) GetAccessPolicy() *AccessPolicy { @@ -8581,7 +8893,7 @@ type UpdateGlobalAccessPolicyRequest struct { func (x *UpdateGlobalAccessPolicyRequest) Reset() { *x = UpdateGlobalAccessPolicyRequest{} - mi := &file_ateapi_proto_msgTypes[119] + mi := &file_ateapi_proto_msgTypes[125] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8593,7 +8905,7 @@ func (x *UpdateGlobalAccessPolicyRequest) String() string { func (*UpdateGlobalAccessPolicyRequest) ProtoMessage() {} func (x *UpdateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[119] + mi := &file_ateapi_proto_msgTypes[125] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8606,7 +8918,7 @@ func (x *UpdateGlobalAccessPolicyRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use UpdateGlobalAccessPolicyRequest.ProtoReflect.Descriptor instead. func (*UpdateGlobalAccessPolicyRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{119} + return file_ateapi_proto_rawDescGZIP(), []int{125} } func (x *UpdateGlobalAccessPolicyRequest) GetAccessPolicy() *AccessPolicy { @@ -8636,7 +8948,7 @@ type GetAtespaceAccessPolicyRequest struct { func (x *GetAtespaceAccessPolicyRequest) Reset() { *x = GetAtespaceAccessPolicyRequest{} - mi := &file_ateapi_proto_msgTypes[120] + mi := &file_ateapi_proto_msgTypes[126] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8648,7 +8960,7 @@ func (x *GetAtespaceAccessPolicyRequest) String() string { func (*GetAtespaceAccessPolicyRequest) ProtoMessage() {} func (x *GetAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[120] + mi := &file_ateapi_proto_msgTypes[126] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8661,7 +8973,7 @@ func (x *GetAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead. func (*GetAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{120} + return file_ateapi_proto_rawDescGZIP(), []int{126} } func (x *GetAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef { @@ -8697,7 +9009,7 @@ type CreateAtespaceAccessPolicyRequest struct { func (x *CreateAtespaceAccessPolicyRequest) Reset() { *x = CreateAtespaceAccessPolicyRequest{} - mi := &file_ateapi_proto_msgTypes[121] + mi := &file_ateapi_proto_msgTypes[127] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8709,7 +9021,7 @@ func (x *CreateAtespaceAccessPolicyRequest) String() string { func (*CreateAtespaceAccessPolicyRequest) ProtoMessage() {} func (x *CreateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[121] + mi := &file_ateapi_proto_msgTypes[127] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8722,7 +9034,7 @@ func (x *CreateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message // Deprecated: Use CreateAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead. func (*CreateAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{121} + return file_ateapi_proto_rawDescGZIP(), []int{127} } func (x *CreateAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef { @@ -8758,7 +9070,7 @@ type UpdateAtespaceAccessPolicyRequest struct { func (x *UpdateAtespaceAccessPolicyRequest) Reset() { *x = UpdateAtespaceAccessPolicyRequest{} - mi := &file_ateapi_proto_msgTypes[122] + mi := &file_ateapi_proto_msgTypes[128] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8770,7 +9082,7 @@ func (x *UpdateAtespaceAccessPolicyRequest) String() string { func (*UpdateAtespaceAccessPolicyRequest) ProtoMessage() {} func (x *UpdateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[122] + mi := &file_ateapi_proto_msgTypes[128] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8783,7 +9095,7 @@ func (x *UpdateAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message // Deprecated: Use UpdateAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead. func (*UpdateAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{122} + return file_ateapi_proto_rawDescGZIP(), []int{128} } func (x *UpdateAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef { @@ -8824,7 +9136,7 @@ type DeleteAtespaceAccessPolicyRequest struct { func (x *DeleteAtespaceAccessPolicyRequest) Reset() { *x = DeleteAtespaceAccessPolicyRequest{} - mi := &file_ateapi_proto_msgTypes[123] + mi := &file_ateapi_proto_msgTypes[129] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -8836,7 +9148,7 @@ func (x *DeleteAtespaceAccessPolicyRequest) String() string { func (*DeleteAtespaceAccessPolicyRequest) ProtoMessage() {} func (x *DeleteAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateapi_proto_msgTypes[123] + mi := &file_ateapi_proto_msgTypes[129] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -8849,7 +9161,7 @@ func (x *DeleteAtespaceAccessPolicyRequest) ProtoReflect() protoreflect.Message // Deprecated: Use DeleteAtespaceAccessPolicyRequest.ProtoReflect.Descriptor instead. func (*DeleteAtespaceAccessPolicyRequest) Descriptor() ([]byte, []int) { - return file_ateapi_proto_rawDescGZIP(), []int{123} + return file_ateapi_proto_rawDescGZIP(), []int{129} } func (x *DeleteAtespaceAccessPolicyRequest) GetAtespace() *ObjectRef { @@ -8873,16 +9185,172 @@ func (x *DeleteAtespaceAccessPolicyRequest) GetOptions() *DeleteOptions { return nil } +// ResourceAnnotation marks a message as an API resource. See +// docs/resource-model.md. +type ResourceAnnotation struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The resource types this resource is a subresource of: another resource's + // message name, or "Global" for the whole Substrate installation. Empty for + // a top-level resource. + Parents []string `protobuf:"bytes,1,rep,name=parents,proto3" json:"parents,omitempty"` + // Whether each parent has at most one instance of this resource. + Singleton bool `protobuf:"varint,2,opt,name=singleton,proto3" json:"singleton,omitempty"` + // Whether a top-level resource belongs to an atespace. Set only on a + // top-level resource: a subresource has its parent's scope. + Scope Scope `protobuf:"varint,3,opt,name=scope,proto3,enum=ateapi.Scope" json:"scope,omitempty"` + // The resource's plural name, as List methods and their response fields + // use it. For example, "AccessPolicies" for AccessPolicy. + Plural string `protobuf:"bytes,4,opt,name=plural,proto3" json:"plural,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ResourceAnnotation) Reset() { + *x = ResourceAnnotation{} + mi := &file_ateapi_proto_msgTypes[130] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ResourceAnnotation) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ResourceAnnotation) ProtoMessage() {} + +func (x *ResourceAnnotation) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[130] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ResourceAnnotation.ProtoReflect.Descriptor instead. +func (*ResourceAnnotation) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{130} +} + +func (x *ResourceAnnotation) GetParents() []string { + if x != nil { + return x.Parents + } + return nil +} + +func (x *ResourceAnnotation) GetSingleton() bool { + if x != nil { + return x.Singleton + } + return false +} + +func (x *ResourceAnnotation) GetScope() Scope { + if x != nil { + return x.Scope + } + return Scope_SCOPE_UNSPECIFIED +} + +func (x *ResourceAnnotation) GetPlural() string { + if x != nil { + return x.Plural + } + return "" +} + +// MethodAnnotation describes an RPC of the API. +type MethodAnnotation struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The message name of the resource the method operates on. For example, + // "Actor" for SuspendActor, or "EgressPolicy" for GetActorEgressPolicy. + Resource string `protobuf:"bytes,1,opt,name=resource,proto3" json:"resource,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *MethodAnnotation) Reset() { + *x = MethodAnnotation{} + mi := &file_ateapi_proto_msgTypes[131] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *MethodAnnotation) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MethodAnnotation) ProtoMessage() {} + +func (x *MethodAnnotation) ProtoReflect() protoreflect.Message { + mi := &file_ateapi_proto_msgTypes[131] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MethodAnnotation.ProtoReflect.Descriptor instead. +func (*MethodAnnotation) Descriptor() ([]byte, []int) { + return file_ateapi_proto_rawDescGZIP(), []int{131} +} + +func (x *MethodAnnotation) GetResource() string { + if x != nil { + return x.Resource + } + return "" +} + +var file_ateapi_proto_extTypes = []protoimpl.ExtensionInfo{ + { + ExtendedType: (*descriptorpb.MessageOptions)(nil), + ExtensionType: (*ResourceAnnotation)(nil), + Field: 50000, + Name: "ateapi.resource", + Tag: "bytes,50000,opt,name=resource", + Filename: "ateapi.proto", + }, + { + ExtendedType: (*descriptorpb.MethodOptions)(nil), + ExtensionType: (*MethodAnnotation)(nil), + Field: 50000, + Name: "ateapi.method", + Tag: "bytes,50000,opt,name=method", + Filename: "ateapi.proto", + }, +} + +// Extension fields to descriptorpb.MessageOptions. +var ( + // optional ateapi.ResourceAnnotation resource = 50000; + E_Resource = &file_ateapi_proto_extTypes[0] +) + +// Extension fields to descriptorpb.MethodOptions. +var ( + // optional ateapi.MethodAnnotation method = 50000; + E_Method = &file_ateapi_proto_extTypes[1] +) + var File_ateapi_proto protoreflect.FileDescriptor const file_ateapi_proto_rawDesc = "" + "\n" + - "\fateapi.proto\x12\x06ateapi\x1a\x1fgoogle/protobuf/timestamp.proto\"3\n" + + "\fateapi.proto\x12\x06ateapi\x1a google/protobuf/descriptor.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"3\n" + "\x0eObjectSnapshot\x12!\n" + "\fsnapshot_uri\x18\x01 \x01(\tR\vsnapshotUri\"x\n" + "\x0fSnapshotStorage\x125\n" + "\x06status\x18\x01 \x01(\x0e2\x1d.ateapi.SnapshotStorageStatusR\x06status\x12.\n" + - "\x06object\x18\x02 \x01(\v2\x16.ateapi.ObjectSnapshotR\x06object\"\xaf\x02\n" + + "\x06object\x18\x02 \x01(\v2\x16.ateapi.ObjectSnapshotR\x06object\"\xf0\x02\n" + "\bSnapshot\x12\x12\n" + "\x04uuid\x18\x01 \x01(\tR\x04uuid\x12\x1e\n" + "\n" + @@ -8892,7 +9360,8 @@ const file_ateapi_proto_rawDesc = "" + "\bfidelity\x18\x04 \x01(\x0e2\x18.ateapi.SnapshotFidelityR\bfidelity\x12,\n" + "\x12actor_template_uid\x18\x05 \x01(\tR\x10actorTemplateUid\x12\x1a\n" + "\blocality\x18\x06 \x01(\tR\blocality\x12B\n" + - "\x10durable_snapshot\x18\a \x01(\v2\x17.ateapi.SnapshotStorageR\x0fdurableSnapshot\"\x90\x01\n" + + "\x10durable_snapshot\x18\a \x01(\v2\x17.ateapi.SnapshotStorageR\x0fdurableSnapshot\x12?\n" + + "\x0fsandbox_runtime\x18\b \x01(\v2\x16.ateapi.SandboxRuntimeR\x0esandboxRuntime\"\x90\x01\n" + "\bSelector\x12D\n" + "\fmatch_labels\x18\x01 \x03(\v2!.ateapi.Selector.MatchLabelsEntryR\vmatchLabels\x1a>\n" + "\x10MatchLabelsEntry\x12\x10\n" + @@ -8906,32 +9375,28 @@ const file_ateapi_proto_rawDesc = "" + "\vcreate_time\x18\x05 \x01(\v2\x1a.google.protobuf.TimestampR\n" + "createTime\x12;\n" + "\vupdate_time\x18\x06 \x01(\v2\x1a.google.protobuf.TimestampR\n" + - "updateTime\"\x9b\x03\n" + + "updateTime\"\x85\x02\n" + "\x0eExternalVolume\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12*\n" + "\x11storage_volume_id\x18\x02 \x01(\tR\x0fstorageVolumeId\x12\x1f\n" + "\vvolume_type\x18\x03 \x01(\tR\n" + - "volumeType\x125\n" + - "\x06status\x18\x04 \x01(\x0e2\x1d.ateapi.ExternalVolume.StatusR\x06status\x12P\n" + + "volumeType\x12P\n" + "\x0evolume_context\x18\x05 \x03(\v2).ateapi.ExternalVolume.VolumeContextEntryR\rvolumeContext\x1a@\n" + "\x12VolumeContextEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"]\n" + - "\x06Status\x12\x16\n" + - "\x12STATUS_UNSPECIFIED\x10\x00\x12\x12\n" + - "\x0eSTATUS_PENDING\x10\x01\x12\x12\n" + - "\x0eSTATUS_CREATED\x10\x02\x12\x13\n" + - "\x0fSTATUS_DELETING\x10\x03\"\x91\x02\n" + + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xa1\x02\n" + "\x05Actor\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x128\n" + "\x0eactor_template\x18\x04 \x01(\v2\x11.ateapi.ObjectRefR\ractorTemplate\x129\n" + "\x0fworker_selector\x18\x05 \x01(\v2\x10.ateapi.SelectorR\x0eworkerSelector\x120\n" + "\n" + "source_tag\x18\x06 \x01(\v2\x11.ateapi.ObjectRefR\tsourceTag\x12+\n" + - "\x06status\x18\a \x01(\v2\x13.ateapi.ActorStatusR\x06status\"n\n" + + "\x06status\x18\a \x01(\v2\x13.ateapi.ActorStatusR\x06status:\x0e\x82\xb5\x18\n" + + "\x18\x01\"\x06Actors\"\x8d\x01\n" + "\fEgressPolicy\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12(\n" + - "\x05rules\x18\x02 \x03(\v2\x12.ateapi.EgressRuleR\x05rules\"\xa0\x01\n" + + "\x05rules\x18\x02 \x03(\v2\x12.ateapi.EgressRuleR\x05rules:\x1d\x82\xb5\x18\x19\n" + + "\x05Actor\x10\x01\"\x0eEgressPolicies\"\xa0\x01\n" + "\n" + "EgressRule\x12$\n" + "\x04http\x18\x01 \x01(\v2\x10.ateapi.HttpRuleR\x04http\x12'\n" + @@ -8976,7 +9441,7 @@ const file_ateapi_proto_rawDesc = "" + "ActorCrash\x12\x18\n" + "\amessage\x18\x01 \x01(\tR\amessage\x129\n" + "\n" + - "crash_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\tcrashTime\"\xb4\x02\n" + + "crash_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\tcrashTime\"\xf5\x02\n" + "\x10WorkerAssignment\x12)\n" + "\x06worker\x18\x06 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\x12)\n" + "\x10worker_namespace\x18\x01 \x01(\tR\x0fworkerNamespace\x12\x1f\n" + @@ -8987,20 +9452,21 @@ const file_ateapi_proto_rawDesc = "" + "\x0eworker_pod_uid\x18\x04 \x01(\tR\fworkerPodUid\x12$\n" + "\x0eworker_pod_ips\x18\x05 \x03(\tR\fworkerPodIps\x12\x1b\n" + "\tnode_name\x18\a \x01(\tR\bnodeName\x12!\n" + - "\fworker_epoch\x18\b \x01(\x03R\vworkerEpoch\"g\n" + + "\fworker_epoch\x18\b \x01(\x03R\vworkerEpoch\x12?\n" + + "\x0fsandbox_runtime\x18\t \x01(\v2\x16.ateapi.SandboxRuntimeR\x0esandboxRuntime\"g\n" + "\tTagStatus\x12,\n" + "\bsnapshot\x18\x01 \x01(\v2\x10.ateapi.SnapshotR\bsnapshot\x12,\n" + - "\x12actor_template_uid\x18\x02 \x01(\tR\x10actorTemplateUid\"\xc4\x01\n" + + "\x12actor_template_uid\x18\x02 \x01(\tR\x10actorTemplateUid\"\xd2\x01\n" + "\x03Tag\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12)\n" + "\x06status\x18\x02 \x01(\v2\x11.ateapi.TagStatusR\x06status\x12&\n" + "\x05scope\x18\x03 \x01(\x0e2\x10.ateapi.TagScopeR\x05scope\x124\n" + - "\fsource_actor\x18\x04 \x01(\v2\x11.ateapi.ObjectRefR\vsourceActor\"@\n" + + "\fsource_actor\x18\x04 \x01(\v2\x11.ateapi.ObjectRefR\vsourceActor:\f\x82\xb5\x18\b\x18\x01\"\x04Tags\"S\n" + "\bAtespace\x124\n" + - "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\";\n" + + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata:\x11\x82\xb5\x18\r\x18\x02\"\tAtespaces\";\n" + "\tObjectRef\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x12\n" + - "\x04name\x18\x02 \x01(\tR\x04name\"\xc2\x03\n" + + "\x04name\x18\x02 \x01(\tR\x04name\"\xda\x03\n" + "\rActorTemplate\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x129\n" + "\x0fworker_selector\x18\x02 \x01(\v2\x10.ateapi.SelectorR\x0eworkerSelector\x121\n" + @@ -9011,7 +9477,7 @@ const file_ateapi_proto_rawDesc = "" + "\x0fsnapshot_config\x18\x05 \x01(\v2\x16.ateapi.SnapshotConfigR\x0esnapshotConfig\x12<\n" + "\x0esandbox_config\x18\x06 \x01(\v2\x15.ateapi.SandboxConfigR\rsandboxConfig\x12/\n" + "\tresources\x18\a \x01(\v2\x11.ateapi.ResourcesR\tresources\x123\n" + - "\x06status\x18\b \x01(\v2\x1b.ateapi.ActorTemplateStatusR\x06status\"3\n" + + "\x06status\x18\b \x01(\v2\x1b.ateapi.ActorTemplateStatusR\x06status:\x16\x82\xb5\x18\x12\x18\x01\"\x0eActorTemplates\"3\n" + "\tResources\x12&\n" + "\x06limits\x18\x01 \x03(\v2\x0e.ateapi.LimitsR\x06limits\"8\n" + "\x06Limits\x12\x12\n" + @@ -9023,9 +9489,9 @@ const file_ateapi_proto_rawDesc = "" + "\rsnapshot_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\fsnapshotTime\x12#\n" + "\rerror_message\x18\x03 \x01(\tR\ferrorMessage\"i\n" + "\x13ActorTemplateStatus\x12R\n" + - "\x16golden_snapshot_status\x18\x01 \x01(\v2\x1c.ateapi.GoldenSnapshotStatusR\x14goldenSnapshotStatus\"k\n" + - "\rSandboxConfig\x129\n" + - "\rsandbox_class\x18\x01 \x01(\x0e2\x14.ateapi.SandboxClassR\fsandboxClass\x12\x1f\n" + + "\x16golden_snapshot_status\x18\x01 \x01(\v2\x1c.ateapi.GoldenSnapshotStatusR\x14goldenSnapshotStatus\"U\n" + + "\rSandboxConfig\x12#\n" + + "\rsandbox_class\x18\x01 \x01(\tR\fsandboxClass\x12\x1f\n" + "\vconfig_name\x18\x02 \x01(\tR\n" + "configName\"\xd8\x01\n" + "\x0eSnapshotConfig\x12)\n" + @@ -9213,13 +9679,21 @@ const file_ateapi_proto_rawDesc = "" + "\x12DrainWorkerRequest\x12)\n" + "\x06worker\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\"=\n" + "\x13DrainWorkerResponse\x12&\n" + + "\x06worker\x18\x01 \x01(\v2\x0e.ateapi.WorkerR\x06worker\"G\n" + + "\x1aMarkWorkerAvailableRequest\x12)\n" + + "\x06worker\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\"E\n" + + "\x1bMarkWorkerAvailableResponse\x12&\n" + + "\x06worker\x18\x01 \x01(\v2\x0e.ateapi.WorkerR\x06worker\"I\n" + + "\x1cMarkWorkerUnavailableRequest\x12)\n" + + "\x06worker\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\"G\n" + + "\x1dMarkWorkerUnavailableResponse\x12&\n" + "\x06worker\x18\x01 \x01(\v2\x0e.ateapi.WorkerR\x06worker\"^\n" + "\x11ListActorsRequest\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12-\n" + "\aoptions\x18\x02 \x01(\v2\x13.ateapi.ListOptionsR\aoptions\"c\n" + "\x12ListActorsResponse\x12%\n" + "\x06actors\x18\x01 \x03(\v2\r.ateapi.ActorR\x06actors\x12&\n" + - "\x0fnext_page_token\x18\x02 \x01(\tR\rnextPageToken\"\xd6\x03\n" + + "\x0fnext_page_token\x18\x02 \x01(\tR\rnextPageToken\"\xe7\x03\n" + "\x06Worker\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12)\n" + "\x10worker_namespace\x18\x02 \x01(\tR\x0fworkerNamespace\x12\x1f\n" + @@ -9236,7 +9710,7 @@ const file_ateapi_proto_rawDesc = "" + "\x06status\x18\v \x01(\v2\x14.ateapi.WorkerStatusR\x06status\x1a9\n" + "\vLabelsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xd6\x02\n" + + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01:\x0f\x82\xb5\x18\v\x18\x02\"\aWorkers\"\xd6\x02\n" + "\fWorkerStatus\x12)\n" + "\x05state\x18\x01 \x01(\x0e2\x13.ateapi.WorkerStateR\x05state\x123\n" + "\bcapacity\x18\x02 \x01(\v2\x17.ateapi.WorkerResourcesR\bcapacity\x125\n" + @@ -9257,13 +9731,14 @@ const file_ateapi_proto_rawDesc = "" + "\x05value\x18\x02 \x01(\tR\x05value\"Z\n" + "\x0fWorkerResources\x12/\n" + "\tresources\x18\x01 \x01(\v2\x11.ateapi.ResourcesR\tresources\x12\x16\n" + - "\x06actors\x18\x02 \x01(\x05R\x06actors\"\xe1\x01\n" + + "\x06actors\x18\x02 \x01(\x05R\x06actors\"\x81\x02\n" + "\x0fActorAssignment\x124\n" + "\bmetadata\x18\x06 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12'\n" + "\x05actor\x18\x02 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" + "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x12/\n" + "\tresources\x18\x05 \x01(\v2\x11.ateapi.ResourcesR\tresources\x12!\n" + - "\fworker_epoch\x18\a \x01(\x03R\vworkerEpoch\"\x81\x02\n" + + "\fworker_epoch\x18\a \x01(\x03R\vworkerEpoch:\x1e\x82\xb5\x18\x1a\n" + + "\x06Worker\"\x10ActorAssignments\"\x81\x02\n" + "\x15RegisterWorkerRequest\x12)\n" + "\x06worker\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\x123\n" + "\bcapacity\x18\x02 \x01(\v2\x17.ateapi.WorkerResourcesR\bcapacity\x12?\n" + @@ -9276,7 +9751,15 @@ const file_ateapi_proto_rawDesc = "" + "\tactor_uid\x18\x02 \x01(\tR\bactorUid\x12>\n" + "\x1bcertificate_signing_request\x18\x03 \x01(\fR\x19certificateSigningRequest\"R\n" + "!MintAteomActorCertificateResponse\x12-\n" + - "\x12actor_certificates\x18\x01 \x03(\fR\x11actorCertificates\"\x8d\x01\n" + + "\x12actor_certificates\x18\x01 \x03(\fR\x11actorCertificates\"~\n" + + "\x18MintAteomActorJwtRequest\x12'\n" + + "\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" + + "\tactor_uid\x18\x02 \x01(\tR\bactorUid\x12\x1c\n" + + "\taudiences\x18\x03 \x03(\tR\taudiences\"z\n" + + "\x19MintAteomActorJwtResponse\x12 \n" + + "\tactor_jwt\x18\x01 \x01(\tB\x03\x80\x01\x01R\bactorJwt\x12;\n" + + "\vexpire_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\n" + + "expireTime\"\x8d\x01\n" + "\x1aRequestActorSuspendRequest\x12)\n" + "\x06worker\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x06worker\x12'\n" + "\x05actor\x18\x02 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" + @@ -9301,10 +9784,12 @@ const file_ateapi_proto_rawDesc = "" + "\bfidelity\x18\x03 \x01(\x0e2\x18.ateapi.SnapshotFidelityR\bfidelity\x12.\n" + "\x06object\x18\x04 \x01(\v2\x16.ateapi.ObjectSnapshotR\x06object\x12F\n" + "\x11local_commit_time\x18\x05 \x01(\v2\x1a.google.protobuf.TimestampR\x0flocalCommitTime\x12N\n" + - "\x15durable_complete_time\x18\x06 \x01(\v2\x1a.google.protobuf.TimestampR\x13durableCompleteTime\"q\n" + + "\x15durable_complete_time\x18\x06 \x01(\v2\x1a.google.protobuf.TimestampR\x13durableCompleteTime\"\x9b\x01\n" + "\fAccessPolicy\x124\n" + "\bmetadata\x18\x01 \x01(\v2\x18.ateapi.ResourceMetadataR\bmetadata\x12+\n" + - "\bbindings\x18\x02 \x03(\v2\x0f.ateapi.BindingR\bbindings\"7\n" + + "\bbindings\x18\x02 \x03(\v2\x0f.ateapi.BindingR\bbindings:(\x82\xb5\x18$\n" + + "\x06Global\n" + + "\bAtespace\x10\x01\"\x0eAccessPolicies\"7\n" + "\aBinding\x12\x12\n" + "\x04role\x18\x01 \x01(\tR\x04role\x12\x18\n" + "\amembers\x18\x02 \x03(\tR\amembers\"2\n" + @@ -9326,7 +9811,14 @@ const file_ateapi_proto_rawDesc = "" + "!DeleteAtespaceAccessPolicyRequest\x12-\n" + "\batespace\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\batespace\x12\x12\n" + "\x04name\x18\x02 \x01(\tR\x04name\x12/\n" + - "\aoptions\x18\x03 \x01(\v2\x15.ateapi.DeleteOptionsR\aoptions*a\n" + + "\aoptions\x18\x03 \x01(\v2\x15.ateapi.DeleteOptionsR\aoptions\"\x89\x01\n" + + "\x12ResourceAnnotation\x12\x18\n" + + "\aparents\x18\x01 \x03(\tR\aparents\x12\x1c\n" + + "\tsingleton\x18\x02 \x01(\bR\tsingleton\x12#\n" + + "\x05scope\x18\x03 \x01(\x0e2\r.ateapi.ScopeR\x05scope\x12\x16\n" + + "\x06plural\x18\x04 \x01(\tR\x06plural\".\n" + + "\x10MethodAnnotation\x12\x1a\n" + + "\bresource\x18\x01 \x01(\tR\bresource*a\n" + "\rSnapshotOwner\x12\x1e\n" + "\x1aSNAPSHOT_OWNER_UNSPECIFIED\x10\x00\x12\x18\n" + "\x14SNAPSHOT_OWNER_ACTOR\x10\x01\x12\x16\n" + @@ -9353,11 +9845,7 @@ const file_ateapi_proto_rawDesc = "" + "\x15ACTOR_STATE_SUSPENDED\x10\x04\x12\x17\n" + "\x13ACTOR_STATE_CRASHED\x10\a\x12\x18\n" + "\x14ACTOR_STATE_DELETING\x10\b\x12\x19\n" + - "\x15ACTOR_STATE_REVERTING\x10\t*b\n" + - "\fSandboxClass\x12\x1d\n" + - "\x19SANDBOX_CLASS_UNSPECIFIED\x10\x00\x12\x18\n" + - "\x14SANDBOX_CLASS_GVISOR\x10\x01\x12\x19\n" + - "\x15SANDBOX_CLASS_MICROVM\x10\x02*\x7f\n" + + "\x15ACTOR_STATE_REVERTING\x10\t*\x7f\n" + "\x12GoldenSnapshotMode\x12$\n" + " GOLDEN_SNAPSHOT_MODE_UNSPECIFIED\x10\x00\x12 \n" + "\x1cGOLDEN_SNAPSHOT_MODE_ENABLED\x10\x01\x12!\n" + @@ -9366,61 +9854,119 @@ const file_ateapi_proto_rawDesc = "" + " ACTOR_METADATA_FIELD_UNSPECIFIED\x10\x00\x12\x1d\n" + "\x19ACTOR_METADATA_FIELD_NAME\x10\x01\x12!\n" + "\x1dACTOR_METADATA_FIELD_ATESPACE\x10\x02\x12\x1c\n" + - "\x18ACTOR_METADATA_FIELD_UID\x10\x03*_\n" + + "\x18ACTOR_METADATA_FIELD_UID\x10\x03*}\n" + "\vWorkerState\x12\x1c\n" + "\x18WORKER_STATE_UNSPECIFIED\x10\x00\x12\x17\n" + "\x13WORKER_STATE_ACTIVE\x10\x01\x12\x19\n" + - "\x15WORKER_STATE_DRAINING\x10\x022\x8b\x19\n" + - "\aControl\x124\n" + - "\bGetActor\x12\x17.ateapi.GetActorRequest\x1a\r.ateapi.Actor\"\x00\x12:\n" + - "\vCreateActor\x12\x1a.ateapi.CreateActorRequest\x1a\r.ateapi.Actor\"\x00\x12:\n" + - "\vUpdateActor\x12\x1a.ateapi.UpdateActorRequest\x1a\r.ateapi.Actor\"\x00\x12K\n" + - "\fSuspendActor\x12\x1b.ateapi.SuspendActorRequest\x1a\x1c.ateapi.SuspendActorResponse\"\x00\x12E\n" + + "\x15WORKER_STATE_DRAINING\x10\x02\x12\x1c\n" + + "\x18WORKER_STATE_UNAVAILABLE\x10\x03*D\n" + + "\x05Scope\x12\x15\n" + + "\x11SCOPE_UNSPECIFIED\x10\x00\x12\x12\n" + + "\x0eSCOPE_ATESPACE\x10\x01\x12\x10\n" + + "\fSCOPE_GLOBAL\x10\x022\xbb\x1f\n" + + "\aControl\x12?\n" + + "\bGetActor\x12\x17.ateapi.GetActorRequest\x1a\r.ateapi.Actor\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12E\n" + + "\vCreateActor\x12\x1a.ateapi.CreateActorRequest\x1a\r.ateapi.Actor\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12E\n" + + "\vUpdateActor\x12\x1a.ateapi.UpdateActorRequest\x1a\r.ateapi.Actor\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12V\n" + + "\fSuspendActor\x12\x1b.ateapi.SuspendActorRequest\x1a\x1c.ateapi.SuspendActorResponse\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12P\n" + + "\n" + + "PauseActor\x12\x19.ateapi.PauseActorRequest\x1a\x1a.ateapi.PauseActorResponse\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12S\n" + + "\vResumeActor\x12\x1a.ateapi.ResumeActorRequest\x1a\x1b.ateapi.ResumeActorResponse\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12S\n" + + "\vRevertActor\x12\x1a.ateapi.RevertActorRequest\x1a\x1b.ateapi.RevertActorResponse\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12E\n" + + "\vDeleteActor\x12\x1a.ateapi.DeleteActorRequest\x1a\r.ateapi.Actor\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12e\n" + + "\x14GetActorEgressPolicy\x12#.ateapi.GetActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fEgressPolicy\x12k\n" + + "\x17CreateActorEgressPolicy\x12&.ateapi.CreateActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fEgressPolicy\x12k\n" + + "\x17UpdateActorEgressPolicy\x12&.ateapi.UpdateActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fEgressPolicy\x12k\n" + + "\x17DeleteActorEgressPolicy\x12&.ateapi.DeleteActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fEgressPolicy\x12V\n" + + "\fMintActorJwt\x12\x1b.ateapi.MintActorJwtRequest\x1a\x1c.ateapi.MintActorJwtResponse\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12n\n" + + "\x14MintActorCertificate\x12#.ateapi.MintActorCertificateRequest\x1a$.ateapi.MintActorCertificateResponse\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12=\n" + + "\tCreateTag\x12\x18.ateapi.CreateTagRequest\x1a\v.ateapi.Tag\"\t\x82\xb5\x18\x05\n" + + "\x03Tag\x127\n" + + "\x06GetTag\x12\x15.ateapi.GetTagRequest\x1a\v.ateapi.Tag\"\t\x82\xb5\x18\x05\n" + + "\x03Tag\x12H\n" + + "\bListTags\x12\x17.ateapi.ListTagsRequest\x1a\x18.ateapi.ListTagsResponse\"\t\x82\xb5\x18\x05\n" + + "\x03Tag\x12=\n" + + "\tUpdateTag\x12\x18.ateapi.UpdateTagRequest\x1a\v.ateapi.Tag\"\t\x82\xb5\x18\x05\n" + + "\x03Tag\x12=\n" + + "\tDeleteTag\x12\x18.ateapi.DeleteTagRequest\x1a\v.ateapi.Tag\"\t\x82\xb5\x18\x05\n" + + "\x03Tag\x12T\n" + + "\vListWorkers\x12\x1a.ateapi.ListWorkersRequest\x1a\x1b.ateapi.ListWorkersResponse\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12C\n" + + "\tGetWorker\x12\x18.ateapi.GetWorkerRequest\x1a\x0e.ateapi.Worker\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12I\n" + + "\fCreateWorker\x12\x1b.ateapi.CreateWorkerRequest\x1a\x0e.ateapi.Worker\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12I\n" + + "\fUpdateWorker\x12\x1b.ateapi.UpdateWorkerRequest\x1a\x0e.ateapi.Worker\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12I\n" + + "\fDeleteWorker\x12\x1b.ateapi.DeleteWorkerRequest\x1a\x0e.ateapi.Worker\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12T\n" + + "\vDrainWorker\x12\x1a.ateapi.DrainWorkerRequest\x1a\x1b.ateapi.DrainWorkerResponse\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12l\n" + + "\x13MarkWorkerAvailable\x12\".ateapi.MarkWorkerAvailableRequest\x1a#.ateapi.MarkWorkerAvailableResponse\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12r\n" + + "\x15MarkWorkerUnavailable\x12$.ateapi.MarkWorkerUnavailableRequest\x1a%.ateapi.MarkWorkerUnavailableResponse\"\f\x82\xb5\x18\b\n" + + "\x06Worker\x12\x8a\x01\n" + + "\x1aListWorkerActorAssignments\x12).ateapi.ListWorkerActorAssignmentsRequest\x1a*.ateapi.ListWorkerActorAssignmentsResponse\"\x15\x82\xb5\x18\x11\n" + + "\x0fActorAssignment\x12P\n" + + "\n" + + "ListActors\x12\x19.ateapi.ListActorsRequest\x1a\x1a.ateapi.ListActorsResponse\"\v\x82\xb5\x18\a\n" + + "\x05Actor\x12Q\n" + + "\x0eCreateAtespace\x12\x1d.ateapi.CreateAtespaceRequest\x1a\x10.ateapi.Atespace\"\x0e\x82\xb5\x18\n" + + "\n" + + "\bAtespace\x12K\n" + + "\vGetAtespace\x12\x1a.ateapi.GetAtespaceRequest\x1a\x10.ateapi.Atespace\"\x0e\x82\xb5\x18\n" + + "\n" + + "\bAtespace\x12\\\n" + + "\rListAtespaces\x12\x1c.ateapi.ListAtespacesRequest\x1a\x1d.ateapi.ListAtespacesResponse\"\x0e\x82\xb5\x18\n" + "\n" + - "PauseActor\x12\x19.ateapi.PauseActorRequest\x1a\x1a.ateapi.PauseActorResponse\"\x00\x12H\n" + - "\vResumeActor\x12\x1a.ateapi.ResumeActorRequest\x1a\x1b.ateapi.ResumeActorResponse\"\x00\x12H\n" + - "\vRevertActor\x12\x1a.ateapi.RevertActorRequest\x1a\x1b.ateapi.RevertActorResponse\"\x00\x12:\n" + - "\vDeleteActor\x12\x1a.ateapi.DeleteActorRequest\x1a\r.ateapi.Actor\"\x00\x12S\n" + - "\x14GetActorEgressPolicy\x12#.ateapi.GetActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x00\x12Y\n" + - "\x17CreateActorEgressPolicy\x12&.ateapi.CreateActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x00\x12Y\n" + - "\x17UpdateActorEgressPolicy\x12&.ateapi.UpdateActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x00\x12Y\n" + - "\x17DeleteActorEgressPolicy\x12&.ateapi.DeleteActorEgressPolicyRequest\x1a\x14.ateapi.EgressPolicy\"\x00\x12K\n" + - "\fMintActorJwt\x12\x1b.ateapi.MintActorJwtRequest\x1a\x1c.ateapi.MintActorJwtResponse\"\x00\x12c\n" + - "\x14MintActorCertificate\x12#.ateapi.MintActorCertificateRequest\x1a$.ateapi.MintActorCertificateResponse\"\x00\x124\n" + - "\tCreateTag\x12\x18.ateapi.CreateTagRequest\x1a\v.ateapi.Tag\"\x00\x12.\n" + - "\x06GetTag\x12\x15.ateapi.GetTagRequest\x1a\v.ateapi.Tag\"\x00\x12?\n" + - "\bListTags\x12\x17.ateapi.ListTagsRequest\x1a\x18.ateapi.ListTagsResponse\"\x00\x124\n" + - "\tUpdateTag\x12\x18.ateapi.UpdateTagRequest\x1a\v.ateapi.Tag\"\x00\x124\n" + - "\tDeleteTag\x12\x18.ateapi.DeleteTagRequest\x1a\v.ateapi.Tag\"\x00\x12H\n" + - "\vListWorkers\x12\x1a.ateapi.ListWorkersRequest\x1a\x1b.ateapi.ListWorkersResponse\"\x00\x127\n" + - "\tGetWorker\x12\x18.ateapi.GetWorkerRequest\x1a\x0e.ateapi.Worker\"\x00\x12=\n" + - "\fCreateWorker\x12\x1b.ateapi.CreateWorkerRequest\x1a\x0e.ateapi.Worker\"\x00\x12=\n" + - "\fUpdateWorker\x12\x1b.ateapi.UpdateWorkerRequest\x1a\x0e.ateapi.Worker\"\x00\x12=\n" + - "\fDeleteWorker\x12\x1b.ateapi.DeleteWorkerRequest\x1a\x0e.ateapi.Worker\"\x00\x12H\n" + - "\vDrainWorker\x12\x1a.ateapi.DrainWorkerRequest\x1a\x1b.ateapi.DrainWorkerResponse\"\x00\x12u\n" + - "\x1aListWorkerActorAssignments\x12).ateapi.ListWorkerActorAssignmentsRequest\x1a*.ateapi.ListWorkerActorAssignmentsResponse\"\x00\x12E\n" + + "\bAtespace\x12Q\n" + + "\x0eDeleteAtespace\x12\x1d.ateapi.DeleteAtespaceRequest\x1a\x10.ateapi.Atespace\"\x0e\x82\xb5\x18\n" + "\n" + - "ListActors\x12\x19.ateapi.ListActorsRequest\x1a\x1a.ateapi.ListActorsResponse\"\x00\x12C\n" + - "\x0eCreateAtespace\x12\x1d.ateapi.CreateAtespaceRequest\x1a\x10.ateapi.Atespace\"\x00\x12=\n" + - "\vGetAtespace\x12\x1a.ateapi.GetAtespaceRequest\x1a\x10.ateapi.Atespace\"\x00\x12N\n" + - "\rListAtespaces\x12\x1c.ateapi.ListAtespacesRequest\x1a\x1d.ateapi.ListAtespacesResponse\"\x00\x12C\n" + - "\x0eDeleteAtespace\x12\x1d.ateapi.DeleteAtespaceRequest\x1a\x10.ateapi.Atespace\"\x00\x12R\n" + - "\x13CreateActorTemplate\x12\".ateapi.CreateActorTemplateRequest\x1a\x15.ateapi.ActorTemplate\"\x00\x12L\n" + - "\x10GetActorTemplate\x12\x1f.ateapi.GetActorTemplateRequest\x1a\x15.ateapi.ActorTemplate\"\x00\x12]\n" + - "\x12ListActorTemplates\x12!.ateapi.ListActorTemplatesRequest\x1a\".ateapi.ListActorTemplatesResponse\"\x00\x12R\n" + - "\x13DeleteActorTemplate\x12\".ateapi.DeleteActorTemplateRequest\x1a\x15.ateapi.ActorTemplate\"\x00\x12U\n" + - "\x15GetGlobalAccessPolicy\x12$.ateapi.GetGlobalAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x00\x12[\n" + - "\x18CreateGlobalAccessPolicy\x12'.ateapi.CreateGlobalAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x00\x12[\n" + - "\x18UpdateGlobalAccessPolicy\x12'.ateapi.UpdateGlobalAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x00\x12Y\n" + - "\x17GetAtespaceAccessPolicy\x12&.ateapi.GetAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x00\x12_\n" + - "\x1aCreateAtespaceAccessPolicy\x12).ateapi.CreateAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x00\x12_\n" + - "\x1aUpdateAtespaceAccessPolicy\x12).ateapi.UpdateAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x00\x12_\n" + - "\x1aDeleteAtespaceAccessPolicy\x12).ateapi.DeleteAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x002\xa1\x03\n" + + "\bAtespace\x12e\n" + + "\x13CreateActorTemplate\x12\".ateapi.CreateActorTemplateRequest\x1a\x15.ateapi.ActorTemplate\"\x13\x82\xb5\x18\x0f\n" + + "\rActorTemplate\x12_\n" + + "\x10GetActorTemplate\x12\x1f.ateapi.GetActorTemplateRequest\x1a\x15.ateapi.ActorTemplate\"\x13\x82\xb5\x18\x0f\n" + + "\rActorTemplate\x12p\n" + + "\x12ListActorTemplates\x12!.ateapi.ListActorTemplatesRequest\x1a\".ateapi.ListActorTemplatesResponse\"\x13\x82\xb5\x18\x0f\n" + + "\rActorTemplate\x12e\n" + + "\x13DeleteActorTemplate\x12\".ateapi.DeleteActorTemplateRequest\x1a\x15.ateapi.ActorTemplate\"\x13\x82\xb5\x18\x0f\n" + + "\rActorTemplate\x12g\n" + + "\x15GetGlobalAccessPolicy\x12$.ateapi.GetGlobalAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fAccessPolicy\x12m\n" + + "\x18CreateGlobalAccessPolicy\x12'.ateapi.CreateGlobalAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fAccessPolicy\x12m\n" + + "\x18UpdateGlobalAccessPolicy\x12'.ateapi.UpdateGlobalAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fAccessPolicy\x12k\n" + + "\x17GetAtespaceAccessPolicy\x12&.ateapi.GetAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fAccessPolicy\x12q\n" + + "\x1aCreateAtespaceAccessPolicy\x12).ateapi.CreateAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fAccessPolicy\x12q\n" + + "\x1aUpdateAtespaceAccessPolicy\x12).ateapi.UpdateAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fAccessPolicy\x12q\n" + + "\x1aDeleteAtespaceAccessPolicy\x12).ateapi.DeleteAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + + "\fAccessPolicy2\xfb\x03\n" + "\rWorkerService\x12O\n" + "\x0eRegisterWorker\x12\x1d.ateapi.RegisterWorkerRequest\x1a\x1e.ateapi.RegisterWorkerResponse\x12p\n" + - "\x19MintAteomActorCertificate\x12(.ateapi.MintAteomActorCertificateRequest\x1a).ateapi.MintAteomActorCertificateResponse\x12^\n" + + "\x19MintAteomActorCertificate\x12(.ateapi.MintAteomActorCertificateRequest\x1a).ateapi.MintAteomActorCertificateResponse\x12X\n" + + "\x11MintAteomActorJwt\x12 .ateapi.MintAteomActorJwtRequest\x1a!.ateapi.MintAteomActorJwtResponse\x12^\n" + "\x13RequestActorSuspend\x12\".ateapi.RequestActorSuspendRequest\x1a#.ateapi.RequestActorSuspendResponse\x12m\n" + - "\x18ReportActorSnapshotState\x12'.ateapi.ReportActorSnapshotStateRequest\x1a(.ateapi.ReportActorSnapshotStateResponseB9Z7github.com/agent-substrate/substrate/pkg/proto/ateapipbb\x06proto3" + "\x18ReportActorSnapshotState\x12'.ateapi.ReportActorSnapshotStateRequest\x1a(.ateapi.ReportActorSnapshotStateResponse:Y\n" + + "\bresource\x12\x1f.google.protobuf.MessageOptions\x18І\x03 \x01(\v2\x1a.ateapi.ResourceAnnotationR\bresource:R\n" + + "\x06method\x12\x1e.google.protobuf.MethodOptions\x18І\x03 \x01(\v2\x18.ateapi.MethodAnnotationR\x06methodB9Z7github.com/agent-substrate/substrate/pkg/proto/ateapipbb\x06proto3" var ( file_ateapi_proto_rawDescOnce sync.Once @@ -9434,419 +9980,445 @@ func file_ateapi_proto_rawDescGZIP() []byte { return file_ateapi_proto_rawDescData } -var file_ateapi_proto_enumTypes = make([]protoimpl.EnumInfo, 10) -var file_ateapi_proto_msgTypes = make([]protoimpl.MessageInfo, 127) +var file_ateapi_proto_enumTypes = make([]protoimpl.EnumInfo, 9) +var file_ateapi_proto_msgTypes = make([]protoimpl.MessageInfo, 135) var file_ateapi_proto_goTypes = []any{ (SnapshotOwner)(0), // 0: ateapi.SnapshotOwner (SnapshotStorageStatus)(0), // 1: ateapi.SnapshotStorageStatus (SnapshotFidelity)(0), // 2: ateapi.SnapshotFidelity (TagScope)(0), // 3: ateapi.TagScope (ActorState)(0), // 4: ateapi.ActorState - (SandboxClass)(0), // 5: ateapi.SandboxClass - (GoldenSnapshotMode)(0), // 6: ateapi.GoldenSnapshotMode - (ActorMetadataField)(0), // 7: ateapi.ActorMetadataField - (WorkerState)(0), // 8: ateapi.WorkerState - (ExternalVolume_Status)(0), // 9: ateapi.ExternalVolume.Status - (*ObjectSnapshot)(nil), // 10: ateapi.ObjectSnapshot - (*SnapshotStorage)(nil), // 11: ateapi.SnapshotStorage - (*Snapshot)(nil), // 12: ateapi.Snapshot - (*Selector)(nil), // 13: ateapi.Selector - (*ResourceMetadata)(nil), // 14: ateapi.ResourceMetadata - (*ExternalVolume)(nil), // 15: ateapi.ExternalVolume - (*Actor)(nil), // 16: ateapi.Actor - (*EgressPolicy)(nil), // 17: ateapi.EgressPolicy - (*EgressRule)(nil), // 18: ateapi.EgressRule - (*HttpRule)(nil), // 19: ateapi.HttpRule - (*HttpsRule)(nil), // 20: ateapi.HttpsRule - (*TlsPassthroughRule)(nil), // 21: ateapi.TlsPassthroughRule - (*Ports)(nil), // 22: ateapi.Ports - (*AllPorts)(nil), // 23: ateapi.AllPorts - (*HttpRuleEffects)(nil), // 24: ateapi.HttpRuleEffects - (*CredentialHeader)(nil), // 25: ateapi.CredentialHeader - (*ActorJwtSource)(nil), // 26: ateapi.ActorJwtSource - (*ActorStatus)(nil), // 27: ateapi.ActorStatus - (*ActorCrash)(nil), // 28: ateapi.ActorCrash - (*WorkerAssignment)(nil), // 29: ateapi.WorkerAssignment - (*TagStatus)(nil), // 30: ateapi.TagStatus - (*Tag)(nil), // 31: ateapi.Tag - (*Atespace)(nil), // 32: ateapi.Atespace - (*ObjectRef)(nil), // 33: ateapi.ObjectRef - (*ActorTemplate)(nil), // 34: ateapi.ActorTemplate - (*Resources)(nil), // 35: ateapi.Resources - (*Limits)(nil), // 36: ateapi.Limits - (*GoldenSnapshotStatus)(nil), // 37: ateapi.GoldenSnapshotStatus - (*ActorTemplateStatus)(nil), // 38: ateapi.ActorTemplateStatus - (*SandboxConfig)(nil), // 39: ateapi.SandboxConfig - (*SnapshotConfig)(nil), // 40: ateapi.SnapshotConfig - (*GoldenSnapshotConfig)(nil), // 41: ateapi.GoldenSnapshotConfig - (*Container)(nil), // 42: ateapi.Container - (*SecurityContext)(nil), // 43: ateapi.SecurityContext - (*Capabilities)(nil), // 44: ateapi.Capabilities - (*EnvVar)(nil), // 45: ateapi.EnvVar - (*WakeupProbe)(nil), // 46: ateapi.WakeupProbe - (*HttpGetAction)(nil), // 47: ateapi.HttpGetAction - (*Volume)(nil), // 48: ateapi.Volume - (*ImageVolumeSource)(nil), // 49: ateapi.ImageVolumeSource - (*DurableDirVolumeSource)(nil), // 50: ateapi.DurableDirVolumeSource - (*ExternalVolumeTemplate)(nil), // 51: ateapi.ExternalVolumeTemplate - (*SystemInfoVolumeSource)(nil), // 52: ateapi.SystemInfoVolumeSource - (*SystemInfoDataSource)(nil), // 53: ateapi.SystemInfoDataSource - (*ActorMetadataDataSource)(nil), // 54: ateapi.ActorMetadataDataSource - (*ActorMetadataItem)(nil), // 55: ateapi.ActorMetadataItem - (*TrustBundleDataSource)(nil), // 56: ateapi.TrustBundleDataSource - (*VolumeMount)(nil), // 57: ateapi.VolumeMount - (*CreateAtespaceRequest)(nil), // 58: ateapi.CreateAtespaceRequest - (*GetAtespaceRequest)(nil), // 59: ateapi.GetAtespaceRequest - (*ListAtespacesRequest)(nil), // 60: ateapi.ListAtespacesRequest - (*ListAtespacesResponse)(nil), // 61: ateapi.ListAtespacesResponse - (*DeleteAtespaceRequest)(nil), // 62: ateapi.DeleteAtespaceRequest - (*CreateActorTemplateRequest)(nil), // 63: ateapi.CreateActorTemplateRequest - (*GetActorTemplateRequest)(nil), // 64: ateapi.GetActorTemplateRequest - (*ListActorTemplatesRequest)(nil), // 65: ateapi.ListActorTemplatesRequest - (*ListActorTemplatesResponse)(nil), // 66: ateapi.ListActorTemplatesResponse - (*DeleteActorTemplateRequest)(nil), // 67: ateapi.DeleteActorTemplateRequest - (*GetActorRequest)(nil), // 68: ateapi.GetActorRequest - (*CreateActorRequest)(nil), // 69: ateapi.CreateActorRequest - (*UpdateActorRequest)(nil), // 70: ateapi.UpdateActorRequest - (*SuspendActorRequest)(nil), // 71: ateapi.SuspendActorRequest - (*SuspendActorResponse)(nil), // 72: ateapi.SuspendActorResponse - (*PauseActorRequest)(nil), // 73: ateapi.PauseActorRequest - (*PauseActorResponse)(nil), // 74: ateapi.PauseActorResponse - (*ResumeActorRequest)(nil), // 75: ateapi.ResumeActorRequest - (*ResumeActorResponse)(nil), // 76: ateapi.ResumeActorResponse - (*RevertActorRequest)(nil), // 77: ateapi.RevertActorRequest - (*RevertActorResponse)(nil), // 78: ateapi.RevertActorResponse - (*DeleteActorRequest)(nil), // 79: ateapi.DeleteActorRequest - (*GetActorEgressPolicyRequest)(nil), // 80: ateapi.GetActorEgressPolicyRequest - (*CreateActorEgressPolicyRequest)(nil), // 81: ateapi.CreateActorEgressPolicyRequest - (*UpdateActorEgressPolicyRequest)(nil), // 82: ateapi.UpdateActorEgressPolicyRequest - (*DeleteActorEgressPolicyRequest)(nil), // 83: ateapi.DeleteActorEgressPolicyRequest - (*GetTagRequest)(nil), // 84: ateapi.GetTagRequest - (*MintActorJwtRequest)(nil), // 85: ateapi.MintActorJwtRequest - (*MintActorJwtResponse)(nil), // 86: ateapi.MintActorJwtResponse - (*MintActorCertificateRequest)(nil), // 87: ateapi.MintActorCertificateRequest - (*MintActorCertificateResponse)(nil), // 88: ateapi.MintActorCertificateResponse - (*ListTagsRequest)(nil), // 89: ateapi.ListTagsRequest - (*ListTagsResponse)(nil), // 90: ateapi.ListTagsResponse - (*CreateTagRequest)(nil), // 91: ateapi.CreateTagRequest - (*UpdateTagRequest)(nil), // 92: ateapi.UpdateTagRequest - (*DeleteTagRequest)(nil), // 93: ateapi.DeleteTagRequest - (*DeleteOptions)(nil), // 94: ateapi.DeleteOptions - (*ListOptions)(nil), // 95: ateapi.ListOptions - (*ListWorkerActorAssignmentsRequest)(nil), // 96: ateapi.ListWorkerActorAssignmentsRequest - (*ListWorkerActorAssignmentsResponse)(nil), // 97: ateapi.ListWorkerActorAssignmentsResponse - (*ListWorkersRequest)(nil), // 98: ateapi.ListWorkersRequest - (*ListWorkersResponse)(nil), // 99: ateapi.ListWorkersResponse - (*GetWorkerRequest)(nil), // 100: ateapi.GetWorkerRequest - (*CreateWorkerRequest)(nil), // 101: ateapi.CreateWorkerRequest - (*UpdateWorkerRequest)(nil), // 102: ateapi.UpdateWorkerRequest - (*DeleteWorkerRequest)(nil), // 103: ateapi.DeleteWorkerRequest - (*DrainWorkerRequest)(nil), // 104: ateapi.DrainWorkerRequest - (*DrainWorkerResponse)(nil), // 105: ateapi.DrainWorkerResponse - (*ListActorsRequest)(nil), // 106: ateapi.ListActorsRequest - (*ListActorsResponse)(nil), // 107: ateapi.ListActorsResponse - (*Worker)(nil), // 108: ateapi.Worker - (*WorkerStatus)(nil), // 109: ateapi.WorkerStatus - (*SandboxRuntime)(nil), // 110: ateapi.SandboxRuntime - (*VersionedSandboxCompat)(nil), // 111: ateapi.VersionedSandboxCompat - (*AttributeEntry)(nil), // 112: ateapi.AttributeEntry - (*WorkerResources)(nil), // 113: ateapi.WorkerResources - (*ActorAssignment)(nil), // 114: ateapi.ActorAssignment - (*RegisterWorkerRequest)(nil), // 115: ateapi.RegisterWorkerRequest - (*RegisterWorkerResponse)(nil), // 116: ateapi.RegisterWorkerResponse - (*MintAteomActorCertificateRequest)(nil), // 117: ateapi.MintAteomActorCertificateRequest - (*MintAteomActorCertificateResponse)(nil), // 118: ateapi.MintAteomActorCertificateResponse - (*RequestActorSuspendRequest)(nil), // 119: ateapi.RequestActorSuspendRequest - (*RequestActorSuspendResponse)(nil), // 120: ateapi.RequestActorSuspendResponse - (*ReportActorSnapshotStateRequest)(nil), // 121: ateapi.ReportActorSnapshotStateRequest - (*ReportActorSnapshotStateResponse)(nil), // 122: ateapi.ReportActorSnapshotStateResponse - (*ActorSnapshotState)(nil), // 123: ateapi.ActorSnapshotState - (*SnapshotReport)(nil), // 124: ateapi.SnapshotReport - (*AccessPolicy)(nil), // 125: ateapi.AccessPolicy - (*Binding)(nil), // 126: ateapi.Binding - (*GetGlobalAccessPolicyRequest)(nil), // 127: ateapi.GetGlobalAccessPolicyRequest - (*CreateGlobalAccessPolicyRequest)(nil), // 128: ateapi.CreateGlobalAccessPolicyRequest - (*UpdateGlobalAccessPolicyRequest)(nil), // 129: ateapi.UpdateGlobalAccessPolicyRequest - (*GetAtespaceAccessPolicyRequest)(nil), // 130: ateapi.GetAtespaceAccessPolicyRequest - (*CreateAtespaceAccessPolicyRequest)(nil), // 131: ateapi.CreateAtespaceAccessPolicyRequest - (*UpdateAtespaceAccessPolicyRequest)(nil), // 132: ateapi.UpdateAtespaceAccessPolicyRequest - (*DeleteAtespaceAccessPolicyRequest)(nil), // 133: ateapi.DeleteAtespaceAccessPolicyRequest - nil, // 134: ateapi.Selector.MatchLabelsEntry - nil, // 135: ateapi.ExternalVolume.VolumeContextEntry - nil, // 136: ateapi.Worker.LabelsEntry - (*timestamppb.Timestamp)(nil), // 137: google.protobuf.Timestamp + (GoldenSnapshotMode)(0), // 5: ateapi.GoldenSnapshotMode + (ActorMetadataField)(0), // 6: ateapi.ActorMetadataField + (WorkerState)(0), // 7: ateapi.WorkerState + (Scope)(0), // 8: ateapi.Scope + (*ObjectSnapshot)(nil), // 9: ateapi.ObjectSnapshot + (*SnapshotStorage)(nil), // 10: ateapi.SnapshotStorage + (*Snapshot)(nil), // 11: ateapi.Snapshot + (*Selector)(nil), // 12: ateapi.Selector + (*ResourceMetadata)(nil), // 13: ateapi.ResourceMetadata + (*ExternalVolume)(nil), // 14: ateapi.ExternalVolume + (*Actor)(nil), // 15: ateapi.Actor + (*EgressPolicy)(nil), // 16: ateapi.EgressPolicy + (*EgressRule)(nil), // 17: ateapi.EgressRule + (*HttpRule)(nil), // 18: ateapi.HttpRule + (*HttpsRule)(nil), // 19: ateapi.HttpsRule + (*TlsPassthroughRule)(nil), // 20: ateapi.TlsPassthroughRule + (*Ports)(nil), // 21: ateapi.Ports + (*AllPorts)(nil), // 22: ateapi.AllPorts + (*HttpRuleEffects)(nil), // 23: ateapi.HttpRuleEffects + (*CredentialHeader)(nil), // 24: ateapi.CredentialHeader + (*ActorJwtSource)(nil), // 25: ateapi.ActorJwtSource + (*ActorStatus)(nil), // 26: ateapi.ActorStatus + (*ActorCrash)(nil), // 27: ateapi.ActorCrash + (*WorkerAssignment)(nil), // 28: ateapi.WorkerAssignment + (*TagStatus)(nil), // 29: ateapi.TagStatus + (*Tag)(nil), // 30: ateapi.Tag + (*Atespace)(nil), // 31: ateapi.Atespace + (*ObjectRef)(nil), // 32: ateapi.ObjectRef + (*ActorTemplate)(nil), // 33: ateapi.ActorTemplate + (*Resources)(nil), // 34: ateapi.Resources + (*Limits)(nil), // 35: ateapi.Limits + (*GoldenSnapshotStatus)(nil), // 36: ateapi.GoldenSnapshotStatus + (*ActorTemplateStatus)(nil), // 37: ateapi.ActorTemplateStatus + (*SandboxConfig)(nil), // 38: ateapi.SandboxConfig + (*SnapshotConfig)(nil), // 39: ateapi.SnapshotConfig + (*GoldenSnapshotConfig)(nil), // 40: ateapi.GoldenSnapshotConfig + (*Container)(nil), // 41: ateapi.Container + (*SecurityContext)(nil), // 42: ateapi.SecurityContext + (*Capabilities)(nil), // 43: ateapi.Capabilities + (*EnvVar)(nil), // 44: ateapi.EnvVar + (*WakeupProbe)(nil), // 45: ateapi.WakeupProbe + (*HttpGetAction)(nil), // 46: ateapi.HttpGetAction + (*Volume)(nil), // 47: ateapi.Volume + (*ImageVolumeSource)(nil), // 48: ateapi.ImageVolumeSource + (*DurableDirVolumeSource)(nil), // 49: ateapi.DurableDirVolumeSource + (*ExternalVolumeTemplate)(nil), // 50: ateapi.ExternalVolumeTemplate + (*SystemInfoVolumeSource)(nil), // 51: ateapi.SystemInfoVolumeSource + (*SystemInfoDataSource)(nil), // 52: ateapi.SystemInfoDataSource + (*ActorMetadataDataSource)(nil), // 53: ateapi.ActorMetadataDataSource + (*ActorMetadataItem)(nil), // 54: ateapi.ActorMetadataItem + (*TrustBundleDataSource)(nil), // 55: ateapi.TrustBundleDataSource + (*VolumeMount)(nil), // 56: ateapi.VolumeMount + (*CreateAtespaceRequest)(nil), // 57: ateapi.CreateAtespaceRequest + (*GetAtespaceRequest)(nil), // 58: ateapi.GetAtespaceRequest + (*ListAtespacesRequest)(nil), // 59: ateapi.ListAtespacesRequest + (*ListAtespacesResponse)(nil), // 60: ateapi.ListAtespacesResponse + (*DeleteAtespaceRequest)(nil), // 61: ateapi.DeleteAtespaceRequest + (*CreateActorTemplateRequest)(nil), // 62: ateapi.CreateActorTemplateRequest + (*GetActorTemplateRequest)(nil), // 63: ateapi.GetActorTemplateRequest + (*ListActorTemplatesRequest)(nil), // 64: ateapi.ListActorTemplatesRequest + (*ListActorTemplatesResponse)(nil), // 65: ateapi.ListActorTemplatesResponse + (*DeleteActorTemplateRequest)(nil), // 66: ateapi.DeleteActorTemplateRequest + (*GetActorRequest)(nil), // 67: ateapi.GetActorRequest + (*CreateActorRequest)(nil), // 68: ateapi.CreateActorRequest + (*UpdateActorRequest)(nil), // 69: ateapi.UpdateActorRequest + (*SuspendActorRequest)(nil), // 70: ateapi.SuspendActorRequest + (*SuspendActorResponse)(nil), // 71: ateapi.SuspendActorResponse + (*PauseActorRequest)(nil), // 72: ateapi.PauseActorRequest + (*PauseActorResponse)(nil), // 73: ateapi.PauseActorResponse + (*ResumeActorRequest)(nil), // 74: ateapi.ResumeActorRequest + (*ResumeActorResponse)(nil), // 75: ateapi.ResumeActorResponse + (*RevertActorRequest)(nil), // 76: ateapi.RevertActorRequest + (*RevertActorResponse)(nil), // 77: ateapi.RevertActorResponse + (*DeleteActorRequest)(nil), // 78: ateapi.DeleteActorRequest + (*GetActorEgressPolicyRequest)(nil), // 79: ateapi.GetActorEgressPolicyRequest + (*CreateActorEgressPolicyRequest)(nil), // 80: ateapi.CreateActorEgressPolicyRequest + (*UpdateActorEgressPolicyRequest)(nil), // 81: ateapi.UpdateActorEgressPolicyRequest + (*DeleteActorEgressPolicyRequest)(nil), // 82: ateapi.DeleteActorEgressPolicyRequest + (*GetTagRequest)(nil), // 83: ateapi.GetTagRequest + (*MintActorJwtRequest)(nil), // 84: ateapi.MintActorJwtRequest + (*MintActorJwtResponse)(nil), // 85: ateapi.MintActorJwtResponse + (*MintActorCertificateRequest)(nil), // 86: ateapi.MintActorCertificateRequest + (*MintActorCertificateResponse)(nil), // 87: ateapi.MintActorCertificateResponse + (*ListTagsRequest)(nil), // 88: ateapi.ListTagsRequest + (*ListTagsResponse)(nil), // 89: ateapi.ListTagsResponse + (*CreateTagRequest)(nil), // 90: ateapi.CreateTagRequest + (*UpdateTagRequest)(nil), // 91: ateapi.UpdateTagRequest + (*DeleteTagRequest)(nil), // 92: ateapi.DeleteTagRequest + (*DeleteOptions)(nil), // 93: ateapi.DeleteOptions + (*ListOptions)(nil), // 94: ateapi.ListOptions + (*ListWorkerActorAssignmentsRequest)(nil), // 95: ateapi.ListWorkerActorAssignmentsRequest + (*ListWorkerActorAssignmentsResponse)(nil), // 96: ateapi.ListWorkerActorAssignmentsResponse + (*ListWorkersRequest)(nil), // 97: ateapi.ListWorkersRequest + (*ListWorkersResponse)(nil), // 98: ateapi.ListWorkersResponse + (*GetWorkerRequest)(nil), // 99: ateapi.GetWorkerRequest + (*CreateWorkerRequest)(nil), // 100: ateapi.CreateWorkerRequest + (*UpdateWorkerRequest)(nil), // 101: ateapi.UpdateWorkerRequest + (*DeleteWorkerRequest)(nil), // 102: ateapi.DeleteWorkerRequest + (*DrainWorkerRequest)(nil), // 103: ateapi.DrainWorkerRequest + (*DrainWorkerResponse)(nil), // 104: ateapi.DrainWorkerResponse + (*MarkWorkerAvailableRequest)(nil), // 105: ateapi.MarkWorkerAvailableRequest + (*MarkWorkerAvailableResponse)(nil), // 106: ateapi.MarkWorkerAvailableResponse + (*MarkWorkerUnavailableRequest)(nil), // 107: ateapi.MarkWorkerUnavailableRequest + (*MarkWorkerUnavailableResponse)(nil), // 108: ateapi.MarkWorkerUnavailableResponse + (*ListActorsRequest)(nil), // 109: ateapi.ListActorsRequest + (*ListActorsResponse)(nil), // 110: ateapi.ListActorsResponse + (*Worker)(nil), // 111: ateapi.Worker + (*WorkerStatus)(nil), // 112: ateapi.WorkerStatus + (*SandboxRuntime)(nil), // 113: ateapi.SandboxRuntime + (*VersionedSandboxCompat)(nil), // 114: ateapi.VersionedSandboxCompat + (*AttributeEntry)(nil), // 115: ateapi.AttributeEntry + (*WorkerResources)(nil), // 116: ateapi.WorkerResources + (*ActorAssignment)(nil), // 117: ateapi.ActorAssignment + (*RegisterWorkerRequest)(nil), // 118: ateapi.RegisterWorkerRequest + (*RegisterWorkerResponse)(nil), // 119: ateapi.RegisterWorkerResponse + (*MintAteomActorCertificateRequest)(nil), // 120: ateapi.MintAteomActorCertificateRequest + (*MintAteomActorCertificateResponse)(nil), // 121: ateapi.MintAteomActorCertificateResponse + (*MintAteomActorJwtRequest)(nil), // 122: ateapi.MintAteomActorJwtRequest + (*MintAteomActorJwtResponse)(nil), // 123: ateapi.MintAteomActorJwtResponse + (*RequestActorSuspendRequest)(nil), // 124: ateapi.RequestActorSuspendRequest + (*RequestActorSuspendResponse)(nil), // 125: ateapi.RequestActorSuspendResponse + (*ReportActorSnapshotStateRequest)(nil), // 126: ateapi.ReportActorSnapshotStateRequest + (*ReportActorSnapshotStateResponse)(nil), // 127: ateapi.ReportActorSnapshotStateResponse + (*ActorSnapshotState)(nil), // 128: ateapi.ActorSnapshotState + (*SnapshotReport)(nil), // 129: ateapi.SnapshotReport + (*AccessPolicy)(nil), // 130: ateapi.AccessPolicy + (*Binding)(nil), // 131: ateapi.Binding + (*GetGlobalAccessPolicyRequest)(nil), // 132: ateapi.GetGlobalAccessPolicyRequest + (*CreateGlobalAccessPolicyRequest)(nil), // 133: ateapi.CreateGlobalAccessPolicyRequest + (*UpdateGlobalAccessPolicyRequest)(nil), // 134: ateapi.UpdateGlobalAccessPolicyRequest + (*GetAtespaceAccessPolicyRequest)(nil), // 135: ateapi.GetAtespaceAccessPolicyRequest + (*CreateAtespaceAccessPolicyRequest)(nil), // 136: ateapi.CreateAtespaceAccessPolicyRequest + (*UpdateAtespaceAccessPolicyRequest)(nil), // 137: ateapi.UpdateAtespaceAccessPolicyRequest + (*DeleteAtespaceAccessPolicyRequest)(nil), // 138: ateapi.DeleteAtespaceAccessPolicyRequest + (*ResourceAnnotation)(nil), // 139: ateapi.ResourceAnnotation + (*MethodAnnotation)(nil), // 140: ateapi.MethodAnnotation + nil, // 141: ateapi.Selector.MatchLabelsEntry + nil, // 142: ateapi.ExternalVolume.VolumeContextEntry + nil, // 143: ateapi.Worker.LabelsEntry + (*timestamppb.Timestamp)(nil), // 144: google.protobuf.Timestamp + (*descriptorpb.MessageOptions)(nil), // 145: google.protobuf.MessageOptions + (*descriptorpb.MethodOptions)(nil), // 146: google.protobuf.MethodOptions } var file_ateapi_proto_depIdxs = []int32{ 1, // 0: ateapi.SnapshotStorage.status:type_name -> ateapi.SnapshotStorageStatus - 10, // 1: ateapi.SnapshotStorage.object:type_name -> ateapi.ObjectSnapshot + 9, // 1: ateapi.SnapshotStorage.object:type_name -> ateapi.ObjectSnapshot 0, // 2: ateapi.Snapshot.owner:type_name -> ateapi.SnapshotOwner 2, // 3: ateapi.Snapshot.fidelity:type_name -> ateapi.SnapshotFidelity - 11, // 4: ateapi.Snapshot.durable_snapshot:type_name -> ateapi.SnapshotStorage - 134, // 5: ateapi.Selector.match_labels:type_name -> ateapi.Selector.MatchLabelsEntry - 137, // 6: ateapi.ResourceMetadata.create_time:type_name -> google.protobuf.Timestamp - 137, // 7: ateapi.ResourceMetadata.update_time:type_name -> google.protobuf.Timestamp - 9, // 8: ateapi.ExternalVolume.status:type_name -> ateapi.ExternalVolume.Status - 135, // 9: ateapi.ExternalVolume.volume_context:type_name -> ateapi.ExternalVolume.VolumeContextEntry - 14, // 10: ateapi.Actor.metadata:type_name -> ateapi.ResourceMetadata - 33, // 11: ateapi.Actor.actor_template:type_name -> ateapi.ObjectRef - 13, // 12: ateapi.Actor.worker_selector:type_name -> ateapi.Selector - 33, // 13: ateapi.Actor.source_tag:type_name -> ateapi.ObjectRef - 27, // 14: ateapi.Actor.status:type_name -> ateapi.ActorStatus - 14, // 15: ateapi.EgressPolicy.metadata:type_name -> ateapi.ResourceMetadata - 18, // 16: ateapi.EgressPolicy.rules:type_name -> ateapi.EgressRule - 19, // 17: ateapi.EgressRule.http:type_name -> ateapi.HttpRule - 20, // 18: ateapi.EgressRule.https:type_name -> ateapi.HttpsRule - 21, // 19: ateapi.EgressRule.tls_passthrough:type_name -> ateapi.TlsPassthroughRule - 22, // 20: ateapi.HttpRule.ports:type_name -> ateapi.Ports - 24, // 21: ateapi.HttpRule.effects:type_name -> ateapi.HttpRuleEffects - 22, // 22: ateapi.HttpsRule.ports:type_name -> ateapi.Ports - 24, // 23: ateapi.HttpsRule.effects:type_name -> ateapi.HttpRuleEffects - 22, // 24: ateapi.TlsPassthroughRule.ports:type_name -> ateapi.Ports - 23, // 25: ateapi.Ports.all:type_name -> ateapi.AllPorts - 25, // 26: ateapi.HttpRuleEffects.replace_headers:type_name -> ateapi.CredentialHeader - 26, // 27: ateapi.CredentialHeader.actor_jwt:type_name -> ateapi.ActorJwtSource + 10, // 4: ateapi.Snapshot.durable_snapshot:type_name -> ateapi.SnapshotStorage + 113, // 5: ateapi.Snapshot.sandbox_runtime:type_name -> ateapi.SandboxRuntime + 141, // 6: ateapi.Selector.match_labels:type_name -> ateapi.Selector.MatchLabelsEntry + 144, // 7: ateapi.ResourceMetadata.create_time:type_name -> google.protobuf.Timestamp + 144, // 8: ateapi.ResourceMetadata.update_time:type_name -> google.protobuf.Timestamp + 142, // 9: ateapi.ExternalVolume.volume_context:type_name -> ateapi.ExternalVolume.VolumeContextEntry + 13, // 10: ateapi.Actor.metadata:type_name -> ateapi.ResourceMetadata + 32, // 11: ateapi.Actor.actor_template:type_name -> ateapi.ObjectRef + 12, // 12: ateapi.Actor.worker_selector:type_name -> ateapi.Selector + 32, // 13: ateapi.Actor.source_tag:type_name -> ateapi.ObjectRef + 26, // 14: ateapi.Actor.status:type_name -> ateapi.ActorStatus + 13, // 15: ateapi.EgressPolicy.metadata:type_name -> ateapi.ResourceMetadata + 17, // 16: ateapi.EgressPolicy.rules:type_name -> ateapi.EgressRule + 18, // 17: ateapi.EgressRule.http:type_name -> ateapi.HttpRule + 19, // 18: ateapi.EgressRule.https:type_name -> ateapi.HttpsRule + 20, // 19: ateapi.EgressRule.tls_passthrough:type_name -> ateapi.TlsPassthroughRule + 21, // 20: ateapi.HttpRule.ports:type_name -> ateapi.Ports + 23, // 21: ateapi.HttpRule.effects:type_name -> ateapi.HttpRuleEffects + 21, // 22: ateapi.HttpsRule.ports:type_name -> ateapi.Ports + 23, // 23: ateapi.HttpsRule.effects:type_name -> ateapi.HttpRuleEffects + 21, // 24: ateapi.TlsPassthroughRule.ports:type_name -> ateapi.Ports + 22, // 25: ateapi.Ports.all:type_name -> ateapi.AllPorts + 24, // 26: ateapi.HttpRuleEffects.replace_headers:type_name -> ateapi.CredentialHeader + 25, // 27: ateapi.CredentialHeader.actor_jwt:type_name -> ateapi.ActorJwtSource 4, // 28: ateapi.ActorStatus.state:type_name -> ateapi.ActorState - 29, // 29: ateapi.ActorStatus.worker_assignment:type_name -> ateapi.WorkerAssignment - 15, // 30: ateapi.ActorStatus.external_volumes:type_name -> ateapi.ExternalVolume - 28, // 31: ateapi.ActorStatus.crash:type_name -> ateapi.ActorCrash - 12, // 32: ateapi.ActorStatus.snapshots:type_name -> ateapi.Snapshot - 137, // 33: ateapi.ActorCrash.crash_time:type_name -> google.protobuf.Timestamp - 33, // 34: ateapi.WorkerAssignment.worker:type_name -> ateapi.ObjectRef - 12, // 35: ateapi.TagStatus.snapshot:type_name -> ateapi.Snapshot - 14, // 36: ateapi.Tag.metadata:type_name -> ateapi.ResourceMetadata - 30, // 37: ateapi.Tag.status:type_name -> ateapi.TagStatus - 3, // 38: ateapi.Tag.scope:type_name -> ateapi.TagScope - 33, // 39: ateapi.Tag.source_actor:type_name -> ateapi.ObjectRef - 14, // 40: ateapi.Atespace.metadata:type_name -> ateapi.ResourceMetadata - 14, // 41: ateapi.ActorTemplate.metadata:type_name -> ateapi.ResourceMetadata - 13, // 42: ateapi.ActorTemplate.worker_selector:type_name -> ateapi.Selector - 42, // 43: ateapi.ActorTemplate.containers:type_name -> ateapi.Container - 48, // 44: ateapi.ActorTemplate.volumes:type_name -> ateapi.Volume - 40, // 45: ateapi.ActorTemplate.snapshot_config:type_name -> ateapi.SnapshotConfig - 39, // 46: ateapi.ActorTemplate.sandbox_config:type_name -> ateapi.SandboxConfig - 35, // 47: ateapi.ActorTemplate.resources:type_name -> ateapi.Resources - 38, // 48: ateapi.ActorTemplate.status:type_name -> ateapi.ActorTemplateStatus - 36, // 49: ateapi.Resources.limits:type_name -> ateapi.Limits - 33, // 50: ateapi.GoldenSnapshotStatus.golden_tag:type_name -> ateapi.ObjectRef - 137, // 51: ateapi.GoldenSnapshotStatus.snapshot_time:type_name -> google.protobuf.Timestamp - 37, // 52: ateapi.ActorTemplateStatus.golden_snapshot_status:type_name -> ateapi.GoldenSnapshotStatus - 5, // 53: ateapi.SandboxConfig.sandbox_class:type_name -> ateapi.SandboxClass + 28, // 29: ateapi.ActorStatus.worker_assignment:type_name -> ateapi.WorkerAssignment + 14, // 30: ateapi.ActorStatus.external_volumes:type_name -> ateapi.ExternalVolume + 27, // 31: ateapi.ActorStatus.crash:type_name -> ateapi.ActorCrash + 11, // 32: ateapi.ActorStatus.snapshots:type_name -> ateapi.Snapshot + 144, // 33: ateapi.ActorCrash.crash_time:type_name -> google.protobuf.Timestamp + 32, // 34: ateapi.WorkerAssignment.worker:type_name -> ateapi.ObjectRef + 113, // 35: ateapi.WorkerAssignment.sandbox_runtime:type_name -> ateapi.SandboxRuntime + 11, // 36: ateapi.TagStatus.snapshot:type_name -> ateapi.Snapshot + 13, // 37: ateapi.Tag.metadata:type_name -> ateapi.ResourceMetadata + 29, // 38: ateapi.Tag.status:type_name -> ateapi.TagStatus + 3, // 39: ateapi.Tag.scope:type_name -> ateapi.TagScope + 32, // 40: ateapi.Tag.source_actor:type_name -> ateapi.ObjectRef + 13, // 41: ateapi.Atespace.metadata:type_name -> ateapi.ResourceMetadata + 13, // 42: ateapi.ActorTemplate.metadata:type_name -> ateapi.ResourceMetadata + 12, // 43: ateapi.ActorTemplate.worker_selector:type_name -> ateapi.Selector + 41, // 44: ateapi.ActorTemplate.containers:type_name -> ateapi.Container + 47, // 45: ateapi.ActorTemplate.volumes:type_name -> ateapi.Volume + 39, // 46: ateapi.ActorTemplate.snapshot_config:type_name -> ateapi.SnapshotConfig + 38, // 47: ateapi.ActorTemplate.sandbox_config:type_name -> ateapi.SandboxConfig + 34, // 48: ateapi.ActorTemplate.resources:type_name -> ateapi.Resources + 37, // 49: ateapi.ActorTemplate.status:type_name -> ateapi.ActorTemplateStatus + 35, // 50: ateapi.Resources.limits:type_name -> ateapi.Limits + 32, // 51: ateapi.GoldenSnapshotStatus.golden_tag:type_name -> ateapi.ObjectRef + 144, // 52: ateapi.GoldenSnapshotStatus.snapshot_time:type_name -> google.protobuf.Timestamp + 36, // 53: ateapi.ActorTemplateStatus.golden_snapshot_status:type_name -> ateapi.GoldenSnapshotStatus 2, // 54: ateapi.SnapshotConfig.preferred_fidelity:type_name -> ateapi.SnapshotFidelity - 41, // 55: ateapi.SnapshotConfig.golden_snapshot_config:type_name -> ateapi.GoldenSnapshotConfig - 6, // 56: ateapi.GoldenSnapshotConfig.mode:type_name -> ateapi.GoldenSnapshotMode - 45, // 57: ateapi.Container.env:type_name -> ateapi.EnvVar - 46, // 58: ateapi.Container.wakeup_probe:type_name -> ateapi.WakeupProbe - 57, // 59: ateapi.Container.volume_mounts:type_name -> ateapi.VolumeMount - 43, // 60: ateapi.Container.security_context:type_name -> ateapi.SecurityContext - 35, // 61: ateapi.Container.resources:type_name -> ateapi.Resources - 44, // 62: ateapi.SecurityContext.capabilities:type_name -> ateapi.Capabilities - 47, // 63: ateapi.WakeupProbe.http_get:type_name -> ateapi.HttpGetAction - 50, // 64: ateapi.Volume.durable_dir:type_name -> ateapi.DurableDirVolumeSource - 51, // 65: ateapi.Volume.external_volume_template:type_name -> ateapi.ExternalVolumeTemplate - 52, // 66: ateapi.Volume.system_info:type_name -> ateapi.SystemInfoVolumeSource - 49, // 67: ateapi.Volume.image:type_name -> ateapi.ImageVolumeSource - 53, // 68: ateapi.SystemInfoVolumeSource.data_sources:type_name -> ateapi.SystemInfoDataSource - 54, // 69: ateapi.SystemInfoDataSource.actor_metadata:type_name -> ateapi.ActorMetadataDataSource - 56, // 70: ateapi.SystemInfoDataSource.trust_bundle:type_name -> ateapi.TrustBundleDataSource - 55, // 71: ateapi.ActorMetadataDataSource.items:type_name -> ateapi.ActorMetadataItem - 7, // 72: ateapi.ActorMetadataItem.field:type_name -> ateapi.ActorMetadataField - 32, // 73: ateapi.CreateAtespaceRequest.atespace:type_name -> ateapi.Atespace - 33, // 74: ateapi.GetAtespaceRequest.atespace:type_name -> ateapi.ObjectRef - 95, // 75: ateapi.ListAtespacesRequest.options:type_name -> ateapi.ListOptions - 32, // 76: ateapi.ListAtespacesResponse.atespaces:type_name -> ateapi.Atespace - 33, // 77: ateapi.DeleteAtespaceRequest.atespace:type_name -> ateapi.ObjectRef - 94, // 78: ateapi.DeleteAtespaceRequest.options:type_name -> ateapi.DeleteOptions - 34, // 79: ateapi.CreateActorTemplateRequest.actor_template:type_name -> ateapi.ActorTemplate - 33, // 80: ateapi.GetActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef - 95, // 81: ateapi.ListActorTemplatesRequest.options:type_name -> ateapi.ListOptions - 34, // 82: ateapi.ListActorTemplatesResponse.actor_templates:type_name -> ateapi.ActorTemplate - 33, // 83: ateapi.DeleteActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef - 94, // 84: ateapi.DeleteActorTemplateRequest.options:type_name -> ateapi.DeleteOptions - 33, // 85: ateapi.GetActorRequest.actor:type_name -> ateapi.ObjectRef - 16, // 86: ateapi.CreateActorRequest.actor:type_name -> ateapi.Actor - 16, // 87: ateapi.UpdateActorRequest.actor:type_name -> ateapi.Actor - 33, // 88: ateapi.SuspendActorRequest.actor:type_name -> ateapi.ObjectRef - 16, // 89: ateapi.SuspendActorResponse.actor:type_name -> ateapi.Actor - 33, // 90: ateapi.PauseActorRequest.actor:type_name -> ateapi.ObjectRef - 16, // 91: ateapi.PauseActorResponse.actor:type_name -> ateapi.Actor - 33, // 92: ateapi.ResumeActorRequest.actor:type_name -> ateapi.ObjectRef - 16, // 93: ateapi.ResumeActorResponse.actor:type_name -> ateapi.Actor - 33, // 94: ateapi.RevertActorRequest.actor:type_name -> ateapi.ObjectRef - 16, // 95: ateapi.RevertActorResponse.actor:type_name -> ateapi.Actor - 33, // 96: ateapi.DeleteActorRequest.actor:type_name -> ateapi.ObjectRef - 94, // 97: ateapi.DeleteActorRequest.options:type_name -> ateapi.DeleteOptions - 33, // 98: ateapi.GetActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef - 33, // 99: ateapi.CreateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef - 17, // 100: ateapi.CreateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy - 33, // 101: ateapi.UpdateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef - 17, // 102: ateapi.UpdateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy - 33, // 103: ateapi.DeleteActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef - 94, // 104: ateapi.DeleteActorEgressPolicyRequest.options:type_name -> ateapi.DeleteOptions - 33, // 105: ateapi.GetTagRequest.tag:type_name -> ateapi.ObjectRef - 33, // 106: ateapi.MintActorJwtRequest.actor:type_name -> ateapi.ObjectRef - 137, // 107: ateapi.MintActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp - 33, // 108: ateapi.MintActorCertificateRequest.actor:type_name -> ateapi.ObjectRef - 95, // 109: ateapi.ListTagsRequest.options:type_name -> ateapi.ListOptions - 31, // 110: ateapi.ListTagsResponse.tags:type_name -> ateapi.Tag - 31, // 111: ateapi.CreateTagRequest.tag:type_name -> ateapi.Tag - 31, // 112: ateapi.UpdateTagRequest.tag:type_name -> ateapi.Tag - 33, // 113: ateapi.DeleteTagRequest.tag:type_name -> ateapi.ObjectRef - 94, // 114: ateapi.DeleteTagRequest.options:type_name -> ateapi.DeleteOptions - 33, // 115: ateapi.ListWorkerActorAssignmentsRequest.worker:type_name -> ateapi.ObjectRef - 95, // 116: ateapi.ListWorkerActorAssignmentsRequest.options:type_name -> ateapi.ListOptions - 114, // 117: ateapi.ListWorkerActorAssignmentsResponse.actor_assignments:type_name -> ateapi.ActorAssignment - 95, // 118: ateapi.ListWorkersRequest.options:type_name -> ateapi.ListOptions - 108, // 119: ateapi.ListWorkersResponse.workers:type_name -> ateapi.Worker - 33, // 120: ateapi.GetWorkerRequest.worker:type_name -> ateapi.ObjectRef - 108, // 121: ateapi.CreateWorkerRequest.worker:type_name -> ateapi.Worker - 108, // 122: ateapi.UpdateWorkerRequest.worker:type_name -> ateapi.Worker - 33, // 123: ateapi.DeleteWorkerRequest.worker:type_name -> ateapi.ObjectRef - 94, // 124: ateapi.DeleteWorkerRequest.options:type_name -> ateapi.DeleteOptions - 33, // 125: ateapi.DrainWorkerRequest.worker:type_name -> ateapi.ObjectRef - 108, // 126: ateapi.DrainWorkerResponse.worker:type_name -> ateapi.Worker - 95, // 127: ateapi.ListActorsRequest.options:type_name -> ateapi.ListOptions - 16, // 128: ateapi.ListActorsResponse.actors:type_name -> ateapi.Actor - 14, // 129: ateapi.Worker.metadata:type_name -> ateapi.ResourceMetadata - 136, // 130: ateapi.Worker.labels:type_name -> ateapi.Worker.LabelsEntry - 109, // 131: ateapi.Worker.status:type_name -> ateapi.WorkerStatus - 8, // 132: ateapi.WorkerStatus.state:type_name -> ateapi.WorkerState - 113, // 133: ateapi.WorkerStatus.capacity:type_name -> ateapi.WorkerResources - 113, // 134: ateapi.WorkerStatus.allocated:type_name -> ateapi.WorkerResources - 110, // 135: ateapi.WorkerStatus.default_runtime:type_name -> ateapi.SandboxRuntime - 110, // 136: ateapi.WorkerStatus.restorable_runtimes:type_name -> ateapi.SandboxRuntime - 111, // 137: ateapi.SandboxRuntime.version:type_name -> ateapi.VersionedSandboxCompat - 112, // 138: ateapi.VersionedSandboxCompat.attributes:type_name -> ateapi.AttributeEntry - 35, // 139: ateapi.WorkerResources.resources:type_name -> ateapi.Resources - 14, // 140: ateapi.ActorAssignment.metadata:type_name -> ateapi.ResourceMetadata - 33, // 141: ateapi.ActorAssignment.actor:type_name -> ateapi.ObjectRef - 35, // 142: ateapi.ActorAssignment.resources:type_name -> ateapi.Resources - 33, // 143: ateapi.RegisterWorkerRequest.worker:type_name -> ateapi.ObjectRef - 113, // 144: ateapi.RegisterWorkerRequest.capacity:type_name -> ateapi.WorkerResources - 110, // 145: ateapi.RegisterWorkerRequest.default_runtime:type_name -> ateapi.SandboxRuntime - 110, // 146: ateapi.RegisterWorkerRequest.restorable_runtimes:type_name -> ateapi.SandboxRuntime - 108, // 147: ateapi.RegisterWorkerResponse.worker:type_name -> ateapi.Worker - 33, // 148: ateapi.MintAteomActorCertificateRequest.actor:type_name -> ateapi.ObjectRef - 33, // 149: ateapi.RequestActorSuspendRequest.worker:type_name -> ateapi.ObjectRef - 33, // 150: ateapi.RequestActorSuspendRequest.actor:type_name -> ateapi.ObjectRef - 16, // 151: ateapi.RequestActorSuspendResponse.actor:type_name -> ateapi.Actor - 33, // 152: ateapi.ReportActorSnapshotStateRequest.actor:type_name -> ateapi.ObjectRef - 123, // 153: ateapi.ReportActorSnapshotStateRequest.state:type_name -> ateapi.ActorSnapshotState - 124, // 154: ateapi.ActorSnapshotState.latest_local:type_name -> ateapi.SnapshotReport - 124, // 155: ateapi.ActorSnapshotState.uploading:type_name -> ateapi.SnapshotReport - 124, // 156: ateapi.ActorSnapshotState.latest_durable:type_name -> ateapi.SnapshotReport - 137, // 157: ateapi.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp - 2, // 158: ateapi.SnapshotReport.fidelity:type_name -> ateapi.SnapshotFidelity - 10, // 159: ateapi.SnapshotReport.object:type_name -> ateapi.ObjectSnapshot - 137, // 160: ateapi.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp - 137, // 161: ateapi.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp - 14, // 162: ateapi.AccessPolicy.metadata:type_name -> ateapi.ResourceMetadata - 126, // 163: ateapi.AccessPolicy.bindings:type_name -> ateapi.Binding - 125, // 164: ateapi.CreateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 125, // 165: ateapi.UpdateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 33, // 166: ateapi.GetAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 33, // 167: ateapi.CreateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 125, // 168: ateapi.CreateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 33, // 169: ateapi.UpdateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 125, // 170: ateapi.UpdateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy - 33, // 171: ateapi.DeleteAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef - 94, // 172: ateapi.DeleteAtespaceAccessPolicyRequest.options:type_name -> ateapi.DeleteOptions - 68, // 173: ateapi.Control.GetActor:input_type -> ateapi.GetActorRequest - 69, // 174: ateapi.Control.CreateActor:input_type -> ateapi.CreateActorRequest - 70, // 175: ateapi.Control.UpdateActor:input_type -> ateapi.UpdateActorRequest - 71, // 176: ateapi.Control.SuspendActor:input_type -> ateapi.SuspendActorRequest - 73, // 177: ateapi.Control.PauseActor:input_type -> ateapi.PauseActorRequest - 75, // 178: ateapi.Control.ResumeActor:input_type -> ateapi.ResumeActorRequest - 77, // 179: ateapi.Control.RevertActor:input_type -> ateapi.RevertActorRequest - 79, // 180: ateapi.Control.DeleteActor:input_type -> ateapi.DeleteActorRequest - 80, // 181: ateapi.Control.GetActorEgressPolicy:input_type -> ateapi.GetActorEgressPolicyRequest - 81, // 182: ateapi.Control.CreateActorEgressPolicy:input_type -> ateapi.CreateActorEgressPolicyRequest - 82, // 183: ateapi.Control.UpdateActorEgressPolicy:input_type -> ateapi.UpdateActorEgressPolicyRequest - 83, // 184: ateapi.Control.DeleteActorEgressPolicy:input_type -> ateapi.DeleteActorEgressPolicyRequest - 85, // 185: ateapi.Control.MintActorJwt:input_type -> ateapi.MintActorJwtRequest - 87, // 186: ateapi.Control.MintActorCertificate:input_type -> ateapi.MintActorCertificateRequest - 91, // 187: ateapi.Control.CreateTag:input_type -> ateapi.CreateTagRequest - 84, // 188: ateapi.Control.GetTag:input_type -> ateapi.GetTagRequest - 89, // 189: ateapi.Control.ListTags:input_type -> ateapi.ListTagsRequest - 92, // 190: ateapi.Control.UpdateTag:input_type -> ateapi.UpdateTagRequest - 93, // 191: ateapi.Control.DeleteTag:input_type -> ateapi.DeleteTagRequest - 98, // 192: ateapi.Control.ListWorkers:input_type -> ateapi.ListWorkersRequest - 100, // 193: ateapi.Control.GetWorker:input_type -> ateapi.GetWorkerRequest - 101, // 194: ateapi.Control.CreateWorker:input_type -> ateapi.CreateWorkerRequest - 102, // 195: ateapi.Control.UpdateWorker:input_type -> ateapi.UpdateWorkerRequest - 103, // 196: ateapi.Control.DeleteWorker:input_type -> ateapi.DeleteWorkerRequest - 104, // 197: ateapi.Control.DrainWorker:input_type -> ateapi.DrainWorkerRequest - 96, // 198: ateapi.Control.ListWorkerActorAssignments:input_type -> ateapi.ListWorkerActorAssignmentsRequest - 106, // 199: ateapi.Control.ListActors:input_type -> ateapi.ListActorsRequest - 58, // 200: ateapi.Control.CreateAtespace:input_type -> ateapi.CreateAtespaceRequest - 59, // 201: ateapi.Control.GetAtespace:input_type -> ateapi.GetAtespaceRequest - 60, // 202: ateapi.Control.ListAtespaces:input_type -> ateapi.ListAtespacesRequest - 62, // 203: ateapi.Control.DeleteAtespace:input_type -> ateapi.DeleteAtespaceRequest - 63, // 204: ateapi.Control.CreateActorTemplate:input_type -> ateapi.CreateActorTemplateRequest - 64, // 205: ateapi.Control.GetActorTemplate:input_type -> ateapi.GetActorTemplateRequest - 65, // 206: ateapi.Control.ListActorTemplates:input_type -> ateapi.ListActorTemplatesRequest - 67, // 207: ateapi.Control.DeleteActorTemplate:input_type -> ateapi.DeleteActorTemplateRequest - 127, // 208: ateapi.Control.GetGlobalAccessPolicy:input_type -> ateapi.GetGlobalAccessPolicyRequest - 128, // 209: ateapi.Control.CreateGlobalAccessPolicy:input_type -> ateapi.CreateGlobalAccessPolicyRequest - 129, // 210: ateapi.Control.UpdateGlobalAccessPolicy:input_type -> ateapi.UpdateGlobalAccessPolicyRequest - 130, // 211: ateapi.Control.GetAtespaceAccessPolicy:input_type -> ateapi.GetAtespaceAccessPolicyRequest - 131, // 212: ateapi.Control.CreateAtespaceAccessPolicy:input_type -> ateapi.CreateAtespaceAccessPolicyRequest - 132, // 213: ateapi.Control.UpdateAtespaceAccessPolicy:input_type -> ateapi.UpdateAtespaceAccessPolicyRequest - 133, // 214: ateapi.Control.DeleteAtespaceAccessPolicy:input_type -> ateapi.DeleteAtespaceAccessPolicyRequest - 115, // 215: ateapi.WorkerService.RegisterWorker:input_type -> ateapi.RegisterWorkerRequest - 117, // 216: ateapi.WorkerService.MintAteomActorCertificate:input_type -> ateapi.MintAteomActorCertificateRequest - 119, // 217: ateapi.WorkerService.RequestActorSuspend:input_type -> ateapi.RequestActorSuspendRequest - 121, // 218: ateapi.WorkerService.ReportActorSnapshotState:input_type -> ateapi.ReportActorSnapshotStateRequest - 16, // 219: ateapi.Control.GetActor:output_type -> ateapi.Actor - 16, // 220: ateapi.Control.CreateActor:output_type -> ateapi.Actor - 16, // 221: ateapi.Control.UpdateActor:output_type -> ateapi.Actor - 72, // 222: ateapi.Control.SuspendActor:output_type -> ateapi.SuspendActorResponse - 74, // 223: ateapi.Control.PauseActor:output_type -> ateapi.PauseActorResponse - 76, // 224: ateapi.Control.ResumeActor:output_type -> ateapi.ResumeActorResponse - 78, // 225: ateapi.Control.RevertActor:output_type -> ateapi.RevertActorResponse - 16, // 226: ateapi.Control.DeleteActor:output_type -> ateapi.Actor - 17, // 227: ateapi.Control.GetActorEgressPolicy:output_type -> ateapi.EgressPolicy - 17, // 228: ateapi.Control.CreateActorEgressPolicy:output_type -> ateapi.EgressPolicy - 17, // 229: ateapi.Control.UpdateActorEgressPolicy:output_type -> ateapi.EgressPolicy - 17, // 230: ateapi.Control.DeleteActorEgressPolicy:output_type -> ateapi.EgressPolicy - 86, // 231: ateapi.Control.MintActorJwt:output_type -> ateapi.MintActorJwtResponse - 88, // 232: ateapi.Control.MintActorCertificate:output_type -> ateapi.MintActorCertificateResponse - 31, // 233: ateapi.Control.CreateTag:output_type -> ateapi.Tag - 31, // 234: ateapi.Control.GetTag:output_type -> ateapi.Tag - 90, // 235: ateapi.Control.ListTags:output_type -> ateapi.ListTagsResponse - 31, // 236: ateapi.Control.UpdateTag:output_type -> ateapi.Tag - 31, // 237: ateapi.Control.DeleteTag:output_type -> ateapi.Tag - 99, // 238: ateapi.Control.ListWorkers:output_type -> ateapi.ListWorkersResponse - 108, // 239: ateapi.Control.GetWorker:output_type -> ateapi.Worker - 108, // 240: ateapi.Control.CreateWorker:output_type -> ateapi.Worker - 108, // 241: ateapi.Control.UpdateWorker:output_type -> ateapi.Worker - 108, // 242: ateapi.Control.DeleteWorker:output_type -> ateapi.Worker - 105, // 243: ateapi.Control.DrainWorker:output_type -> ateapi.DrainWorkerResponse - 97, // 244: ateapi.Control.ListWorkerActorAssignments:output_type -> ateapi.ListWorkerActorAssignmentsResponse - 107, // 245: ateapi.Control.ListActors:output_type -> ateapi.ListActorsResponse - 32, // 246: ateapi.Control.CreateAtespace:output_type -> ateapi.Atespace - 32, // 247: ateapi.Control.GetAtespace:output_type -> ateapi.Atespace - 61, // 248: ateapi.Control.ListAtespaces:output_type -> ateapi.ListAtespacesResponse - 32, // 249: ateapi.Control.DeleteAtespace:output_type -> ateapi.Atespace - 34, // 250: ateapi.Control.CreateActorTemplate:output_type -> ateapi.ActorTemplate - 34, // 251: ateapi.Control.GetActorTemplate:output_type -> ateapi.ActorTemplate - 66, // 252: ateapi.Control.ListActorTemplates:output_type -> ateapi.ListActorTemplatesResponse - 34, // 253: ateapi.Control.DeleteActorTemplate:output_type -> ateapi.ActorTemplate - 125, // 254: ateapi.Control.GetGlobalAccessPolicy:output_type -> ateapi.AccessPolicy - 125, // 255: ateapi.Control.CreateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy - 125, // 256: ateapi.Control.UpdateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy - 125, // 257: ateapi.Control.GetAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 125, // 258: ateapi.Control.CreateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 125, // 259: ateapi.Control.UpdateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 125, // 260: ateapi.Control.DeleteAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy - 116, // 261: ateapi.WorkerService.RegisterWorker:output_type -> ateapi.RegisterWorkerResponse - 118, // 262: ateapi.WorkerService.MintAteomActorCertificate:output_type -> ateapi.MintAteomActorCertificateResponse - 120, // 263: ateapi.WorkerService.RequestActorSuspend:output_type -> ateapi.RequestActorSuspendResponse - 122, // 264: ateapi.WorkerService.ReportActorSnapshotState:output_type -> ateapi.ReportActorSnapshotStateResponse - 219, // [219:265] is the sub-list for method output_type - 173, // [173:219] is the sub-list for method input_type - 173, // [173:173] is the sub-list for extension type_name - 173, // [173:173] is the sub-list for extension extendee - 0, // [0:173] is the sub-list for field type_name + 40, // 55: ateapi.SnapshotConfig.golden_snapshot_config:type_name -> ateapi.GoldenSnapshotConfig + 5, // 56: ateapi.GoldenSnapshotConfig.mode:type_name -> ateapi.GoldenSnapshotMode + 44, // 57: ateapi.Container.env:type_name -> ateapi.EnvVar + 45, // 58: ateapi.Container.wakeup_probe:type_name -> ateapi.WakeupProbe + 56, // 59: ateapi.Container.volume_mounts:type_name -> ateapi.VolumeMount + 42, // 60: ateapi.Container.security_context:type_name -> ateapi.SecurityContext + 34, // 61: ateapi.Container.resources:type_name -> ateapi.Resources + 43, // 62: ateapi.SecurityContext.capabilities:type_name -> ateapi.Capabilities + 46, // 63: ateapi.WakeupProbe.http_get:type_name -> ateapi.HttpGetAction + 49, // 64: ateapi.Volume.durable_dir:type_name -> ateapi.DurableDirVolumeSource + 50, // 65: ateapi.Volume.external_volume_template:type_name -> ateapi.ExternalVolumeTemplate + 51, // 66: ateapi.Volume.system_info:type_name -> ateapi.SystemInfoVolumeSource + 48, // 67: ateapi.Volume.image:type_name -> ateapi.ImageVolumeSource + 52, // 68: ateapi.SystemInfoVolumeSource.data_sources:type_name -> ateapi.SystemInfoDataSource + 53, // 69: ateapi.SystemInfoDataSource.actor_metadata:type_name -> ateapi.ActorMetadataDataSource + 55, // 70: ateapi.SystemInfoDataSource.trust_bundle:type_name -> ateapi.TrustBundleDataSource + 54, // 71: ateapi.ActorMetadataDataSource.items:type_name -> ateapi.ActorMetadataItem + 6, // 72: ateapi.ActorMetadataItem.field:type_name -> ateapi.ActorMetadataField + 31, // 73: ateapi.CreateAtespaceRequest.atespace:type_name -> ateapi.Atespace + 32, // 74: ateapi.GetAtespaceRequest.atespace:type_name -> ateapi.ObjectRef + 94, // 75: ateapi.ListAtespacesRequest.options:type_name -> ateapi.ListOptions + 31, // 76: ateapi.ListAtespacesResponse.atespaces:type_name -> ateapi.Atespace + 32, // 77: ateapi.DeleteAtespaceRequest.atespace:type_name -> ateapi.ObjectRef + 93, // 78: ateapi.DeleteAtespaceRequest.options:type_name -> ateapi.DeleteOptions + 33, // 79: ateapi.CreateActorTemplateRequest.actor_template:type_name -> ateapi.ActorTemplate + 32, // 80: ateapi.GetActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef + 94, // 81: ateapi.ListActorTemplatesRequest.options:type_name -> ateapi.ListOptions + 33, // 82: ateapi.ListActorTemplatesResponse.actor_templates:type_name -> ateapi.ActorTemplate + 32, // 83: ateapi.DeleteActorTemplateRequest.actor_template:type_name -> ateapi.ObjectRef + 93, // 84: ateapi.DeleteActorTemplateRequest.options:type_name -> ateapi.DeleteOptions + 32, // 85: ateapi.GetActorRequest.actor:type_name -> ateapi.ObjectRef + 15, // 86: ateapi.CreateActorRequest.actor:type_name -> ateapi.Actor + 15, // 87: ateapi.UpdateActorRequest.actor:type_name -> ateapi.Actor + 32, // 88: ateapi.SuspendActorRequest.actor:type_name -> ateapi.ObjectRef + 15, // 89: ateapi.SuspendActorResponse.actor:type_name -> ateapi.Actor + 32, // 90: ateapi.PauseActorRequest.actor:type_name -> ateapi.ObjectRef + 15, // 91: ateapi.PauseActorResponse.actor:type_name -> ateapi.Actor + 32, // 92: ateapi.ResumeActorRequest.actor:type_name -> ateapi.ObjectRef + 15, // 93: ateapi.ResumeActorResponse.actor:type_name -> ateapi.Actor + 32, // 94: ateapi.RevertActorRequest.actor:type_name -> ateapi.ObjectRef + 15, // 95: ateapi.RevertActorResponse.actor:type_name -> ateapi.Actor + 32, // 96: ateapi.DeleteActorRequest.actor:type_name -> ateapi.ObjectRef + 93, // 97: ateapi.DeleteActorRequest.options:type_name -> ateapi.DeleteOptions + 32, // 98: ateapi.GetActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef + 32, // 99: ateapi.CreateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef + 16, // 100: ateapi.CreateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy + 32, // 101: ateapi.UpdateActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef + 16, // 102: ateapi.UpdateActorEgressPolicyRequest.egress_policy:type_name -> ateapi.EgressPolicy + 32, // 103: ateapi.DeleteActorEgressPolicyRequest.actor:type_name -> ateapi.ObjectRef + 93, // 104: ateapi.DeleteActorEgressPolicyRequest.options:type_name -> ateapi.DeleteOptions + 32, // 105: ateapi.GetTagRequest.tag:type_name -> ateapi.ObjectRef + 32, // 106: ateapi.MintActorJwtRequest.actor:type_name -> ateapi.ObjectRef + 144, // 107: ateapi.MintActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp + 32, // 108: ateapi.MintActorCertificateRequest.actor:type_name -> ateapi.ObjectRef + 94, // 109: ateapi.ListTagsRequest.options:type_name -> ateapi.ListOptions + 30, // 110: ateapi.ListTagsResponse.tags:type_name -> ateapi.Tag + 30, // 111: ateapi.CreateTagRequest.tag:type_name -> ateapi.Tag + 30, // 112: ateapi.UpdateTagRequest.tag:type_name -> ateapi.Tag + 32, // 113: ateapi.DeleteTagRequest.tag:type_name -> ateapi.ObjectRef + 93, // 114: ateapi.DeleteTagRequest.options:type_name -> ateapi.DeleteOptions + 32, // 115: ateapi.ListWorkerActorAssignmentsRequest.worker:type_name -> ateapi.ObjectRef + 94, // 116: ateapi.ListWorkerActorAssignmentsRequest.options:type_name -> ateapi.ListOptions + 117, // 117: ateapi.ListWorkerActorAssignmentsResponse.actor_assignments:type_name -> ateapi.ActorAssignment + 94, // 118: ateapi.ListWorkersRequest.options:type_name -> ateapi.ListOptions + 111, // 119: ateapi.ListWorkersResponse.workers:type_name -> ateapi.Worker + 32, // 120: ateapi.GetWorkerRequest.worker:type_name -> ateapi.ObjectRef + 111, // 121: ateapi.CreateWorkerRequest.worker:type_name -> ateapi.Worker + 111, // 122: ateapi.UpdateWorkerRequest.worker:type_name -> ateapi.Worker + 32, // 123: ateapi.DeleteWorkerRequest.worker:type_name -> ateapi.ObjectRef + 93, // 124: ateapi.DeleteWorkerRequest.options:type_name -> ateapi.DeleteOptions + 32, // 125: ateapi.DrainWorkerRequest.worker:type_name -> ateapi.ObjectRef + 111, // 126: ateapi.DrainWorkerResponse.worker:type_name -> ateapi.Worker + 32, // 127: ateapi.MarkWorkerAvailableRequest.worker:type_name -> ateapi.ObjectRef + 111, // 128: ateapi.MarkWorkerAvailableResponse.worker:type_name -> ateapi.Worker + 32, // 129: ateapi.MarkWorkerUnavailableRequest.worker:type_name -> ateapi.ObjectRef + 111, // 130: ateapi.MarkWorkerUnavailableResponse.worker:type_name -> ateapi.Worker + 94, // 131: ateapi.ListActorsRequest.options:type_name -> ateapi.ListOptions + 15, // 132: ateapi.ListActorsResponse.actors:type_name -> ateapi.Actor + 13, // 133: ateapi.Worker.metadata:type_name -> ateapi.ResourceMetadata + 143, // 134: ateapi.Worker.labels:type_name -> ateapi.Worker.LabelsEntry + 112, // 135: ateapi.Worker.status:type_name -> ateapi.WorkerStatus + 7, // 136: ateapi.WorkerStatus.state:type_name -> ateapi.WorkerState + 116, // 137: ateapi.WorkerStatus.capacity:type_name -> ateapi.WorkerResources + 116, // 138: ateapi.WorkerStatus.allocated:type_name -> ateapi.WorkerResources + 113, // 139: ateapi.WorkerStatus.default_runtime:type_name -> ateapi.SandboxRuntime + 113, // 140: ateapi.WorkerStatus.restorable_runtimes:type_name -> ateapi.SandboxRuntime + 114, // 141: ateapi.SandboxRuntime.version:type_name -> ateapi.VersionedSandboxCompat + 115, // 142: ateapi.VersionedSandboxCompat.attributes:type_name -> ateapi.AttributeEntry + 34, // 143: ateapi.WorkerResources.resources:type_name -> ateapi.Resources + 13, // 144: ateapi.ActorAssignment.metadata:type_name -> ateapi.ResourceMetadata + 32, // 145: ateapi.ActorAssignment.actor:type_name -> ateapi.ObjectRef + 34, // 146: ateapi.ActorAssignment.resources:type_name -> ateapi.Resources + 32, // 147: ateapi.RegisterWorkerRequest.worker:type_name -> ateapi.ObjectRef + 116, // 148: ateapi.RegisterWorkerRequest.capacity:type_name -> ateapi.WorkerResources + 113, // 149: ateapi.RegisterWorkerRequest.default_runtime:type_name -> ateapi.SandboxRuntime + 113, // 150: ateapi.RegisterWorkerRequest.restorable_runtimes:type_name -> ateapi.SandboxRuntime + 111, // 151: ateapi.RegisterWorkerResponse.worker:type_name -> ateapi.Worker + 32, // 152: ateapi.MintAteomActorCertificateRequest.actor:type_name -> ateapi.ObjectRef + 32, // 153: ateapi.MintAteomActorJwtRequest.actor:type_name -> ateapi.ObjectRef + 144, // 154: ateapi.MintAteomActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp + 32, // 155: ateapi.RequestActorSuspendRequest.worker:type_name -> ateapi.ObjectRef + 32, // 156: ateapi.RequestActorSuspendRequest.actor:type_name -> ateapi.ObjectRef + 15, // 157: ateapi.RequestActorSuspendResponse.actor:type_name -> ateapi.Actor + 32, // 158: ateapi.ReportActorSnapshotStateRequest.actor:type_name -> ateapi.ObjectRef + 128, // 159: ateapi.ReportActorSnapshotStateRequest.state:type_name -> ateapi.ActorSnapshotState + 129, // 160: ateapi.ActorSnapshotState.latest_local:type_name -> ateapi.SnapshotReport + 129, // 161: ateapi.ActorSnapshotState.uploading:type_name -> ateapi.SnapshotReport + 129, // 162: ateapi.ActorSnapshotState.latest_durable:type_name -> ateapi.SnapshotReport + 144, // 163: ateapi.ActorSnapshotState.upload_due_time:type_name -> google.protobuf.Timestamp + 2, // 164: ateapi.SnapshotReport.fidelity:type_name -> ateapi.SnapshotFidelity + 9, // 165: ateapi.SnapshotReport.object:type_name -> ateapi.ObjectSnapshot + 144, // 166: ateapi.SnapshotReport.local_commit_time:type_name -> google.protobuf.Timestamp + 144, // 167: ateapi.SnapshotReport.durable_complete_time:type_name -> google.protobuf.Timestamp + 13, // 168: ateapi.AccessPolicy.metadata:type_name -> ateapi.ResourceMetadata + 131, // 169: ateapi.AccessPolicy.bindings:type_name -> ateapi.Binding + 130, // 170: ateapi.CreateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 130, // 171: ateapi.UpdateGlobalAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 32, // 172: ateapi.GetAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 32, // 173: ateapi.CreateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 130, // 174: ateapi.CreateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 32, // 175: ateapi.UpdateAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 130, // 176: ateapi.UpdateAtespaceAccessPolicyRequest.access_policy:type_name -> ateapi.AccessPolicy + 32, // 177: ateapi.DeleteAtespaceAccessPolicyRequest.atespace:type_name -> ateapi.ObjectRef + 93, // 178: ateapi.DeleteAtespaceAccessPolicyRequest.options:type_name -> ateapi.DeleteOptions + 8, // 179: ateapi.ResourceAnnotation.scope:type_name -> ateapi.Scope + 145, // 180: ateapi.resource:extendee -> google.protobuf.MessageOptions + 146, // 181: ateapi.method:extendee -> google.protobuf.MethodOptions + 139, // 182: ateapi.resource:type_name -> ateapi.ResourceAnnotation + 140, // 183: ateapi.method:type_name -> ateapi.MethodAnnotation + 67, // 184: ateapi.Control.GetActor:input_type -> ateapi.GetActorRequest + 68, // 185: ateapi.Control.CreateActor:input_type -> ateapi.CreateActorRequest + 69, // 186: ateapi.Control.UpdateActor:input_type -> ateapi.UpdateActorRequest + 70, // 187: ateapi.Control.SuspendActor:input_type -> ateapi.SuspendActorRequest + 72, // 188: ateapi.Control.PauseActor:input_type -> ateapi.PauseActorRequest + 74, // 189: ateapi.Control.ResumeActor:input_type -> ateapi.ResumeActorRequest + 76, // 190: ateapi.Control.RevertActor:input_type -> ateapi.RevertActorRequest + 78, // 191: ateapi.Control.DeleteActor:input_type -> ateapi.DeleteActorRequest + 79, // 192: ateapi.Control.GetActorEgressPolicy:input_type -> ateapi.GetActorEgressPolicyRequest + 80, // 193: ateapi.Control.CreateActorEgressPolicy:input_type -> ateapi.CreateActorEgressPolicyRequest + 81, // 194: ateapi.Control.UpdateActorEgressPolicy:input_type -> ateapi.UpdateActorEgressPolicyRequest + 82, // 195: ateapi.Control.DeleteActorEgressPolicy:input_type -> ateapi.DeleteActorEgressPolicyRequest + 84, // 196: ateapi.Control.MintActorJwt:input_type -> ateapi.MintActorJwtRequest + 86, // 197: ateapi.Control.MintActorCertificate:input_type -> ateapi.MintActorCertificateRequest + 90, // 198: ateapi.Control.CreateTag:input_type -> ateapi.CreateTagRequest + 83, // 199: ateapi.Control.GetTag:input_type -> ateapi.GetTagRequest + 88, // 200: ateapi.Control.ListTags:input_type -> ateapi.ListTagsRequest + 91, // 201: ateapi.Control.UpdateTag:input_type -> ateapi.UpdateTagRequest + 92, // 202: ateapi.Control.DeleteTag:input_type -> ateapi.DeleteTagRequest + 97, // 203: ateapi.Control.ListWorkers:input_type -> ateapi.ListWorkersRequest + 99, // 204: ateapi.Control.GetWorker:input_type -> ateapi.GetWorkerRequest + 100, // 205: ateapi.Control.CreateWorker:input_type -> ateapi.CreateWorkerRequest + 101, // 206: ateapi.Control.UpdateWorker:input_type -> ateapi.UpdateWorkerRequest + 102, // 207: ateapi.Control.DeleteWorker:input_type -> ateapi.DeleteWorkerRequest + 103, // 208: ateapi.Control.DrainWorker:input_type -> ateapi.DrainWorkerRequest + 105, // 209: ateapi.Control.MarkWorkerAvailable:input_type -> ateapi.MarkWorkerAvailableRequest + 107, // 210: ateapi.Control.MarkWorkerUnavailable:input_type -> ateapi.MarkWorkerUnavailableRequest + 95, // 211: ateapi.Control.ListWorkerActorAssignments:input_type -> ateapi.ListWorkerActorAssignmentsRequest + 109, // 212: ateapi.Control.ListActors:input_type -> ateapi.ListActorsRequest + 57, // 213: ateapi.Control.CreateAtespace:input_type -> ateapi.CreateAtespaceRequest + 58, // 214: ateapi.Control.GetAtespace:input_type -> ateapi.GetAtespaceRequest + 59, // 215: ateapi.Control.ListAtespaces:input_type -> ateapi.ListAtespacesRequest + 61, // 216: ateapi.Control.DeleteAtespace:input_type -> ateapi.DeleteAtespaceRequest + 62, // 217: ateapi.Control.CreateActorTemplate:input_type -> ateapi.CreateActorTemplateRequest + 63, // 218: ateapi.Control.GetActorTemplate:input_type -> ateapi.GetActorTemplateRequest + 64, // 219: ateapi.Control.ListActorTemplates:input_type -> ateapi.ListActorTemplatesRequest + 66, // 220: ateapi.Control.DeleteActorTemplate:input_type -> ateapi.DeleteActorTemplateRequest + 132, // 221: ateapi.Control.GetGlobalAccessPolicy:input_type -> ateapi.GetGlobalAccessPolicyRequest + 133, // 222: ateapi.Control.CreateGlobalAccessPolicy:input_type -> ateapi.CreateGlobalAccessPolicyRequest + 134, // 223: ateapi.Control.UpdateGlobalAccessPolicy:input_type -> ateapi.UpdateGlobalAccessPolicyRequest + 135, // 224: ateapi.Control.GetAtespaceAccessPolicy:input_type -> ateapi.GetAtespaceAccessPolicyRequest + 136, // 225: ateapi.Control.CreateAtespaceAccessPolicy:input_type -> ateapi.CreateAtespaceAccessPolicyRequest + 137, // 226: ateapi.Control.UpdateAtespaceAccessPolicy:input_type -> ateapi.UpdateAtespaceAccessPolicyRequest + 138, // 227: ateapi.Control.DeleteAtespaceAccessPolicy:input_type -> ateapi.DeleteAtespaceAccessPolicyRequest + 118, // 228: ateapi.WorkerService.RegisterWorker:input_type -> ateapi.RegisterWorkerRequest + 120, // 229: ateapi.WorkerService.MintAteomActorCertificate:input_type -> ateapi.MintAteomActorCertificateRequest + 122, // 230: ateapi.WorkerService.MintAteomActorJwt:input_type -> ateapi.MintAteomActorJwtRequest + 124, // 231: ateapi.WorkerService.RequestActorSuspend:input_type -> ateapi.RequestActorSuspendRequest + 126, // 232: ateapi.WorkerService.ReportActorSnapshotState:input_type -> ateapi.ReportActorSnapshotStateRequest + 15, // 233: ateapi.Control.GetActor:output_type -> ateapi.Actor + 15, // 234: ateapi.Control.CreateActor:output_type -> ateapi.Actor + 15, // 235: ateapi.Control.UpdateActor:output_type -> ateapi.Actor + 71, // 236: ateapi.Control.SuspendActor:output_type -> ateapi.SuspendActorResponse + 73, // 237: ateapi.Control.PauseActor:output_type -> ateapi.PauseActorResponse + 75, // 238: ateapi.Control.ResumeActor:output_type -> ateapi.ResumeActorResponse + 77, // 239: ateapi.Control.RevertActor:output_type -> ateapi.RevertActorResponse + 15, // 240: ateapi.Control.DeleteActor:output_type -> ateapi.Actor + 16, // 241: ateapi.Control.GetActorEgressPolicy:output_type -> ateapi.EgressPolicy + 16, // 242: ateapi.Control.CreateActorEgressPolicy:output_type -> ateapi.EgressPolicy + 16, // 243: ateapi.Control.UpdateActorEgressPolicy:output_type -> ateapi.EgressPolicy + 16, // 244: ateapi.Control.DeleteActorEgressPolicy:output_type -> ateapi.EgressPolicy + 85, // 245: ateapi.Control.MintActorJwt:output_type -> ateapi.MintActorJwtResponse + 87, // 246: ateapi.Control.MintActorCertificate:output_type -> ateapi.MintActorCertificateResponse + 30, // 247: ateapi.Control.CreateTag:output_type -> ateapi.Tag + 30, // 248: ateapi.Control.GetTag:output_type -> ateapi.Tag + 89, // 249: ateapi.Control.ListTags:output_type -> ateapi.ListTagsResponse + 30, // 250: ateapi.Control.UpdateTag:output_type -> ateapi.Tag + 30, // 251: ateapi.Control.DeleteTag:output_type -> ateapi.Tag + 98, // 252: ateapi.Control.ListWorkers:output_type -> ateapi.ListWorkersResponse + 111, // 253: ateapi.Control.GetWorker:output_type -> ateapi.Worker + 111, // 254: ateapi.Control.CreateWorker:output_type -> ateapi.Worker + 111, // 255: ateapi.Control.UpdateWorker:output_type -> ateapi.Worker + 111, // 256: ateapi.Control.DeleteWorker:output_type -> ateapi.Worker + 104, // 257: ateapi.Control.DrainWorker:output_type -> ateapi.DrainWorkerResponse + 106, // 258: ateapi.Control.MarkWorkerAvailable:output_type -> ateapi.MarkWorkerAvailableResponse + 108, // 259: ateapi.Control.MarkWorkerUnavailable:output_type -> ateapi.MarkWorkerUnavailableResponse + 96, // 260: ateapi.Control.ListWorkerActorAssignments:output_type -> ateapi.ListWorkerActorAssignmentsResponse + 110, // 261: ateapi.Control.ListActors:output_type -> ateapi.ListActorsResponse + 31, // 262: ateapi.Control.CreateAtespace:output_type -> ateapi.Atespace + 31, // 263: ateapi.Control.GetAtespace:output_type -> ateapi.Atespace + 60, // 264: ateapi.Control.ListAtespaces:output_type -> ateapi.ListAtespacesResponse + 31, // 265: ateapi.Control.DeleteAtespace:output_type -> ateapi.Atespace + 33, // 266: ateapi.Control.CreateActorTemplate:output_type -> ateapi.ActorTemplate + 33, // 267: ateapi.Control.GetActorTemplate:output_type -> ateapi.ActorTemplate + 65, // 268: ateapi.Control.ListActorTemplates:output_type -> ateapi.ListActorTemplatesResponse + 33, // 269: ateapi.Control.DeleteActorTemplate:output_type -> ateapi.ActorTemplate + 130, // 270: ateapi.Control.GetGlobalAccessPolicy:output_type -> ateapi.AccessPolicy + 130, // 271: ateapi.Control.CreateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy + 130, // 272: ateapi.Control.UpdateGlobalAccessPolicy:output_type -> ateapi.AccessPolicy + 130, // 273: ateapi.Control.GetAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 130, // 274: ateapi.Control.CreateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 130, // 275: ateapi.Control.UpdateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 130, // 276: ateapi.Control.DeleteAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy + 119, // 277: ateapi.WorkerService.RegisterWorker:output_type -> ateapi.RegisterWorkerResponse + 121, // 278: ateapi.WorkerService.MintAteomActorCertificate:output_type -> ateapi.MintAteomActorCertificateResponse + 123, // 279: ateapi.WorkerService.MintAteomActorJwt:output_type -> ateapi.MintAteomActorJwtResponse + 125, // 280: ateapi.WorkerService.RequestActorSuspend:output_type -> ateapi.RequestActorSuspendResponse + 127, // 281: ateapi.WorkerService.ReportActorSnapshotState:output_type -> ateapi.ReportActorSnapshotStateResponse + 233, // [233:282] is the sub-list for method output_type + 184, // [184:233] is the sub-list for method input_type + 182, // [182:184] is the sub-list for extension type_name + 180, // [180:182] is the sub-list for extension extendee + 0, // [0:180] is the sub-list for field type_name } func init() { file_ateapi_proto_init() } @@ -9859,15 +10431,16 @@ func file_ateapi_proto_init() { File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_ateapi_proto_rawDesc), len(file_ateapi_proto_rawDesc)), - NumEnums: 10, - NumMessages: 127, - NumExtensions: 0, + NumEnums: 9, + NumMessages: 135, + NumExtensions: 2, NumServices: 2, }, GoTypes: file_ateapi_proto_goTypes, DependencyIndexes: file_ateapi_proto_depIdxs, EnumInfos: file_ateapi_proto_enumTypes, MessageInfos: file_ateapi_proto_msgTypes, + ExtensionInfos: file_ateapi_proto_extTypes, }.Build() File_ateapi_proto = out.File file_ateapi_proto_goTypes = nil diff --git a/pkg/proto/ateapipb/ateapi.proto b/pkg/proto/ateapipb/ateapi.proto index 58a2cfccd8..7d5ecde1e9 100644 --- a/pkg/proto/ateapipb/ateapi.proto +++ b/pkg/proto/ateapipb/ateapi.proto @@ -16,6 +16,7 @@ syntax = "proto3"; package ateapi; +import "google/protobuf/descriptor.proto"; import "google/protobuf/timestamp.proto"; option go_package = "github.com/agent-substrate/substrate/pkg/proto/ateapipb"; @@ -23,51 +24,77 @@ option go_package = "github.com/agent-substrate/substrate/pkg/proto/ateapipb"; // Control is the primary RPC interface for Agentic Substrate. service Control { // Get an Actor. - rpc GetActor(GetActorRequest) returns (Actor) {} + rpc GetActor(GetActorRequest) returns (Actor) { + option (method).resource = "Actor"; + } // Create a new Actor deriving from a given ActorTemplate. - rpc CreateActor(CreateActorRequest) returns (Actor) {} + rpc CreateActor(CreateActorRequest) returns (Actor) { + option (method).resource = "Actor"; + } // Update mutable fields on an existing Actor. - rpc UpdateActor(UpdateActorRequest) returns (Actor) {} + rpc UpdateActor(UpdateActorRequest) returns (Actor) { + option (method).resource = "Actor"; + } // Suspend a given actor to a new snapshot. A running actor is checkpointed // on its worker; a paused actor's node-local snapshot is uploaded, narrowed // to the template's commit scope where required (Full capture, Data commit). - rpc SuspendActor(SuspendActorRequest) returns (SuspendActorResponse) {} + rpc SuspendActor(SuspendActorRequest) returns (SuspendActorResponse) { + option (method).resource = "Actor"; + } // Pause a given actor: checkpoint it to its worker's node and leave it // SUSPENDED with that node-local snapshot, pinned to the node. A paused // actor is told from a suspended one by its snapshots, not its state. - rpc PauseActor(PauseActorRequest) returns (PauseActorResponse) {} + rpc PauseActor(PauseActorRequest) returns (PauseActorResponse) { + option (method).resource = "Actor"; + } // Resume an actor from its latest snapshot. - rpc ResumeActor(ResumeActorRequest) returns (ResumeActorResponse) {} + rpc ResumeActor(ResumeActorRequest) returns (ResumeActorResponse) { + option (method).resource = "Actor"; + } // Revert an actor to SUSPENDED state at its durable snapshot. // Only crashed, running or paused actors can be reverted. - rpc RevertActor(RevertActorRequest) returns (RevertActorResponse) {} + rpc RevertActor(RevertActorRequest) returns (RevertActorResponse) { + option (method).resource = "Actor"; + } // Delete an actor. Only suspended or crashed actors can be deleted unless // any_state is set. - rpc DeleteActor(DeleteActorRequest) returns (Actor) {} + rpc DeleteActor(DeleteActorRequest) returns (Actor) { + option (method).resource = "Actor"; + } // Get the egress policy resource nested under an Actor. - rpc GetActorEgressPolicy(GetActorEgressPolicyRequest) returns (EgressPolicy) {} + rpc GetActorEgressPolicy(GetActorEgressPolicyRequest) returns (EgressPolicy) { + option (method).resource = "EgressPolicy"; + } // Create the egress policy resource nested under an Actor. - rpc CreateActorEgressPolicy(CreateActorEgressPolicyRequest) returns (EgressPolicy) {} + rpc CreateActorEgressPolicy(CreateActorEgressPolicyRequest) returns (EgressPolicy) { + option (method).resource = "EgressPolicy"; + } // Replace the egress policy resource nested under an Actor. - rpc UpdateActorEgressPolicy(UpdateActorEgressPolicyRequest) returns (EgressPolicy) {} + rpc UpdateActorEgressPolicy(UpdateActorEgressPolicyRequest) returns (EgressPolicy) { + option (method).resource = "EgressPolicy"; + } // Delete the egress policy resource nested under an Actor. - rpc DeleteActorEgressPolicy(DeleteActorEgressPolicyRequest) returns (EgressPolicy) {} + rpc DeleteActorEgressPolicy(DeleteActorEgressPolicyRequest) returns (EgressPolicy) { + option (method).resource = "EgressPolicy"; + } // Create a Substrate-issued JWT asserting the actor identity. // // * Called by the egress gateway when actor JWT injection is configured for outbound requests. - rpc MintActorJwt(MintActorJwtRequest) returns (MintActorJwtResponse) {} + rpc MintActorJwt(MintActorJwtRequest) returns (MintActorJwtResponse) { + option (method).resource = "Actor"; + } // Create a Substrate-issued SPIFFE certificate asserting the actor identity. // @@ -75,102 +102,179 @@ service Control { // // * Called by the egress gateway when actor client certificate injection is // configured for outbound requests. - rpc MintActorCertificate(MintActorCertificateRequest) returns (MintActorCertificateResponse) {} + rpc MintActorCertificate(MintActorCertificateRequest) returns (MintActorCertificateResponse) { + option (method).resource = "Actor"; + } // Tag the external snapshot a suspended Actor holds. The tag gets its own // copy of that snapshot, so suspending or deleting the Actor afterwards // cannot collect it. - rpc CreateTag(CreateTagRequest) returns (Tag) {} + rpc CreateTag(CreateTagRequest) returns (Tag) { + option (method).resource = "Tag"; + } // Get a Tag. - rpc GetTag(GetTagRequest) returns (Tag) {} + rpc GetTag(GetTagRequest) returns (Tag) { + option (method).resource = "Tag"; + } // List Tags. - rpc ListTags(ListTagsRequest) returns (ListTagsResponse) {} + rpc ListTags(ListTagsRequest) returns (ListTagsResponse) { + option (method).resource = "Tag"; + } // Publish or unpublish a Tag without changing its address. - rpc UpdateTag(UpdateTagRequest) returns (Tag) {} + rpc UpdateTag(UpdateTagRequest) returns (Tag) { + option (method).resource = "Tag"; + } // Delete a Tag and the external snapshot it owns. Actors created from the // tag that have not yet been suspended still point at that external snapshot // and become unrecoverable, so do not delete a tag while such Actors exist. - rpc DeleteTag(DeleteTagRequest) returns (Tag) {} + rpc DeleteTag(DeleteTagRequest) returns (Tag) { + option (method).resource = "Tag"; + } // List Workers. - rpc ListWorkers(ListWorkersRequest) returns (ListWorkersResponse) {} + rpc ListWorkers(ListWorkersRequest) returns (ListWorkersResponse) { + option (method).resource = "Worker"; + } // Get a Worker. - rpc GetWorker(GetWorkerRequest) returns (Worker) {} + rpc GetWorker(GetWorkerRequest) returns (Worker) { + option (method).resource = "Worker"; + } - // Register a Worker. Called once its Pod is Ready and has an IP. - rpc CreateWorker(CreateWorkerRequest) returns (Worker) {} + // Register a Worker. Called once its Pod has an IP. A Worker is registered + // UNAVAILABLE and takes no Actors until MarkWorkerAvailable. + rpc CreateWorker(CreateWorkerRequest) returns (Worker) { + option (method).resource = "Worker"; + } // Update observed pool state on a Worker. - rpc UpdateWorker(UpdateWorkerRequest) returns (Worker) {} + rpc UpdateWorker(UpdateWorkerRequest) returns (Worker) { + option (method).resource = "Worker"; + } // Deregister a Worker. Does not cascade: the caller is responsible for // cleaning up related resources first. - rpc DeleteWorker(DeleteWorkerRequest) returns (Worker) {} + rpc DeleteWorker(DeleteWorkerRequest) returns (Worker) { + option (method).resource = "Worker"; + } // Mark a Worker as terminating so the scheduler stops routing new Actors to // it. Idempotent; one-way. Deliberately leaves any bound Actor alone. // Returns ABORTED if another write lands on the Worker first; retry. - rpc DrainWorker(DrainWorkerRequest) returns (DrainWorkerResponse) {} + rpc DrainWorker(DrainWorkerRequest) returns (DrainWorkerResponse) { + option (method).resource = "Worker"; + } + + // Mark an UNAVAILABLE Worker ACTIVE so the scheduler places Actors on it. + // Idempotent. Returns FAILED_PRECONDITION for a DRAINING Worker, and ABORTED + // if another write lands on the Worker first; retry. + // + // Every Actor placed after this call is stamped with the Worker's epoch as + // it stands then, so a caller raises epoch for a restarted ateom first. + rpc MarkWorkerAvailable(MarkWorkerAvailableRequest) returns (MarkWorkerAvailableResponse) { + option (method).resource = "Worker"; + } + + // Mark an ACTIVE Worker UNAVAILABLE so the scheduler stops placing Actors on + // it, as while its Pod is not Ready. Idempotent. Leaves any bound Actor + // alone. Returns FAILED_PRECONDITION for a DRAINING Worker, and ABORTED if + // another write lands on the Worker first; retry. + rpc MarkWorkerUnavailable(MarkWorkerUnavailableRequest) returns (MarkWorkerUnavailableResponse) { + option (method).resource = "Worker"; + } // List the Actors hosted by a given Worker. - rpc ListWorkerActorAssignments(ListWorkerActorAssignmentsRequest) returns (ListWorkerActorAssignmentsResponse) {} + rpc ListWorkerActorAssignments(ListWorkerActorAssignmentsRequest) returns (ListWorkerActorAssignmentsResponse) { + option (method).resource = "ActorAssignment"; + } // List Actors. - rpc ListActors(ListActorsRequest) returns (ListActorsResponse) {} + rpc ListActors(ListActorsRequest) returns (ListActorsResponse) { + option (method).resource = "Actor"; + } // Create a new Atespace. Substrate-native, stored in database. - rpc CreateAtespace(CreateAtespaceRequest) returns (Atespace) {} + rpc CreateAtespace(CreateAtespaceRequest) returns (Atespace) { + option (method).resource = "Atespace"; + } // Get an Atespace by name. - rpc GetAtespace(GetAtespaceRequest) returns (Atespace) {} + rpc GetAtespace(GetAtespaceRequest) returns (Atespace) { + option (method).resource = "Atespace"; + } // List Atespaces. - rpc ListAtespaces(ListAtespacesRequest) returns (ListAtespacesResponse) {} + rpc ListAtespaces(ListAtespacesRequest) returns (ListAtespacesResponse) { + option (method).resource = "Atespace"; + } // Delete an empty Atespace. Rejects (FailedPrecondition) if any Actors or // Tags remain. - rpc DeleteAtespace(DeleteAtespaceRequest) returns (Atespace) {} + rpc DeleteAtespace(DeleteAtespaceRequest) returns (Atespace) { + option (method).resource = "Atespace"; + } - rpc CreateActorTemplate(CreateActorTemplateRequest) returns (ActorTemplate) {} + rpc CreateActorTemplate(CreateActorTemplateRequest) returns (ActorTemplate) { + option (method).resource = "ActorTemplate"; + } - rpc GetActorTemplate(GetActorTemplateRequest) returns (ActorTemplate) {} + rpc GetActorTemplate(GetActorTemplateRequest) returns (ActorTemplate) { + option (method).resource = "ActorTemplate"; + } - rpc ListActorTemplates(ListActorTemplatesRequest) returns (ListActorTemplatesResponse) {} + rpc ListActorTemplates(ListActorTemplatesRequest) returns (ListActorTemplatesResponse) { + option (method).resource = "ActorTemplate"; + } // Delete an ActorTemplate together with its golden actor and golden // tag in the reserved ate-golden atespace. - rpc DeleteActorTemplate(DeleteActorTemplateRequest) returns (ActorTemplate) {} + rpc DeleteActorTemplate(DeleteActorTemplateRequest) returns (ActorTemplate) { + option (method).resource = "ActorTemplate"; + } // Get the deployment-wide global access policy singleton. Returns NotFound // until the policy has been created. - rpc GetGlobalAccessPolicy(GetGlobalAccessPolicyRequest) returns (AccessPolicy) {} + rpc GetGlobalAccessPolicy(GetGlobalAccessPolicyRequest) returns (AccessPolicy) { + option (method).resource = "AccessPolicy"; + } // Create the deployment-wide global access policy singleton. The server's // configured bootstrap owners are global owners whether or not this policy // exists, so they can create it. - rpc CreateGlobalAccessPolicy(CreateGlobalAccessPolicyRequest) returns (AccessPolicy) {} + rpc CreateGlobalAccessPolicy(CreateGlobalAccessPolicyRequest) returns (AccessPolicy) { + option (method).resource = "AccessPolicy"; + } // Replace the deployment-wide global access policy singleton. Empty // bindings revoke every grant held through the policy; bootstrap owners // keep access. - rpc UpdateGlobalAccessPolicy(UpdateGlobalAccessPolicyRequest) returns (AccessPolicy) {} + rpc UpdateGlobalAccessPolicy(UpdateGlobalAccessPolicyRequest) returns (AccessPolicy) { + option (method).resource = "AccessPolicy"; + } // Get the access policy resource nested under an Atespace. - rpc GetAtespaceAccessPolicy(GetAtespaceAccessPolicyRequest) returns (AccessPolicy) {} + rpc GetAtespaceAccessPolicy(GetAtespaceAccessPolicyRequest) returns (AccessPolicy) { + option (method).resource = "AccessPolicy"; + } // Create the access policy resource nested under an Atespace. - rpc CreateAtespaceAccessPolicy(CreateAtespaceAccessPolicyRequest) returns (AccessPolicy) {} + rpc CreateAtespaceAccessPolicy(CreateAtespaceAccessPolicyRequest) returns (AccessPolicy) { + option (method).resource = "AccessPolicy"; + } // Replace the access policy resource nested under an Atespace. - rpc UpdateAtespaceAccessPolicy(UpdateAtespaceAccessPolicyRequest) returns (AccessPolicy) {} + rpc UpdateAtespaceAccessPolicy(UpdateAtespaceAccessPolicyRequest) returns (AccessPolicy) { + option (method).resource = "AccessPolicy"; + } // Delete the access policy resource nested under an Atespace. - rpc DeleteAtespaceAccessPolicy(DeleteAtespaceAccessPolicyRequest) returns (AccessPolicy) {} + rpc DeleteAtespaceAccessPolicy(DeleteAtespaceAccessPolicyRequest) returns (AccessPolicy) { + option (method).resource = "AccessPolicy"; + } } // ObjectSnapshot addresses a snapshot held in object storage. @@ -269,6 +373,12 @@ message Snapshot { // // +k8s:optional SnapshotStorage durable_snapshot = 7; + + // sandbox_runtime is the runtime the actor ran under when the snapshot was + // taken. + // + // +k8s:optional + SandboxRuntime sandbox_runtime = 8; } // SnapshotFidelity orders the state layers a snapshot can hold, lowest to @@ -409,22 +519,6 @@ message ExternalVolume { // +k8s:customValidation string volume_type = 3; - enum Status { - STATUS_UNSPECIFIED = 0; - // Volume creation pending in the storage system. - STATUS_PENDING = 1; - // Volume successfully created in the storage system. - STATUS_CREATED = 2; - // Volume being deleted from the storage system. - STATUS_DELETING = 3; - // Keep this in sync with ExternalVolume.status's maximum. - } - - // +k8s:optional - // +k8s:minimum=1 - // +k8s:maximum=3 # keep this in sync with the Status enum - Status status = 4; - // volume_context contains metadata returned by the CSI driver during volume // provisioning, needed by the node plugin for mounting (e.g. attachment // info). Keys and values are the driver's own, so the map is only bounded, @@ -436,6 +530,11 @@ message ExternalVolume { } message Actor { + option (resource) = { + scope : SCOPE_ATESPACE + plural : "Actors" + }; + // Common resource metadata: atespace, name, uid, version, timestamps. // // +k8s:required @@ -476,6 +575,12 @@ message Actor { // at most one egress policy resource, named "default". // message EgressPolicy { + option (resource) = { + parents : "Actor" + singleton : true + plural : "EgressPolicies" + }; + // Standard resource metadata. Atespace must match the parent Actor and name // must be "default". Both are caller-specified on create and immutable. // UID, version, create_time, and update_time are server-managed. @@ -895,6 +1000,12 @@ message WorkerAssignment { // +k8s:optional // +k8s:minimum=0 int64 worker_epoch = 8; + + // sandbox_runtime is the sandbox runtime and compatibility version the Actor + // was started or restored under on this Worker. + // + // +k8s:optional + SandboxRuntime sandbox_runtime = 9; } // TagStatus is the server-owned state of a Tag. @@ -925,6 +1036,11 @@ message TagStatus { // snapshot is copied when the tag is created, and deleted when the tag is. Its // owning Atespace cannot be deleted until the tag is removed. message Tag { + option (resource) = { + scope : SCOPE_ATESPACE + plural : "Tags" + }; + // Common resource metadata: atespace, name, uid, version, timestamps. // // +k8s:required @@ -956,6 +1072,11 @@ message Tag { // Atespace is the isolation boundary an Actor is created into. Global-scoped: // metadata.atespace is always empty; the atespace's identity is metadata.name. message Atespace { + option (resource) = { + scope : SCOPE_GLOBAL + plural : "Atespaces" + }; + // Common resource metadata: name, uid, version, timestamps. // // +k8s:required @@ -982,16 +1103,12 @@ message ObjectRef { string name = 2; } -// SandboxClass selects the sandbox runtime family. Snapshots are not portable -// across classes. -enum SandboxClass { - SANDBOX_CLASS_UNSPECIFIED = 0; - SANDBOX_CLASS_GVISOR = 1; - SANDBOX_CLASS_MICROVM = 2; - // Keep this in sync with SandboxConfig.sandbox_class's maximum. -} - message ActorTemplate { + option (resource) = { + scope : SCOPE_ATESPACE + plural : "ActorTemplates" + }; + // Common resource metadata: atespace, name, uid, version, timestamps. // // +k8s:required @@ -1087,12 +1204,13 @@ message ActorTemplateStatus { // SandboxConfig selects the sandbox runtime for an ActorTemplate. message SandboxConfig { - // sandbox_class selects the sandbox runtime family. + // sandbox_class selects the sandbox runtime family: "gvisor" or "microvm". + // Snapshots are not portable across classes. // // +k8s:required - // +k8s:minimum=1 - // +k8s:maximum=2 # keep this in sync with the SandboxClass enum - SandboxClass sandbox_class = 1; + // +k8s:maxLength=63 + // +k8s:customValidation # must be "gvisor" or "microvm" + string sandbox_class = 1; // config_name names the cluster-scoped SandboxConfig Kubernetes object // supplying the sandbox binaries. Required; must match sandbox_class. @@ -1120,7 +1238,7 @@ message SnapshotConfig { // captured or is too expensive to capture at the time, for example memory // under host pressure. A process resumed in place on its worker keeps its // memory regardless of this setting. ROOTFS requires - // sandbox_config.sandbox_class SANDBOX_CLASS_MICROVM. Defaults to MEMORY + // sandbox_config.sandbox_class "microvm". Defaults to MEMORY // when unset. // // +k8s:required @@ -1800,8 +1918,13 @@ message MintActorJwtResponse { // * exp, nbf, iat: Numeric Unix timestamps. nbf is five minutes before iat // to allow for clock skew. // * jti: A random token ID. - // * `ate.dev`: A JSON object with the actor's `atespace`, `actorName`, and - // `actorUID`. + // * `actor.ate.dev`: A JSON object with the actor's `atespace`, + // `actorName`, and `actorUID`. + // + // The subject and claim name say the token speaks for the actor itself. + // A token minted for a Substrate component acting on the actor's behalf + // (WorkerService.MintAteomActorJwt) uses `ateom-for-actor` in both places + // instead, so a relying party can grant the two different access. string actor_jwt = 1 [ debug_redact = true ]; // expire_time is when actor_jwt expires. Equal to its exp claim. @@ -2045,6 +2168,36 @@ message DrainWorkerResponse { Worker worker = 1; } +// MarkWorkerAvailableRequest names the Worker to mark ACTIVE. +message MarkWorkerAvailableRequest { + // The Worker to mark ACTIVE. + // + // +k8s:required + // +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix + ObjectRef worker = 1; +} + +// MarkWorkerAvailableResponse holds the Worker marked ACTIVE. +message MarkWorkerAvailableResponse { + // The Worker marked ACTIVE. + Worker worker = 1; +} + +// MarkWorkerUnavailableRequest names the Worker to mark UNAVAILABLE. +message MarkWorkerUnavailableRequest { + // The Worker to mark UNAVAILABLE. + // + // +k8s:required + // +k8s:beta(since: "0.0")=+k8s:subfield(atespace)=+k8s:forbidden # TODO: get rid of beta prefix + ObjectRef worker = 1; +} + +// MarkWorkerUnavailableResponse holds the Worker marked UNAVAILABLE. +message MarkWorkerUnavailableResponse { + // The Worker marked UNAVAILABLE. + Worker worker = 1; +} + // List actors with pagination. Provides "soft" guarantees: actors may be // missed or duplicated if the system state changes during pagination. message ListActorsRequest { @@ -2079,6 +2232,11 @@ message ListActorsResponse { // an immutable field that a request changes — including by omitting it, which // would clear it — is rejected with INVALID_ARGUMENT. message Worker { + option (resource) = { + scope : SCOPE_GLOBAL + plural : "Workers" + }; + // Output-only: name, uid, version and timestamps are all server-assigned. // uid and version are echoed back on UpdateWorker as preconditions. // @@ -2122,6 +2280,7 @@ message Worker { // // +k8s:optional // +k8s:maxLength=63 + // +k8s:customValidation # gvisor or microvm // +k8s:immutable string sandbox_class = 8; @@ -2151,8 +2310,9 @@ message Worker { int64 epoch = 12; // Output-only server-managed state. Absent from Create/Update request - // payloads; whatever a request carries here is ignored. DrainWorker is the - // only way a client moves state, and the assignments are the scheduler's. + // payloads; whatever a request carries here is ignored. A client moves + // state only through DrainWorker, MarkWorkerAvailable and + // MarkWorkerUnavailable, and the assignments are the scheduler's. // // +k8s:optional WorkerStatus status = 11; @@ -2162,15 +2322,19 @@ enum WorkerState { WORKER_STATE_UNSPECIFIED = 0; // Ready; schedulable. WORKER_STATE_ACTIVE = 1; - // Pod terminating. Not schedulable. + // Pod terminating. Not schedulable. Final: a draining Worker never becomes + // active or unavailable again. WORKER_STATE_DRAINING = 2; + // Registered but not Ready, or its ateom restarting. Not schedulable. Every + // Worker starts here until MarkWorkerAvailable. + WORKER_STATE_UNAVAILABLE = 3; // Keep this in sync with WorkerStatus.state's maximum. } message WorkerStatus { // +k8s:required // +k8s:minimum=1 - // +k8s:maximum=2 # keep this in sync with the WorkerState enum + // +k8s:maximum=3 # keep this in sync with the WorkerState enum WorkerState state = 1; // The total resource capacity of the Worker. @@ -2223,6 +2387,7 @@ message SandboxRuntime { // // +k8s:required // +k8s:maxLength=63 + // +k8s:customValidation # gvisor or microvm string sandbox_class = 1; // Primary compatibility version stamped onto an actor's snapshot when a @@ -2252,8 +2417,10 @@ message VersionedSandboxCompat { // features after masking (e.g. "architecture", "cpu_features"). // Matched against a snapshot under the same schema_version: every attribute // on the snapshot must be present on the worker with an equal value; keys - // only the worker has are ignored. Bump schema_version when a key's meaning - // changes or older snapshots must no longer match. + // only the worker has are ignored, so a worker can gain keys under one + // schema_version without invalidating older snapshots. Removing or + // recomputing a key is a new schema_version, under which older snapshots + // no longer match. // Order as sent does not matter: ate-api-server sorts attributes by key // before recording them. // @@ -2300,6 +2467,11 @@ message WorkerResources { // ActorAssignment binds an Actor to a Worker. This is a subresource of the // Worker. message ActorAssignment { + option (resource) = { + parents : "Worker" + plural : "ActorAssignments" + }; + // Metadata about the assignment. // // The atespace field is always empty, since it corresponds to a Worker, and @@ -2355,6 +2527,18 @@ service WorkerService { // SPIFFE URI: spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor} rpc MintAteomActorCertificate(MintAteomActorCertificateRequest) returns (MintAteomActorCertificateResponse); + // MintAteomActorJwt creates a Substrate-issued JWT that asserts an ateom + // acting on behalf of a particular actor: the JWT counterpart of + // MintAteomActorCertificate. atelet calls it with its own client + // certificate and hands the JWT to node-side plugins, which exchange it for + // credentials scoped to that actor. + // + // The token speaks for the ateom, not the actor: its subject is + // `ateom-for-actor//` and its claims sit under + // `ateom-for-actor.ate.dev`, see MintAteomActorJwtResponse. The lifetime is + // fixed at one hour. + rpc MintAteomActorJwt(MintAteomActorJwtRequest) returns (MintAteomActorJwtResponse); + // RequestActorSuspend asks the control plane to suspend an Actor that the // calling Worker hosts. Only the Worker can observe what makes an Actor // worth reclaiming -- whether its workload still has anything to do -- and @@ -2461,6 +2645,52 @@ message MintAteomActorCertificateResponse { repeated bytes actor_certificates = 1; } +// Request message for MintAteomActorJwt. +message MintAteomActorJwtRequest { + // The actor for which the JWT should be issued. + // + // Must be a valid actor that currently exists according to the actor store. + // + // +k8s:required + // +k8s:subfield(atespace)=+k8s:required + ObjectRef actor = 1; + + // The UID of the actor --- used to guard against deletion and recreation of + // an actor with the same name. + // + // +k8s:required + // +k8s:format=k8s-uuid + string actor_uid = 2; + + // The audiences the minted JWT is bound to. Tokens are only issued with + // audience bindings, so at least one is required. + // + // +k8s:required + // +k8s:maxItems=16 # guardrail; tokens realistically bind a handful of audiences + // +k8s:listType=set + // +k8s:eachVal=+k8s:minLength=1 + // +k8s:eachVal=+k8s:maxLength=512 # audiences are caller-defined URIs; bound only + repeated string audiences = 3; +} + +// Response message for MintAteomActorJwt. +message MintAteomActorJwtResponse { + // The signed JWT. Its claims are those described on + // MintActorJwtResponse.actor_jwt, except that it speaks for an ateom + // acting on behalf of the actor rather than for the actor itself: + // + // * sub: `ateom-for-actor//`. + // * `ateom-for-actor.ate.dev`: A JSON object with the actor's `atespace`, + // `actorName`, and `actorUID`. + // + // A relying party that trusts actor JWTs must not accept this token as the + // actor, and vice versa. + string actor_jwt = 1 [ debug_redact = true ]; + + // expire_time is when actor_jwt expires. Equal to its exp claim. + google.protobuf.Timestamp expire_time = 2; +} + message RequestActorSuspendRequest { // The Worker making the request. atespace is always empty; Workers are // global-scoped. atelet fills this in from its caller's certificate, so it @@ -2585,6 +2815,12 @@ message SnapshotReport { // the global singleton installation). Every parent resource has at most one // access policy resource, named "default". message AccessPolicy { + option (resource) = { + parents : [ "Global", "Atespace" ] + singleton : true + plural : "AccessPolicies" + }; + // Standard resource metadata. Atespace must match the parent resource (empty // for global-scoped parents like Global and Atespace) and name must be // "default". Both are caller-specified on create and immutable. UID, version, @@ -2717,3 +2953,48 @@ message DeleteAtespaceAccessPolicyRequest { // +k8s:optional DeleteOptions options = 3; } + +// ResourceAnnotation marks a message as an API resource. See +// docs/resource-model.md. +message ResourceAnnotation { + // The resource types this resource is a subresource of: another resource's + // message name, or "Global" for the whole Substrate installation. Empty for + // a top-level resource. + repeated string parents = 1; + + // Whether each parent has at most one instance of this resource. + bool singleton = 2; + + // Whether a top-level resource belongs to an atespace. Set only on a + // top-level resource: a subresource has its parent's scope. + Scope scope = 3; + + // The resource's plural name, as List methods and their response fields + // use it. For example, "AccessPolicies" for AccessPolicy. + string plural = 4; +} + +// Scope is whether a top-level resource belongs to an atespace. +enum Scope { + // The scope is not set. + SCOPE_UNSPECIFIED = 0; + // The resource belongs to an atespace, identified by (atespace, name). + SCOPE_ATESPACE = 1; + // The resource belongs to no atespace, identified by its name alone. + SCOPE_GLOBAL = 2; +} + +// MethodAnnotation describes an RPC of the API. +message MethodAnnotation { + // The message name of the resource the method operates on. For example, + // "Actor" for SuspendActor, or "EgressPolicy" for GetActorEgressPolicy. + string resource = 1; +} + +extend google.protobuf.MessageOptions { + ResourceAnnotation resource = 50000; +} + +extend google.protobuf.MethodOptions { + MethodAnnotation method = 50000; +} diff --git a/pkg/proto/ateapipb/ateapi_grpc.pb.go b/pkg/proto/ateapipb/ateapi_grpc.pb.go index 421bced7f9..c8e48710b7 100644 --- a/pkg/proto/ateapipb/ateapi_grpc.pb.go +++ b/pkg/proto/ateapipb/ateapi_grpc.pb.go @@ -58,6 +58,8 @@ const ( Control_UpdateWorker_FullMethodName = "/ateapi.Control/UpdateWorker" Control_DeleteWorker_FullMethodName = "/ateapi.Control/DeleteWorker" Control_DrainWorker_FullMethodName = "/ateapi.Control/DrainWorker" + Control_MarkWorkerAvailable_FullMethodName = "/ateapi.Control/MarkWorkerAvailable" + Control_MarkWorkerUnavailable_FullMethodName = "/ateapi.Control/MarkWorkerUnavailable" Control_ListWorkerActorAssignments_FullMethodName = "/ateapi.Control/ListWorkerActorAssignments" Control_ListActors_FullMethodName = "/ateapi.Control/ListActors" Control_CreateAtespace_FullMethodName = "/ateapi.Control/CreateAtespace" @@ -142,7 +144,8 @@ type ControlClient interface { ListWorkers(ctx context.Context, in *ListWorkersRequest, opts ...grpc.CallOption) (*ListWorkersResponse, error) // Get a Worker. GetWorker(ctx context.Context, in *GetWorkerRequest, opts ...grpc.CallOption) (*Worker, error) - // Register a Worker. Called once its Pod is Ready and has an IP. + // Register a Worker. Called once its Pod has an IP. A Worker is registered + // UNAVAILABLE and takes no Actors until MarkWorkerAvailable. CreateWorker(ctx context.Context, in *CreateWorkerRequest, opts ...grpc.CallOption) (*Worker, error) // Update observed pool state on a Worker. UpdateWorker(ctx context.Context, in *UpdateWorkerRequest, opts ...grpc.CallOption) (*Worker, error) @@ -153,6 +156,18 @@ type ControlClient interface { // it. Idempotent; one-way. Deliberately leaves any bound Actor alone. // Returns ABORTED if another write lands on the Worker first; retry. DrainWorker(ctx context.Context, in *DrainWorkerRequest, opts ...grpc.CallOption) (*DrainWorkerResponse, error) + // Mark an UNAVAILABLE Worker ACTIVE so the scheduler places Actors on it. + // Idempotent. Returns FAILED_PRECONDITION for a DRAINING Worker, and ABORTED + // if another write lands on the Worker first; retry. + // + // Every Actor placed after this call is stamped with the Worker's epoch as + // it stands then, so a caller raises epoch for a restarted ateom first. + MarkWorkerAvailable(ctx context.Context, in *MarkWorkerAvailableRequest, opts ...grpc.CallOption) (*MarkWorkerAvailableResponse, error) + // Mark an ACTIVE Worker UNAVAILABLE so the scheduler stops placing Actors on + // it, as while its Pod is not Ready. Idempotent. Leaves any bound Actor + // alone. Returns FAILED_PRECONDITION for a DRAINING Worker, and ABORTED if + // another write lands on the Worker first; retry. + MarkWorkerUnavailable(ctx context.Context, in *MarkWorkerUnavailableRequest, opts ...grpc.CallOption) (*MarkWorkerUnavailableResponse, error) // List the Actors hosted by a given Worker. ListWorkerActorAssignments(ctx context.Context, in *ListWorkerActorAssignmentsRequest, opts ...grpc.CallOption) (*ListWorkerActorAssignmentsResponse, error) // List Actors. @@ -451,6 +466,26 @@ func (c *controlClient) DrainWorker(ctx context.Context, in *DrainWorkerRequest, return out, nil } +func (c *controlClient) MarkWorkerAvailable(ctx context.Context, in *MarkWorkerAvailableRequest, opts ...grpc.CallOption) (*MarkWorkerAvailableResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(MarkWorkerAvailableResponse) + err := c.cc.Invoke(ctx, Control_MarkWorkerAvailable_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *controlClient) MarkWorkerUnavailable(ctx context.Context, in *MarkWorkerUnavailableRequest, opts ...grpc.CallOption) (*MarkWorkerUnavailableResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(MarkWorkerUnavailableResponse) + err := c.cc.Invoke(ctx, Control_MarkWorkerUnavailable_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + func (c *controlClient) ListWorkerActorAssignments(ctx context.Context, in *ListWorkerActorAssignmentsRequest, opts ...grpc.CallOption) (*ListWorkerActorAssignmentsResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(ListWorkerActorAssignmentsResponse) @@ -686,7 +721,8 @@ type ControlServer interface { ListWorkers(context.Context, *ListWorkersRequest) (*ListWorkersResponse, error) // Get a Worker. GetWorker(context.Context, *GetWorkerRequest) (*Worker, error) - // Register a Worker. Called once its Pod is Ready and has an IP. + // Register a Worker. Called once its Pod has an IP. A Worker is registered + // UNAVAILABLE and takes no Actors until MarkWorkerAvailable. CreateWorker(context.Context, *CreateWorkerRequest) (*Worker, error) // Update observed pool state on a Worker. UpdateWorker(context.Context, *UpdateWorkerRequest) (*Worker, error) @@ -697,6 +733,18 @@ type ControlServer interface { // it. Idempotent; one-way. Deliberately leaves any bound Actor alone. // Returns ABORTED if another write lands on the Worker first; retry. DrainWorker(context.Context, *DrainWorkerRequest) (*DrainWorkerResponse, error) + // Mark an UNAVAILABLE Worker ACTIVE so the scheduler places Actors on it. + // Idempotent. Returns FAILED_PRECONDITION for a DRAINING Worker, and ABORTED + // if another write lands on the Worker first; retry. + // + // Every Actor placed after this call is stamped with the Worker's epoch as + // it stands then, so a caller raises epoch for a restarted ateom first. + MarkWorkerAvailable(context.Context, *MarkWorkerAvailableRequest) (*MarkWorkerAvailableResponse, error) + // Mark an ACTIVE Worker UNAVAILABLE so the scheduler stops placing Actors on + // it, as while its Pod is not Ready. Idempotent. Leaves any bound Actor + // alone. Returns FAILED_PRECONDITION for a DRAINING Worker, and ABORTED if + // another write lands on the Worker first; retry. + MarkWorkerUnavailable(context.Context, *MarkWorkerUnavailableRequest) (*MarkWorkerUnavailableResponse, error) // List the Actors hosted by a given Worker. ListWorkerActorAssignments(context.Context, *ListWorkerActorAssignmentsRequest) (*ListWorkerActorAssignmentsResponse, error) // List Actors. @@ -820,6 +868,12 @@ func (UnimplementedControlServer) DeleteWorker(context.Context, *DeleteWorkerReq func (UnimplementedControlServer) DrainWorker(context.Context, *DrainWorkerRequest) (*DrainWorkerResponse, error) { return nil, status.Error(codes.Unimplemented, "method DrainWorker not implemented") } +func (UnimplementedControlServer) MarkWorkerAvailable(context.Context, *MarkWorkerAvailableRequest) (*MarkWorkerAvailableResponse, error) { + return nil, status.Error(codes.Unimplemented, "method MarkWorkerAvailable not implemented") +} +func (UnimplementedControlServer) MarkWorkerUnavailable(context.Context, *MarkWorkerUnavailableRequest) (*MarkWorkerUnavailableResponse, error) { + return nil, status.Error(codes.Unimplemented, "method MarkWorkerUnavailable not implemented") +} func (UnimplementedControlServer) ListWorkerActorAssignments(context.Context, *ListWorkerActorAssignmentsRequest) (*ListWorkerActorAssignmentsResponse, error) { return nil, status.Error(codes.Unimplemented, "method ListWorkerActorAssignments not implemented") } @@ -1342,6 +1396,42 @@ func _Control_DrainWorker_Handler(srv interface{}, ctx context.Context, dec func return interceptor(ctx, in, info, handler) } +func _Control_MarkWorkerAvailable_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(MarkWorkerAvailableRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(ControlServer).MarkWorkerAvailable(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: Control_MarkWorkerAvailable_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(ControlServer).MarkWorkerAvailable(ctx, req.(*MarkWorkerAvailableRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _Control_MarkWorkerUnavailable_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(MarkWorkerUnavailableRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(ControlServer).MarkWorkerUnavailable(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: Control_MarkWorkerUnavailable_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(ControlServer).MarkWorkerUnavailable(ctx, req.(*MarkWorkerUnavailableRequest)) + } + return interceptor(ctx, in, info, handler) +} + func _Control_ListWorkerActorAssignments_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(ListWorkerActorAssignmentsRequest) if err := dec(in); err != nil { @@ -1755,6 +1845,14 @@ var Control_ServiceDesc = grpc.ServiceDesc{ MethodName: "DrainWorker", Handler: _Control_DrainWorker_Handler, }, + { + MethodName: "MarkWorkerAvailable", + Handler: _Control_MarkWorkerAvailable_Handler, + }, + { + MethodName: "MarkWorkerUnavailable", + Handler: _Control_MarkWorkerUnavailable_Handler, + }, { MethodName: "ListWorkerActorAssignments", Handler: _Control_ListWorkerActorAssignments_Handler, @@ -1831,6 +1929,7 @@ var Control_ServiceDesc = grpc.ServiceDesc{ const ( WorkerService_RegisterWorker_FullMethodName = "/ateapi.WorkerService/RegisterWorker" WorkerService_MintAteomActorCertificate_FullMethodName = "/ateapi.WorkerService/MintAteomActorCertificate" + WorkerService_MintAteomActorJwt_FullMethodName = "/ateapi.WorkerService/MintAteomActorJwt" WorkerService_RequestActorSuspend_FullMethodName = "/ateapi.WorkerService/RequestActorSuspend" WorkerService_ReportActorSnapshotState_FullMethodName = "/ateapi.WorkerService/ReportActorSnapshotState" ) @@ -1860,6 +1959,17 @@ type WorkerServiceClient interface { // // SPIFFE URI: spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor} MintAteomActorCertificate(ctx context.Context, in *MintAteomActorCertificateRequest, opts ...grpc.CallOption) (*MintAteomActorCertificateResponse, error) + // MintAteomActorJwt creates a Substrate-issued JWT that asserts an ateom + // acting on behalf of a particular actor: the JWT counterpart of + // MintAteomActorCertificate. atelet calls it with its own client + // certificate and hands the JWT to node-side plugins, which exchange it for + // credentials scoped to that actor. + // + // The token speaks for the ateom, not the actor: its subject is + // `ateom-for-actor//` and its claims sit under + // `ateom-for-actor.ate.dev`, see MintAteomActorJwtResponse. The lifetime is + // fixed at one hour. + MintAteomActorJwt(ctx context.Context, in *MintAteomActorJwtRequest, opts ...grpc.CallOption) (*MintAteomActorJwtResponse, error) // RequestActorSuspend asks the control plane to suspend an Actor that the // calling Worker hosts. Only the Worker can observe what makes an Actor // worth reclaiming -- whether its workload still has anything to do -- and @@ -1923,6 +2033,16 @@ func (c *workerServiceClient) MintAteomActorCertificate(ctx context.Context, in return out, nil } +func (c *workerServiceClient) MintAteomActorJwt(ctx context.Context, in *MintAteomActorJwtRequest, opts ...grpc.CallOption) (*MintAteomActorJwtResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(MintAteomActorJwtResponse) + err := c.cc.Invoke(ctx, WorkerService_MintAteomActorJwt_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + func (c *workerServiceClient) RequestActorSuspend(ctx context.Context, in *RequestActorSuspendRequest, opts ...grpc.CallOption) (*RequestActorSuspendResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(RequestActorSuspendResponse) @@ -1968,6 +2088,17 @@ type WorkerServiceServer interface { // // SPIFFE URI: spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor} MintAteomActorCertificate(context.Context, *MintAteomActorCertificateRequest) (*MintAteomActorCertificateResponse, error) + // MintAteomActorJwt creates a Substrate-issued JWT that asserts an ateom + // acting on behalf of a particular actor: the JWT counterpart of + // MintAteomActorCertificate. atelet calls it with its own client + // certificate and hands the JWT to node-side plugins, which exchange it for + // credentials scoped to that actor. + // + // The token speaks for the ateom, not the actor: its subject is + // `ateom-for-actor//` and its claims sit under + // `ateom-for-actor.ate.dev`, see MintAteomActorJwtResponse. The lifetime is + // fixed at one hour. + MintAteomActorJwt(context.Context, *MintAteomActorJwtRequest) (*MintAteomActorJwtResponse, error) // RequestActorSuspend asks the control plane to suspend an Actor that the // calling Worker hosts. Only the Worker can observe what makes an Actor // worth reclaiming -- whether its workload still has anything to do -- and @@ -2017,6 +2148,9 @@ func (UnimplementedWorkerServiceServer) RegisterWorker(context.Context, *Registe func (UnimplementedWorkerServiceServer) MintAteomActorCertificate(context.Context, *MintAteomActorCertificateRequest) (*MintAteomActorCertificateResponse, error) { return nil, status.Error(codes.Unimplemented, "method MintAteomActorCertificate not implemented") } +func (UnimplementedWorkerServiceServer) MintAteomActorJwt(context.Context, *MintAteomActorJwtRequest) (*MintAteomActorJwtResponse, error) { + return nil, status.Error(codes.Unimplemented, "method MintAteomActorJwt not implemented") +} func (UnimplementedWorkerServiceServer) RequestActorSuspend(context.Context, *RequestActorSuspendRequest) (*RequestActorSuspendResponse, error) { return nil, status.Error(codes.Unimplemented, "method RequestActorSuspend not implemented") } @@ -2080,6 +2214,24 @@ func _WorkerService_MintAteomActorCertificate_Handler(srv interface{}, ctx conte return interceptor(ctx, in, info, handler) } +func _WorkerService_MintAteomActorJwt_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(MintAteomActorJwtRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(WorkerServiceServer).MintAteomActorJwt(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: WorkerService_MintAteomActorJwt_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(WorkerServiceServer).MintAteomActorJwt(ctx, req.(*MintAteomActorJwtRequest)) + } + return interceptor(ctx, in, info, handler) +} + func _WorkerService_RequestActorSuspend_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(RequestActorSuspendRequest) if err := dec(in); err != nil { @@ -2131,6 +2283,10 @@ var WorkerService_ServiceDesc = grpc.ServiceDesc{ MethodName: "MintAteomActorCertificate", Handler: _WorkerService_MintAteomActorCertificate_Handler, }, + { + MethodName: "MintAteomActorJwt", + Handler: _WorkerService_MintAteomActorJwt_Handler, + }, { MethodName: "RequestActorSuspend", Handler: _WorkerService_RequestActorSuspend_Handler, diff --git a/pkg/proto/objectstorepb/v1/objectstore.proto b/pkg/proto/objectstorepb/v1/objectstore.proto index 34ecf429cb..0f589229ea 100644 --- a/pkg/proto/objectstorepb/v1/objectstore.proto +++ b/pkg/proto/objectstorepb/v1/objectstore.proto @@ -126,8 +126,10 @@ message CopySnapshotResponse {} // every node, next to NodeProvider, to serve sandbox runtime assets (the files // a SandboxConfig names, such as a micro-VM kernel and guest image). atelet // calls it for every asset. It reads a gs:// asset anonymously only if the -// plugin returns NOT_FOUND, INVALID_ARGUMENT, UNIMPLEMENTED or UNAVAILABLE for -// it; any other failure is final. +// plugin returns NOT_FOUND, INVALID_ARGUMENT, UNIMPLEMENTED, UNAVAILABLE or +// INTERNAL for it; any other failure is final. A plugin should return INTERNAL +// only for failures it does not recognize, since atelet then tries the asset +// anonymously. // // An asset is pinned by its sha256, so the plugin hands back an object only if // the caller already knows its content. The plugin must also refuse to read diff --git a/pkg/proto/objectstorepb/v1/objectstore_grpc.pb.go b/pkg/proto/objectstorepb/v1/objectstore_grpc.pb.go index 989147ceec..4bd992a7a8 100644 --- a/pkg/proto/objectstorepb/v1/objectstore_grpc.pb.go +++ b/pkg/proto/objectstorepb/v1/objectstore_grpc.pb.go @@ -392,8 +392,10 @@ const ( // every node, next to NodeProvider, to serve sandbox runtime assets (the files // a SandboxConfig names, such as a micro-VM kernel and guest image). atelet // calls it for every asset. It reads a gs:// asset anonymously only if the -// plugin returns NOT_FOUND, INVALID_ARGUMENT, UNIMPLEMENTED or UNAVAILABLE for -// it; any other failure is final. +// plugin returns NOT_FOUND, INVALID_ARGUMENT, UNIMPLEMENTED, UNAVAILABLE or +// INTERNAL for it; any other failure is final. A plugin should return INTERNAL +// only for failures it does not recognize, since atelet then tries the asset +// anonymously. // // An asset is pinned by its sha256, so the plugin hands back an object only if // the caller already knows its content. The plugin must also refuse to read @@ -436,8 +438,10 @@ func (c *assetProviderClient) FetchAsset(ctx context.Context, in *FetchAssetRequ // every node, next to NodeProvider, to serve sandbox runtime assets (the files // a SandboxConfig names, such as a micro-VM kernel and guest image). atelet // calls it for every asset. It reads a gs:// asset anonymously only if the -// plugin returns NOT_FOUND, INVALID_ARGUMENT, UNIMPLEMENTED or UNAVAILABLE for -// it; any other failure is final. +// plugin returns NOT_FOUND, INVALID_ARGUMENT, UNIMPLEMENTED, UNAVAILABLE or +// INTERNAL for it; any other failure is final. A plugin should return INTERNAL +// only for failures it does not recognize, since atelet then tries the asset +// anonymously. // // An asset is pinned by its sha256, so the plugin hands back an object only if // the caller already knows its content. The plugin must also refuse to read diff --git a/pkg/sandboxclass/sandboxclass.go b/pkg/sandboxclass/sandboxclass.go new file mode 100644 index 0000000000..533c05a3af --- /dev/null +++ b/pkg/sandboxclass/sandboxclass.go @@ -0,0 +1,40 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package sandboxclass names the sandbox runtime families: the values a +// WorkerPool, a SandboxConfig, and an ActorTemplate can select. It imports +// nothing from Kubernetes, so any package can name a class without pulling in +// the ate.dev/v1alpha1 API types. +package sandboxclass + +import "slices" + +const ( + // GVisor is the gVisor/runsc runtime (cmd/ateom-gvisor). Default. + GVisor = "gvisor" + // MicroVM is the micro-VM runtime (cmd/ateom-microvm). It needs /dev/kvm + // and vhost devices. + MicroVM = "microvm" +) + +// All returns every sandbox class. Keep it in sync with the +// +kubebuilder:validation:Enum markers in pkg/api/v1alpha1. +func All() []string { + return []string{GVisor, MicroVM} +} + +// Valid reports whether class is one of All. The match is case-sensitive. +func Valid(class string) bool { + return slices.Contains(All(), class) +} diff --git a/pkg/sandboxclass/sandboxclass_test.go b/pkg/sandboxclass/sandboxclass_test.go new file mode 100644 index 0000000000..7ef930ca0a --- /dev/null +++ b/pkg/sandboxclass/sandboxclass_test.go @@ -0,0 +1,65 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package sandboxclass + +import "testing" + +func TestValid(t *testing.T) { + tests := []struct { + name string + class string + want bool + }{ + { + name: "gvisor", + class: GVisor, + want: true, + }, + { + name: "microvm", + class: MicroVM, + want: true, + }, + { + name: "empty", + class: "", + want: false, + }, + { + name: "unknown", + class: "kata", + want: false, + }, + { + name: "case-sensitive", + class: "GVISOR", + want: false, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := Valid(tt.class); got != tt.want { + t.Errorf("Valid(%q) = %v, want %v", tt.class, got, tt.want) + } + }) + } +} + +func TestAllReturnsCopy(t *testing.T) { + All()[0] = "mutated" + if !Valid(GVisor) { + t.Errorf("mutating the result of All changed Valid(%q)", GVisor) + } +} diff --git a/tools/apitool/exemptions.json b/tools/apitool/exemptions.json index 597a6c6bf6..1664fa468b 100644 --- a/tools/apitool/exemptions.json +++ b/tools/apitool/exemptions.json @@ -2,7 +2,7 @@ { "rule": "delete-request-shape", "subject": "ateapi.Control.DeleteActor", - "message": "field \"any_state\" is not a message, want ateapi.ObjectRef or ateapi.DeleteOptions - non-resource control fields belong in ateapi.DeleteOptions" + "message": "request field \"any_state\" is not allowed - want only \"actor\" ateapi.ObjectRef and optional \"options\" ateapi.DeleteOptions" }, { "rule": "documented", @@ -154,16 +154,6 @@ "subject": "ateapi.ExternalVolume", "message": "message has no doc comment" }, - { - "rule": "documented", - "subject": "ateapi.ExternalVolume.Status", - "message": "enum has no doc comment" - }, - { - "rule": "documented", - "subject": "ateapi.ExternalVolume.status", - "message": "field has no doc comment" - }, { "rule": "documented", "subject": "ateapi.GetActorRequest", @@ -415,13 +405,8 @@ "message": "field has no doc comment" }, { - "rule": "list-method-shape", - "subject": "ateapi.Control.ListWorkerActorAssignments", - "message": "field \"worker\" is not part of the standard List request shape - only atespace and an options field of type ateapi.ListOptions are allowed (sorting/filtering per AIP-132 is not supported)" - }, - { - "rule": "list-method-shape", - "subject": "ateapi.Control.ListWorkerActorAssignments", - "message": "repeated field is named \"actor_assignments\", want \"worker_actor_assignments\" (the plural resource name, matching the method name)" + "rule": "resource-metadata", + "subject": "ateapi.ActorAssignment", + "message": "\"metadata\" field is number 6, want 1" } ] diff --git a/tools/apitool/internal/cmd/validate.go b/tools/apitool/internal/cmd/validate.go index 75ad49358e..31e81cada2 100644 --- a/tools/apitool/internal/cmd/validate.go +++ b/tools/apitool/internal/cmd/validate.go @@ -94,6 +94,7 @@ func runValidate(cmd *cobra.Command, args []string) error { failedRules++ total += len(shown) fmt.Fprintf(out, " %s %s\n", failIcon(), r.Rule.Name) + fmt.Fprintf(out, " %s\n", r.Rule.Description) for _, f := range shown { fmt.Fprintf(out, " %s: %s\n", f.Subject, f.Message) } diff --git a/tools/apitool/internal/lint/create.go b/tools/apitool/internal/lint/create.go deleted file mode 100644 index 29d4aea922..0000000000 --- a/tools/apitool/internal/lint/create.go +++ /dev/null @@ -1,91 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -const createOptionsTypeFullName = "ateapi.CreateOptions" - -// CreateRequestShape requires a Create method's request to embed the -// resource with exactly one field, named after the resource's snake_case -// type name, plus an optional CreateOptions field for non-resource controls. -var CreateRequestShape = Rule{ - Name: "create-request-shape", - Description: "A Create method's request has exactly one field, of the resource's own type and named after its snake_case type name, plus an optional CreateOptions field - nothing else.", - Check: checkCreateRequestShape, -} - -func checkCreateRequestShape(api *model.API) ([]Finding, error) { - resources, err := model.Resources(api) - if err != nil { - return nil, err - } - messagesByName := api.MessagesByFullName() - - var findings []Finding - for _, rg := range resources { - for _, m := range rg.Methods { - verb, ok := standardVerbFor(m.Name, rg.Message.Name) - if !ok || verb != "Create" { - continue - } - subject := m.ServiceFullName + "." + m.Name - - req, ok := messagesByName[m.InputName] - if !ok { - findings = append(findings, Finding{Subject: subject, Message: "request type " + m.InputName + " not found"}) - continue - } - - var resourceFields, optionsFields int - for _, f := range req.Fields { - switch { - case f.TypeKind == "message" && f.TypeFullName == rg.Message.FullName: - resourceFields++ - if want := fieldNameForResource(rg.Message.Name); f.Name != want { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("resource field is named %q, want %q", f.Name, want), - }) - } - case f.TypeKind == "message" && f.TypeFullName == createOptionsTypeFullName: - optionsFields++ - default: - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("field %q is %s, want %s or %s - non-resource control fields belong in %s", f.Name, fieldTypeDescription(f), rg.Message.FullName, createOptionsTypeFullName, createOptionsTypeFullName), - }) - } - } - if resourceFields != 1 { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request has %d field(s) of type %s, want exactly 1 embedding the resource", resourceFields, rg.Message.FullName), - }) - } - if optionsFields > 1 { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request has %d field(s) of type %s, want at most 1", optionsFields, createOptionsTypeFullName), - }) - } - } - } - return findings, nil -} diff --git a/tools/apitool/internal/lint/create_test.go b/tools/apitool/internal/lint/create_test.go deleted file mode 100644 index 029b15cc57..0000000000 --- a/tools/apitool/internal/lint/create_test.go +++ /dev/null @@ -1,67 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -func TestCreateRequestShape(t *testing.T) { - resourceField := model.Field{Name: "atespace", TypeKind: "message", TypeFullName: "test.Atespace"} - misnamedResourceField := model.Field{Name: "space", TypeKind: "message", TypeFullName: "test.Atespace"} - optionsField := model.Field{Name: "options", TypeKind: "message", TypeFullName: "ateapi.CreateOptions"} - - tests := []struct { - name string - reqFields []model.Field - wantFinding bool - }{ - {"resource field alone", []model.Field{resourceField}, false}, - {"resource field plus CreateOptions", []model.Field{resourceField, optionsField}, false}, - {"no fields", nil, true}, - {"two resource fields", []model.Field{resourceField, resourceField}, true}, - {"two CreateOptions fields", []model.Field{resourceField, optionsField, optionsField}, true}, - {"loose control field outside CreateOptions", []model.Field{resourceField, {Name: "dry_run"}}, true}, - {"resource field, wrong name", []model.Field{misnamedResourceField}, true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.CreateRequestShape.Check(createMethodAPI(tt.reqFields)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} - -// TestCreateRequestShape_IgnoresGetDelete confirms the rule ignores -// Get/Delete, which reference the resource via ObjectRef instead. -func TestCreateRequestShape_IgnoresGetDelete(t *testing.T) { - findings, err := lint.CreateRequestShape.Check(deleteMethodAPI([]model.Field{ - {Name: "atespace", TypeKind: "message", TypeFullName: "ateapi.ObjectRef"}, - })) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if len(findings) != 0 { - t.Errorf("Check() = %+v, want no findings for a Delete method", findings) - } -} diff --git a/tools/apitool/internal/lint/delete.go b/tools/apitool/internal/lint/delete.go deleted file mode 100644 index 246fc94890..0000000000 --- a/tools/apitool/internal/lint/delete.go +++ /dev/null @@ -1,135 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -const deleteOptionsTypeFullName = "ateapi.DeleteOptions" - -// DeleteRequestShape requires a Delete method's request to identify the -// resource with exactly one ObjectRef field, named after the resource's -// snake_case type name, plus an optional DeleteOptions field for -// preconditions. -var DeleteRequestShape = Rule{ - Name: "delete-request-shape", - Description: "A Delete method's request has exactly one ObjectRef field, named after the resource's snake_case type name, plus an optional DeleteOptions field - nothing else.", - Check: checkDeleteRequestShape, -} - -func checkDeleteRequestShape(api *model.API) ([]Finding, error) { - resources, err := model.Resources(api) - if err != nil { - return nil, err - } - messagesByName := api.MessagesByFullName() - - var findings []Finding - for _, rg := range resources { - for _, m := range rg.Methods { - verb, ok := standardVerbFor(m.Name, rg.Message.Name) - if !ok || verb != "Delete" { - continue - } - subject := m.ServiceFullName + "." + m.Name - - req, ok := messagesByName[m.InputName] - if !ok { - findings = append(findings, Finding{Subject: subject, Message: "request type " + m.InputName + " not found"}) - continue - } - - var objectRefFields, optionsFields int - for _, f := range req.Fields { - switch { - case f.TypeKind == "message" && f.TypeFullName == objectRefTypeFullName: - objectRefFields++ - if want := fieldNameForResource(rg.Message.Name); f.Name != want { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("resource field is named %q, want %q", f.Name, want), - }) - } - case f.TypeKind == "message" && f.TypeFullName == deleteOptionsTypeFullName: - optionsFields++ - default: - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("field %q is %s, want %s or %s - non-resource control fields belong in %s", f.Name, fieldTypeDescription(f), objectRefTypeFullName, deleteOptionsTypeFullName, deleteOptionsTypeFullName), - }) - } - } - if objectRefFields != 1 { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request has %d field(s) of type %s, want exactly 1 identifying the resource", objectRefFields, objectRefTypeFullName), - }) - } - if optionsFields > 1 { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request has %d field(s) of type %s, want at most 1", optionsFields, deleteOptionsTypeFullName), - }) - } - } - } - return findings, nil -} - -// DeleteOptionsShape requires the shared ateapi.DeleteOptions message to -// have only its two documented precondition fields: version (int64) and -// uid (string) - nothing else. Presence of either field is not required; -// this rule only bounds what's allowed. -var DeleteOptionsShape = Rule{ - Name: "delete-options-shape", - Description: "The shared ateapi.DeleteOptions message has only version (int64) and uid (string) fields - nothing else.", - Check: checkDeleteOptionsShape, -} - -func checkDeleteOptionsShape(api *model.API) ([]Finding, error) { - opts, ok := api.MessagesByFullName()[deleteOptionsTypeFullName] - if !ok { - return nil, nil // DeleteOptions isn't declared in this API - nothing to check. - } - - var findings []Finding - for _, f := range opts.Fields { - switch f.Name { - case "version": - if f.TypeDisplay != "int64" { - findings = append(findings, Finding{ - Subject: opts.FullName, - Message: fmt.Sprintf("%q field is %s, want int64", f.Name, f.TypeDisplay), - }) - } - case "uid": - if f.TypeDisplay != "string" { - findings = append(findings, Finding{ - Subject: opts.FullName, - Message: fmt.Sprintf("%q field is %s, want string", f.Name, f.TypeDisplay), - }) - } - default: - findings = append(findings, Finding{ - Subject: opts.FullName, - Message: fmt.Sprintf("field %q is not part of DeleteOptions - only version (int64) and uid (string) are allowed", f.Name), - }) - } - } - return findings, nil -} diff --git a/tools/apitool/internal/lint/delete_test.go b/tools/apitool/internal/lint/delete_test.go deleted file mode 100644 index 8c42afda64..0000000000 --- a/tools/apitool/internal/lint/delete_test.go +++ /dev/null @@ -1,117 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -func TestDeleteRequestShape(t *testing.T) { - objectRefField := model.Field{Name: "atespace", TypeKind: "message", TypeFullName: "ateapi.ObjectRef"} - misnamedObjectRefField := model.Field{Name: "space", TypeKind: "message", TypeFullName: "ateapi.ObjectRef"} - optionsField := model.Field{Name: "options", TypeKind: "message", TypeFullName: "ateapi.DeleteOptions"} - - tests := []struct { - name string - reqFields []model.Field - wantFinding bool - }{ - {"ObjectRef alone", []model.Field{objectRefField}, false}, - {"ObjectRef plus DeleteOptions", []model.Field{objectRefField, optionsField}, false}, - {"no fields", nil, true}, - {"two ObjectRef fields", []model.Field{objectRefField, objectRefField}, true}, - {"two DeleteOptions fields", []model.Field{objectRefField, optionsField, optionsField}, true}, - {"loose control field outside DeleteOptions", []model.Field{objectRefField, {Name: "dry_run"}}, true}, - {"ObjectRef field, wrong name", []model.Field{misnamedObjectRefField}, true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.DeleteRequestShape.Check(deleteMethodAPI(tt.reqFields)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} - -// TestDeleteRequestShape_IgnoresCreateUpdate confirms the rule ignores -// Create/Update, which legitimately embed the full resource. -func TestDeleteRequestShape_IgnoresCreateUpdate(t *testing.T) { - findings, err := lint.DeleteRequestShape.Check(createAtespaceAPI()) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if len(findings) != 0 { - t.Errorf("Check() = %+v, want no findings for a Create method", findings) - } -} - -// deleteOptionsAPI builds a minimal API with just the shared -// ateapi.DeleteOptions message, with the given fields. -func deleteOptionsAPI(fields []model.Field) *model.API { - return &model.API{ - Messages: []model.Message{ - {FullName: "ateapi.DeleteOptions", Name: "DeleteOptions", Fields: fields}, - }, - } -} - -func TestDeleteOptionsShape(t *testing.T) { - version := model.Field{Name: "version", TypeDisplay: "int64"} - uid := model.Field{Name: "uid", TypeDisplay: "string"} - - tests := []struct { - name string - fields []model.Field - wantFinding bool - }{ - {"version and uid", []model.Field{version, uid}, false}, - {"version alone - presence not required", []model.Field{version}, false}, - {"uid alone - presence not required", []model.Field{uid}, false}, - {"no fields at all", nil, false}, - {"version wrong type", []model.Field{{Name: "version", TypeDisplay: "string"}, uid}, true}, - {"uid wrong type", []model.Field{version, {Name: "uid", TypeDisplay: "int64"}}, true}, - {"extra field beyond version/uid", []model.Field{version, uid, {Name: "dry_run", TypeDisplay: "bool"}}, true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.DeleteOptionsShape.Check(deleteOptionsAPI(tt.fields)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} - -// TestDeleteOptionsShape_NotDeclared confirms the rule is a no-op when the -// API doesn't declare ateapi.DeleteOptions at all. -func TestDeleteOptionsShape_NotDeclared(t *testing.T) { - findings, err := lint.DeleteOptionsShape.Check(&model.API{}) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if len(findings) != 0 { - t.Errorf("Check() = %+v, want no findings when DeleteOptions isn't declared", findings) - } -} diff --git a/tools/apitool/internal/lint/documented.go b/tools/apitool/internal/lint/documented.go index 8dc4036914..8ed3298c51 100644 --- a/tools/apitool/internal/lint/documented.go +++ b/tools/apitool/internal/lint/documented.go @@ -20,39 +20,43 @@ import ( "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) -// Documented requires every message, enum, message field, and RPC method -// to have a doc comment. var Documented = Rule{ Name: "documented", Description: "Every message, enum, message field, and RPC method has a doc comment.", - Check: checkDocumented, + Check: allOf( + forEachMessage(checkMessageDocumented), + forEachField(checkFieldDocumented), + forEachEnum(checkEnumDocumented), + forEachMethod(checkMethodDocumented), + ), } -func checkDocumented(api *model.API) ([]Finding, error) { - var findings []Finding - for _, m := range api.Messages { - if docText(m.Comment) == "" { - findings = append(findings, Finding{Subject: m.FullName, Message: "message has no doc comment"}) - } - for _, f := range m.Fields { - if docText(f.Comment) == "" { - findings = append(findings, Finding{Subject: m.FullName + "." + f.Name, Message: "field has no doc comment"}) - } - } +func checkMessageDocumented(m model.Message) []Finding { + if docText(m.Comment) == "" { + return []Finding{findingf(m.FullName, "message has no doc comment")} } - for _, e := range api.Enums { - if docText(e.Comment) == "" { - findings = append(findings, Finding{Subject: e.FullName, Message: "enum has no doc comment"}) - } + return nil +} + +func checkFieldDocumented(m model.Message, f model.Field) []Finding { + if docText(f.Comment) == "" { + return []Finding{findingf(m.FullName.Append(f.Name), "field has no doc comment")} } - for _, svc := range api.Services { - for _, method := range svc.Methods { - if docText(method.Comment) == "" { - findings = append(findings, Finding{Subject: method.ServiceFullName + "." + method.Name, Message: "method has no doc comment"}) - } - } + return nil +} + +func checkEnumDocumented(e model.Enum) []Finding { + if docText(e.Comment) == "" { + return []Finding{findingf(e.FullName, "enum has no doc comment")} + } + return nil +} + +func checkMethodDocumented(rpc model.Method) []Finding { + if docText(rpc.Comment) == "" { + return []Finding{findingf(subjectOf(rpc), "method has no doc comment")} } - return findings, nil + return nil } func docText(comment string) string { diff --git a/tools/apitool/internal/lint/documented_test.go b/tools/apitool/internal/lint/documented_test.go index 78fdc9b965..5896c171c0 100644 --- a/tools/apitool/internal/lint/documented_test.go +++ b/tools/apitool/internal/lint/documented_test.go @@ -17,6 +17,8 @@ package lint_test import ( "testing" + "github.com/google/go-cmp/cmp" + "github.com/agent-substrate/substrate/tools/apitool/internal/lint" "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) @@ -28,18 +30,110 @@ func TestDocumented(t *testing.T) { enumComment string fieldComment string methodComment string - wantFindings int + want []lint.Finding }{ - {"all documented", "a request", "a state", "does a thing", "does another thing", 0}, - {"message undocumented", "", "a state", "does a thing", "does another thing", 1}, - {"enum undocumented", "a request", "", "does a thing", "does another thing", 1}, - {"field undocumented", "a request", "a state", "", "does another thing", 1}, - {"method undocumented", "a request", "a state", "does a thing", "", 1}, - {"nothing documented", "", "", "", "", 4}, - {"whitespace-only comment counts as undocumented", "\t", "\t", " ", "\n", 4}, - {"empty comment lines", "a request", "a state", "\n\n", "does another thing", 1}, - {"tags only", "+k8s:required", "+k8s:optional", "+k8s:required\n+k8s:format=k8s-short-name", "does another thing", 3}, - {"prose and tags", "a request\n\n+k8s:required", "a state", "does a thing\n\n+k8s:immutable", "does another thing", 0}, + { + name: "all documented", + messageComment: "a request", + enumComment: "a state", + fieldComment: "does a thing", + methodComment: "does another thing", + }, + { + name: "message undocumented", + messageComment: "", + enumComment: "a state", + fieldComment: "does a thing", + methodComment: "does another thing", + want: []lint.Finding{ + {Subject: "test.DoThingRequest", Message: "message has no doc comment"}, + }, + }, + { + name: "enum undocumented", + messageComment: "a request", + enumComment: "", + fieldComment: "does a thing", + methodComment: "does another thing", + want: []lint.Finding{ + {Subject: "test.Widget.State", Message: "enum has no doc comment"}, + }, + }, + { + name: "field undocumented", + messageComment: "a request", + enumComment: "a state", + fieldComment: "", + methodComment: "does another thing", + want: []lint.Finding{ + {Subject: "test.DoThingRequest.widget", Message: "field has no doc comment"}, + }, + }, + { + name: "method undocumented", + messageComment: "a request", + enumComment: "a state", + fieldComment: "does a thing", + methodComment: "", + want: []lint.Finding{ + {Subject: "test.Control.DoThing", Message: "method has no doc comment"}, + }, + }, + { + name: "nothing documented", + messageComment: "", + enumComment: "", + fieldComment: "", + methodComment: "", + want: []lint.Finding{ + {Subject: "test.DoThingRequest", Message: "message has no doc comment"}, + {Subject: "test.DoThingRequest.widget", Message: "field has no doc comment"}, + {Subject: "test.Widget.State", Message: "enum has no doc comment"}, + {Subject: "test.Control.DoThing", Message: "method has no doc comment"}, + }, + }, + { + name: "whitespace-only comment counts as undocumented", + messageComment: "\t", + enumComment: "\t", + fieldComment: " ", + methodComment: "\n", + want: []lint.Finding{ + {Subject: "test.DoThingRequest", Message: "message has no doc comment"}, + {Subject: "test.DoThingRequest.widget", Message: "field has no doc comment"}, + {Subject: "test.Widget.State", Message: "enum has no doc comment"}, + {Subject: "test.Control.DoThing", Message: "method has no doc comment"}, + }, + }, + { + name: "empty comment lines", + messageComment: "a request", + enumComment: "a state", + fieldComment: "\n\n", + methodComment: "does another thing", + want: []lint.Finding{ + {Subject: "test.DoThingRequest.widget", Message: "field has no doc comment"}, + }, + }, + { + name: "tags only", + messageComment: "+k8s:required", + enumComment: "+k8s:optional", + fieldComment: "+k8s:required\n+k8s:format=k8s-short-name", + methodComment: "does another thing", + want: []lint.Finding{ + {Subject: "test.DoThingRequest", Message: "message has no doc comment"}, + {Subject: "test.DoThingRequest.widget", Message: "field has no doc comment"}, + {Subject: "test.Widget.State", Message: "enum has no doc comment"}, + }, + }, + { + name: "prose and tags", + messageComment: "a request\n\n+k8s:required", + enumComment: "a state", + fieldComment: "does a thing\n\n+k8s:immutable", + methodComment: "does another thing", + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -47,7 +141,7 @@ func TestDocumented(t *testing.T) { Services: []model.Service{{ Name: "Control", Methods: []model.Method{ - {Name: "DoThing", ServiceName: "Control", Comment: tt.methodComment, InputName: "test.DoThingRequest", OutputName: "test.DoThingResponse"}, + {Name: "DoThing", ServiceFullName: "test.Control", ServiceName: "Control", Comment: tt.methodComment, InputName: "test.DoThingRequest", OutputName: "test.DoThingResponse"}, }, }}, Messages: []model.Message{ @@ -63,8 +157,8 @@ func TestDocumented(t *testing.T) { if err != nil { t.Fatalf("Check() error = %v", err) } - if len(findings) != tt.wantFindings { - t.Errorf("Check() = %+v, want %d finding(s)", findings, tt.wantFindings) + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) } }) } diff --git a/tools/apitool/internal/lint/enum_naming.go b/tools/apitool/internal/lint/enum_naming.go deleted file mode 100644 index 5603adb6c0..0000000000 --- a/tools/apitool/internal/lint/enum_naming.go +++ /dev/null @@ -1,79 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - "strings" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -// EnumZeroValueUnspecified requires every enum's zero value to be named -// "{EnumName}_UNSPECIFIED". -var EnumZeroValueUnspecified = Rule{ - Name: "enum-zero-value-unspecified", - Description: `Every enum's zero value is named "{EnumName}_UNSPECIFIED".`, - Check: checkEnumZeroValueUnspecified, -} - -// EnumValuesPrefixed requires every enum's values - top-level or nested - -// to be prefixed with the enum's own name. -var EnumValuesPrefixed = Rule{ - Name: "enum-values-prefixed", - Description: `Every enum's values are prefixed with "{ENUM_NAME}_".`, - Check: checkEnumValuesPrefixed, -} - -func checkEnumZeroValueUnspecified(api *model.API) ([]Finding, error) { - var findings []Finding - for _, e := range api.Enums { - want := strings.ToUpper(snakeCase(lastNameSegment(e.Name))) + "_UNSPECIFIED" - zero := e.ValueByNumber(0) - switch { - case zero == nil: - findings = append(findings, Finding{Subject: e.FullName, Message: "has no value numbered 0"}) - case zero.Name != want: - findings = append(findings, Finding{ - Subject: e.FullName, - Message: fmt.Sprintf("zero value is named %q, want %q", zero.Name, want), - }) - } - } - return findings, nil -} - -func checkEnumValuesPrefixed(api *model.API) ([]Finding, error) { - var findings []Finding - for _, e := range api.Enums { - prefix := strings.ToUpper(snakeCase(lastNameSegment(e.Name))) + "_" - for _, v := range e.Values { - if !strings.HasPrefix(v.Name, prefix) { - findings = append(findings, Finding{ - Subject: e.FullName + "." + v.Name, - Message: fmt.Sprintf("not prefixed with %q", prefix), - }) - } - } - } - return findings, nil -} - -func lastNameSegment(name string) string { - if i := strings.LastIndex(name, "."); i >= 0 { - return name[i+1:] - } - return name -} diff --git a/tools/apitool/internal/lint/enum_naming_test.go b/tools/apitool/internal/lint/enum_naming_test.go deleted file mode 100644 index d849122cda..0000000000 --- a/tools/apitool/internal/lint/enum_naming_test.go +++ /dev/null @@ -1,128 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -func enumAPI(enums []model.Enum) *model.API { - return &model.API{Enums: enums} -} - -func TestEnumZeroValueUnspecified(t *testing.T) { - tests := []struct { - name string - enum model.Enum - wantFinding bool - }{ - { - name: "compliant top-level", - enum: model.Enum{FullName: "test.ActorState", Name: "ActorState", Values: []model.EnumValue{ - {Name: "ACTOR_STATE_UNSPECIFIED", Number: 0}, - {Name: "ACTOR_STATE_RUNNING", Number: 1}, - }}, - wantFinding: false, - }, - { - name: "compliant nested", - enum: model.Enum{FullName: "test.Worker.State", Name: "Worker.State", ParentFullName: "test.Worker", Values: []model.EnumValue{ - {Name: "STATE_UNSPECIFIED", Number: 0}, - {Name: "STATE_ACTIVE", Number: 1}, - }}, - wantFinding: false, - }, - { - name: "wrong zero-value name", - enum: model.Enum{FullName: "test.ActorState", Name: "ActorState", Values: []model.EnumValue{ - {Name: "UNKNOWN", Number: 0}, - }}, - wantFinding: true, - }, - { - name: "no zero value at all", - enum: model.Enum{FullName: "test.ActorState", Name: "ActorState", Values: []model.EnumValue{ - {Name: "ACTOR_STATE_RUNNING", Number: 1}, - }}, - wantFinding: true, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.EnumZeroValueUnspecified.Check(enumAPI([]model.Enum{tt.enum})) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} - -func TestEnumValuesPrefixed(t *testing.T) { - tests := []struct { - name string - enum model.Enum - wantFinding bool - }{ - { - name: "compliant top-level", - enum: model.Enum{FullName: "test.SandboxClass", Name: "SandboxClass", Values: []model.EnumValue{ - {Name: "SANDBOX_CLASS_UNSPECIFIED", Number: 0}, - {Name: "SANDBOX_CLASS_GVISOR", Number: 1}, - }}, - wantFinding: false, - }, - { - name: "unprefixed top-level value", - enum: model.Enum{FullName: "test.SandboxClass", Name: "SandboxClass", Values: []model.EnumValue{ - {Name: "UNSPECIFIED", Number: 0}, - {Name: "GVISOR", Number: 1}, - }}, - wantFinding: true, - }, - { - name: "compliant nested", - enum: model.Enum{FullName: "test.Worker.State", Name: "Worker.State", ParentFullName: "test.Worker", Values: []model.EnumValue{ - {Name: "STATE_UNSPECIFIED", Number: 0}, - {Name: "STATE_ACTIVE", Number: 1}, - }}, - wantFinding: false, - }, - { - name: "unprefixed nested value", - enum: model.Enum{FullName: "test.Worker.State", Name: "Worker.State", ParentFullName: "test.Worker", Values: []model.EnumValue{ - {Name: "UNSPECIFIED", Number: 0}, - {Name: "ACTIVE", Number: 1}, - }}, - wantFinding: true, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.EnumValuesPrefixed.Check(enumAPI([]model.Enum{tt.enum})) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} diff --git a/tools/apitool/internal/lint/fields.go b/tools/apitool/internal/lint/fields.go new file mode 100644 index 0000000000..13b73b6040 --- /dev/null +++ b/tools/apitool/internal/lint/fields.go @@ -0,0 +1,174 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package lint + +import ( + "fmt" + "slices" + "strings" + + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/agent-substrate/substrate/tools/apitool/internal/model" +) + +// typeMatcher reports whether a field has the wanted type, including +// whether it's repeated, and names that type in findings. +type typeMatcher struct { + // label names the type in findings. For example, "string" or + // "ateapi.ObjectRef". + label string + matches func(model.Field) bool +} + +func scalar(kind protoreflect.Kind) typeMatcher { + return typeMatcher{kind.String(), func(f model.Field) bool { + return !f.Repeated && f.Kind == kind + }} +} + +func message(typeFullName protoreflect.FullName) typeMatcher { + return typeMatcher{string(typeFullName), func(f model.Field) bool { + return isMessageOf(f, typeFullName) + }} +} + +func repeatedMessage(typeFullName protoreflect.FullName) typeMatcher { + return typeMatcher{"repeated " + string(typeFullName), func(f model.Field) bool { + return f.Repeated && f.Kind == protoreflect.MessageKind && f.TypeFullName == typeFullName + }} +} + +// wantField is one field a message should have. +type wantField struct { + // name is the field's wanted name. + name string + typ typeMatcher + // optional lets the field be absent. + optional bool +} + +// wantFields is some of the fields a message should have. +type wantFields []wantField + +// required wants exactly one field named name, of type t. +func required(name string, t typeMatcher) wantFields { + return wantFields{{name: name, typ: t}} +} + +// optional wants at most one field named name, of type t. +func optional(name string, t typeMatcher) wantFields { + return wantFields{{name: name, typ: t, optional: true}} +} + +func (w wantField) String() string { + var b strings.Builder + if w.optional { + b.WriteString("optional ") + } + fmt.Fprintf(&b, "%q %s", w.name, w.typ.label) + return b.String() +} + +// checkFields checks that msg has exactly the fields want lists, and +// nothing else. role names msg in findings, such as "request". +func checkFields(subject protoreflect.FullName, role string, msg model.Message, want ...wantFields) []Finding { + all := slices.Concat(want...) + + var findings []Finding + counts := make([]int, len(all)) + assigned := make([]bool, len(msg.Fields)) + + // A field with a wanted name is that wantField's, whatever its type. + for fi, f := range msg.Fields { + for wi, w := range all { + if w.name == string(f.Name) { + assigned[fi] = true + counts[wi]++ + if !w.typ.matches(f) { + findings = append(findings, findingf(subject, "%s field %q is %s, want %s", role, f.Name, typeOf(f), w.typ.label)) + } + break + } + } + } + for fi, f := range msg.Fields { + if assigned[fi] { + continue + } + if wi := matchByType(all, counts, f); wi == -1 { + findings = append(findings, findingf(subject, "%s field %q is not allowed - want only %s", role, f.Name, joinAnd(wantStrings(all)))) + } else { + counts[wi]++ + findings = append(findings, findingf(subject, "%s field %q is %s, want it named %q", role, f.Name, typeOf(f), all[wi].name)) + } + } + + for wi, w := range all { + switch { + case counts[wi] == 0 && !w.optional: + findings = append(findings, findingf(subject, "%s has no %s field", role, w)) + case counts[wi] > 1: + findings = append(findings, findingf(subject, "%s has %d %s fields, want at most 1", role, counts[wi], strings.TrimPrefix(w.String(), "optional "))) + } + } + return findings +} + +// matchByType returns the index of the wantField f is misnamed for: one of +// f's type that no field has filled yet. -1 if there's none. +func matchByType(want wantFields, counts []int, f model.Field) int { + for wi, w := range want { + if counts[wi] == 0 && w.typ.matches(f) { + return wi + } + } + return -1 +} + +func wantStrings(want wantFields) []string { + out := make([]string, len(want)) + for i, w := range want { + out[i] = w.String() + } + return out +} + +// typeOf describes f's type in findings. For example, "string", +// "ateapi.ObjectRef", or "repeated ateapi.Actor". +func typeOf(f model.Field) string { + if f.TypeFullName == "" { + return f.TypeDisplay + } + if f.Repeated { + return "repeated " + string(f.TypeFullName) + } + return string(f.TypeFullName) +} + +// joinAnd joins names as an English list. For example, "a and b", or +// "a, b, and c". +func joinAnd(names []string) string { + if len(names) <= 2 { + return strings.Join(names, " and ") + } + return strings.Join(names[:len(names)-1], ", ") + ", and " + names[len(names)-1] +} + +// isMessageOf reports whether f is a singular field of the message type +// typeFullName. +func isMessageOf(f model.Field, typeFullName protoreflect.FullName) bool { + return !f.Repeated && f.Kind == protoreflect.MessageKind && f.TypeFullName == typeFullName +} diff --git a/tools/apitool/internal/lint/fields_test.go b/tools/apitool/internal/lint/fields_test.go new file mode 100644 index 0000000000..dc8fda8f54 --- /dev/null +++ b/tools/apitool/internal/lint/fields_test.go @@ -0,0 +1,121 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package lint + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/agent-substrate/substrate/tools/apitool/internal/model" +) + +func TestCheckFields(t *testing.T) { + ref := model.Field{Name: "gadget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + name := model.Field{Name: "name", Kind: protoreflect.StringKind, TypeDisplay: "string"} + opts := model.Field{Name: "options", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.DeleteOptions"} + want := []wantFields{ + required("gadget", message("ateapi.ObjectRef")), + required("name", scalar(protoreflect.StringKind)), + optional("options", message("ateapi.DeleteOptions")), + } + + tests := []struct { + name string + fields []model.Field + want []Finding + }{ + { + name: "exactly the wanted fields", + fields: []model.Field{ref, name, opts}, + }, + { + name: "optional field absent", + fields: []model.Field{ref, name}, + }, + { + name: "required field missing", + fields: []model.Field{ref}, + want: []Finding{ + {Subject: "s", Message: `request has no "name" string field`}, + }, + }, + { + name: "named field of the wrong type", + fields: []model.Field{ref, {Name: "name", Kind: protoreflect.Int64Kind, TypeDisplay: "int64"}}, + want: []Finding{ + {Subject: "s", Message: `request field "name" is int64, want string`}, + }, + }, + { + name: "field of a required type, misnamed", + fields: []model.Field{{Name: "parent", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"}, name}, + want: []Finding{ + {Subject: "s", Message: `request field "parent" is ateapi.ObjectRef, want it named "gadget"`}, + }, + }, + { + name: "field of an already filled wanted field's type is stray", + fields: []model.Field{ref, name, {Name: "other", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"}}, + want: []Finding{ + {Subject: "s", Message: `request field "other" is not allowed - want only "gadget" ateapi.ObjectRef, "name" string, and optional "options" ateapi.DeleteOptions`}, + }, + }, + { + name: "stray field", + fields: []model.Field{ref, name, {Name: "dry_run", Kind: protoreflect.BoolKind, TypeDisplay: "bool"}}, + want: []Finding{ + {Subject: "s", Message: `request field "dry_run" is not allowed - want only "gadget" ateapi.ObjectRef, "name" string, and optional "options" ateapi.DeleteOptions`}, + }, + }, + { + name: "optional field twice", + fields: []model.Field{ref, name, opts, opts}, + want: []Finding{ + {Subject: "s", Message: `request has 2 "options" ateapi.DeleteOptions fields, want at most 1`}, + }, + }, + { + name: "named field twice", + fields: []model.Field{ref, ref, name}, + want: []Finding{ + {Subject: "s", Message: `request has 2 "gadget" ateapi.ObjectRef fields, want at most 1`}, + }, + }, + { + name: "repeated field where a singular message is wanted", + fields: []model.Field{{Name: "gadget", Repeated: true, Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"}, name}, + want: []Finding{ + {Subject: "s", Message: `request field "gadget" is repeated ateapi.ObjectRef, want ateapi.ObjectRef`}, + }, + }, + { + name: "repeated field where a singular scalar is wanted", + fields: []model.Field{ref, {Name: "name", Repeated: true, Kind: protoreflect.StringKind, TypeDisplay: "repeated string"}}, + want: []Finding{ + {Subject: "s", Message: `request field "name" is repeated string, want string`}, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := checkFields("s", "request", model.Message{Fields: tt.fields}, want...) + if diff := cmp.Diff(tt.want, got); diff != "" { + t.Errorf("checkFields() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/tools/apitool/internal/lint/fixtures_test.go b/tools/apitool/internal/lint/fixtures_test.go index a2bc7090bc..8675de2e5f 100644 --- a/tools/apitool/internal/lint/fixtures_test.go +++ b/tools/apitool/internal/lint/fixtures_test.go @@ -12,92 +12,75 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Fixture builders shared by more than one verb's test file - e.g. a -// Create fixture used both by create_test.go and by another verb's test -// confirming it ignores Create methods. +// Builders for the APIs rule tests check. They declare every message in +// package "test", and every method on service "test.Control". package lint_test import ( + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) -// standardMethodAPI builds a minimal API with one Control.GetAtespace -// method (see resourceAPI's comment) whose response is outputName, -// targeting an "Atespace" resource with the given fields. -func standardMethodAPI(outputName string, atespaceFields []model.Field) *model.API { +// methodAPI builds an API whose test.Control service has the one method +// rpc, plus messages. +func methodAPI(rpc model.Method, messages ...model.Message) *model.API { + rpc.ServiceFullName = "test.Control" + rpc.ServiceName = "Control" return &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "GetAtespace", ServiceName: "Control", InputName: "test.GetAtespaceRequest", OutputName: outputName}, - }, - }}, - Messages: []model.Message{ - {FullName: "test.Atespace", Name: "Atespace", Fields: atespaceFields}, - {FullName: "test.GetAtespaceRequest", Name: "GetAtespaceRequest", Fields: []model.Field{ - {Name: "atespace", TypeKind: "message", TypeFullName: "ateapi.ObjectRef"}, - }}, - {FullName: "test.GetAtespaceResponse", Name: "GetAtespaceResponse"}, - }, + Services: []model.Service{{Name: "Control", Methods: []model.Method{rpc}}}, + Messages: messages, } } -// deleteMethodAPI builds a minimal API with one Control.DeleteAtespace -// method targeting an "Atespace" resource, with the given request fields. -func deleteMethodAPI(reqFields []model.Field) *model.API { - return &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "DeleteAtespace", ServiceName: "Control", InputName: "test.DeleteAtespaceRequest", OutputName: "test.Atespace"}, - }, - }}, - Messages: []model.Message{ - {FullName: "test.Atespace", Name: "Atespace"}, - {FullName: "test.DeleteAtespaceRequest", Name: "DeleteAtespaceRequest", Fields: reqFields}, - }, - } +// messagesAPI builds an API with just messages. +func messagesAPI(messages ...model.Message) *model.API { + return &model.API{Messages: messages} } -// createAtespaceAPI builds a minimal API with one Control.CreateAtespace -// method whose request embeds the resource (not an ObjectRef) - used to -// confirm Get/Delete-only rules ignore Create/Update. -func createAtespaceAPI() *model.API { - return createMethodAPI([]model.Field{ - {Name: "atespace", TypeKind: "message", TypeFullName: "test.Atespace"}, - }) +// rpc returns a method named name, annotated with resource, taking +// test.{name}Request and returning test.{output}. +func rpc(name, resource, output string) model.Method { + return model.Method{Name: protoreflect.Name(name), Resource: resource, InputName: protoreflect.FullName("test." + name + "Request"), OutputName: protoreflect.FullName("test." + output)} } -// createMethodAPI builds a minimal API with one Control.CreateAtespace -// method targeting an "Atespace" resource, with the given request fields. -func createMethodAPI(reqFields []model.Field) *model.API { - return &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "CreateAtespace", ServiceName: "Control", InputName: "test.CreateAtespaceRequest", OutputName: "test.Atespace"}, - }, - }}, - Messages: []model.Message{ - {FullName: "test.Atespace", Name: "Atespace"}, - {FullName: "test.CreateAtespaceRequest", Name: "CreateAtespaceRequest", Fields: reqFields}, - }, +// resource returns a resource message test.{name}, with the given parents +// and fields, and plural {name}s. A top-level one, with no parents, is +// atespace-scoped. +func resource(name string, parents []string, fields ...model.Field) model.Message { + a := &model.ResourceAnnotation{Parents: parents, Plural: name + "s"} + if len(parents) == 0 { + a.Scope = ateapipb.Scope_SCOPE_ATESPACE } + return model.Message{FullName: protoreflect.FullName("test." + name), Name: name, Fields: fields, ResourceAnnotation: a} } -// updateMethodAPI builds a minimal API with one Control.UpdateAtespace -// method targeting an "Atespace" resource, with the given request fields. -func updateMethodAPI(reqFields []model.Field) *model.API { - return &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "UpdateAtespace", ServiceName: "Control", InputName: "test.UpdateAtespaceRequest", OutputName: "test.Atespace"}, - }, - }}, - Messages: []model.Message{ - {FullName: "test.Atespace", Name: "Atespace"}, - {FullName: "test.UpdateAtespaceRequest", Name: "UpdateAtespaceRequest", Fields: reqFields}, - }, - } +// globalResource returns a global-scoped, top-level resource message +// test.{name}, with the given fields. +func globalResource(name string, fields ...model.Field) model.Message { + m := resource(name, nil, fields...) + m.ResourceAnnotation.Scope = ateapipb.Scope_SCOPE_GLOBAL + return m +} + +// msg returns a message test.{name} with the given fields. +func msg(name string, fields ...model.Field) model.Message { + return model.Message{FullName: protoreflect.FullName("test." + name), Name: name, Fields: fields} +} + +// singletonResource returns a singleton resource message test.{name}, with +// the given parents. +func singletonResource(name string, parents []string) model.Message { + m := resource(name, parents) + m.ResourceAnnotation.Singleton = true + return m +} + +// pluralResource returns a resource message test.{name}, with the given +// plural and parents. +func pluralResource(name, plural string, parents []string) model.Message { + m := resource(name, parents) + m.ResourceAnnotation.Plural = plural + return m } diff --git a/tools/apitool/internal/lint/get.go b/tools/apitool/internal/lint/get.go deleted file mode 100644 index 82fe631f3f..0000000000 --- a/tools/apitool/internal/lint/get.go +++ /dev/null @@ -1,77 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -// GetRequestSingleObjectRef requires a Get method's request to identify the -// resource with exactly one field, of type ObjectRef and named after the -// resource's snake_case type name. -var GetRequestSingleObjectRef = Rule{ - Name: "get-request-single-objectref", - Description: "A Get method's request has exactly one field, of type ObjectRef and named after the resource's snake_case type name, identifying the resource.", - Check: checkGetRequestSingleObjectRef, -} - -func checkGetRequestSingleObjectRef(api *model.API) ([]Finding, error) { - resources, err := model.Resources(api) - if err != nil { - return nil, err - } - messagesByName := api.MessagesByFullName() - - var findings []Finding - for _, rg := range resources { - for _, m := range rg.Methods { - verb, ok := standardVerbFor(m.Name, rg.Message.Name) - if !ok || verb != "Get" { - continue - } - subject := m.ServiceFullName + "." + m.Name - - req, ok := messagesByName[m.InputName] - if !ok { - findings = append(findings, Finding{Subject: subject, Message: "request type " + m.InputName + " not found"}) - continue - } - if len(req.Fields) != 1 { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request has %d field(s), want exactly 1 (an ObjectRef)", len(req.Fields)), - }) - continue - } - f := req.Fields[0] - if f.TypeKind != "message" || f.TypeFullName != objectRefTypeFullName { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request's only field is %s, want %s", fieldTypeDescription(f), objectRefTypeFullName), - }) - continue - } - if want := fieldNameForResource(rg.Message.Name); f.Name != want { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("resource field is named %q, want %q", f.Name, want), - }) - } - } - } - return findings, nil -} diff --git a/tools/apitool/internal/lint/get_test.go b/tools/apitool/internal/lint/get_test.go deleted file mode 100644 index 3bde26a08e..0000000000 --- a/tools/apitool/internal/lint/get_test.go +++ /dev/null @@ -1,64 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -func TestGetRequestSingleObjectRef(t *testing.T) { - objectRefField := model.Field{Name: "atespace", TypeKind: "message", TypeFullName: "ateapi.ObjectRef"} - misnamedObjectRefField := model.Field{Name: "space", TypeKind: "message", TypeFullName: "ateapi.ObjectRef"} - - tests := []struct { - name string - reqFields []model.Field - wantFinding bool - }{ - {"single ObjectRef field", []model.Field{objectRefField}, false}, - {"no fields", nil, true}, - {"two fields", []model.Field{objectRefField, {Name: "extra"}}, true}, - {"single field, wrong type", []model.Field{{Name: "name"}}, true}, - {"single field, wrong name", []model.Field{misnamedObjectRefField}, true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - api := standardMethodAPI("test.Atespace", nil) - api.Messages[1].Fields = tt.reqFields // test.GetAtespaceRequest - findings, err := lint.GetRequestSingleObjectRef.Check(api) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} - -// TestGetRequestSingleObjectRef_IgnoresCreateUpdate confirms the rule -// ignores Create/Update, which legitimately embed the full resource. -func TestGetRequestSingleObjectRef_IgnoresCreateUpdate(t *testing.T) { - findings, err := lint.GetRequestSingleObjectRef.Check(createAtespaceAPI()) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if len(findings) != 0 { - t.Errorf("Check() = %+v, want no findings for a Create method", findings) - } -} diff --git a/tools/apitool/internal/lint/lint.go b/tools/apitool/internal/lint/lint.go index 385347cfd4..9cf17d302f 100644 --- a/tools/apitool/internal/lint/lint.go +++ b/tools/apitool/internal/lint/lint.go @@ -28,10 +28,33 @@ type Finding struct { // Rule is one predicate over the API model. Check returns one Finding per // violation (nil if api satisfies the rule), or a non-nil error if the rule -// could not be evaluated at all - a failure of the rule's own machinery, -// not a finding about api. +// could not be evaluated at all: api is malformed in a way no rule can +// check past, such as a standard method that doesn't parse. type Rule struct { Name string Description string Check func(api *model.API) ([]Finding, error) } + +// All is every rule `apitool validate` runs, in the order their findings +// are reported. +var All = []Rule{ + ResourceMetadata, + ListMethodShape, + RequestNameMatchesMethod, + StandardMethodReturnsResource, + GetRequestShape, + DeleteRequestShape, + DeleteOptionsShape, + CreateRequestShape, + UpdateRequestShape, + ListResponseNameMatchesMethod, + CustomResponseNameMatchesMethod, + SingletonNoList, + SubresourceNaming, + EnumZeroValueUnspecified, + EnumValuesPrefixed, + ResourceStatusFieldShape, + NoOneofs, + Documented, +} diff --git a/tools/apitool/internal/lint/list.go b/tools/apitool/internal/lint/list.go deleted file mode 100644 index 4159865315..0000000000 --- a/tools/apitool/internal/lint/list.go +++ /dev/null @@ -1,153 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - "strings" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -const listOptionsTypeFullName = "ateapi.ListOptions" - -// ListMethodShape requires a "List*" method's request to have exactly an -// options field of type ListOptions plus an optional atespace field, and its -// response to have exactly a repeated message field plus a next_page_token -// field - nothing else. -var ListMethodShape = Rule{ - Name: "list-method-shape", - Description: `A "List*" method's request has a ListOptions options field (plus an optional atespace field), and its ` + - "response has a repeated message field, named after the method's own plural, plus a next_page_token field - nothing else.", - Check: checkListMethodShape, -} - -// ListResponseNameMatchesMethod requires a "List*" method's response to be -// named "{MethodName}Response". -var ListResponseNameMatchesMethod = Rule{ - Name: "list-response-name-matches-method", - Description: `Every "List*" method's response message is named "{MethodName}Response".`, - Check: checkListResponseNameMatchesMethod, -} - -func checkListMethodShape(api *model.API) ([]Finding, error) { - messagesByName := api.MessagesByFullName() - - var findings []Finding - for _, svc := range api.Services { - for _, method := range svc.Methods { - if !strings.HasPrefix(method.Name, "List") { - continue - } - subject := method.ServiceFullName + "." + method.Name - findings = append(findings, checkListRequest(messagesByName, subject, method.InputName)...) - findings = append(findings, checkListResponse(messagesByName, subject, method.Name, method.OutputName)...) - } - } - return findings, nil -} - -func checkListRequest(messagesByName map[string]model.Message, subject, inputName string) []Finding { - req, ok := messagesByName[inputName] - if !ok { - return []Finding{{Subject: subject, Message: "request type " + inputName + " not found"}} - } - - // Sorting and filtering per AIP-132 are not supported, so nothing besides - // atespace and options belongs here. - var findings []Finding - var optionsFields int - for _, f := range req.Fields { - switch { - case f.Name == "atespace": - case f.Name == "options" && f.TypeKind == "message" && f.TypeFullName == listOptionsTypeFullName: - optionsFields++ - default: - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("field %q is not part of the standard List request shape - only atespace and an options field of type %s are allowed (sorting/filtering per AIP-132 is not supported)", f.Name, listOptionsTypeFullName), - }) - } - } - if optionsFields == 0 { - findings = append(findings, Finding{Subject: subject, Message: "request has no options field of type " + listOptionsTypeFullName}) - } - return findings -} - -func checkListResponse(messagesByName map[string]model.Message, subject, methodName, outputName string) []Finding { - resp, ok := messagesByName[outputName] - if !ok { - return []Finding{{Subject: subject, Message: "response type " + outputName + " not found"}} - } - - var findings []Finding - repeatedIdx := -1 - for i := range resp.Fields { - if resp.Fields[i].Repeated && resp.Fields[i].TypeKind == "message" { - repeatedIdx = i - break - } - } - if repeatedIdx == -1 { - findings = append(findings, Finding{Subject: subject, Message: "response has no repeated message field for the page of results"}) - } else if want := repeatedFieldNameForList(methodName); resp.Fields[repeatedIdx].Name != want { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("repeated field is named %q, want %q (the plural resource name, matching the method name)", resp.Fields[repeatedIdx].Name, want), - }) - } - if findFieldByName(resp, "next_page_token") == nil { - findings = append(findings, Finding{Subject: subject, Message: "response has no next_page_token field"}) - } - for i, f := range resp.Fields { - if i == repeatedIdx || f.Name == "next_page_token" { - continue - } - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("field %q is not part of the standard List response shape - only the repeated resource field and next_page_token are allowed", f.Name), - }) - } - return findings -} - -func checkListResponseNameMatchesMethod(api *model.API) ([]Finding, error) { - messagesByName := api.MessagesByFullName() - - var findings []Finding - for _, svc := range api.Services { - for _, method := range svc.Methods { - if !strings.HasPrefix(method.Name, "List") { - continue - } - subject := method.ServiceFullName + "." + method.Name - want := method.Name + "Response" - - resp, ok := messagesByName[method.OutputName] - if !ok { - findings = append(findings, Finding{Subject: subject, Message: "response type " + method.OutputName + " not found"}) - continue - } - if resp.Name != want { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("response message is named %q, want %q", resp.Name, want), - }) - } - } - } - return findings, nil -} diff --git a/tools/apitool/internal/lint/list_test.go b/tools/apitool/internal/lint/list_test.go deleted file mode 100644 index 0a76c03045..0000000000 --- a/tools/apitool/internal/lint/list_test.go +++ /dev/null @@ -1,178 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -func listAPI(reqFields, respFields []model.Field) *model.API { - return &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "ListWidgets", ServiceName: "Control", InputName: "test.ListWidgetsRequest", OutputName: "test.ListWidgetsResponse"}, - }, - }}, - Messages: []model.Message{ - {FullName: "test.ListWidgetsRequest", Name: "ListWidgetsRequest", Fields: reqFields}, - {FullName: "test.ListWidgetsResponse", Name: "ListWidgetsResponse", Fields: respFields}, - }, - } -} - -func TestListMethodShape(t *testing.T) { - compliantReq := []model.Field{{Name: "options", TypeKind: "message", TypeFullName: "ateapi.ListOptions"}} - compliantResp := []model.Field{ - {Name: "widgets", Repeated: true, TypeKind: "message", TypeFullName: "test.Widget"}, - {Name: "next_page_token"}, - } - - tests := []struct { - name string - reqFields []model.Field - respFields []model.Field - wantViolations int - }{ - {"compliant", compliantReq, compliantResp, 0}, - { - name: "missing options", - reqFields: nil, - respFields: compliantResp, - wantViolations: 1, - }, - { - name: "loose page_size and page_token outside ListOptions", - reqFields: []model.Field{{Name: "page_size"}, {Name: "page_token"}}, - respFields: compliantResp, - wantViolations: 3, - }, - { - name: "options field of the wrong type", - reqFields: []model.Field{{Name: "options", TypeKind: "message", TypeFullName: "ateapi.DeleteOptions"}}, - respFields: compliantResp, - wantViolations: 2, - }, - { - name: "missing repeated field and next_page_token", - reqFields: compliantReq, - respFields: []model.Field{ - {Name: "widget", TypeKind: "message", TypeFullName: "test.Widget"}, // not repeated - }, - // no repeated message field found, no next_page_token, plus "widget" - // itself is a stray field since it doesn't qualify as either. - wantViolations: 3, - }, - { - name: "repeated scalar field doesn't satisfy the rule", - reqFields: compliantReq, - respFields: []model.Field{ - {Name: "ids", Repeated: true}, // repeated, but not a message - {Name: "next_page_token"}, - }, - // no repeated message field found, plus "ids" is itself a stray field. - wantViolations: 2, - }, - { - name: "repeated field misnamed - doesn't match the method's plural", - reqFields: compliantReq, - respFields: []model.Field{ - {Name: "items", Repeated: true, TypeKind: "message", TypeFullName: "test.Widget"}, // want "widgets" - {Name: "next_page_token"}, - }, - wantViolations: 1, - }, - { - name: "atespace field is allowed on the request", - reqFields: append([]model.Field{{Name: "atespace"}}, compliantReq...), - respFields: compliantResp, - wantViolations: 0, - }, - { - name: "stray field on the request - sorting/filtering is not supported", - reqFields: append(append([]model.Field{}, compliantReq...), model.Field{Name: "filter"}), - respFields: compliantResp, - wantViolations: 1, - }, - { - name: "stray field on the response", - reqFields: compliantReq, - respFields: append(append([]model.Field{}, compliantResp...), model.Field{Name: "total_count"}), - wantViolations: 1, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.ListMethodShape.Check(listAPI(tt.reqFields, tt.respFields)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if len(findings) != tt.wantViolations { - t.Errorf("Check() = %+v, want %d finding(s)", findings, tt.wantViolations) - } - }) - } -} - -// TestListMethodShape_IgnoresNonListMethods confirms non-"List*" methods -// aren't checked. -func TestListMethodShape_IgnoresNonListMethods(t *testing.T) { - api := &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "GetWidget", ServiceName: "Control", InputName: "test.GetWidgetRequest", OutputName: "test.Widget"}, - }, - }}, - Messages: []model.Message{ - {FullName: "test.GetWidgetRequest", Name: "GetWidgetRequest"}, - {FullName: "test.Widget", Name: "Widget"}, - }, - } - findings, err := lint.ListMethodShape.Check(api) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if len(findings) != 0 { - t.Errorf("Check() = %+v, want no findings for a non-List method", findings) - } -} - -func TestListResponseNameMatchesMethod(t *testing.T) { - tests := []struct { - name string - responseName string - wantFinding bool - }{ - {"compliant", "ListWidgetsResponse", false}, - {"wrong name", "WidgetsResponse", true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - api := listAPI(nil, nil) - api.Messages[1].Name = tt.responseName // test.ListWidgetsResponse - findings, err := lint.ListResponseNameMatchesMethod.Check(api) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} diff --git a/tools/apitool/internal/lint/method.go b/tools/apitool/internal/lint/method.go new file mode 100644 index 0000000000..8ad7a2522a --- /dev/null +++ b/tools/apitool/internal/lint/method.go @@ -0,0 +1,235 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package lint + +import ( + "fmt" + "slices" + "strings" + + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/agent-substrate/substrate/tools/apitool/internal/model" +) + +// method is a standard method with everything a rule needs resolved: its +// resource, how its name reads against that resource, and its messages. +type method struct { + // verb is the standard verb the name starts with. For example, verbGet + // for GetActorEgressPolicy. + verb verb + // parent is the parent the name names after the verb, for a + // subresource method, or "" for a top-level resource's. For example, + // "Actor" for GetActorEgressPolicy, or "Global" for + // GetGlobalAccessPolicy. + parent string + rpc model.Method + resource model.Message + request model.Message + response model.Message +} + +func (m method) isSubresource() bool { + return m.parent != "" +} + +// findingf reports a finding about m. +func (m method) findingf(format string, args ...any) Finding { + return findingf(subjectOf(m.rpc), format, args...) +} + +func findingf(subject protoreflect.FullName, format string, args ...any) Finding { + return Finding{Subject: string(subject), Message: fmt.Sprintf(format, args...)} +} + +// subjectOf names m in findings. For example, "ateapi.Control.GetActor". +func subjectOf(m model.Method) protoreflect.FullName { + return m.ServiceFullName.Append(m.Name) +} + +// forEachStandardMethod returns a Rule.Check running check on every +// standard method whose verb is one of verbs. +func forEachStandardMethod(check func(method) []Finding, verbs ...verb) func(*model.API) ([]Finding, error) { + return func(api *model.API) ([]Finding, error) { + resources, err := model.Resources(api) + if err != nil { + return nil, err + } + messagesByName := api.MessagesByFullName() + + var findings []Finding + for _, r := range resources { + for _, rpc := range r.Methods { + prefix, ok := standardVerbPrefix(rpc.Name) + if !ok { + continue + } + v, parent, ok := parseStandardMethod(rpc.Name, r.Message) + if !ok { + return nil, fmt.Errorf("%s: a method named with %s is a standard method - %s, or name it with a custom verb", subjectOf(rpc), prefix, wantStandardName(prefix, r.Message)) + } + if !slices.Contains(verbs, v) { + continue + } + m := method{verb: v, parent: parent, rpc: rpc, resource: r.Message} + if m.request, ok = messagesByName[rpc.InputName]; !ok { + m.request = model.Message{FullName: rpc.InputName, Name: string(rpc.InputName.Name())} + } + if m.response, ok = messagesByName[rpc.OutputName]; !ok { + m.response = model.Message{FullName: rpc.OutputName, Name: string(rpc.OutputName.Name())} + } + findings = append(findings, check(m)...) + } + } + return findings, nil + } +} + +// forEachResource returns a Rule.Check running check on every resource. +func forEachResource(check func(model.Resource) []Finding) func(*model.API) ([]Finding, error) { + return func(api *model.API) ([]Finding, error) { + resources, err := model.Resources(api) + if err != nil { + return nil, err + } + var findings []Finding + for _, r := range resources { + findings = append(findings, check(r)...) + } + return findings, nil + } +} + +// forEachMessage returns a Rule.Check running check on every message. +func forEachMessage(check func(model.Message) []Finding) func(*model.API) ([]Finding, error) { + return func(api *model.API) ([]Finding, error) { + var findings []Finding + for _, m := range api.Messages { + findings = append(findings, check(m)...) + } + return findings, nil + } +} + +// forEachField returns a Rule.Check running check on every field of every +// message. +func forEachField(check func(model.Message, model.Field) []Finding) func(*model.API) ([]Finding, error) { + return func(api *model.API) ([]Finding, error) { + var findings []Finding + for _, m := range api.Messages { + for _, f := range m.Fields { + findings = append(findings, check(m, f)...) + } + } + return findings, nil + } +} + +// forEachMethod returns a Rule.Check running check on every method of +// every service. +func forEachMethod(check func(model.Method) []Finding) func(*model.API) ([]Finding, error) { + return func(api *model.API) ([]Finding, error) { + var findings []Finding + for _, svc := range api.Services { + for _, rpc := range svc.Methods { + findings = append(findings, check(rpc)...) + } + } + return findings, nil + } +} + +// allOf returns a Rule.Check running every one of checks, in order, and +// joining their findings. +func allOf(checks ...func(*model.API) ([]Finding, error)) func(*model.API) ([]Finding, error) { + return func(api *model.API) ([]Finding, error) { + var findings []Finding + for _, check := range checks { + f, err := check(api) + if err != nil { + return nil, err + } + findings = append(findings, f...) + } + return findings, nil + } +} + +// forEachEnum returns a Rule.Check running check on every enum. +func forEachEnum(check func(model.Enum) []Finding) func(*model.API) ([]Finding, error) { + return func(api *model.API) ([]Finding, error) { + var findings []Finding + for _, e := range api.Enums { + findings = append(findings, check(e)...) + } + return findings, nil + } +} + +// verb is a standard method's verb, which its name starts with (style +// guide section #3). +type verb string + +const ( + verbGet verb = "Get" + verbList verb = "List" + verbCreate verb = "Create" + verbUpdate verb = "Update" + verbDelete verb = "Delete" +) + +// standardVerbs lists every standard verb. +var standardVerbs = []verb{verbGet, verbList, verbCreate, verbUpdate, verbDelete} + +// parseStandardMethod parses methodName as a standard method on resource: +// the verb, then one of resource's parents (none for a top-level +// resource), then resource as resourceNameFor names it. It returns the +// verb and the parent ("" for a top-level resource). ok is false for a +// custom method, and for a standard method named any other way. +func parseStandardMethod(methodName protoreflect.Name, resource model.Message) (v verb, parent string, ok bool) { + parents := resource.ResourceAnnotation.Parents + if len(parents) == 0 { + parents = []string{""} + } + for _, p := range parents { + for _, sv := range standardVerbs { + if string(methodName) == string(sv)+p+resourceNameFor(sv, resource) { + return sv, p, true + } + } + } + return "", "", false +} + +// wantStandardName describes how a standard method with verb v on resource +// is named. For example, "want Get, then one of the parents Global/Atespace, +// then AccessPolicy". +func wantStandardName(v verb, resource model.Message) string { + parents := resource.ResourceAnnotation.Parents + if len(parents) == 0 { + return fmt.Sprintf("want %s, then %s", v, resourceNameFor(v, resource)) + } + return fmt.Sprintf("want %s, then one of the parents %s, then %s", v, strings.Join(parents, "/"), resourceNameFor(v, resource)) +} + +// resourceNameFor is how a method with verb v names resource: by its +// plural for List, and by its name otherwise. For example, "Actors" in +// ListActors, but "Actor" in GetActor. +func resourceNameFor(v verb, resource model.Message) string { + if v == verbList { + return resource.ResourceAnnotation.Plural + } + return resource.Name +} diff --git a/tools/apitool/internal/lint/method_test.go b/tools/apitool/internal/lint/method_test.go new file mode 100644 index 0000000000..5daab4ff6e --- /dev/null +++ b/tools/apitool/internal/lint/method_test.go @@ -0,0 +1,64 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package lint + +import ( + "errors" + "testing" + + "github.com/google/go-cmp/cmp" + + "github.com/agent-substrate/substrate/tools/apitool/internal/model" +) + +func TestAllOf(t *testing.T) { + finds := func(subject string) func(*model.API) ([]Finding, error) { + return func(*model.API) ([]Finding, error) { + return []Finding{{Subject: subject, Message: "m"}}, nil + } + } + fails := func(*model.API) ([]Finding, error) { + return nil, errors.New("boom") + } + + tests := []struct { + name string + checks []func(*model.API) ([]Finding, error) + want []Finding + wantErr bool + }{ + { + name: "joins findings in order", + checks: []func(*model.API) ([]Finding, error){finds("a"), finds("b")}, + want: []Finding{{Subject: "a", Message: "m"}, {Subject: "b", Message: "m"}}, + }, + { + name: "an error drops every finding", + checks: []func(*model.API) ([]Finding, error){finds("a"), fails}, + wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := allOf(tt.checks...)(&model.API{}) + if (err != nil) != tt.wantErr { + t.Fatalf("allOf() error = %v, want an error: %v", err, tt.wantErr) + } + if diff := cmp.Diff(tt.want, got); diff != "" { + t.Errorf("allOf() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/tools/apitool/internal/lint/naming.go b/tools/apitool/internal/lint/naming.go index 1dbdcb361c..c144f5f64f 100644 --- a/tools/apitool/internal/lint/naming.go +++ b/tools/apitool/internal/lint/naming.go @@ -17,18 +17,114 @@ package lint import ( "regexp" "strings" + "unicode" + "unicode/utf8" + + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) -var pascalBoundary = regexp.MustCompile(`([a-z0-9])([A-Z])`) +var RequestNameMatchesMethod = Rule{ + Name: "request-name-matches-method", + Description: `Every method's request message is named "{MethodName}Request".`, + Check: forEachMethod(checkRequestNameMatchesMethod), +} -func snakeCase(s string) string { - return strings.ToLower(pascalBoundary.ReplaceAllString(s, "${1}_${2}")) +func checkRequestNameMatchesMethod(rpc model.Method) []Finding { + if got, want := rpc.InputName.Name(), rpc.Name+"Request"; got != want { + return []Finding{findingf(subjectOf(rpc), "request message is named %q, want %q", got, want)} + } + return nil +} + +var CustomResponseNameMatchesMethod = Rule{ + Name: "custom-response-name-matches-method", + Description: `Every custom method's response message is named "{MethodName}Response", not the resource itself.`, + Check: forEachResource(checkCustomResponseName), +} + +func checkCustomResponseName(r model.Resource) []Finding { + var findings []Finding + for _, rpc := range r.Methods { + if _, ok := standardVerbPrefix(rpc.Name); ok { + continue + } + if got, want := rpc.OutputName.Name(), rpc.Name+"Response"; got != want { + findings = append(findings, findingf(subjectOf(rpc), "response message is named %q, want %q", got, want)) + } + } + return findings +} + +var SubresourceNaming = Rule{ + Name: "subresource-naming", + Description: "A subresource's message is named without its parent type.", + Check: forEachResource(checkSubresourceNaming), +} + +func checkSubresourceNaming(r model.Resource) []Finding { + var findings []Finding + for _, parent := range r.Message.ResourceAnnotation.Parents { + if strings.HasPrefix(r.Message.Name, parent) { + findings = append(findings, findingf(r.Message.FullName, "message is named with its parent type %s - name it for the subresource alone", parent)) + } + } + return findings +} + +// standardVerbPrefix returns the standard verb methodName starts with, if +// it's followed by another word. For example, "Get" for +// "GetActorEgressPolicy", but nothing for "Getaway". +func standardVerbPrefix(methodName protoreflect.Name) (verb, bool) { + for _, v := range standardVerbs { + rest, ok := strings.CutPrefix(string(methodName), string(v)) + if r, _ := utf8.DecodeRuneInString(rest); ok && unicode.IsUpper(r) { + return v, true + } + } + return "", false +} + +var EnumZeroValueUnspecified = Rule{ + Name: "enum-zero-value-unspecified", + Description: `Every enum's zero value is named "{EnumName}_UNSPECIFIED".`, + Check: forEachEnum(checkEnumZeroValueUnspecified), +} + +var EnumValuesPrefixed = Rule{ + Name: "enum-values-prefixed", + Description: `Every enum's values are prefixed with "{ENUM_NAME}_".`, + Check: forEachEnum(checkEnumValuesPrefixed), } -func fieldNameForResource(resourceName string) string { - return snakeCase(resourceName) +func checkEnumZeroValueUnspecified(e model.Enum) []Finding { + want := strings.ToUpper(snakeCase(string(e.FullName.Name()))) + "_UNSPECIFIED" + zero := e.ValueByNumber(0) + switch { + case zero == nil: + return []Finding{findingf(e.FullName, "has no value numbered 0")} + case string(zero.Name) != want: + return []Finding{findingf(e.FullName, "zero value is named %q, want %q", zero.Name, want)} + } + return nil } -func repeatedFieldNameForList(methodName string) string { - return snakeCase(strings.TrimPrefix(methodName, "List")) +func checkEnumValuesPrefixed(e model.Enum) []Finding { + prefix := strings.ToUpper(snakeCase(string(e.FullName.Name()))) + "_" + var findings []Finding + for _, v := range e.Values { + if !strings.HasPrefix(string(v.Name), prefix) { + findings = append(findings, findingf(e.FullName.Append(v.Name), "not prefixed with %q", prefix)) + } + } + return findings +} + +var pascalBoundary = regexp.MustCompile(`([a-z0-9])([A-Z])`) + +// snakeCase converts a PascalCase name to snake_case. For example, +// "actor_assignment" for "ActorAssignment". +func snakeCase(s string) string { + return strings.ToLower(pascalBoundary.ReplaceAllString(s, "${1}_${2}")) } diff --git a/tools/apitool/internal/lint/naming_test.go b/tools/apitool/internal/lint/naming_test.go new file mode 100644 index 0000000000..aa96ab65ec --- /dev/null +++ b/tools/apitool/internal/lint/naming_test.go @@ -0,0 +1,420 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package lint_test + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + + "github.com/agent-substrate/substrate/tools/apitool/internal/lint" + "github.com/agent-substrate/substrate/tools/apitool/internal/model" +) + +func TestRequestNameMatchesMethod(t *testing.T) { + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "compliant", + api: methodAPI(model.Method{Name: "DoThing", InputName: "test.DoThingRequest", OutputName: "test.DoThingResponse"}, + msg("DoThingRequest"), + msg("DoThingResponse")), + }, + { + name: "wrong suffix", + api: methodAPI(model.Method{Name: "DoThing", InputName: "test.DoThingReq", OutputName: "test.DoThingResponse"}, + msg("DoThingReq"), + msg("DoThingResponse")), + want: []lint.Finding{ + {Subject: "test.Control.DoThing", Message: `request message is named "DoThingReq", want "DoThingRequest"`}, + }, + }, + { + name: "wrong verb", + api: methodAPI(model.Method{Name: "DoThing", InputName: "test.GetThingRequest", OutputName: "test.DoThingResponse"}, + msg("GetThingRequest"), + msg("DoThingResponse")), + want: []lint.Finding{ + {Subject: "test.Control.DoThing", Message: `request message is named "GetThingRequest", want "DoThingRequest"`}, + }, + }, + { + name: "request declared outside this API", + api: methodAPI(model.Method{Name: "DoThing", InputName: "google.protobuf.Empty", OutputName: "test.DoThingResponse"}, + msg("DoThingResponse")), + want: []lint.Finding{ + {Subject: "test.Control.DoThing", Message: `request message is named "Empty", want "DoThingRequest"`}, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.RequestNameMatchesMethod.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestCustomResponseNameMatchesMethod(t *testing.T) { + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "compliant", + api: methodAPI(rpc("DrainWidget", "Widget", "DrainWidgetResponse"), + resource("Widget", nil)), + }, + { + name: "returns the resource", + api: methodAPI(rpc("DrainWidget", "Widget", "Widget"), + resource("Widget", nil)), + want: []lint.Finding{ + {Subject: "test.Control.DrainWidget", Message: `response message is named "Widget", want "DrainWidgetResponse"`}, + }, + }, + { + name: "wrong name", + api: methodAPI(rpc("DrainWidget", "Widget", "DrainResponse"), + resource("Widget", nil)), + want: []lint.Finding{ + {Subject: "test.Control.DrainWidget", Message: `response message is named "DrainResponse", want "DrainWidgetResponse"`}, + }, + }, + { + name: "a standard method is not checked", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil)), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.CustomResponseNameMatchesMethod.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestStandardMethodNaming(t *testing.T) { + tests := []struct { + name string + api *model.API + wantErr string + }{ + { + name: "Get naming a parent", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil)), + }, + { + name: "Get naming Global", + api: methodAPI(rpc("GetGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"})), + }, + { + name: "Get naming the second of two parents", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Global", "Gadget"}), + resource("Gadget", nil)), + }, + { + name: "Get naming no parent", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil)), + wantErr: "test.Control.GetWidget: a method named with Get is a standard method - want Get, then one of the parents Gadget, then Widget, or name it with a custom verb", + }, + { + name: "Get naming a resource that isn't a parent", + api: methodAPI(rpc("GetGadgetBoxWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("GadgetBox", nil), + resource("Gadget", nil)), + wantErr: "test.Control.GetGadgetBoxWidget: a method named with Get is a standard method - want Get, then one of the parents Gadget, then Widget, or name it with a custom verb", + }, + { + name: "Get naming neither of two parents", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", []string{"Global", "Gadget"}), + resource("Gadget", nil)), + wantErr: "test.Control.GetWidget: a method named with Get is a standard method - want Get, then one of the parents Global/Gadget, then Widget, or name it with a custom verb", + }, + { + name: "List naming a parent and the plural", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil)), + }, + { + name: "List with the singular", + api: methodAPI(rpc("ListGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil)), + wantErr: "test.Control.ListGadgetWidget: a method named with List is a standard method - want List, then one of the parents Gadget, then Widgets, or name it with a custom verb", + }, + { + name: "List naming no parent", + api: methodAPI(rpc("ListWidgets", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil)), + wantErr: "test.Control.ListWidgets: a method named with List is a standard method - want List, then one of the parents Gadget, then Widgets, or name it with a custom verb", + }, + { + name: "List with the declared plural", + api: methodAPI(rpc("ListGadgetPeople", "Person", "Person"), + pluralResource("Person", "People", []string{"Gadget"}), + resource("Gadget", nil)), + }, + { + name: "List with a plural other than the declared one", + api: methodAPI(rpc("ListGadgetPersons", "Person", "Person"), + pluralResource("Person", "People", []string{"Gadget"}), + resource("Gadget", nil)), + wantErr: "test.Control.ListGadgetPersons: a method named with List is a standard method - want List, then one of the parents Gadget, then People, or name it with a custom verb", + }, + { + name: "custom method", + api: methodAPI(rpc("DrainWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil)), + }, + { + name: "word starting with a verb is not a verb", + api: methodAPI(rpc("Getaway", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil)), + }, + { + name: "top-level Get", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil)), + }, + { + name: "top-level List", + api: methodAPI(rpc("ListWidgets", "Widget", "Widget"), + resource("Widget", nil)), + }, + { + name: "top-level Get with the plural", + api: methodAPI(rpc("GetWidgets", "Widget", "Widget"), + resource("Widget", nil)), + wantErr: "test.Control.GetWidgets: a method named with Get is a standard method - want Get, then Widget, or name it with a custom verb", + }, + { + name: "top-level List with the singular", + api: methodAPI(rpc("ListWidget", "Widget", "Widget"), + resource("Widget", nil)), + wantErr: "test.Control.ListWidget: a method named with List is a standard method - want List, then Widgets, or name it with a custom verb", + }, + { + name: "top-level Get naming a parent it doesn't have", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", nil), + resource("Gadget", nil)), + wantErr: "test.Control.GetGadgetWidget: a method named with Get is a standard method - want Get, then Widget, or name it with a custom verb", + }, + { + name: "message named with its parent type", + api: methodAPI(rpc("GetGadgetGadgetWidget", "GadgetWidget", "GadgetWidget"), + resource("GadgetWidget", []string{"Gadget"}), + resource("Gadget", nil)), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + // Every rule over standard methods parses them the same way; + // StandardMethodReturnsResource stands in for all of them. + _, err := lint.StandardMethodReturnsResource.Check(tt.api) + var gotErr string + if err != nil { + gotErr = err.Error() + } + if gotErr != tt.wantErr { + t.Errorf("Check() error = %q, want %q", gotErr, tt.wantErr) + } + }) + } +} + +func TestSubresourceNaming(t *testing.T) { + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "message named with its parent type", + api: methodAPI(rpc("GetGadgetGadgetWidget", "GadgetWidget", "GadgetWidget"), + resource("GadgetWidget", []string{"Gadget"}), + resource("Gadget", nil)), + want: []lint.Finding{ + {Subject: "test.GadgetWidget", Message: "message is named with its parent type Gadget - name it for the subresource alone"}, + }, + }, + { + name: "message named with Global", + api: methodAPI(rpc("GetGlobalGlobalWidget", "GlobalWidget", "GlobalWidget"), + resource("GlobalWidget", []string{"Global", "Gadget"}), + resource("Gadget", nil)), + want: []lint.Finding{ + {Subject: "test.GlobalWidget", Message: "message is named with its parent type Global - name it for the subresource alone"}, + }, + }, + { + name: "top-level resource named after another resource", + api: methodAPI(rpc("GetGadgetWidget", "GadgetWidget", "GadgetWidget"), + resource("GadgetWidget", nil), + resource("Gadget", nil)), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.SubresourceNaming.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func enumAPI(enums []model.Enum) *model.API { + return &model.API{Enums: enums} +} + +func TestEnumZeroValueUnspecified(t *testing.T) { + tests := []struct { + name string + enum model.Enum + want []lint.Finding + }{ + { + name: "compliant top-level", + enum: model.Enum{FullName: "test.ActorState", Name: "ActorState", Values: []model.EnumValue{ + {Name: "ACTOR_STATE_UNSPECIFIED", Number: 0}, + {Name: "ACTOR_STATE_RUNNING", Number: 1}, + }}, + }, + { + name: "compliant nested", + enum: model.Enum{FullName: "test.Worker.State", Name: "Worker.State", ParentFullName: "test.Worker", Values: []model.EnumValue{ + {Name: "STATE_UNSPECIFIED", Number: 0}, + {Name: "STATE_ACTIVE", Number: 1}, + }}, + }, + { + name: "wrong zero-value name", + enum: model.Enum{FullName: "test.ActorState", Name: "ActorState", Values: []model.EnumValue{ + {Name: "UNKNOWN", Number: 0}, + }}, + want: []lint.Finding{ + {Subject: "test.ActorState", Message: `zero value is named "UNKNOWN", want "ACTOR_STATE_UNSPECIFIED"`}, + }, + }, + { + name: "no zero value at all", + enum: model.Enum{FullName: "test.ActorState", Name: "ActorState", Values: []model.EnumValue{ + {Name: "ACTOR_STATE_RUNNING", Number: 1}, + }}, + want: []lint.Finding{ + {Subject: "test.ActorState", Message: "has no value numbered 0"}, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.EnumZeroValueUnspecified.Check(enumAPI([]model.Enum{tt.enum})) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestEnumValuesPrefixed(t *testing.T) { + tests := []struct { + name string + enum model.Enum + want []lint.Finding + }{ + { + name: "compliant top-level", + enum: model.Enum{FullName: "test.SandboxClass", Name: "SandboxClass", Values: []model.EnumValue{ + {Name: "SANDBOX_CLASS_UNSPECIFIED", Number: 0}, + {Name: "SANDBOX_CLASS_GVISOR", Number: 1}, + }}, + }, + { + name: "unprefixed top-level value", + enum: model.Enum{FullName: "test.SandboxClass", Name: "SandboxClass", Values: []model.EnumValue{ + {Name: "UNSPECIFIED", Number: 0}, + {Name: "GVISOR", Number: 1}, + }}, + want: []lint.Finding{ + {Subject: "test.SandboxClass.UNSPECIFIED", Message: `not prefixed with "SANDBOX_CLASS_"`}, + {Subject: "test.SandboxClass.GVISOR", Message: `not prefixed with "SANDBOX_CLASS_"`}, + }, + }, + { + name: "compliant nested", + enum: model.Enum{FullName: "test.Worker.State", Name: "Worker.State", ParentFullName: "test.Worker", Values: []model.EnumValue{ + {Name: "STATE_UNSPECIFIED", Number: 0}, + {Name: "STATE_ACTIVE", Number: 1}, + }}, + }, + { + name: "unprefixed nested value", + enum: model.Enum{FullName: "test.Worker.State", Name: "Worker.State", ParentFullName: "test.Worker", Values: []model.EnumValue{ + {Name: "UNSPECIFIED", Number: 0}, + {Name: "ACTIVE", Number: 1}, + }}, + want: []lint.Finding{ + {Subject: "test.Worker.State.UNSPECIFIED", Message: `not prefixed with "STATE_"`}, + {Subject: "test.Worker.State.ACTIVE", Message: `not prefixed with "STATE_"`}, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.EnumValuesPrefixed.Check(enumAPI([]model.Enum{tt.enum})) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/tools/apitool/internal/lint/no_oneofs.go b/tools/apitool/internal/lint/no_oneofs.go index 843461c569..d9cae9c0b2 100644 --- a/tools/apitool/internal/lint/no_oneofs.go +++ b/tools/apitool/internal/lint/no_oneofs.go @@ -15,29 +15,18 @@ package lint import ( - "fmt" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) -// NoOneofs requires that no field belong to an explicit `oneof` group. var NoOneofs = Rule{ Name: "no-oneofs", Description: "No field belongs to an explicit oneof group.", - Check: checkNoOneofs, + Check: forEachField(checkNoOneofs), } -func checkNoOneofs(api *model.API) ([]Finding, error) { - var findings []Finding - for _, m := range api.Messages { - for _, f := range m.Fields { - if f.OneofName != "" { - findings = append(findings, Finding{ - Subject: m.FullName + "." + f.Name, - Message: fmt.Sprintf("belongs to oneof %q - oneofs are not used in this API", f.OneofName), - }) - } - } +func checkNoOneofs(m model.Message, f model.Field) []Finding { + if f.OneofName == "" { + return nil } - return findings, nil + return []Finding{findingf(m.FullName.Append(f.Name), "belongs to oneof %q - oneofs are not used in this API", f.OneofName)} } diff --git a/tools/apitool/internal/lint/no_oneofs_test.go b/tools/apitool/internal/lint/no_oneofs_test.go index c66869a4ce..d2bc11f030 100644 --- a/tools/apitool/internal/lint/no_oneofs_test.go +++ b/tools/apitool/internal/lint/no_oneofs_test.go @@ -17,31 +17,42 @@ package lint_test import ( "testing" + "github.com/google/go-cmp/cmp" + "github.com/agent-substrate/substrate/tools/apitool/internal/lint" "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) func TestNoOneofs(t *testing.T) { tests := []struct { - name string - field model.Field - wantFinding bool + name string + api *model.API + want []lint.Finding }{ - {"plain field", model.Field{Name: "widget"}, false}, - {"proto3 optional scalar - not a real oneof", model.Field{Name: "widget", Proto3Optional: true}, false}, - {"member of a real oneof", model.Field{Name: "widget", OneofName: "reference"}, true}, + { + name: "plain field", + api: messagesAPI(msg("Widget", model.Field{Name: "gadget"})), + }, + { + name: "proto3 optional scalar - not a real oneof", + api: messagesAPI(msg("Widget", model.Field{Name: "gadget", Proto3Optional: true})), + }, + { + name: "member of a real oneof", + api: messagesAPI(msg("Widget", model.Field{Name: "gadget", OneofName: "reference"})), + want: []lint.Finding{ + {Subject: "test.Widget.gadget", Message: `belongs to oneof "reference" - oneofs are not used in this API`}, + }, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - api := &model.API{Messages: []model.Message{ - {FullName: "test.Foo", Name: "Foo", Fields: []model.Field{tt.field}}, - }} - findings, err := lint.NoOneofs.Check(api) + findings, err := lint.NoOneofs.Check(tt.api) if err != nil { t.Fatalf("Check() error = %v", err) } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) } }) } diff --git a/tools/apitool/internal/lint/request_naming.go b/tools/apitool/internal/lint/request_naming.go deleted file mode 100644 index 51ef9e8b44..0000000000 --- a/tools/apitool/internal/lint/request_naming.go +++ /dev/null @@ -1,54 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -// RequestNameMatchesMethod requires every method's request to be named -// "{MethodName}Request". -var RequestNameMatchesMethod = Rule{ - Name: "request-name-matches-method", - Description: `Every method's request message is named "{MethodName}Request".`, - Check: checkRequestNameMatchesMethod, -} - -func checkRequestNameMatchesMethod(api *model.API) ([]Finding, error) { - messagesByName := api.MessagesByFullName() - - var findings []Finding - for _, svc := range api.Services { - for _, method := range svc.Methods { - subject := method.ServiceFullName + "." + method.Name - want := method.Name + "Request" - - req, ok := messagesByName[method.InputName] - if !ok { - findings = append(findings, Finding{Subject: subject, Message: "request type " + method.InputName + " not found"}) - continue - } - if req.Name != want { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request message is named %q, want %q", req.Name, want), - }) - } - } - } - return findings, nil -} diff --git a/tools/apitool/internal/lint/request_naming_test.go b/tools/apitool/internal/lint/request_naming_test.go deleted file mode 100644 index 71dda9f766..0000000000 --- a/tools/apitool/internal/lint/request_naming_test.go +++ /dev/null @@ -1,62 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -// requestNamingAPI builds a minimal API with one Control.DoThing method -// whose request is named requestName. -func requestNamingAPI(requestName string) *model.API { - return &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "DoThing", ServiceName: "Control", InputName: "test." + requestName, OutputName: "test.DoThingResponse"}, - }, - }}, - Messages: []model.Message{ - {FullName: "test." + requestName, Name: requestName}, - {FullName: "test.DoThingResponse", Name: "DoThingResponse"}, - }, - } -} - -func TestRequestNameMatchesMethod(t *testing.T) { - tests := []struct { - name string - requestName string - wantFinding bool - }{ - {"compliant", "DoThingRequest", false}, - {"wrong suffix", "DoThingReq", true}, - {"wrong verb", "GetThingRequest", true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.RequestNameMatchesMethod.Check(requestNamingAPI(tt.requestName)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} diff --git a/tools/apitool/internal/lint/resource_metadata.go b/tools/apitool/internal/lint/resource_metadata.go deleted file mode 100644 index 79531f3518..0000000000 --- a/tools/apitool/internal/lint/resource_metadata.go +++ /dev/null @@ -1,77 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -const resourceMetadataTypeFullName = "ateapi.ResourceMetadata" - -// ResourceMetadata requires every resource to declare a ResourceMetadata -// field named "metadata" as field 1. -var ResourceMetadata = Rule{ - Name: "resource-metadata", - Description: `Every resource message has a "ResourceMetadata metadata = 1" field.`, - Check: checkResourceMetadata, -} - -func checkResourceMetadata(api *model.API) ([]Finding, error) { - resources, err := model.Resources(api) - if err != nil { - return nil, err - } - - var findings []Finding - for _, rg := range resources { - field := findFieldByName(rg.Message, "metadata") - switch { - case field == nil: - findings = append(findings, Finding{ - Subject: rg.Message.FullName, - Message: `has no "metadata" field`, - }) - case field.TypeKind != "message" || field.TypeFullName != resourceMetadataTypeFullName: - findings = append(findings, Finding{ - Subject: rg.Message.FullName, - Message: fmt.Sprintf(`"metadata" field is %s, want %s`, fieldTypeDescription(*field), resourceMetadataTypeFullName), - }) - case field.Number != 1: - findings = append(findings, Finding{ - Subject: rg.Message.FullName, - Message: fmt.Sprintf(`"metadata" field is number %d, want 1`, field.Number), - }) - } - } - return findings, nil -} - -func findFieldByName(m model.Message, name string) *model.Field { - for i := range m.Fields { - if m.Fields[i].Name == name { - return &m.Fields[i] - } - } - return nil -} - -func fieldTypeDescription(f model.Field) string { - if f.TypeKind == "" { - return "not a message" - } - return f.TypeFullName -} diff --git a/tools/apitool/internal/lint/resource_metadata_test.go b/tools/apitool/internal/lint/resource_metadata_test.go deleted file mode 100644 index fc7d07ef59..0000000000 --- a/tools/apitool/internal/lint/resource_metadata_test.go +++ /dev/null @@ -1,87 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -// resourceAPI builds a minimal API with one Control.GetAtespace method -// resolving to an "Atespace" resource with the given fields. The method -// name must contain a real entry in resourceNames (internal/model/resources.go) - -// Resources rejects anything else. -func resourceAPI(fields []model.Field) *model.API { - return &model.API{ - Services: []model.Service{{ - Name: "Control", - Methods: []model.Method{ - {Name: "GetAtespace", ServiceName: "Control", InputName: "test.GetAtespaceRequest", OutputName: "test.Atespace"}, - }, - }}, - Messages: []model.Message{ - {FullName: "test.Atespace", Name: "Atespace", Fields: fields}, - {FullName: "test.GetAtespaceRequest", Name: "GetAtespaceRequest"}, - }, - } -} - -func TestResourceMetadata(t *testing.T) { - tests := []struct { - name string - fields []model.Field - wantFindig bool - }{ - { - name: "compliant", - fields: []model.Field{ - {Name: "metadata", Number: 1, TypeKind: "message", TypeFullName: "ateapi.ResourceMetadata"}, - }, - wantFindig: false, - }, - { - name: "missing entirely", - fields: []model.Field{{Name: "name", Number: 1}}, - wantFindig: true, - }, - { - name: "wrong type", - fields: []model.Field{ - {Name: "metadata", Number: 1, TypeKind: "message", TypeFullName: "ateapi.ObjectRef"}, - }, - wantFindig: true, - }, - { - name: "wrong field number", - fields: []model.Field{ - {Name: "metadata", Number: 2, TypeKind: "message", TypeFullName: "ateapi.ResourceMetadata"}, - }, - wantFindig: true, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.ResourceMetadata.Check(resourceAPI(tt.fields)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFindig { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFindig) - } - }) - } -} diff --git a/tools/apitool/internal/lint/resource_status.go b/tools/apitool/internal/lint/resource_status.go deleted file mode 100644 index 11a2a5027d..0000000000 --- a/tools/apitool/internal/lint/resource_status.go +++ /dev/null @@ -1,60 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - "strings" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -// ResourceStatusFieldShape requires a resource's "status" field, if -// present, to be typed "{ResourceName}Status". -var ResourceStatusFieldShape = Rule{ - Name: "resource-status-field-shape", - Description: `A resource's "status" field, if present, is typed "{ResourceName}Status".`, - Check: checkResourceStatusFieldShape, -} - -func checkResourceStatusFieldShape(api *model.API) ([]Finding, error) { - resources, err := model.Resources(api) - if err != nil { - return nil, err - } - - var findings []Finding - for _, rg := range resources { - field := findFieldByName(rg.Message, "status") - if field == nil { - continue - } - want := expectedStatusTypeFullName(rg.Message.FullName, rg.Message.Name) - if field.TypeKind != "message" || field.TypeFullName != want { - findings = append(findings, Finding{ - Subject: rg.Message.FullName, - Message: fmt.Sprintf(`"status" field is %s, want %s`, fieldTypeDescription(*field), want), - }) - } - } - return findings, nil -} - -// expectedStatusTypeFullName derives "{pkg}.{ResourceName}Status" from the -// resource's own full and short names. -func expectedStatusTypeFullName(resourceFullName, resourceName string) string { - pkgPrefix := strings.TrimSuffix(resourceFullName, resourceName) - return pkgPrefix + resourceName + "Status" -} diff --git a/tools/apitool/internal/lint/resource_status_test.go b/tools/apitool/internal/lint/resource_status_test.go deleted file mode 100644 index 6cb720752e..0000000000 --- a/tools/apitool/internal/lint/resource_status_test.go +++ /dev/null @@ -1,59 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -// TestResourceStatusFieldShape reuses resourceAPI (resource_metadata_test.go). -func TestResourceStatusFieldShape(t *testing.T) { - tests := []struct { - name string - fields []model.Field - wantFinding bool - }{ - {"no status field at all", nil, false}, - { - name: "correctly typed status field", - fields: []model.Field{{Name: "status", TypeKind: "message", TypeFullName: "test.AtespaceStatus"}}, - wantFinding: false, - }, - { - name: "wrongly typed status field", - fields: []model.Field{{Name: "status", TypeKind: "message", TypeFullName: "test.WrongStatus"}}, - wantFinding: true, - }, - { - name: "status field is a scalar", - fields: []model.Field{{Name: "status", TypeDisplay: "string"}}, - wantFinding: true, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.ResourceStatusFieldShape.Check(resourceAPI(tt.fields)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} diff --git a/tools/apitool/internal/lint/resources.go b/tools/apitool/internal/lint/resources.go new file mode 100644 index 0000000000..017801f7c9 --- /dev/null +++ b/tools/apitool/internal/lint/resources.go @@ -0,0 +1,68 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package lint + +import ( + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/agent-substrate/substrate/tools/apitool/internal/model" +) + +const resourceMetadataTypeFullName protoreflect.FullName = "ateapi.ResourceMetadata" + +var ResourceMetadata = Rule{ + Name: "resource-metadata", + Description: `Every resource message has a "ResourceMetadata metadata = 1" field.`, + Check: forEachResource(checkResourceMetadata), +} + +func checkResourceMetadata(r model.Resource) []Finding { + field := findFieldByName(r.Message, "metadata") + switch { + case field == nil: + return []Finding{findingf(r.Message.FullName, `has no "metadata" field`)} + case !isMessageOf(*field, resourceMetadataTypeFullName): + return []Finding{findingf(r.Message.FullName, `"metadata" field is %s, want %s`, typeOf(*field), resourceMetadataTypeFullName)} + case field.Number != 1: + return []Finding{findingf(r.Message.FullName, `"metadata" field is number %d, want 1`, field.Number)} + } + return nil +} + +var ResourceStatusFieldShape = Rule{ + Name: "resource-status-field-shape", + Description: `A resource's "status" field, if present, is typed "{ResourceName}Status".`, + Check: forEachResource(checkResourceStatusField), +} + +func checkResourceStatusField(r model.Resource) []Finding { + field := findFieldByName(r.Message, "status") + if field == nil { + return nil + } + if want := r.Message.FullName + "Status"; !isMessageOf(*field, want) { + return []Finding{findingf(r.Message.FullName, `"status" field is %s, want %s`, typeOf(*field), want)} + } + return nil +} + +func findFieldByName(m model.Message, name protoreflect.Name) *model.Field { + for i := range m.Fields { + if m.Fields[i].Name == name { + return &m.Fields[i] + } + } + return nil +} diff --git a/tools/apitool/internal/lint/resources_test.go b/tools/apitool/internal/lint/resources_test.go new file mode 100644 index 0000000000..1dce851caf --- /dev/null +++ b/tools/apitool/internal/lint/resources_test.go @@ -0,0 +1,147 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package lint_test + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + "google.golang.org/protobuf/reflect/protoreflect" + + "github.com/agent-substrate/substrate/tools/apitool/internal/lint" + "github.com/agent-substrate/substrate/tools/apitool/internal/model" +) + +func TestResourceMetadata(t *testing.T) { + tests := []struct { + name string + api *model.API + want []lint.Finding + wantErr bool + }{ + { + name: "compliant", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "metadata", Number: 1, Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ResourceMetadata"})), + }, + { + name: "missing entirely", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "name", Number: 1})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `has no "metadata" field`}, + }, + }, + { + name: "wrong type", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "metadata", Number: 1, Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `"metadata" field is ateapi.ObjectRef, want ateapi.ResourceMetadata`}, + }, + }, + { + name: "scalar", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "metadata", Number: 1, Kind: protoreflect.StringKind, TypeDisplay: "string"})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `"metadata" field is string, want ateapi.ResourceMetadata`}, + }, + }, + { + name: "repeated", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "metadata", Number: 1, Repeated: true, Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ResourceMetadata"})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `"metadata" field is repeated ateapi.ResourceMetadata, want ateapi.ResourceMetadata`}, + }, + }, + { + name: "wrong field number", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "metadata", Number: 2, Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ResourceMetadata"})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `"metadata" field is number 2, want 1`}, + }, + }, + { + name: "a message that isn't a resource is not checked", + api: messagesAPI(msg("Widget")), + }, + { + name: "parent that isn't a resource", + api: messagesAPI(resource("Widget", []string{"Gadget"}, model.Field{Name: "metadata", Number: 1, Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ResourceMetadata"})), + wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.ResourceMetadata.Check(tt.api) + if (err != nil) != tt.wantErr { + t.Fatalf("Check() error = %v, want an error: %v", err, tt.wantErr) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestResourceStatusFieldShape(t *testing.T) { + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "no status field at all", + api: messagesAPI(resource("Widget", nil)), + }, + { + name: "correctly typed status field", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "status", Kind: protoreflect.MessageKind, TypeFullName: "test.WidgetStatus"})), + }, + { + name: "wrongly typed status field", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "status", Kind: protoreflect.MessageKind, TypeFullName: "test.WrongStatus"})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `"status" field is test.WrongStatus, want test.WidgetStatus`}, + }, + }, + { + name: "status field is repeated", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "status", Repeated: true, Kind: protoreflect.MessageKind, TypeFullName: "test.WidgetStatus"})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `"status" field is repeated test.WidgetStatus, want test.WidgetStatus`}, + }, + }, + { + name: "status field is a scalar", + api: messagesAPI(resource("Widget", nil, model.Field{Name: "status", Kind: protoreflect.StringKind, TypeDisplay: "string"})), + want: []lint.Finding{ + {Subject: "test.Widget", Message: `"status" field is string, want test.WidgetStatus`}, + }, + }, + { + name: "a message that isn't a resource is not checked", + api: messagesAPI(msg("Widget", model.Field{Name: "status", Kind: protoreflect.StringKind, TypeDisplay: "string"})), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.ResourceStatusFieldShape.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/tools/apitool/internal/lint/rules.go b/tools/apitool/internal/lint/rules.go deleted file mode 100644 index b6c8e7db88..0000000000 --- a/tools/apitool/internal/lint/rules.go +++ /dev/null @@ -1,35 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -// All is every rule `apitool validate` runs, in the order their findings -// are reported. -var All = []Rule{ - ResourceMetadata, - ListMethodShape, - RequestNameMatchesMethod, - StandardMethodReturnsResource, - GetRequestSingleObjectRef, - DeleteRequestShape, - DeleteOptionsShape, - CreateRequestShape, - UpdateRequestShape, - ListResponseNameMatchesMethod, - EnumZeroValueUnspecified, - EnumValuesPrefixed, - ResourceStatusFieldShape, - NoOneofs, - Documented, -} diff --git a/tools/apitool/internal/lint/standard_methods.go b/tools/apitool/internal/lint/standard_methods.go index 362c5c073c..b2152d05a1 100644 --- a/tools/apitool/internal/lint/standard_methods.go +++ b/tools/apitool/internal/lint/standard_methods.go @@ -15,50 +15,195 @@ package lint import ( - "fmt" + "google.golang.org/protobuf/reflect/protoreflect" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) -const objectRefTypeFullName = "ateapi.ObjectRef" - -// StandardMethodReturnsResource requires a Get/Create/Update/Delete -// method's response to be the resource itself, never a wrapper message. var StandardMethodReturnsResource = Rule{ Name: "standard-method-returns-resource", Description: "A Get/Create/Update/Delete method's response is the resource itself, not a wrapper message.", - Check: checkStandardMethodReturnsResource, + Check: forEachStandardMethod(checkReturnsResource, verbGet, verbCreate, verbUpdate, verbDelete), +} + +func checkReturnsResource(m method) []Finding { + if m.response.FullName == m.resource.FullName { + return nil + } + return []Finding{m.findingf("%s method returns %s, want the resource itself (%s)", m.verb, m.response.FullName, m.resource.FullName)} +} + +var GetRequestShape = Rule{ + Name: "get-request-shape", + Description: "A Get method's request has exactly one field, of type ObjectRef and named after the resource's snake_case type name, identifying the resource. " + + `For a subresource, it has the parent's ObjectRef, named after the parent (none for Global), plus a string "name" field - nothing else.`, + Check: forEachStandardMethod(checkGetRequest, verbGet), +} + +func checkGetRequest(m method) []Finding { + return m.requestHas(m.resourceRefFields()) +} + +var ListMethodShape = Rule{ + Name: "list-method-shape", + Description: "A List method's request has a ListOptions options field, plus an atespace field for an atespace-scoped top-level resource, " + + "or for a subresource an ObjectRef field named after the parent (none for Global). Its response has a repeated field of the resource, " + + "named after the resource's plural, plus a next_page_token field - nothing else.", + Check: forEachStandardMethod(checkListMethod, verbList), +} + +var ListResponseNameMatchesMethod = Rule{ + Name: "list-response-name-matches-method", + Description: `Every "List*" method's response message is named "{MethodName}Response".`, + Check: forEachStandardMethod(checkListResponseName, verbList), +} + +func checkListMethod(m method) []Finding { + return append( + m.requestHas(m.listScopeField(), listOptions), + m.responseHas(m.resourcePageField(), nextPageToken)..., + ) +} + +func checkListResponseName(m method) []Finding { + if got, want := m.response.FullName.Name(), m.rpc.Name+"Response"; got != want { + return []Finding{m.findingf("response message is named %q, want %q", got, want)} + } + return nil +} + +var CreateRequestShape = Rule{ + Name: "create-request-shape", + Description: "A Create method's request has exactly one field, of the resource's own type and named after its snake_case type name, plus an optional CreateOptions field named \"options\" - nothing else. " + + "For a subresource, it also has the parent's ObjectRef, named after the parent (none for Global).", + Check: forEachStandardMethod(checkCreateRequest, verbCreate), +} + +func checkCreateRequest(m method) []Finding { + return m.requestHas(m.parentRefField(), m.embeddedResourceField(), optionalCreateOptions) +} + +var UpdateRequestShape = Rule{ + Name: "update-request-shape", + Description: "An Update method's request has exactly one field, of the resource's own type and named after its snake_case type name - nothing else. " + + "For a subresource, it also has the parent's ObjectRef, named after the parent (none for Global).", + Check: forEachStandardMethod(checkUpdateRequest, verbUpdate), +} + +func checkUpdateRequest(m method) []Finding { + return m.requestHas(m.parentRefField(), m.embeddedResourceField()) +} + +var DeleteRequestShape = Rule{ + Name: "delete-request-shape", + Description: "A Delete method's request has exactly one ObjectRef field, named after the resource's snake_case type name, plus an optional DeleteOptions field named \"options\" - nothing else. " + + `For a subresource, the parent's ObjectRef, named after the parent (none for Global), plus a string "name" field replace the resource's ObjectRef.`, + Check: forEachStandardMethod(checkDeleteRequest, verbDelete), +} + +func checkDeleteRequest(m method) []Finding { + return m.requestHas(m.resourceRefFields(), optionalDeleteOptions) +} + +var DeleteOptionsShape = Rule{ + Name: "delete-options-shape", + Description: "The shared ateapi.DeleteOptions message has only version (int64) and uid (string) fields - nothing else.", + Check: checkDeleteOptionsShape, +} + +func checkDeleteOptionsShape(api *model.API) ([]Finding, error) { + opts, ok := api.MessagesByFullName()[deleteOptionsTypeFullName] + if !ok { + return nil, nil // DeleteOptions isn't declared in this API - nothing to check. + } + return checkFields(opts.FullName, "DeleteOptions", opts, + optional("version", scalar(protoreflect.Int64Kind)), + optional("uid", scalar(protoreflect.StringKind)), + ), nil +} + +var SingletonNoList = Rule{ + Name: "singleton-no-list", + Description: "A singleton resource has no List method.", + Check: forEachStandardMethod(checkSingletonList, verbList), +} + +func checkSingletonList(m method) []Finding { + if !m.resource.ResourceAnnotation.Singleton { + return nil + } + return []Finding{m.findingf("lists %s, which is a singleton - a singleton has no List method", m.resource.FullName)} } -func checkStandardMethodReturnsResource(api *model.API) ([]Finding, error) { - resources, err := model.Resources(api) - if err != nil { - return nil, err +// Fields that every request or response of their kind has the same way. +var ( + listOptions = required("options", message(listOptionsTypeFullName)) + nextPageToken = required("next_page_token", scalar(protoreflect.StringKind)) + optionalCreateOptions = optional("options", message(createOptionsTypeFullName)) + optionalDeleteOptions = optional("options", message(deleteOptionsTypeFullName)) +) + +// requestHas checks that m's request has exactly the fields want lists. +func (m method) requestHas(want ...wantFields) []Finding { + return checkFields(subjectOf(m.rpc), "request", m.request, want...) +} + +// responseHas checks that m's response has exactly the fields want lists. +func (m method) responseHas(want ...wantFields) []Finding { + return checkFields(subjectOf(m.rpc), "response", m.response, want...) +} + +// resourceRefFields are how m's request references m's resource: the +// resource's own ObjectRef, or for a subresource, the parent's plus a name +// (style guide section #2.4). +func (m method) resourceRefFields() wantFields { + if !m.isSubresource() { + return required(snakeCase(m.resource.Name), message(objectRefTypeFullName)) } + return append(m.parentRefField(), required("name", scalar(protoreflect.StringKind))...) +} - var findings []Finding - for _, rg := range resources { - for _, m := range rg.Methods { - verb, ok := standardVerbFor(m.Name, rg.Message.Name) - if !ok { - continue // a custom or List method - not this rule's concern. - } - if m.OutputName != rg.Message.FullName { - findings = append(findings, Finding{ - Subject: m.ServiceFullName + "." + m.Name, - Message: fmt.Sprintf("%s method returns %s, want the resource itself (%s)", verb, m.OutputName, rg.Message.FullName), - }) - } - } +// parentRefField is how m's request identifies m's parent: the parent's +// ObjectRef, named after it. None for a top-level resource, which has no +// parent, or for a subresource of Global, which has no identity. +func (m method) parentRefField() wantFields { + if !m.isSubresource() || m.parent == model.GlobalParent { + return nil } - return findings, nil + return required(snakeCase(m.parent), message(objectRefTypeFullName)) } -func standardVerbFor(methodName, resourceName string) (verb string, ok bool) { - for _, v := range []string{"Get", "Create", "Update", "Delete"} { - if methodName == v+resourceName { - return v, true - } +// embeddedResourceField is m's resource, embedded whole in a Create or +// Update request. +func (m method) embeddedResourceField() wantFields { + return required(snakeCase(m.resource.Name), message(m.resource.FullName)) +} + +// listScopeField is how m's List request scopes the listing: an atespace +// for an atespace-scoped top-level resource, nothing for a global one, or the +// parent's ObjectRef for a subresource. Sorting and filtering per AIP-132 are +// not supported. +func (m method) listScopeField() wantFields { + switch { + case m.isSubresource(): + return m.parentRefField() + case m.resource.ResourceAnnotation.Scope == ateapipb.Scope_SCOPE_ATESPACE: + return required("atespace", scalar(protoreflect.StringKind)) + default: + return nil } - return "", false } + +// resourcePageField is the repeated field of m's List response holding a +// page of resources, named after their plural. +func (m method) resourcePageField() wantFields { + return required(snakeCase(m.resource.ResourceAnnotation.Plural), repeatedMessage(m.resource.FullName)) +} + +const ( + objectRefTypeFullName protoreflect.FullName = "ateapi.ObjectRef" + createOptionsTypeFullName protoreflect.FullName = "ateapi.CreateOptions" + deleteOptionsTypeFullName protoreflect.FullName = "ateapi.DeleteOptions" + listOptionsTypeFullName protoreflect.FullName = "ateapi.ListOptions" +) diff --git a/tools/apitool/internal/lint/standard_methods_test.go b/tools/apitool/internal/lint/standard_methods_test.go index 529a08a21b..d61ef968d2 100644 --- a/tools/apitool/internal/lint/standard_methods_test.go +++ b/tools/apitool/internal/lint/standard_methods_test.go @@ -17,26 +17,1119 @@ package lint_test import ( "testing" + "github.com/google/go-cmp/cmp" + "google.golang.org/protobuf/reflect/protoreflect" + "github.com/agent-substrate/substrate/tools/apitool/internal/lint" + "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) func TestStandardMethodReturnsResource(t *testing.T) { tests := []struct { - name string - outputName string - wantFinding bool + name string + api *model.API + want []lint.Finding + }{ + { + name: "Get returns the resource directly", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest")), + }, + { + name: "Get returns a wrapper instead", + api: methodAPI(rpc("GetWidget", "Widget", "GetWidgetResponse"), + resource("Widget", nil), + msg("GetWidgetRequest"), + msg("GetWidgetResponse")), + want: []lint.Finding{ + {Subject: "test.Control.GetWidget", Message: "Get method returns test.GetWidgetResponse, want the resource itself (test.Widget)"}, + }, + }, + { + name: "Create returns a wrapper instead", + api: methodAPI(rpc("CreateWidget", "Widget", "CreateWidgetResponse"), + resource("Widget", nil), + msg("CreateWidgetRequest"), + msg("CreateWidgetResponse")), + want: []lint.Finding{ + {Subject: "test.Control.CreateWidget", Message: "Create method returns test.CreateWidgetResponse, want the resource itself (test.Widget)"}, + }, + }, + { + name: "Update returns a wrapper instead", + api: methodAPI(rpc("UpdateWidget", "Widget", "UpdateWidgetResponse"), + resource("Widget", nil), + msg("UpdateWidgetRequest"), + msg("UpdateWidgetResponse")), + want: []lint.Finding{ + {Subject: "test.Control.UpdateWidget", Message: "Update method returns test.UpdateWidgetResponse, want the resource itself (test.Widget)"}, + }, + }, + { + name: "Delete returns a wrapper instead", + api: methodAPI(rpc("DeleteWidget", "Widget", "DeleteWidgetResponse"), + resource("Widget", nil), + msg("DeleteWidgetRequest"), + msg("DeleteWidgetResponse")), + want: []lint.Finding{ + {Subject: "test.Control.DeleteWidget", Message: "Delete method returns test.DeleteWidgetResponse, want the resource itself (test.Widget)"}, + }, + }, + { + name: "a List method is not checked", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest"), + msg("ListWidgetsResponse")), + }, + { + name: "returns a type not declared in this API", + api: methodAPI(model.Method{Name: "GetWidget", Resource: "Widget", InputName: "test.GetWidgetRequest", OutputName: "google.protobuf.Empty"}, + resource("Widget", nil), + msg("GetWidgetRequest")), + want: []lint.Finding{ + {Subject: "test.Control.GetWidget", Message: "Get method returns google.protobuf.Empty, want the resource itself (test.Widget)"}, + }, + }, + { + name: "subresource: returns the resource directly", + api: methodAPI(rpc("GetGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"}), + msg("GetGlobalWidgetRequest")), + }, + { + name: "subresource: returns a wrapper instead", + api: methodAPI(rpc("GetGlobalWidget", "Widget", "GetGlobalWidgetResponse"), + resource("Widget", []string{"Global"}), + msg("GetGlobalWidgetRequest"), + msg("GetGlobalWidgetResponse")), + want: []lint.Finding{ + {Subject: "test.Control.GetGlobalWidget", Message: "Get method returns test.GetGlobalWidgetResponse, want the resource itself (test.Widget)"}, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.StandardMethodReturnsResource.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestGetRequestShape(t *testing.T) { + widgetRef := model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + misnamedWidgetRef := model.Field{Name: "thing", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + gadgetRef := model.Field{Name: "gadget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + name := model.Field{Name: "name", Kind: protoreflect.StringKind, TypeDisplay: "string"} + + tests := []struct { + name string + api *model.API + want []lint.Finding + wantErr bool + }{ + { + name: "single ObjectRef field", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest", widgetRef)), + }, + { + name: "no fields", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest")), + want: []lint.Finding{ + {Subject: "test.Control.GetWidget", Message: `request has no "widget" ateapi.ObjectRef field`}, + }, + }, + { + name: "stray field", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest", widgetRef, model.Field{Name: "extra"})), + want: []lint.Finding{ + {Subject: "test.Control.GetWidget", Message: `request field "extra" is not allowed - want only "widget" ateapi.ObjectRef`}, + }, + }, + { + name: "single field, wrong type", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest", name)), + want: []lint.Finding{ + {Subject: "test.Control.GetWidget", Message: `request field "name" is not allowed - want only "widget" ateapi.ObjectRef`}, + {Subject: "test.Control.GetWidget", Message: `request has no "widget" ateapi.ObjectRef field`}, + }, + }, + { + name: "single field, wrong name", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest", misnamedWidgetRef)), + want: []lint.Finding{ + {Subject: "test.Control.GetWidget", Message: `request field "thing" is ateapi.ObjectRef, want it named "widget"`}, + }, + }, + { + name: "a Create method is not checked", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"})), + }, + { + name: "subresource: parent ObjectRef and name", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest", gadgetRef, name)), + }, + { + name: "subresource: no name", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest", gadgetRef)), + want: []lint.Finding{ + {Subject: "test.Control.GetGadgetWidget", Message: `request has no "name" string field`}, + }, + }, + { + name: "subresource: name not a string", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest", gadgetRef, model.Field{Name: "name", Kind: protoreflect.Int64Kind, TypeDisplay: "int64"})), + want: []lint.Finding{ + {Subject: "test.Control.GetGadgetWidget", Message: `request field "name" is int64, want string`}, + }, + }, + { + name: "subresource: no parent ObjectRef", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest", name)), + want: []lint.Finding{ + {Subject: "test.Control.GetGadgetWidget", Message: `request has no "gadget" ateapi.ObjectRef field`}, + }, + }, + { + name: "subresource: parent field not an ObjectRef", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest", model.Field{Name: "gadget", Kind: protoreflect.StringKind, TypeDisplay: "string"}, name)), + want: []lint.Finding{ + {Subject: "test.Control.GetGadgetWidget", Message: `request field "gadget" is string, want ateapi.ObjectRef`}, + }, + }, + { + name: "subresource: the resource's own ObjectRef instead", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest", widgetRef)), + want: []lint.Finding{ + {Subject: "test.Control.GetGadgetWidget", Message: `request field "widget" is ateapi.ObjectRef, want it named "gadget"`}, + {Subject: "test.Control.GetGadgetWidget", Message: `request has no "name" string field`}, + }, + }, + { + name: "subresource: stray field", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest", gadgetRef, name, model.Field{Name: "extra"})), + want: []lint.Finding{ + {Subject: "test.Control.GetGadgetWidget", Message: `request field "extra" is not allowed - want only "gadget" ateapi.ObjectRef and "name" string`}, + }, + }, + { + name: "subresource of Global: name alone", + api: methodAPI(rpc("GetGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"}), + msg("GetGlobalWidgetRequest", name)), + }, + { + name: "subresource of Global: no name", + api: methodAPI(rpc("GetGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"}), + msg("GetGlobalWidgetRequest")), + want: []lint.Finding{ + {Subject: "test.Control.GetGlobalWidget", Message: `request has no "name" string field`}, + }, + }, + { + name: "subresource of Global: a parent ObjectRef is a stray field", + api: methodAPI(rpc("GetGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"}), + msg("GetGlobalWidgetRequest", model.Field{Name: "global", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"}, name)), + want: []lint.Finding{ + {Subject: "test.Control.GetGlobalWidget", Message: `request field "global" is not allowed - want only "name" string`}, + }, + }, + { + name: "method with no resource annotation", + api: methodAPI(rpc("GetWidget", "", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest", widgetRef)), + wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.GetRequestShape.Check(tt.api) + if (err != nil) != tt.wantErr { + t.Fatalf("Check() error = %v, want an error: %v", err, tt.wantErr) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestListMethodShape(t *testing.T) { + options := model.Field{Name: "options", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ListOptions"} + atespace := model.Field{Name: "atespace", Kind: protoreflect.StringKind, TypeDisplay: "string"} + gadgetRef := model.Field{Name: "gadget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + widgets := model.Field{Name: "widgets", Repeated: true, Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"} + nextPageToken := model.Field{Name: "next_page_token", Kind: protoreflect.StringKind, TypeDisplay: "string"} + + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "compliant", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options), + msg("ListWidgetsResponse", widgets, nextPageToken)), + }, + { + name: "missing options", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace), + msg("ListWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `request has no "options" ateapi.ListOptions field`}, + }, + }, + { + name: "loose page_size and page_token outside ListOptions", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, + model.Field{Name: "page_size", Kind: protoreflect.Int32Kind, TypeDisplay: "int32"}, + model.Field{Name: "page_token", Kind: protoreflect.StringKind, TypeDisplay: "string"}), + msg("ListWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `request field "page_size" is not allowed - want only "atespace" string and "options" ateapi.ListOptions`}, + {Subject: "test.Control.ListWidgets", Message: `request field "page_token" is not allowed - want only "atespace" string and "options" ateapi.ListOptions`}, + {Subject: "test.Control.ListWidgets", Message: `request has no "options" ateapi.ListOptions field`}, + }, + }, + { + name: "missing repeated field and next_page_token", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options), + msg("ListWidgetsResponse", model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"})), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `response field "widget" is not allowed - want only "widgets" repeated test.Widget and "next_page_token" string`}, + {Subject: "test.Control.ListWidgets", Message: `response has no "widgets" repeated test.Widget field`}, + {Subject: "test.Control.ListWidgets", Message: `response has no "next_page_token" string field`}, + }, + }, + { + name: "repeated scalar field doesn't satisfy the rule", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options), + msg("ListWidgetsResponse", model.Field{Name: "ids", Repeated: true}, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `response field "ids" is not allowed - want only "widgets" repeated test.Widget and "next_page_token" string`}, + {Subject: "test.Control.ListWidgets", Message: `response has no "widgets" repeated test.Widget field`}, + }, + }, + { + name: "repeated field misnamed - doesn't match the resource's plural", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options), + msg("ListWidgetsResponse", model.Field{Name: "items", Repeated: true, Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"}, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `response field "items" is repeated test.Widget, want it named "widgets"`}, + }, + }, + { + name: "repeated field of another message", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options), + msg("ListWidgetsResponse", model.Field{Name: "widgets", Repeated: true, Kind: protoreflect.MessageKind, TypeFullName: "test.Gadget"}, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `response field "widgets" is repeated test.Gadget, want repeated test.Widget`}, + }, + }, + { + name: "atespace-scoped resource without atespace", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", options), + msg("ListWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `request has no "atespace" string field`}, + }, + }, + { + name: "global resource", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + globalResource("Widget"), + msg("ListWidgetsRequest", options), + msg("ListWidgetsResponse", widgets, nextPageToken)), + }, + { + name: "global resource with atespace", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + globalResource("Widget"), + msg("ListWidgetsRequest", atespace, options), + msg("ListWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `request field "atespace" is not allowed - want only "options" ateapi.ListOptions`}, + }, + }, + { + name: "stray field on the request - sorting/filtering is not supported", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options, model.Field{Name: "filter"}), + msg("ListWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `request field "filter" is not allowed - want only "atespace" string and "options" ateapi.ListOptions`}, + }, + }, + { + name: "stray field on the response", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options), + msg("ListWidgetsResponse", widgets, nextPageToken, model.Field{Name: "total_count"})), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `response field "total_count" is not allowed - want only "widgets" repeated test.Widget and "next_page_token" string`}, + }, + }, + { + name: "a Get method is not checked", + api: methodAPI(rpc("GetWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("GetWidgetRequest")), + }, + { + name: "request not declared in this API", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `request has no "atespace" string field`}, + {Subject: "test.Control.ListWidgets", Message: `request has no "options" ateapi.ListOptions field`}, + }, + }, + { + name: "response not declared in this API", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest", atespace, options)), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `response has no "widgets" repeated test.Widget field`}, + {Subject: "test.Control.ListWidgets", Message: `response has no "next_page_token" string field`}, + }, + }, + { + name: "subresource: compliant", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "ListGadgetWidgetsResponse"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("ListGadgetWidgetsRequest", gadgetRef, options), + msg("ListGadgetWidgetsResponse", widgets, nextPageToken)), + }, + { + name: "subresource: missing parent field", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "ListGadgetWidgetsResponse"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("ListGadgetWidgetsRequest", options), + msg("ListGadgetWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListGadgetWidgets", Message: `request has no "gadget" ateapi.ObjectRef field`}, + }, + }, + { + name: "subresource: parent field is not an ObjectRef", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "ListGadgetWidgetsResponse"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("ListGadgetWidgetsRequest", model.Field{Name: "gadget", Kind: protoreflect.StringKind, TypeDisplay: "string"}, options), + msg("ListGadgetWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListGadgetWidgets", Message: `request field "gadget" is string, want ateapi.ObjectRef`}, + }, + }, + { + name: "subresource: atespace field is not allowed", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "ListGadgetWidgetsResponse"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("ListGadgetWidgetsRequest", gadgetRef, atespace, options), + msg("ListGadgetWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListGadgetWidgets", Message: `request field "atespace" is not allowed - want only "gadget" ateapi.ObjectRef and "options" ateapi.ListOptions`}, + }, + }, + { + name: "subresource: repeated field named after the parent too", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "ListGadgetWidgetsResponse"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("ListGadgetWidgetsRequest", gadgetRef, options), + msg("ListGadgetWidgetsResponse", model.Field{Name: "gadget_widgets", Repeated: true, Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"}, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListGadgetWidgets", Message: `response field "gadget_widgets" is repeated test.Widget, want it named "widgets"`}, + }, + }, + { + name: "subresource of Global: compliant", + api: methodAPI(rpc("ListGlobalWidgets", "Widget", "ListGlobalWidgetsResponse"), + resource("Widget", []string{"Global"}), + msg("ListGlobalWidgetsRequest", options), + msg("ListGlobalWidgetsResponse", widgets, nextPageToken)), + }, + { + name: "subresource of Global: parent field is not allowed", + api: methodAPI(rpc("ListGlobalWidgets", "Widget", "ListGlobalWidgetsResponse"), + resource("Widget", []string{"Global"}), + msg("ListGlobalWidgetsRequest", model.Field{Name: "global", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"}, options), + msg("ListGlobalWidgetsResponse", widgets, nextPageToken)), + want: []lint.Finding{ + {Subject: "test.Control.ListGlobalWidgets", Message: `request field "global" is not allowed - want only "options" ateapi.ListOptions`}, + }, + }, + { + name: "subresource: a parent whose name starts with another parent's", + api: methodAPI(rpc("ListGadgetBoxWidgets", "Widget", "ListGadgetBoxWidgetsResponse"), + resource("Widget", []string{"Gadget", "GadgetBox"}), + resource("Gadget", nil), + resource("GadgetBox", nil), + msg("ListGadgetBoxWidgetsRequest", model.Field{Name: "gadget_box", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"}, options), + msg("ListGadgetBoxWidgetsResponse", widgets, nextPageToken)), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.ListMethodShape.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestListResponseNameMatchesMethod(t *testing.T) { + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "compliant", + api: methodAPI(rpc("ListWidgets", "Widget", "ListWidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest"), + msg("ListWidgetsResponse")), + }, + { + name: "wrong name", + api: methodAPI(rpc("ListWidgets", "Widget", "WidgetsResponse"), + resource("Widget", nil), + msg("ListWidgetsRequest"), + msg("WidgetsResponse")), + want: []lint.Finding{ + {Subject: "test.Control.ListWidgets", Message: `response message is named "WidgetsResponse", want "ListWidgetsResponse"`}, + }, + }, + { + name: "a Get method is not checked", + api: methodAPI(rpc("GetWidget", "Widget", "GetWidgetResponse"), + resource("Widget", nil), + msg("GetWidgetRequest"), + msg("GetWidgetResponse")), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.ListResponseNameMatchesMethod.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestCreateRequestShape(t *testing.T) { + widget := model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"} + misnamedWidget := model.Field{Name: "thing", Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"} + options := model.Field{Name: "options", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.CreateOptions"} + gadgetRef := model.Field{Name: "gadget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "resource field alone", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", widget)), + }, + { + name: "resource field plus CreateOptions", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", widget, options)), + }, + { + name: "no fields", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest")), + want: []lint.Finding{ + {Subject: "test.Control.CreateWidget", Message: `request has no "widget" test.Widget field`}, + }, + }, + { + name: "two resource fields", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", widget, widget)), + want: []lint.Finding{ + {Subject: "test.Control.CreateWidget", Message: `request has 2 "widget" test.Widget fields, want at most 1`}, + }, + }, + { + name: "two CreateOptions fields", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", widget, options, options)), + want: []lint.Finding{ + {Subject: "test.Control.CreateWidget", Message: `request has 2 "options" ateapi.CreateOptions fields, want at most 1`}, + }, + }, + { + name: "CreateOptions field misnamed", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", widget, model.Field{Name: "opts", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.CreateOptions"})), + want: []lint.Finding{ + {Subject: "test.Control.CreateWidget", Message: `request field "opts" is ateapi.CreateOptions, want it named "options"`}, + }, + }, + { + name: "loose control field outside CreateOptions", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", widget, model.Field{Name: "dry_run"})), + want: []lint.Finding{ + {Subject: "test.Control.CreateWidget", Message: `request field "dry_run" is not allowed - want only "widget" test.Widget and optional "options" ateapi.CreateOptions`}, + }, + }, + { + name: "resource field, wrong name", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", misnamedWidget)), + want: []lint.Finding{ + {Subject: "test.Control.CreateWidget", Message: `request field "thing" is test.Widget, want it named "widget"`}, + }, + }, + { + name: "a Delete method is not checked", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"})), + }, + { + name: "subresource: parent ObjectRef and resource field", + api: methodAPI(rpc("CreateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("CreateGadgetWidgetRequest", gadgetRef, widget)), + }, + { + name: "subresource: parent ObjectRef, resource field, and CreateOptions", + api: methodAPI(rpc("CreateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("CreateGadgetWidgetRequest", gadgetRef, widget, options)), + }, + { + name: "subresource: no parent ObjectRef", + api: methodAPI(rpc("CreateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("CreateGadgetWidgetRequest", widget)), + want: []lint.Finding{ + {Subject: "test.Control.CreateGadgetWidget", Message: `request has no "gadget" ateapi.ObjectRef field`}, + }, + }, + { + name: "subresource: parent field not an ObjectRef", + api: methodAPI(rpc("CreateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("CreateGadgetWidgetRequest", model.Field{Name: "gadget", Kind: protoreflect.StringKind, TypeDisplay: "string"}, widget)), + want: []lint.Finding{ + {Subject: "test.Control.CreateGadgetWidget", Message: `request field "gadget" is string, want ateapi.ObjectRef`}, + }, + }, + { + name: "subresource: no resource field", + api: methodAPI(rpc("CreateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("CreateGadgetWidgetRequest", gadgetRef)), + want: []lint.Finding{ + {Subject: "test.Control.CreateGadgetWidget", Message: `request has no "widget" test.Widget field`}, + }, + }, + { + name: "subresource of Global: resource field alone", + api: methodAPI(rpc("CreateGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"}), + msg("CreateGlobalWidgetRequest", widget)), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.CreateRequestShape.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestUpdateRequestShape(t *testing.T) { + widget := model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"} + misnamedWidget := model.Field{Name: "thing", Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"} + gadgetRef := model.Field{Name: "gadget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "resource field alone", + api: methodAPI(rpc("UpdateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("UpdateWidgetRequest", widget)), + }, + { + name: "no fields", + api: methodAPI(rpc("UpdateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("UpdateWidgetRequest")), + want: []lint.Finding{ + {Subject: "test.Control.UpdateWidget", Message: `request has no "widget" test.Widget field`}, + }, + }, + { + name: "two resource fields", + api: methodAPI(rpc("UpdateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("UpdateWidgetRequest", widget, widget)), + want: []lint.Finding{ + {Subject: "test.Control.UpdateWidget", Message: `request has 2 "widget" test.Widget fields, want at most 1`}, + }, + }, + { + name: "loose control field outside the resource", + api: methodAPI(rpc("UpdateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("UpdateWidgetRequest", widget, model.Field{Name: "dry_run"})), + want: []lint.Finding{ + {Subject: "test.Control.UpdateWidget", Message: `request field "dry_run" is not allowed - want only "widget" test.Widget`}, + }, + }, + { + name: "resource field, wrong name", + api: methodAPI(rpc("UpdateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("UpdateWidgetRequest", misnamedWidget)), + want: []lint.Finding{ + {Subject: "test.Control.UpdateWidget", Message: `request field "thing" is test.Widget, want it named "widget"`}, + }, + }, + { + name: "a Delete method is not checked", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"})), + }, + { + name: "subresource: parent ObjectRef and resource field", + api: methodAPI(rpc("UpdateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("UpdateGadgetWidgetRequest", gadgetRef, widget)), + }, + { + name: "subresource: no parent ObjectRef", + api: methodAPI(rpc("UpdateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("UpdateGadgetWidgetRequest", widget)), + want: []lint.Finding{ + {Subject: "test.Control.UpdateGadgetWidget", Message: `request has no "gadget" ateapi.ObjectRef field`}, + }, + }, + { + name: "subresource: parent field not an ObjectRef", + api: methodAPI(rpc("UpdateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("UpdateGadgetWidgetRequest", model.Field{Name: "gadget", Kind: protoreflect.StringKind, TypeDisplay: "string"}, widget)), + want: []lint.Finding{ + {Subject: "test.Control.UpdateGadgetWidget", Message: `request field "gadget" is string, want ateapi.ObjectRef`}, + }, + }, + { + name: "subresource: no resource field", + api: methodAPI(rpc("UpdateGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("UpdateGadgetWidgetRequest", gadgetRef)), + want: []lint.Finding{ + {Subject: "test.Control.UpdateGadgetWidget", Message: `request has no "widget" test.Widget field`}, + }, + }, + { + name: "subresource of Global: resource field alone", + api: methodAPI(rpc("UpdateGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"}), + msg("UpdateGlobalWidgetRequest", widget)), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.UpdateRequestShape.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestDeleteRequestShape(t *testing.T) { + widgetRef := model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + misnamedWidgetRef := model.Field{Name: "thing", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + options := model.Field{Name: "options", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.DeleteOptions"} + gadgetRef := model.Field{Name: "gadget", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.ObjectRef"} + name := model.Field{Name: "name", Kind: protoreflect.StringKind, TypeDisplay: "string"} + + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "ObjectRef alone", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", widgetRef)), + }, + { + name: "ObjectRef plus DeleteOptions", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", widgetRef, options)), + }, + { + name: "no fields", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest")), + want: []lint.Finding{ + {Subject: "test.Control.DeleteWidget", Message: `request has no "widget" ateapi.ObjectRef field`}, + }, + }, + { + name: "two ObjectRef fields", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", widgetRef, widgetRef)), + want: []lint.Finding{ + {Subject: "test.Control.DeleteWidget", Message: `request has 2 "widget" ateapi.ObjectRef fields, want at most 1`}, + }, + }, + { + name: "two DeleteOptions fields", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", widgetRef, options, options)), + want: []lint.Finding{ + {Subject: "test.Control.DeleteWidget", Message: `request has 2 "options" ateapi.DeleteOptions fields, want at most 1`}, + }, + }, + { + name: "DeleteOptions field misnamed", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", widgetRef, model.Field{Name: "opts", Kind: protoreflect.MessageKind, TypeFullName: "ateapi.DeleteOptions"})), + want: []lint.Finding{ + {Subject: "test.Control.DeleteWidget", Message: `request field "opts" is ateapi.DeleteOptions, want it named "options"`}, + }, + }, + { + name: "loose control field outside DeleteOptions", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", widgetRef, model.Field{Name: "dry_run"})), + want: []lint.Finding{ + {Subject: "test.Control.DeleteWidget", Message: `request field "dry_run" is not allowed - want only "widget" ateapi.ObjectRef and optional "options" ateapi.DeleteOptions`}, + }, + }, + { + name: "ObjectRef field, wrong name", + api: methodAPI(rpc("DeleteWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("DeleteWidgetRequest", misnamedWidgetRef)), + want: []lint.Finding{ + {Subject: "test.Control.DeleteWidget", Message: `request field "thing" is ateapi.ObjectRef, want it named "widget"`}, + }, + }, + { + name: "a Create method is not checked", + api: methodAPI(rpc("CreateWidget", "Widget", "Widget"), + resource("Widget", nil), + msg("CreateWidgetRequest", model.Field{Name: "widget", Kind: protoreflect.MessageKind, TypeFullName: "test.Widget"})), + }, + { + name: "subresource: parent ObjectRef and name", + api: methodAPI(rpc("DeleteGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("DeleteGadgetWidgetRequest", gadgetRef, name)), + }, + { + name: "subresource: parent ObjectRef, name, and DeleteOptions", + api: methodAPI(rpc("DeleteGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("DeleteGadgetWidgetRequest", gadgetRef, name, options)), + }, + { + name: "subresource: no name", + api: methodAPI(rpc("DeleteGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("DeleteGadgetWidgetRequest", gadgetRef, options)), + want: []lint.Finding{ + {Subject: "test.Control.DeleteGadgetWidget", Message: `request has no "name" string field`}, + }, + }, + { + name: "subresource: no parent ObjectRef", + api: methodAPI(rpc("DeleteGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("DeleteGadgetWidgetRequest", name, options)), + want: []lint.Finding{ + {Subject: "test.Control.DeleteGadgetWidget", Message: `request has no "gadget" ateapi.ObjectRef field`}, + }, + }, + { + name: "subresource: the resource's own ObjectRef instead", + api: methodAPI(rpc("DeleteGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("DeleteGadgetWidgetRequest", widgetRef, options)), + want: []lint.Finding{ + {Subject: "test.Control.DeleteGadgetWidget", Message: `request field "widget" is ateapi.ObjectRef, want it named "gadget"`}, + {Subject: "test.Control.DeleteGadgetWidget", Message: `request has no "name" string field`}, + }, + }, + { + name: "subresource: two DeleteOptions fields", + api: methodAPI(rpc("DeleteGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("DeleteGadgetWidgetRequest", gadgetRef, name, options, options)), + want: []lint.Finding{ + {Subject: "test.Control.DeleteGadgetWidget", Message: `request has 2 "options" ateapi.DeleteOptions fields, want at most 1`}, + }, + }, + { + name: "subresource: loose control field outside DeleteOptions", + api: methodAPI(rpc("DeleteGadgetWidget", "Widget", "Widget"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("DeleteGadgetWidgetRequest", gadgetRef, name, model.Field{Name: "dry_run"})), + want: []lint.Finding{ + {Subject: "test.Control.DeleteGadgetWidget", Message: `request field "dry_run" is not allowed - want only "gadget" ateapi.ObjectRef, "name" string, and optional "options" ateapi.DeleteOptions`}, + }, + }, + { + name: "subresource of Global: name and DeleteOptions", + api: methodAPI(rpc("DeleteGlobalWidget", "Widget", "Widget"), + resource("Widget", []string{"Global"}), + msg("DeleteGlobalWidgetRequest", name, options)), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.DeleteRequestShape.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestDeleteOptionsShape(t *testing.T) { + version := model.Field{Name: "version", Kind: protoreflect.Int64Kind, TypeDisplay: "int64"} + uid := model.Field{Name: "uid", Kind: protoreflect.StringKind, TypeDisplay: "string"} + deleteOptions := func(fields ...model.Field) model.Message { + return model.Message{FullName: "ateapi.DeleteOptions", Name: "DeleteOptions", Fields: fields} + } + + tests := []struct { + name string + api *model.API + want []lint.Finding + }{ + { + name: "version and uid", + api: messagesAPI(deleteOptions(version, uid)), + }, + { + name: "version alone - presence not required", + api: messagesAPI(deleteOptions(version)), + }, + { + name: "uid alone - presence not required", + api: messagesAPI(deleteOptions(uid)), + }, + { + name: "no fields at all", + api: messagesAPI(deleteOptions()), + }, + { + name: "version wrong type", + api: messagesAPI(deleteOptions(model.Field{Name: "version", Kind: protoreflect.StringKind, TypeDisplay: "string"}, uid)), + want: []lint.Finding{ + {Subject: "ateapi.DeleteOptions", Message: `DeleteOptions field "version" is string, want int64`}, + }, + }, + { + name: "uid wrong type", + api: messagesAPI(deleteOptions(version, model.Field{Name: "uid", Kind: protoreflect.Int64Kind, TypeDisplay: "int64"})), + want: []lint.Finding{ + {Subject: "ateapi.DeleteOptions", Message: `DeleteOptions field "uid" is int64, want string`}, + }, + }, + { + name: "extra field beyond version/uid", + api: messagesAPI(deleteOptions(version, uid, model.Field{Name: "dry_run", Kind: protoreflect.BoolKind, TypeDisplay: "bool"})), + want: []lint.Finding{ + {Subject: "ateapi.DeleteOptions", Message: `DeleteOptions field "dry_run" is not allowed - want only optional "version" int64 and optional "uid" string`}, + }, + }, + { + name: "DeleteOptions not declared in this API", + api: &model.API{}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + findings, err := lint.DeleteOptionsShape.Check(tt.api) + if err != nil { + t.Fatalf("Check() error = %v", err) + } + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestSingletonNoList(t *testing.T) { + tests := []struct { + name string + api *model.API + want []lint.Finding }{ - {"returns the resource directly", "test.Atespace", false}, - {"returns a wrapper instead", "test.GetAtespaceResponse", true}, + { + name: "singleton with a List method", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "ListGadgetWidgetsResponse"), + singletonResource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("ListGadgetWidgetsRequest"), + msg("ListGadgetWidgetsResponse")), + want: []lint.Finding{ + {Subject: "test.Control.ListGadgetWidgets", Message: "lists test.Widget, which is a singleton - a singleton has no List method"}, + }, + }, + { + name: "singleton with a Get method", + api: methodAPI(rpc("GetGadgetWidget", "Widget", "Widget"), + singletonResource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("GetGadgetWidgetRequest")), + }, + { + name: "non-singleton with a List method", + api: methodAPI(rpc("ListGadgetWidgets", "Widget", "ListGadgetWidgetsResponse"), + resource("Widget", []string{"Gadget"}), + resource("Gadget", nil), + msg("ListGadgetWidgetsRequest"), + msg("ListGadgetWidgetsResponse")), + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - findings, err := lint.StandardMethodReturnsResource.Check(standardMethodAPI(tt.outputName, nil)) + findings, err := lint.SingletonNoList.Check(tt.api) if err != nil { t.Fatalf("Check() error = %v", err) } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) + if diff := cmp.Diff(tt.want, findings); diff != "" { + t.Errorf("Check() findings mismatch (-want +got):\n%s", diff) } }) } diff --git a/tools/apitool/internal/lint/update.go b/tools/apitool/internal/lint/update.go deleted file mode 100644 index 3de12a7cab..0000000000 --- a/tools/apitool/internal/lint/update.go +++ /dev/null @@ -1,86 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint - -import ( - "fmt" - - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -const fieldMaskTypeFullName = "google.protobuf.FieldMask" - -// UpdateRequestShape requires an Update method's request to embed the -// resource with exactly one field, named after the resource's snake_case -// type name - nothing else, aside from a google.protobuf.FieldMask field, -// which this rule does not police since field masks are being phased out. -var UpdateRequestShape = Rule{ - Name: "update-request-shape", - Description: "An Update method's request has exactly one field, of the resource's own type and named after its snake_case type name - nothing else (field masks, where still present, are not policed by this rule).", - Check: checkUpdateRequestShape, -} - -func checkUpdateRequestShape(api *model.API) ([]Finding, error) { - resources, err := model.Resources(api) - if err != nil { - return nil, err - } - messagesByName := api.MessagesByFullName() - - var findings []Finding - for _, rg := range resources { - for _, m := range rg.Methods { - verb, ok := standardVerbFor(m.Name, rg.Message.Name) - if !ok || verb != "Update" { - continue - } - subject := m.ServiceFullName + "." + m.Name - - req, ok := messagesByName[m.InputName] - if !ok { - findings = append(findings, Finding{Subject: subject, Message: "request type " + m.InputName + " not found"}) - continue - } - - var resourceFields int - for _, f := range req.Fields { - switch { - case f.TypeKind == "message" && f.TypeFullName == rg.Message.FullName: - resourceFields++ - if want := fieldNameForResource(rg.Message.Name); f.Name != want { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("resource field is named %q, want %q", f.Name, want), - }) - } - case f.TypeKind == "message" && f.TypeFullName == fieldMaskTypeFullName: - // Field masks are being phased out; not policed by this rule. - default: - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("field %q is %s, want %s - non-resource control fields belong elsewhere", f.Name, fieldTypeDescription(f), rg.Message.FullName), - }) - } - } - if resourceFields != 1 { - findings = append(findings, Finding{ - Subject: subject, - Message: fmt.Sprintf("request has %d field(s) of type %s, want exactly 1 embedding the resource", resourceFields, rg.Message.FullName), - }) - } - } - } - return findings, nil -} diff --git a/tools/apitool/internal/lint/update_test.go b/tools/apitool/internal/lint/update_test.go deleted file mode 100644 index 3311e6534d..0000000000 --- a/tools/apitool/internal/lint/update_test.go +++ /dev/null @@ -1,70 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package lint_test - -import ( - "testing" - - "github.com/agent-substrate/substrate/tools/apitool/internal/lint" - "github.com/agent-substrate/substrate/tools/apitool/internal/model" -) - -func TestUpdateRequestShape(t *testing.T) { - resourceField := model.Field{Name: "atespace", TypeKind: "message", TypeFullName: "test.Atespace"} - misnamedResourceField := model.Field{Name: "space", TypeKind: "message", TypeFullName: "test.Atespace"} - maskField := model.Field{Name: "update_mask", TypeKind: "message", TypeFullName: "google.protobuf.FieldMask"} - oddlyNamedMaskField := model.Field{Name: "mask", TypeKind: "message", TypeFullName: "google.protobuf.FieldMask"} - - tests := []struct { - name string - reqFields []model.Field - wantFinding bool - }{ - {"resource field alone", []model.Field{resourceField}, false}, - {"resource field plus a field mask", []model.Field{resourceField, maskField}, false}, - {"resource field plus an oddly-named field mask - name not policed", []model.Field{resourceField, oddlyNamedMaskField}, false}, - {"resource field plus repeated field masks - count not policed", []model.Field{resourceField, maskField, maskField}, false}, - {"no fields", nil, true}, - {"only a field mask, no resource field", []model.Field{maskField}, true}, - {"two resource fields", []model.Field{resourceField, resourceField, maskField}, true}, - {"loose control field outside the resource/a field mask", []model.Field{resourceField, maskField, {Name: "dry_run"}}, true}, - {"resource field, wrong name", []model.Field{misnamedResourceField, maskField}, true}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - findings, err := lint.UpdateRequestShape.Check(updateMethodAPI(tt.reqFields)) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if got := len(findings) > 0; got != tt.wantFinding { - t.Errorf("Check() findings = %+v, want a finding: %v", findings, tt.wantFinding) - } - }) - } -} - -// TestUpdateRequestShape_IgnoresGetDelete confirms the rule ignores -// Get/Delete, which reference the resource via ObjectRef instead. -func TestUpdateRequestShape_IgnoresGetDelete(t *testing.T) { - findings, err := lint.UpdateRequestShape.Check(deleteMethodAPI([]model.Field{ - {Name: "atespace", TypeKind: "message", TypeFullName: "ateapi.ObjectRef"}, - })) - if err != nil { - t.Fatalf("Check() error = %v", err) - } - if len(findings) != 0 { - t.Errorf("Check() = %+v, want no findings for a Delete method", findings) - } -} diff --git a/tools/apitool/internal/model/model.go b/tools/apitool/internal/model/model.go index 7a664e341e..38760d86d6 100644 --- a/tools/apitool/internal/model/model.go +++ b/tools/apitool/internal/model/model.go @@ -22,7 +22,11 @@ import ( "strings" "github.com/bufbuild/protocompile" + "google.golang.org/protobuf/proto" "google.golang.org/protobuf/reflect/protoreflect" + "google.golang.org/protobuf/reflect/protoregistry" + + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) // sourceFileName is an arbitrary internal label: Build only ever compiles @@ -30,6 +34,10 @@ import ( // and appearing in error messages. const sourceFileName = "source.proto" +// GlobalParent is the ResourceAnnotation parent that stands for the whole +// Substrate installation rather than another resource. +const GlobalParent = "Global" + // API is the whole documented surface of one proto file. type API struct { // Services lists every RPC service declared in the file. For example, @@ -40,16 +48,15 @@ type API struct { // "ateapi.ResourceMetadata" both appear here as top-level entries. Messages []Message // Enums lists every enum declared in the file, top-level and nested, - // flattened the same way as Messages. For example, both the top-level - // "ateapi.ActorState" and the nested "ateapi.ExternalVolume.Status" - // appear here. + // flattened the same way as Messages. For example, the top-level + // "ateapi.ActorState" appears here. Enums []Enum } // MessagesByFullName indexes a.Messages by their FullName, for looking up // a Method's InputName/OutputName or a Field's TypeFullName. -func (a API) MessagesByFullName() map[string]Message { - byName := make(map[string]Message, len(a.Messages)) +func (a API) MessagesByFullName() map[protoreflect.FullName]Message { + byName := make(map[protoreflect.FullName]Message, len(a.Messages)) for _, m := range a.Messages { byName[m.FullName] = m } @@ -59,9 +66,9 @@ func (a API) MessagesByFullName() map[string]Message { // Service is one RPC service. For example, ateapi.proto's "Control" service. type Service struct { // FullName is the service's proto full name. For example, "ateapi.Control". - FullName string + FullName protoreflect.FullName // Name is the service's short name. For example, "Control". - Name string + Name protoreflect.Name // Comment is the service's leading doc comment. For example, "Control // is the primary API for Agent Substrate." Empty if fd wasn't built // with comments - see Build. @@ -74,40 +81,44 @@ type Service struct { // Method is one RPC declared on a Service. type Method struct { // Name is the RPC's name. For example, "CreateActor". - Name string + Name protoreflect.Name // Comment is the method's leading doc comment. For example, "Create a // new Actor deriving from a given ActorTemplate." Comment string // InputName is the full name of the request message, for linking. For // example, "ateapi.CreateActorRequest" for CreateActor. - InputName string + InputName protoreflect.FullName // OutputName is the full name of the response message, for linking. // For example, "ateapi.Actor" for CreateActor - most standard methods // return the resource itself rather than a wrapper message. - OutputName string + OutputName protoreflect.FullName + + // Resource is the resource the method's (ateapi.method) annotation + // names, or "" if the method has none. For example, "Actor" for + // SuspendActor, or "EgressPolicy" for GetActorEgressPolicy. + Resource string // ServiceFullName is the declaring service's full name. For example, // "ateapi.Control". - ServiceFullName string + ServiceFullName protoreflect.FullName // ServiceName is the declaring service's short name. For example, "Control". - ServiceName string + ServiceName protoreflect.Name } // Message is one message declared in the file, top-level or nested. type Message struct { // FullName is the message's proto full name. For example, // "ateapi.ResourceMetadata". - FullName string + FullName protoreflect.FullName // Name is the message's short name, with its enclosing message's name // dotted in front for a nested type. For example, "ResourceMetadata" - // for a top-level message. ateapi.proto has no nested message today - // (only a nested enum, ExternalVolume.Status - see Enum.Name for that - // shape); a nested message named "Child" inside "Parent" would read - // "Parent.Child" here. + // for a top-level message. ateapi.proto has no nested message today; a + // nested message named "Child" inside "Parent" would read "Parent.Child" + // here. Name string // ParentFullName is the enclosing message's full name, or "" for a // top-level message. For example, "" for "ateapi.ResourceMetadata". - ParentFullName string + ParentFullName protoreflect.FullName // Comment is the message's leading doc comment. For example, // "ResourceMetadata holds the common fields carried by every Substrate // resource." @@ -116,13 +127,41 @@ type Message struct { // example, ResourceMetadata's atespace, name, uid, version, // create_time, update_time. Fields []Field + // ResourceAnnotation is the message's (ateapi.resource) annotation, or + // nil if the message isn't a resource. For example, non-nil for + // "ateapi.Actor"; nil for "ateapi.ResourceMetadata". + ResourceAnnotation *ResourceAnnotation +} + +// IsResource reports whether m is an API resource. +func (m Message) IsResource() bool { + return m.ResourceAnnotation != nil +} + +// ResourceAnnotation is a resource message's (ateapi.resource) annotation. +type ResourceAnnotation struct { + // Parents lists the resource types this resource is a subresource of: + // another resource's message name, or GlobalParent. For example, + // ["Global", "Atespace"] for AccessPolicy; empty for the top-level + // Actor. + Parents []string + // Singleton is true if each parent has at most one instance of this + // resource. For example, true for EgressPolicy. + Singleton bool + // Scope is whether a top-level resource belongs to an atespace. For + // example, SCOPE_ATESPACE for Actor, or SCOPE_GLOBAL for Worker. Set + // only on a top-level resource: a subresource has its parent's scope. + Scope ateapipb.Scope + // Plural is the resource's plural name. For example, "AccessPolicies" + // for AccessPolicy. + Plural string } // Field is one field declared on a Message. type Field struct { // Name is the field's proto name. For example, "atespace" for // ResourceMetadata.atespace. - Name string + Name protoreflect.Name // Number is the field's proto field number. For example, 4 for // ResourceMetadata.version (`int64 version = 4;`). Number int32 @@ -143,15 +182,17 @@ type Field struct { // ActorStatus.state, or "google.protobuf.Timestamp" for // ResourceMetadata.create_time. TypeDisplay string - // TypeFullName and TypeKind ("message" or "enum") are set only for a - // singular or repeated message/enum field. Empty for scalars and maps - - // see MapValueFullName/MapValueKind for a map's value type. For - // example, for Actor.metadata (`ResourceMetadata metadata = 1;`), - // TypeFullName is "ateapi.ResourceMetadata" and TypeKind is "message"; - // for ActorStatus.state (`ActorState state = 1;`), TypeFullName is - // "ateapi.ActorState" and TypeKind is "enum". - TypeFullName string - TypeKind string + // Kind is the field's kind, as protoreflect reports it. For example, + // StringKind for ResourceMetadata.name, MessageKind for Actor.metadata, + // or EnumKind for ActorStatus.state. A map field is MessageKind, since + // its entries are messages - see MapValueKind for its value's kind. + Kind protoreflect.Kind + // TypeFullName is set only for a singular or repeated message/enum + // field. Empty for scalars and maps - see MapValueFullName for a map's + // value type. For example, "ateapi.ResourceMetadata" for Actor.metadata + // (`ResourceMetadata metadata = 1;`), or "ateapi.ActorState" for + // ActorStatus.state (`ActorState state = 1;`). + TypeFullName protoreflect.FullName // TypeIsExternal is true when the type isn't declared in this proto // file (for example, an imported well-known type). For example, true // for ResourceMetadata.create_time (google.protobuf.Timestamp, @@ -160,20 +201,21 @@ type Field struct { // itself). TypeIsExternal bool - // MapValueFullName and MapValueKind mirror TypeFullName/TypeKind, but - // for a map field's value type. For example, for SandboxAssets.assets - // (`map assets = 2;`), MapValueFullName is - // "ateapi.ArchAssets" and MapValueKind is "message"; for - // Selector.match_labels (`map match_labels = 1;`), - // both are empty, since the map's value is a scalar. - MapValueFullName string - MapValueKind string + // MapValueKind and MapValueFullName mirror Kind/TypeFullName, but for a + // map field's value type. Both are empty for a field that isn't a map. + // For example, for SandboxAssets.assets (`map + // assets = 2;`), MapValueKind is MessageKind and MapValueFullName is + // "ateapi.ArchAssets"; for Selector.match_labels (`map + // match_labels = 1;`), MapValueKind is StringKind and MapValueFullName + // is empty, since the map's value is a scalar. + MapValueKind protoreflect.Kind + MapValueFullName protoreflect.FullName // OneofName is set only for a member of a real, explicit `oneof` // group. ateapi.proto has none today - the API forbids real oneofs by // convention (see lint.NoOneofs) - so OneofName is always "" for every // field this model currently produces. - OneofName string + OneofName protoreflect.Name // Proto3Optional is true for a proto3 `optional` scalar - protoreflect // models it as a synthetic oneof, not a real one. ateapi.proto // declares no `optional` field today, so this is always false for @@ -184,26 +226,24 @@ type Field struct { // Enum is one enum declared in the file, top-level or nested. type Enum struct { // FullName is the enum's proto full name. For example, "ateapi.ActorState" - // for a top-level enum, or "ateapi.ExternalVolume.Status" for one - // nested inside the ExternalVolume message. - FullName string + // for a top-level enum, or "ateapi.Parent.Status" for one nested inside + // a Parent message. + FullName protoreflect.FullName // Name is the enum's short name, with its enclosing message's name // dotted in front for a nested enum. For example, "ActorState" for - // the top-level enum, or "ExternalVolume.Status" for the one nested - // inside ExternalVolume. + // the top-level enum, or "Parent.Status" for one nested inside Parent. Name string // ParentFullName is the enclosing message's full name, or "" for a // top-level enum. For example, "" for "ateapi.ActorState", or - // "ateapi.ExternalVolume" for "ateapi.ExternalVolume.Status". - ParentFullName string + // "ateapi.Parent" for "ateapi.Parent.Status". + ParentFullName protoreflect.FullName // Comment is the enum's leading doc comment. ateapi.proto's ActorState - // and ExternalVolume.Status have none today, so this is often "" in - // practice even though the field is populated the same way as for a - // message or field. + // has none today, so this is often "" in practice even though the field + // is populated the same way as for a message or field. Comment string // Values lists the enum's values, in declaration order. For example, - // ExternalVolume.Status's STATUS_UNSPECIFIED, STATUS_PENDING, - // STATUS_CREATED, STATUS_DELETING. + // ActorState's ACTOR_STATE_UNSPECIFIED, ACTOR_STATE_RESUMING, + // ACTOR_STATE_RUNNING, and so on. Values []EnumValue } @@ -220,20 +260,21 @@ func (e Enum) ValueByNumber(number int32) *EnumValue { // EnumValue is one value declared on an Enum. type EnumValue struct { - // Name is the value's proto name. For example, "STATUS_PENDING" for - // ExternalVolume.Status. - Name string + // Name is the value's proto name. For example, "ACTOR_STATE_RESUMING" + // for ActorState. + Name protoreflect.Name // Number is the value's proto number. For example, 1 for - // `STATUS_PENDING = 1;`. + // `ACTOR_STATE_RESUMING = 1;`. Number int32 - // Comment is the value's leading doc comment. For example, "Volume - // creation pending in the storage system." for ExternalVolume.Status's - // STATUS_PENDING. + // Comment is the value's leading doc comment, or "" when it has none. Comment string } // Build compiles source - a single, self-contained proto3 file's raw text - -// with source info retained, and walks the result into an API. +// with source info retained, and walks the result into an API. Annotations +// are read from the generated descriptors compiled into this binary (see +// generatedOptions), so a message or method that has none, such as one in a +// test fixture, comes back unannotated. func Build(ctx context.Context, source string) (*API, error) { compiler := protocompile.Compiler{ Resolver: protocompile.WithStandardImports(&protocompile.SourceResolver{ @@ -268,9 +309,9 @@ func fromDescriptor(fd protoreflect.FileDescriptor) *API { // ScopeToService returns a copy of api with only the service named // serviceName, plus every message/enum reachable from its methods' -// input/output types (including through map values). Empty result if -// serviceName doesn't exist. -func ScopeToService(api *API, serviceName string) *API { +// input/output types (including through map values and resource parents). +// Empty result if serviceName doesn't exist. +func ScopeToService(api *API, serviceName protoreflect.Name) *API { var svc *Service for i := range api.Services { if api.Services[i].Name == serviceName { @@ -282,16 +323,20 @@ func ScopeToService(api *API, serviceName string) *API { return &API{} } - messagesByName := make(map[string]*Message, len(api.Messages)) + messagesByName := make(map[protoreflect.FullName]*Message, len(api.Messages)) + resourcesByName := map[string]protoreflect.FullName{} for i := range api.Messages { messagesByName[api.Messages[i].FullName] = &api.Messages[i] + if api.Messages[i].IsResource() { + resourcesByName[api.Messages[i].Name] = api.Messages[i].FullName + } } - reachableMessages := map[string]bool{} - reachableEnums := map[string]bool{} + reachableMessages := map[protoreflect.FullName]bool{} + reachableEnums := map[protoreflect.FullName]bool{} - var visitMessage func(name string) - visitMessage = func(name string) { + var visitMessage func(name protoreflect.FullName) + visitMessage = func(name protoreflect.FullName) { if reachableMessages[name] { return } @@ -300,17 +345,24 @@ func ScopeToService(api *API, serviceName string) *API { return } reachableMessages[name] = true + if msg.IsResource() { + for _, parent := range msg.ResourceAnnotation.Parents { + if full, ok := resourcesByName[parent]; ok { + visitMessage(full) + } + } + } for _, f := range msg.Fields { - switch f.TypeKind { - case "message": + switch f.Kind { + case protoreflect.MessageKind: visitMessage(f.TypeFullName) - case "enum": + case protoreflect.EnumKind: reachableEnums[f.TypeFullName] = true } switch f.MapValueKind { - case "message": + case protoreflect.MessageKind: visitMessage(f.MapValueFullName) - case "enum": + case protoreflect.EnumKind: reachableEnums[f.MapValueFullName] = true } } @@ -355,18 +407,19 @@ func collectEnums(fd protoreflect.FileDescriptor, eds protoreflect.EnumDescripto func buildService(fd protoreflect.FileDescriptor, sd protoreflect.ServiceDescriptor) Service { svc := Service{ - FullName: string(sd.FullName()), - Name: string(sd.Name()), + FullName: sd.FullName(), + Name: sd.Name(), Comment: comment(fd, sd), } methods := sd.Methods() for i := range methods.Len() { md := methods.Get(i) svc.Methods = append(svc.Methods, Method{ - Name: string(md.Name()), + Name: md.Name(), Comment: comment(fd, md), - InputName: string(md.Input().FullName()), - OutputName: string(md.Output().FullName()), + InputName: md.Input().FullName(), + OutputName: md.Output().FullName(), + Resource: methodResource(md), ServiceFullName: svc.FullName, ServiceName: svc.Name, }) @@ -376,10 +429,11 @@ func buildService(fd protoreflect.FileDescriptor, sd protoreflect.ServiceDescrip func buildMessage(fd protoreflect.FileDescriptor, md protoreflect.MessageDescriptor) Message { m := Message{ - FullName: string(md.FullName()), - Name: shortName(fd, md.FullName()), - ParentFullName: string(parentFullName(md)), - Comment: comment(fd, md), + FullName: md.FullName(), + Name: shortName(fd, md.FullName()), + ParentFullName: parentFullName(md), + Comment: comment(fd, md), + ResourceAnnotation: resourceAnnotation(md), } fields := md.Fields() for i := range fields.Len() { @@ -388,35 +442,68 @@ func buildMessage(fd protoreflect.FileDescriptor, md protoreflect.MessageDescrip return m } +// generatedOptions returns the options of the generated descriptor compiled +// into this binary under d's full name, or nil if there's none. Build reads +// annotations from there rather than from d: protocompile, compiling the +// annotations' own definitions from source, fills them in as dynamic +// messages that the generated ateapipb extension types can't read. +func generatedOptions(d protoreflect.Descriptor) proto.Message { + g, err := protoregistry.GlobalFiles.FindDescriptorByName(d.FullName()) + if err != nil { + return nil + } + return g.Options() +} + +func methodResource(md protoreflect.MethodDescriptor) string { + opts := generatedOptions(md) + if opts == nil { + return "" + } + return proto.GetExtension(opts, ateapipb.E_Method).(*ateapipb.MethodAnnotation).GetResource() +} + +func resourceAnnotation(md protoreflect.MessageDescriptor) *ResourceAnnotation { + opts := generatedOptions(md) + if opts == nil || !proto.HasExtension(opts, ateapipb.E_Resource) { + return nil + } + a := proto.GetExtension(opts, ateapipb.E_Resource).(*ateapipb.ResourceAnnotation) + return &ResourceAnnotation{ + Parents: a.GetParents(), + Singleton: a.GetSingleton(), + Scope: a.GetScope(), + Plural: a.GetPlural(), + } +} + func buildField(fd protoreflect.FileDescriptor, field protoreflect.FieldDescriptor) Field { f := Field{ - Name: string(field.Name()), + Name: field.Name(), Number: int32(field.Number()), Comment: comment(fd, field), Repeated: field.IsList(), + Kind: field.Kind(), } switch { case field.IsMap(): f.TypeDisplay = fmt.Sprintf("map<%s, %s>", scalarOrTypeName(fd, field.MapKey()), scalarOrTypeName(fd, field.MapValue())) - switch field.MapValue().Kind() { + f.MapValueKind = field.MapValue().Kind() + switch f.MapValueKind { case protoreflect.MessageKind, protoreflect.GroupKind: - f.MapValueFullName = string(field.MapValue().Message().FullName()) - f.MapValueKind = "message" + f.MapValueFullName = field.MapValue().Message().FullName() case protoreflect.EnumKind: - f.MapValueFullName = string(field.MapValue().Enum().FullName()) - f.MapValueKind = "enum" + f.MapValueFullName = field.MapValue().Enum().FullName() } case field.Kind() == protoreflect.MessageKind || field.Kind() == protoreflect.GroupKind: name := shortName(fd, field.Message().FullName()) - f.TypeFullName = string(field.Message().FullName()) - f.TypeKind = "message" + f.TypeFullName = field.Message().FullName() f.TypeIsExternal = field.Message().ParentFile() != fd f.TypeDisplay = repeatedPrefix(field) + name case field.Kind() == protoreflect.EnumKind: name := shortName(fd, field.Enum().FullName()) - f.TypeFullName = string(field.Enum().FullName()) - f.TypeKind = "enum" + f.TypeFullName = field.Enum().FullName() f.TypeIsExternal = field.Enum().ParentFile() != fd f.TypeDisplay = repeatedPrefix(field) + name default: @@ -427,7 +514,7 @@ func buildField(fd protoreflect.FileDescriptor, field protoreflect.FieldDescript if oneof.IsSynthetic() { f.Proto3Optional = true } else { - f.OneofName = string(oneof.Name()) + f.OneofName = oneof.Name() } } @@ -436,16 +523,16 @@ func buildField(fd protoreflect.FileDescriptor, field protoreflect.FieldDescript func buildEnum(fd protoreflect.FileDescriptor, ed protoreflect.EnumDescriptor) Enum { e := Enum{ - FullName: string(ed.FullName()), + FullName: ed.FullName(), Name: shortName(fd, ed.FullName()), - ParentFullName: string(parentFullName(ed)), + ParentFullName: parentFullName(ed), Comment: comment(fd, ed), } values := ed.Values() for i := range values.Len() { v := values.Get(i) e.Values = append(e.Values, EnumValue{ - Name: string(v.Name()), + Name: v.Name(), Number: int32(v.Number()), Comment: comment(fd, v), }) @@ -492,97 +579,73 @@ func comment(fd protoreflect.FileDescriptor, d protoreflect.Descriptor) string { return strings.TrimSpace(fd.SourceLocations().ByDescriptor(d).LeadingComments) } -// Resource pairs a message with every method (across services) that -// resourceForMethodName matches it to, in encounter order. +// Resource pairs a resource message with every method (across services) +// whose annotation names it, in encounter order. type Resource struct { Message Message Methods []Method } -// Resources partitions api's methods by resource, matching each method's -// name against resourceNames (see resourceForMethodName), in api.Messages -// order. Fails if a method matches no resource name or multiple -// equally-specific ones, or if a matched name has no corresponding message. +// Resources groups api's methods by the resource their annotation names, +// one group per annotated message, in api.Messages order. Fails if a method +// has no annotation or names a message that isn't a resource, if a resource +// names a parent that is neither GlobalParent nor another resource, if a +// resource has no plural, or if a top-level resource has no scope or a +// subresource has one. func Resources(api *API) ([]Resource, error) { - // Pass 1: resolve every method to a resource name, and collect the - // names actually referenced. - methodResource := make(map[string]string, len(api.Services)) - referencedNames := map[string]bool{} - for _, svc := range api.Services { - for _, method := range svc.Methods { - resource, err := resourceForMethodName(method.Name) - if err != nil { - return nil, fmt.Errorf("%s.%s: %w", svc.Name, method.Name, err) - } - methodResource[svc.Name+"."+method.Name] = resource - referencedNames[resource] = true - } - } - - // Pass 2: one group per referenced resource, in api.Messages order, - // deleting each name as matched so any left over names an unknown - // resource. - groups := make([]Resource, 0, len(referencedNames)) + var resources []Resource for _, m := range api.Messages { - if referencedNames[m.Name] { - groups = append(groups, Resource{Message: m}) - delete(referencedNames, m.Name) + if m.IsResource() { + resources = append(resources, Resource{Message: m}) } } - for name := range referencedNames { - return nil, fmt.Errorf("resource name %q, matched from a method name, has no corresponding message in this API", name) + + // resources's length is fixed by now, so taking addresses into it is safe. + byName := make(map[string]*Resource, len(resources)) + for i := range resources { + byName[resources[i].Message.Name] = &resources[i] } - // Pass 3: append each method to its group. groups's length is fixed by - // now, so taking addresses into it is safe. - byName := make(map[string]*Resource, len(groups)) - for i := range groups { - byName[groups[i].Message.Name] = &groups[i] + for _, r := range resources { + if err := validateAnnotation(r.Message, byName); err != nil { + return nil, err + } } + for _, svc := range api.Services { for _, method := range svc.Methods { - resource := methodResource[svc.Name+"."+method.Name] - byName[resource].Methods = append(byName[resource].Methods, method) + if method.Resource == "" { + return nil, fmt.Errorf("%s.%s: no resource annotation - add `option (ateapi.method).resource = \"\";`", svc.Name, method.Name) + } + r, ok := byName[method.Resource] + if !ok { + return nil, fmt.Errorf("%s.%s: annotated resource %q is not a resource in this API - annotate its message with (ateapi.resource)", svc.Name, method.Name, method.Resource) + } + r.Methods = append(r.Methods, method) } } - return groups, nil + return resources, nil } -// TODO: We should consider adding proto options to attach this (and other) -// metadata in the proto service descriptor itself. -var resourceNames = []string{ - "AccessPolicy", - "Actor", - "ActorSnapshot", - "Tag", - "ActorTemplate", - "Atespace", - "Worker", -} - -func resourceForMethodName(methodName string) (string, error) { - var matches []string - for _, name := range resourceNames { - if strings.Contains(methodName, name) { - matches = append(matches, name) +// validateAnnotation checks resource's (ateapi.resource) annotation: each +// parent is GlobalParent or a resource in byName, the plural is set, and a +// scope is set exactly when resource is top-level. +func validateAnnotation(resource Message, byName map[string]*Resource) error { + annotation := resource.ResourceAnnotation + for _, parent := range annotation.Parents { + if _, ok := byName[parent]; !ok && parent != GlobalParent { + return fmt.Errorf("%s: parent %q is neither %q nor a resource in this API", resource.FullName, parent, GlobalParent) } } - if len(matches) == 0 { - return "", fmt.Errorf("no name in resourceNames is contained in %q - add one naming the resource this method operates on", methodName) + if annotation.Plural == "" { + return fmt.Errorf("%s: resource has no plural - set `plural` in its (ateapi.resource) annotation", resource.FullName) } - - longest := matches[:1] - for _, m := range matches[1:] { - switch { - case len(m) > len(longest[0]): - longest = []string{m} - case len(m) == len(longest[0]): - longest = append(longest, m) - } + if len(annotation.Parents) == 0 && annotation.Scope == ateapipb.Scope_SCOPE_UNSPECIFIED { + return fmt.Errorf("%s: top-level resource has no scope - set `scope` in its (ateapi.resource) annotation", resource.FullName) } - if len(longest) > 1 { - return "", fmt.Errorf("%q matches multiple equally-specific resource names %v - rename the method or the resource so only one matches", methodName, longest) + if len(annotation.Parents) > 0 && annotation.Scope != ateapipb.Scope_SCOPE_UNSPECIFIED { + return fmt.Errorf("%s: subresource has scope %s - a subresource has its parent's scope, so leave `scope` unset", resource.FullName, annotation.Scope) } - return longest[0], nil + return nil } diff --git a/tools/apitool/internal/model/model_test.go b/tools/apitool/internal/model/model_test.go index 3d01e08d8d..7071fd4628 100644 --- a/tools/apitool/internal/model/model_test.go +++ b/tools/apitool/internal/model/model_test.go @@ -15,10 +15,13 @@ package model_test import ( + "strings" "testing" "github.com/google/go-cmp/cmp" + "google.golang.org/protobuf/reflect/protoreflect" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/agent-substrate/substrate/tools/apitool/internal/model" ) @@ -42,7 +45,7 @@ message Widget { want := &model.API{ Messages: []model.Message{ {FullName: "fixture.Widget", Name: "Widget", Fields: []model.Field{ - {Name: "labels", Number: 1, TypeDisplay: "map"}, + {Name: "labels", Number: 1, Kind: protoreflect.MessageKind, TypeDisplay: "map", MapValueKind: protoreflect.StringKind}, }}, }, } @@ -65,10 +68,10 @@ message Gadget { want := &model.API{ Messages: []model.Message{ {FullName: "fixture.Widget", Name: "Widget", Fields: []model.Field{ - {Name: "gadgets", Number: 1, Repeated: true, TypeDisplay: "repeated Gadget", TypeFullName: "fixture.Gadget", TypeKind: "message"}, + {Name: "gadgets", Number: 1, Repeated: true, TypeDisplay: "repeated Gadget", TypeFullName: "fixture.Gadget", Kind: protoreflect.MessageKind}, }}, {FullName: "fixture.Gadget", Name: "Gadget", Fields: []model.Field{ - {Name: "id", Number: 1, TypeDisplay: "string"}, + {Name: "id", Number: 1, Kind: protoreflect.StringKind, TypeDisplay: "string"}, }}, }, } @@ -91,10 +94,10 @@ message Gadget { want := &model.API{ Messages: []model.Message{ {FullName: "fixture.Widget", Name: "Widget", Fields: []model.Field{ - {Name: "gadgets_by_id", Number: 1, TypeDisplay: "map", MapValueFullName: "fixture.Gadget", MapValueKind: "message"}, + {Name: "gadgets_by_id", Number: 1, Kind: protoreflect.MessageKind, TypeDisplay: "map", MapValueFullName: "fixture.Gadget", MapValueKind: protoreflect.MessageKind}, }}, {FullName: "fixture.Gadget", Name: "Gadget", Fields: []model.Field{ - {Name: "id", Number: 1, TypeDisplay: "string"}, + {Name: "id", Number: 1, Kind: protoreflect.StringKind, TypeDisplay: "string"}, }}, }, } @@ -116,8 +119,8 @@ message Ref { want := &model.API{ Messages: []model.Message{ {FullName: "fixture.Ref", Name: "Ref", Fields: []model.Field{ - {Name: "snapshot", Number: 1, TypeDisplay: "string", OneofName: "reference"}, - {Name: "tag", Number: 2, TypeDisplay: "string", OneofName: "reference"}, + {Name: "snapshot", Number: 1, Kind: protoreflect.StringKind, TypeDisplay: "string", OneofName: "reference"}, + {Name: "tag", Number: 2, Kind: protoreflect.StringKind, TypeDisplay: "string", OneofName: "reference"}, }}, }, } @@ -141,7 +144,7 @@ message Widget { want := &model.API{ Messages: []model.Message{ {FullName: "fixture.Widget", Name: "Widget", Fields: []model.Field{ - {Name: "status", Number: 1, TypeDisplay: "Widget.Status", TypeFullName: "fixture.Widget.Status", TypeKind: "enum"}, + {Name: "status", Number: 1, TypeDisplay: "Widget.Status", TypeFullName: "fixture.Widget.Status", Kind: protoreflect.EnumKind}, }}, }, Enums: []model.Enum{ @@ -173,7 +176,7 @@ message Widget { { FullName: "fixture.Widget", Name: "Widget", Comment: "Widget's doc comment.", Fields: []model.Field{ - {Name: "name", Number: 1, TypeDisplay: "string", Comment: "name's doc comment."}, + {Name: "name", Number: 1, Kind: protoreflect.StringKind, TypeDisplay: "string", Comment: "name's doc comment."}, }, }, }, @@ -193,7 +196,7 @@ message Widget { want := &model.API{ Messages: []model.Message{ {FullName: "fixture.Widget", Name: "Widget", Fields: []model.Field{ - {Name: "nickname", Number: 1, TypeDisplay: "string", Proto3Optional: true}, + {Name: "nickname", Number: 1, Kind: protoreflect.StringKind, TypeDisplay: "string", Proto3Optional: true}, }}, }, } @@ -202,76 +205,191 @@ message Widget { } } -// The resource names below ("Actor", "Worker") aren't arbitrary - -// model.Resources itself hardcodes that vocabulary (see resourceNames in -// model.go), so this fixture has to use it too. -func TestResources(t *testing.T) { - api := buildAPI(t, ` -service FixtureService { - rpc GetActor(GetActorRequest) returns (Actor); - rpc CreateActor(CreateActorRequest) returns (Actor); - rpc GetWorker(GetWorkerRequest) returns (Worker); -} +func TestBuild_Annotations(t *testing.T) { + api, err := model.Build(t.Context(), ateapipb.Source) + if err != nil { + t.Fatalf("Build() error = %v", err) + } -message GetActorRequest { string name = 1; } -message CreateActorRequest { string name = 1; } -message GetWorkerRequest { string name = 1; } + gotResources := map[string]*model.ResourceAnnotation{} + for _, m := range api.Messages { + if m.IsResource() { + gotResources[m.Name] = m.ResourceAnnotation + } + } + wantResources := map[string]*model.ResourceAnnotation{ + "Actor": {Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "Actors"}, + "EgressPolicy": {Parents: []string{"Actor"}, Singleton: true, Plural: "EgressPolicies"}, + "Tag": {Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "Tags"}, + "Atespace": {Scope: ateapipb.Scope_SCOPE_GLOBAL, Plural: "Atespaces"}, + "ActorTemplate": {Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "ActorTemplates"}, + "Worker": {Scope: ateapipb.Scope_SCOPE_GLOBAL, Plural: "Workers"}, + "ActorAssignment": {Parents: []string{"Worker"}, Plural: "ActorAssignments"}, + "AccessPolicy": {Parents: []string{"Global", "Atespace"}, Singleton: true, Plural: "AccessPolicies"}, + } + if diff := cmp.Diff(wantResources, gotResources); diff != "" { + t.Errorf("resource annotations mismatch (-want +got):\n%s", diff) + } -message Actor { string name = 1; } -message Worker { string name = 1; } -`) + wantMethods := map[protoreflect.Name]string{ + "GetActor": "Actor", + "SuspendActor": "Actor", + "GetActorEgressPolicy": "EgressPolicy", + "GetGlobalAccessPolicy": "AccessPolicy", + "GetAtespaceAccessPolicy": "AccessPolicy", + "ListWorkerActorAssignments": "ActorAssignment", + } + gotMethods := map[protoreflect.Name]string{} + for _, svc := range api.Services { + for _, m := range svc.Methods { + if _, ok := wantMethods[m.Name]; ok { + gotMethods[m.Name] = m.Resource + } + } + } + if diff := cmp.Diff(wantMethods, gotMethods); diff != "" { + t.Errorf("method annotations mismatch (-want +got):\n%s", diff) + } +} + +func TestResources(t *testing.T) { + api := &model.API{ + Services: []model.Service{{ + Name: "Control", + Methods: []model.Method{ + {Name: "GetActor", ServiceName: "Control", Resource: "Actor"}, + {Name: "GetWorker", ServiceName: "Control", Resource: "Worker"}, + {Name: "SuspendActor", ServiceName: "Control", Resource: "Actor"}, + }, + }}, + Messages: []model.Message{ + {FullName: "test.GetActorRequest", Name: "GetActorRequest"}, + {FullName: "test.Worker", Name: "Worker", ResourceAnnotation: &model.ResourceAnnotation{Scope: ateapipb.Scope_SCOPE_GLOBAL, Plural: "Workers"}}, + {FullName: "test.Actor", Name: "Actor", ResourceAnnotation: &model.ResourceAnnotation{Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "Actors"}}, + {FullName: "test.EgressPolicy", Name: "EgressPolicy", ResourceAnnotation: &model.ResourceAnnotation{Parents: []string{"Global", "Actor"}, Plural: "EgressPolicies"}}, + }, + } groups, err := model.Resources(api) if err != nil { t.Fatalf("Resources() error = %v", err) } - var actorGroup *model.Resource - for i := range groups { - if groups[i].Message.Name == "Actor" { - actorGroup = &groups[i] + got := map[string][]protoreflect.Name{} + var gotOrder []string + for _, g := range groups { + gotOrder = append(gotOrder, g.Message.Name) + got[g.Message.Name] = []protoreflect.Name{} + for _, m := range g.Methods { + got[g.Message.Name] = append(got[g.Message.Name], m.Name) } } - if actorGroup == nil { - t.Fatalf("no Resource for Actor; groups = %+v", groups) - } - - var gotNames []string - for _, m := range actorGroup.Methods { - gotNames = append(gotNames, m.Name) + if diff := cmp.Diff([]string{"Worker", "Actor", "EgressPolicy"}, gotOrder); diff != "" { + t.Errorf("resource order mismatch (-want +got):\n%s", diff) } - wantNames := []string{"GetActor", "CreateActor"} - if len(gotNames) != len(wantNames) { - t.Fatalf("Actor group methods = %v, want %v", gotNames, wantNames) + want := map[string][]protoreflect.Name{ + "Worker": {"GetWorker"}, + "Actor": {"GetActor", "SuspendActor"}, + "EgressPolicy": {}, } - for i, want := range wantNames { - if gotNames[i] != want { - t.Errorf("Actor group methods[%d] = %q, want %q (declaration order matters)", i, gotNames[i], want) - } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("methods by resource mismatch (-want +got):\n%s", diff) } } -func TestResources_InvalidMethodName(t *testing.T) { +func TestResources_Invalid(t *testing.T) { tests := []struct { - name string - methodName string + name string + api *model.API + wantErr string }{ - {"no resource name matches", "DoThing"}, - {"two equally-specific resource names match", "ActorSnapshotAndActorTemplate"}, + { + name: "method without annotation", + api: &model.API{ + Services: []model.Service{{Name: "Control", Methods: []model.Method{{Name: "GetActor"}}}}, + Messages: []model.Message{{FullName: "test.Actor", Name: "Actor", ResourceAnnotation: &model.ResourceAnnotation{Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "Actors"}}}, + }, + wantErr: "Control.GetActor: no resource annotation", + }, + { + name: "method annotated with a message that isn't a resource", + api: &model.API{ + Services: []model.Service{{Name: "Control", Methods: []model.Method{{Name: "GetActor", Resource: "Actor"}}}}, + Messages: []model.Message{{FullName: "test.Actor", Name: "Actor"}}, + }, + wantErr: `annotated resource "Actor" is not a resource in this API`, + }, + { + name: "method annotated with an unknown resource", + api: &model.API{ + Services: []model.Service{{Name: "Control", Methods: []model.Method{{Name: "GetActor", Resource: "Actor"}}}}, + }, + wantErr: `annotated resource "Actor" is not a resource in this API`, + }, + { + name: "unknown parent", + api: &model.API{ + Messages: []model.Message{ + {FullName: "test.EgressPolicy", Name: "EgressPolicy", ResourceAnnotation: &model.ResourceAnnotation{Parents: []string{"Actor"}, Plural: "EgressPolicies"}}, + }, + }, + wantErr: `test.EgressPolicy: parent "Actor" is neither "Global" nor a resource in this API`, + }, + { + name: "resource without plural", + api: &model.API{ + Messages: []model.Message{{FullName: "test.Actor", Name: "Actor", ResourceAnnotation: &model.ResourceAnnotation{Scope: ateapipb.Scope_SCOPE_ATESPACE}}}, + }, + wantErr: "test.Actor: resource has no plural", + }, + { + name: "top-level resource without scope", + api: &model.API{ + Messages: []model.Message{{FullName: "test.Actor", Name: "Actor", ResourceAnnotation: &model.ResourceAnnotation{Plural: "Actors"}}}, + }, + wantErr: "test.Actor: top-level resource has no scope", + }, + { + name: "subresource with scope", + api: &model.API{ + Messages: []model.Message{ + {FullName: "test.Actor", Name: "Actor", ResourceAnnotation: &model.ResourceAnnotation{Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "Actors"}}, + {FullName: "test.EgressPolicy", Name: "EgressPolicy", ResourceAnnotation: &model.ResourceAnnotation{Parents: []string{"Actor"}, Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "EgressPolicies"}}, + }, + }, + wantErr: "test.EgressPolicy: subresource has scope SCOPE_ATESPACE", + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - api := &model.API{ - Services: []model.Service{{ - Name: "Bogus", - Methods: []model.Method{ - {Name: tt.methodName, ServiceName: "Bogus"}, - }, - }}, - } - if _, err := model.Resources(api); err == nil { - t.Errorf("Resources() error = nil for method %q, want an error", tt.methodName) + _, err := model.Resources(tt.api) + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("Resources() error = %v, want one containing %q", err, tt.wantErr) } }) } } + +func TestScopeToService(t *testing.T) { + control := model.Service{Name: "Control", Methods: []model.Method{ + {Name: "GetActorEgressPolicy", ServiceName: "Control", InputName: "test.GetActorEgressPolicyRequest", OutputName: "test.EgressPolicy", Resource: "EgressPolicy"}, + }} + actor := model.Message{FullName: "test.Actor", Name: "Actor", ResourceAnnotation: &model.ResourceAnnotation{Scope: ateapipb.Scope_SCOPE_ATESPACE, Plural: "Actors"}} + egressPolicy := model.Message{FullName: "test.EgressPolicy", Name: "EgressPolicy", ResourceAnnotation: &model.ResourceAnnotation{Parents: []string{"Global", "Actor"}, Plural: "EgressPolicies"}} + request := model.Message{FullName: "test.GetActorEgressPolicyRequest", Name: "GetActorEgressPolicyRequest"} + unused := model.Message{FullName: "test.Unused", Name: "Unused"} + api := &model.API{ + Services: []model.Service{control, {Name: "Other"}}, + Messages: []model.Message{actor, unused, egressPolicy, request}, + } + + got := model.ScopeToService(api, "Control") + + want := &model.API{ + Services: []model.Service{control}, + Messages: []model.Message{actor, egressPolicy, request}, + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("ScopeToService() mismatch (-want +got):\n%s", diff) + } +}