Repository navigation
Sync upstream actor JWT discovery and sandbox asset updates - #87
Merged
Merged
Conversation
…gent-substrate#2267) Part of agent-substrate#1743. This is the doc I said I would send in agent-substrate#1942. Three parts: - **API style guide, new section 12.** A field that holds a secret must carry `debug_redact`. When to label it (tokens, keys, secret bytes, user values that may hold secrets), when not (references to a secret, identifiers, page tokens), and to prefer the leaf field over the whole message. Examples are all from the public API. - **New `docs/dev/best-practices/logging.md`**, the logging counterpart of the metrics and tracing guides, with a section "Logging protos that hold secrets". How masking works in the Go servers: the handler `serverboot.InitLogger` installs, what it masks, and what it can't see (protos in slices, maps or structs, protos turned into strings, `google.protobuf.Any`, spans, metrics, errors, actor event OTLP). I checked that protobuf-go's `String()`, `%v`, `prototext` and `protojson` all print labeled fields in clear, so the advice is to log the proto as its own attribute. - **`observability.md`, a short "Secrets in Component Logs" note** for people reading the logs: component logs show `[REDACTED]` for these fields, and actor container output is not touched. It links to the guide. I split it this way because the API guide is about the proto surface, and the logging details are Go implementation that will change. They sit next to the other o11y developer guides.
…agent-substrate#2699) Steps 1 and 2 of agent-substrate#2675: sandbox assets move onto the node-local file cache, with every actor holding hard links to its assets under its own directory. No change to how assets are resolved; Run and Restore still take them from the request. The runtime-hash index and the prewarm rework are later PRs. **Reads well commit by commit.** 1. **`filecache: serve directory trees and keep execute bits`** — library only. `GetDirTo` adds directory entries: a `DirFetcher` fills a directory the store creates, publication admits only directories and regular files, and a hit mirrors the tree at the destination as directories the consumer owns with every file a hard link, built under the hit lock. Published files keep the execute bits the fetcher set (0555 for a binary), and a fetcher that leaves a symlink is rejected. Eviction treats a tree as linked while any file under it is linked; retirement stays whole-entry, so a published tree is never missing a file. Serving a tree entry through a file method, or the reverse, is an error rather than a miss. 2. **`atelet: stage sandbox assets per actor from the file cache`** — `ensureSandboxAssets` takes a destination and links each asset from the cache into `<actor dir>/sandbox-assets/<name>`; the gVisor release is a tree with runsc next to its helpers. One download per node, shared by concurrent callers; a running actor protects its assets by link count. Run fetches into a staging dir before it resets the actor's other directories, so a failed fetch leaves a live activation untouched, and moves the staged set into place together with the sandbox record after the reset, so the two never disagree. An actor keeps its staged links, and so its binaries in the cache, while it stays resumable on this node through its local snapshot; Restore empties the dir before staging, since it may bring another runtime. Two calls staging the same assets for one actor at once both succeed: a hard link is complete the moment it exists and the cache places a tree by rename. Both workers derive the actor's sandbox assets dir from `actor_uid` and reject a request naming any other, so a request cannot choose the root its binary paths are checked against. `ActorDirs` gains `sandbox_assets_dir`; the kata process cleanup identifies a sandbox's VMM by the same derived directory. Startup probes a hard link from the cache dir into the actors dir and refuses to start if it fails, since every hit would fail the same way. The static-files layout and `nodepath.StaticFilesDir` are removed. Prewarm stages into a scratch directory and removes it. Flags: `--sandbox-asset-cache-dir`, `--sandbox-asset-fetch-timeout` (5m), and `--sandbox-asset-cache-min-age` (1h, recorded for the GC pass to come). **Roll together.** atelet and both ateoms change the asset path contract in the same commit. Old nodes keep a stale `static-files` directory until recreated; nothing cleans it up, per the pre-1.0 compatibility policy. **Tests.** Eleven new filecache tests (tree miss/hit with inode and mode checks, singleflight, kind mismatch, exec bits, tree eviction). In atelet: two concurrent Runs share one download and end up with one inode; the gVisor release path is covered end to end for the first time, including that the tarball is not published with the tree; a canceled caller no longer stops a download others can use, which replaces the old canceled-request test whose premise does not hold with detached fetches. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…t-substrate#2752) Populate CheckpointRequest.Snapshot.Object on both immediate and deferred checkpoints so atelet knows where to upload a paused actor's local snapshot in the background. PauseActor derives the URI from the persisted snapshot UUID rather than recording a pending DurableSnapshot on the actor, so an in-progress pause stays distinguishable from an in-progress suspend. Both workflows validate the template's storage location before marking the actor SUSPENDING. Part of agent-substrate#2069 and agent-substrate#2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…bs (agent-substrate#2666) Fixes agent-substrate#2419 1. In `internal/proto/ateletpb/atelet.proto`: - Rename `EnvEntry` -> `EnvVar`. - Rename `DurableDirVolume` -> `DurableDirVolumeSource` and `SystemInfoVolume` -> `SystemInfoVolumeSource`. 2. In `internal/proto/ateompb/ateom.proto`: - Rename `Container.csi_volume_mounts` -> `external_volume_mounts`. - Rename `ActorDirs.volumes_dir` -> `external_volume_mounts_dir`. - Consolidate `VolumeMount`, `DurableDirVolumeMount`, `SystemInfoVolumeMount`, and `ImageVolumeMount` into a single `VolumeMount { string name = 1; string mount_path = 2; }` message (matching `ateapi.VolumeMount` and `atelet.VolumeMount`).
…tor JWTs (agent-substrate#2099) Part of agent-substrate#1756. - Adds `cmd/ate-idp-server`, which serves `/.well-known/openid-configuration` and `/openid/v1/jwks` over HTTPS with a servicedns credential bundle, so relying parties can verify actor JWTs. - Reads the actor JWT authority pool from `--actor-id-jwt-pool` with `localjwtauthority.RefreshingPool`, the reader ate-api-server signs with, and builds each document from it per request. - Builds both documents at startup and exits if it can't. After that, a request whose document can't be built gets a 500 rather than an empty or stale set. - Requires `--issuer` rather than deriving a default, so it can't disagree with ateapi, and accepts only `https://<host>`, so the routes are fixed. The check is the new `actoridjwt.ValidateIssuer`; the discovery document moves out of `internal/oidcdiscovery` into the server, its only user. - Makes the EC case in `localjwtauthority.JWKS` explicit: P-256 with fixed 32-byte coordinates and an error for any other curve, rather than reading the coordinate size off the encoded point. - Records in `docs/metrics/substrate.yaml` that it emits no metrics. For reviewers: the pool holds the signing keys, so ate-idp-server will mount the same Secret ate-api-server does, in place of the controller and ConfigMap in agent-substrate#2134. ate-api-server and ate-setup still accept an issuer with a path until the installer deploys ate-idp-server. It serves `/healthz` but no `/readyz`. Nothing deploys it yet. Testing: unit tests for routing, the host-only issuer check, the discovery document, startup failures, key changes, 500s on a broken set, the P-256 encoding, and required flags, plus a local run over TLS against a generated pool. E2E can't cover it until it's deployed. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…t-substrate#2461) Part of agent-substrate#207 Drops the temporary exclusions and fixes what they hid. Field docs start with the JSON name. `sandboxClass` is optional, the `gvisor` default is unchanged. Written with AI assistance. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Bring in per-actor sandbox asset staging, checkpoint storage URI validation, and actor JWT discovery. Keep the fork's runtime integration, synchronize chart CRDs, and publish the discovery server image with the other components. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
marked this pull request as ready for review
October 16, 2026 20:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge six upstream commits through agent-substrate/substrate@d22a4bb. This brings in actor JWT discovery and JWKS serving, per-actor sandbox assets staged from the node cache, required storage URIs for every checkpoint, consistent internal volume/environment types, and the SandboxConfig defaulting fix.
Add
ate-idp-serverto the release image builds and synchronize the Helm CRDs. The discovery server is not deployed by upstream or the chart yet, so this upgrade does not expose a discovery endpoint. Preserve the fork's agentgateway dataplane, optional atelet TLS, workflow deadlines, and immediate worker-cache updates; resolve the prewarm test conflict using the new asset cache while retaining the fork's separate pause-image coverage.Breaking change
Upgrade the API server, atelet, and workers together. Checkpoints now require an object-storage URI even when a pause keeps the snapshot local; actor templates need a usable
snapshotConfig.storageLocation. Custom worker integrations must also use the updated asset paths and internal protobuf types. See the chart upgrade notes.Please apply the
breaking-changelabel.Validation:
env -u NO_COLOR go test -race -p 1 ./..., all repository verifier checks, Helm lint, all 18 chart tests, manifest rendering, and CRD parity passed locally. The verification driver stopped at lint's package-loading timeout; lint passed separately with--timeout=15mafter a shared temporary-disk exhaustion retry, and the remaining verifiers passed. All five CI checks passed, including the full test job, Helm E2E, and agentgateway E2E with gVisor and microVM in both preview configurations. Local runtime E2E has not been run.Merge this PR with a merge commit to preserve upstream ancestry for the next sync.