Skip to content

feat: add Boat (boat.dev) remote execution environment provider - #1601

Open
zozo123 wants to merge 11 commits into
stacklok:mainfrom
zozo123:feat/box-remote-environment-v2
Open

zozo123 wants to merge 11 commits into
stacklok:mainfrom
zozo123:feat/box-remote-environment-v2

Conversation

@zozo123

@zozo123 zozo123 commented Sep 15, 2026 •

Copy link
Copy Markdown

Summary

Adds a Boat (boat.dev) remote execution-environment provider on top of mecatl's existing placement / durable EnvironmentRef seam, in internal/adapter/boatenv.

  • provisions a fresh sandbox for the deployment-default placement and reattaches the exact persisted sandbox by opaque ID
  • keeps model/provider/GitHub/SSH account secrets out of guests by forcing noEnv: true on every create and resume
  • binds Workspace and CommandRunner to the same sandbox namespace and exposes the server affinity proof
  • implements the full versioned Workspace + WorkspaceNamespace contract and passes both shared fsconformance tables (30/30, offline and live)
  • implements tool.AuthorityResourceResolver, so authority-bound sessions can use the file tools
  • implements the side-effect-free server.PlacementValidator, so daemon startup never provisions a billable sandbox
  • preserves the no-FS attenuation without provisioning remote compute
  • uses a non-secret configuration fingerprint as EnvironmentRef.Revision; the API credential never enters refs, logs, metadata, or source

Naming

The vendor formerly known as Ascii Box now ships as boat.dev. This PR targets the current API (https://boat.dev/api/v1, the /sandboxes resource family); the old ascii.dev/api/box/v1 endpoint sunsets 2026-10-31. EnvironmentKind is "boat".

How it matches the sibling backends

The adapter is held to osfs's behavior, not just the interface:

  • Glob runs doublestar.GlobWalk with osfs's exact options (WithNoFollow, symlink leaves dropped, leading / and ./ stripped). It walks a guest listing that never follows symlinks and is fetched in one round trip, bounded to the pattern's depth.
  • Grep compiles with Go's regexp (RE2, the syntax the tool advertises). It translates the parsed pattern into an equivalent Python pattern, with every class and case fold expanded to explicit ranges, and runs it next to the files. Lines split on \n, binary files are skipped, and osfs's budgets and 201-match cap apply.
  • Symlinks: Remove/Rename act on the link itself, ReadDir uses lstat and reports fs.ModeSymlink, and a write through an escaping link returns ErrPathEscape.
  • CAS: a guest-side lock plus SHA-256 versions. A zero FileVersion is never a wildcard. New files are published atomically.

These are tested against an osfs workspace on the same directory, so osfs itself is the oracle.

Composition, and coordination with #1614

The adapter is composed through the real app.Build with app.Config.PlacementProvider set to the Boat provider, and driven by scripted sessions (offline and live). That surfaced three composition gaps:

  1. Startup preflight provisioned a sandbox. This PR adds server.PlacementValidator with the same name, signature and call site as feat: draft native Kubernetes execution provider #1614, so the two merge cleanly whichever lands first. Boat implements it as one GET /me. A non-sentinel validator error keeps its cause and is classified as ErrPlacementUnavailable.
  2. Authority-bound file tools were denied. Fixed by AuthorityResourceResolver (above), resolved in the guest under the same confinement as file access.
  3. Shell needs a local workspace and shell even when the placement supplies the runner. feat: draft native Kubernetes execution provider #1614's RemoteExecution posture (runner != nil || remote) removes that gate, and this PR does not duplicate it. Until feat: draft native Kubernetes execution provider #1614 lands, the Shell composition test switches Shell on with a throwaway host workspace and asserts nothing is written there.

This PR adds no operator flag or config grammar while #580 and #1614 settle that surface.

Lifecycle and limits (measured against boat.dev)

Fact (live) How the adapter handles it
A command string over ~128 KiB fails with E2BIG File content never rides the command line. Large requests are staged through the files API.
Files API writes are capped at 5 MiB per call Staging is chunked.
Command stdout is capped at 8 MiB, with stdoutTruncated set Large helper output spills to a guest snapshot fetched in ranges. Truncation is reported, never passed off as complete.
timeoutSeconds runs 1–600; a 75s command is honored The limit follows the caller's deadline. There is no client-wide HTTP timeout.
A signal-killed command returns exitCode: null plus a signal Reported as exit −1, with the signal named.
A missing cwd is rejected A non-default Workdir is created on first use.
PATCH ttlSeconds re-arms archival relative to now Close schedules archival after a 5-minute grace instead of archiving at once, so the run that follows CreateSession finds the sandbox warm. Readiness re-arms the full TTL.

Reattach costs one GET. The resume, if needed, happens on the first operation. A 409 from a command, or from a concurrent resume, is retried once after readiness, and the command never runs twice. Redirects are refused, so the bearer credential is never replayed. The helper runs as python3 -I, so workspace files cannot shadow its imports.

Tests

The offline fake enforces the live limits above (argv size, files-API size, missing cwd, stream caps, signals, timeouts, 409). An adapter that only works against a lenient fake fails here first. Coverage:

  • both fsconformance tables
  • osfs parity for Glob and Grep (13 glob and 20 grep patterns on one tree) and for symlink semantics
  • 300 KiB and 6 MiB round trips
  • timeout derivation, signal mapping, redirect refusal, error-envelope reasons
  • lazy reattach, a sandbox archived behind the adapter's back, and a concurrent resume race
  • app.Build composition
  • the server PlacementValidator seam

The regression tests were mutation-checked: reintroducing any of four original defects fails them.

go test passes for ./internal/adapter/boatenv/, ./internal/adapter/server/..., ./internal/app/..., ./cmd/mecated/, ./cmd/mecak8s/ and ./docs/lint. -race passes on the changed packages, the pinned golangci-lint v2.13.1 reports 0 issues, and matlatl check --strict is clean. The cloud-native inventory (ADR 0027, List 1) gains row 76 for the provider's client, handles and guest staging.

Live verification

Opt-in, under the repo's e2e build tag and driven only by BOAT_API_KEY:

BOAT_API_KEY=... go test -tags e2e ./internal/adapter/boatenv/ -run TestLive -v -timeout 45m

Run against boat.dev on 2026-09-23:

Test Covers Result
TestLiveBoat create → versioned write/read → stale-CAS rejection → replace → grep → Shell/Workspace agreement → archive → exact reattach with the filesystem intact. Plus a 70s command under a 150s deadline, a signal-kill, a 6 MiB round trip, Remove on a symlink, a brace glob, a POSIX-class grep, and stale-revision rejection PASS, 107.6s
TestLiveBoatConformance both shared conformance tables on one sandbox, one directory per case 30/30 PASS, 31.6s
TestLiveBoatComposition real app.Build: GET /me preflight, exactly one sandbox, authority-checked Write/Edit/Grep/Read plus Shell, nothing on the host PASS, 4.2s

Observed lifecycle: provisioning -> idle -> archiving -> archived -> provisioned -> ready.

🤖 Generated with Claude Code

@jhrozek

jhrozek commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@zozo123 thanks for the PR! I think this is exciting in the sense that the PR does show how the harness can be decoupled from the execution environment.

I'm not familiar with Box but checking the wiring with mecatl I think this is missing a builder composition (something like boxenv.New?) to be testable? Also, check out PR #1614 that adds a native k8s execution environment - I would suggest to coordinate e.g. internal/adapter/executioncontroller/podexec.go is a good place to look there for inspiration.

Thanks again for the contribution! Feel free to jump on Discord or GH discussions if you want to discuss anything!

@zozo123 zozo123 changed the title feat: add Box remote execution environment provider feat: add Boat (boat.dev) remote execution environment provider Sep 23, 2026
@zozo123

zozo123 commented Sep 23, 2026

Copy link
Copy Markdown
Author

@jhrozek to answer your question on #1604: this is the current PR. I've closed #1604; #1598 and #1603 were already closed.

New since your review (6a22253): the vendor rebranded from Ascii Box to boat.dev, so the adapter now lives at internal/adapter/boatenv and uses EnvironmentKind "boat". It targets the current API at https://boat.dev/api/v1, where the resource is /sandboxes. The old ascii.dev/api/box/v1 endpoint shuts down on 2026-10-31.

On the live API, the opt-in contract smoke TestLiveBoat passes in 84.7s. It creates a sandbox, archives it, reattaches the exact same sandbox and checks the file is still there. It also confirms that a stale version is rejected on write and that a stale revision is rejected on reattach. The offline suite and the pinned golangci-lint v2.13.1 both pass locally.

Your builder-composition point and lining this up with the k8s environment in #1614 (podexec.go) are next. I'll follow up here.

🤖 Generated with Claude Code

zozo123 and others added 8 commits September 23, 2026 11:31
The vendor formerly known as Ascii Box now ships as boat.dev, and the
old ascii.dev/api/box/v1 endpoint sunsets 2026-10-31. Rename the package
internal/adapter/boxenv -> internal/adapter/boatenv and move every call
onto the current API, verified against the live service:

- base URL https://boat.dev/api/v1 and the /sandboxes resource family
  (create, get, resume, stop, commands) replacing /boxes
- response envelope key "sandbox" replacing "box"
- EnvironmentKind "boat", revision prefix boat-v1-, workspace root
  boat:<sandbox-id>:<workdir>, live-smoke credential BOAT_API_KEY
- Config.BoxType -> Config.MachineType (the API's machine-size field)

Two behavioural fixes found while retargeting:

- ensureReady no longer lets waitReady issue a second resume for a
  sandbox it just resumed; boat.dev reports the transitional states
  provisioning/provisioned/resuming/archiving, which are now polled
  through rather than acted on.
- the opt-in live smoke now proves the durability claim end to end:
  archive the provisional sandbox, reattach the persisted ref, and read
  the file back, plus stale-CAS and stale-revision rejection.

Also clears two lint findings the package carried (unchecked
resp.Body.Close, unused versionOf).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nce tables

Answers the review ask for a builder composition that makes the adapter
testable end to end, and lines it up with stacklok#1614.

Composition (internal/adapter/boatenv/composition_test.go): drive scripted
sessions through the real app.Build with the Boat provider as the
deployment PlacementProvider. Doing so surfaced three gaps that library
tests could not see, all fixed here:

- Startup preflight provisioned a sandbox. configuredPlacementBinder
  validated the default placement by binding it, which for Boat creates
  and archives a billable sandbox on every daemon start. Add the
  side-effect-free server.PlacementValidator seam, with the same name,
  signature and call site as stacklok#1614 so the two merge cleanly, and implement
  it for Boat as GET /me. Providers without the seam keep the legacy
  bind-to-validate path (both pinned in placement_validator_test.go).
- File tools were denied by authority: the workspace could not derive an
  authority resource identity. Implement tool.AuthorityResourceResolver
  with a guest round trip that resolves symlinks under the same
  confinement as file access, so policy and access name the same file.
- Shell needs a local workspace and shell on main even when the
  placement supplies the runner. stacklok#1614's RemoteExecution posture removes
  that gate; until it lands, the Shell composition test enables it with a
  throwaway host workspace and proves nothing runs there.

Conformance: run engine/adapter/fsconformance Run and RunNamespace
against the Boat workspace, offline over the fake API and live over one
sandbox with a directory per case. It caught a contract bug: ReplaceFile
with a zero FileVersion on a missing path reported a version mismatch
instead of fs.ErrNotExist. A zero version still goes to the guest, which
now reports a missing file before a stale version.

Live, against boat.dev: TestLiveBoat (lifecycle, CAS, archive and exact
reattach), TestLiveBoatConformance (30/30) and TestLiveBoatComposition
(app.Build, one sandbox, Write/Edit/Grep/Shell/Read) all pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zozo123
zozo123 force-pushed the feat/box-remote-environment-v2 branch from 6a22253 to d4c006b Compare September 23, 2026 08:53
@zozo123

zozo123 commented Sep 23, 2026

Copy link
Copy Markdown
Author

@jhrozek following up on your two points (d4c006b):

Builder composition. The adapter is now composed through the real app.Build, with app.Config.PlacementProvider set to the Boat provider, and driven by scripted sessions (internal/adapter/boatenv/composition_test.go, plus a live variant). Composing it end to end found three gaps that the library-level tests missed:

  1. Startup preflight bound the default placement, so every daemon start provisioned and archived a billable sandbox.
  2. Authority-bound file tools were denied because the workspace couldn't derive a resource identity.
  3. Shell only appears when the deployment has a local workspace and shell.

Coordinating with #1614.

The workspace now also passes the shared fsconformance tables, 30/30 live. That caught a real contract bug: a zero-version ReplaceFile on a missing path returned a version mismatch instead of fs.ErrNotExist. It's fixed.

All three live suites pass against boat.dev; the numbers are in the updated description.

🤖 Generated with Claude Code

… boat.dev

A six-lens review of this PR, with three independent skeptics per finding,
confirmed defects the earlier tests could not see. Each is fixed here and
pinned by a regression test; the live limits they depend on were measured
against boat.dev before changing code.

Commands
- The guest time limit now follows the caller's deadline (clamped to the
  API's 1-600s) instead of a hard 60s cap, and the client has no
  client-wide HTTP timeout. A command over 600s reports Boat's ceiling
  rather than the caller's value.
- A signal-killed process (live: exitCode null, signal SIGKILL) reports
  exit -1 and names the signal instead of passing as exit 0.
- API errors carry the server's code and message; redirects are refused
  so the bearer credential is never replayed; responses have a size bound
  with an explicit error; truncated stdout is flagged.

Files
- Content no longer travels on the command line (live: commands over
  ~128 KiB fail with E2BIG). Large requests are staged through the files
  API (5 MiB per call, chunked); large reads come back as a guest snapshot
  fetched in ranges and are checked against their version hash.
- The helper runs as python3 -I, so workspace files cannot shadow its
  imports, and it now lives in helper.py (embedded).
- Symlinks follow the osfs contract: Remove and Rename act on the link,
  ReadDir uses lstat and reports ModeSymlink, a dangling link no longer
  fails a listing, and writes through an escaping link report
  ErrPathEscape. New files are published atomically.
- Glob runs doublestar with osfs's exact options over a guest listing that
  never follows symlinks, so brace patterns, a leading "/" and "dir/**"
  behave as on osfs. Grep compiles with Go's RE2, translates the parsed
  pattern into an equivalent Python pattern (explicit ranges, no reliance
  on Python's \w, [[:space:]] or (?i)), splits lines on "\n", skips binary
  files, and applies osfs's budgets and 201-match cap. Both are checked
  against an osfs workspace on the same directory.

Lifecycle
- Reattach costs one GET; readiness (resume, workdir creation) happens on
  the first operation and re-arms the TTL. A non-default Workdir is now
  created (live: a missing cwd is rejected).
- Close schedules archival after a 5-minute grace instead of archiving at
  once, so the run that follows CreateSession finds the sandbox warm. The
  live app.Build composition went from 46.8s to 4.2s.
- A 409 from a command or a concurrent resume is retried once after
  readiness; the command never ran twice.

Server seam: a validator's non-sentinel error keeps its cause and is
classified as ErrPlacementUnavailable.

Live tests now build under the repo's e2e tag, use dedicated workdirs, and
register sandbox cleanup the moment a sandbox exists. The cloud-native
inventory gains row 76 for the provider's client, handles and guest staging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zozo123

zozo123 commented Sep 23, 2026

Copy link
Copy Markdown
Author

Update (c7f3a0a): before asking for review, I ran an adversarial review of this PR and fixed what it found. Each fix has a regression test, and the tests were mutation-checked.

  • Shell timeouts now follow the caller's deadline (Boat allows 1–600s). There was a hard 60s cap plus a 70s HTTP client timeout.
  • A signal-killed command reports exit −1 and names the signal. It used to decode Boat's exitCode: null as exit 0.
  • Large files: content no longer goes on the command line. The live API rejects any command string over about 128 KiB, so writes are staged through the files API and large reads are paged. 6 MiB round trips pass live.
  • osfs parity: Glob uses doublestar with osfs's exact options. Grep compiles RE2 and translates it faithfully, instead of passing the pattern to Python re. Symlink Remove/Rename/ReadDir follow osfs. These are tested against an osfs workspace on the same directory.
  • Lifecycle: Reattach is a single GET and readiness happens lazily. A non-default Workdir is now created. Close sets a 5-minute grace TTL instead of archiving immediately, so a new session's first run finds its sandbox warm. The live app.Build composition went from 46.8s to 4.2s.

Live against boat.dev: TestLiveBoat, TestLiveBoatConformance (30/30) and TestLiveBoatComposition all pass. They now build under the repo's e2e tag.

CI: pull-request workflows on this PR are still waiting for approval because it comes from a fork. The same workflows ran on my fork at this exact commit: CI run, with 29 jobs passing and 1 skipped (race partitions, lint, analysis, docs, domain model, api-compat and vuln all green). Deslop also passed. Details are in the updated description.

🤖 Generated with Claude Code

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey! Thanks a lot for pushing this. I'm working on refining the definitions for execution environments. I'll get to this tomorrow and hopefully it'll be in better shape then. Super excited to see this!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants