[earakely-scale]

What

An environment artifact can now hold several files, each kept by its path. Every environment artifact today is the one-file case of that, and is stored, read and loaded exactly as before.

• One file whose path is its filename is stored as the file_artifact_ref every environment artifact had, document for document, whichever put wrote it. Golden documents are unchanged.
• Several files, or one inside a folder, are stored as file_artifact_refs: {path: ref}, the shape a FileArtifactUniverse uses.
• A load hands each file back to the env's data/add with its path, as data/get returned them.

Issue: FD-4818. Design (approved): Environment Artifacts With Several Files. This is the prerequisite for #423 (FD-4382), which is rebuilt on it next.

How

• EnvironmentArtifact
  • put takes file_artifact or file_artifacts (by path), exactly one. A single file whose path is its filename is written as file_artifact.
  • Paths must be plain file names joined by / (utils.paths.validate_file_path).
  • get_file_artifacts() reads either shape: {filename: file} for one file, legacy unpinned documents included.
  • get_file_artifact() is unchanged for one file and refuses a collection, naming get_file_artifacts.
• Loads over data/add. MCP server envs, website envs and signed-URL loads for a plugin's env send one file part per file, each with its path in the data-objects marker (_deployment.file_parts).
  • A one-file artifact sends exactly the part it always has, with no marker, staged at /data/<filename> as before; its filename is still checked as one plain name.
  • Several files are staged at their paths inside a new folder of the load's own (/data/.agentenv-load-<8 hex>/, /tmp/data/... for a website), removed after the load however it ends (_deployment.staged_paths, _remove_from_container). So a file an earlier load left can't be in the way of this load's folders, and repeated loads leave nothing behind.
  • An MCP server env sizes the load timeout from the files' total size. A signed-URL load signs a URL for each file.
  • The pre-v1 loads take one file and refuse a collection, naming it (legacy_load_file).
• Staging into an agent, a container or a sandbox (load_artifact with agent_name, container_name or sandbox_name):
  • one file: today's payload expansion, unchanged;
  • several files: each file written at its path through stage_files_into_container, the helper that stages a file universe into an agent.
• verify_universe_load_export_roundtrip compares an environment of several files export to export only, as it does a non-JSON one, since it isn't one JSON state. The agent-judge bundle puts each of its files at original/<name>/<path>, and both judge prompts describe that folder: the judge reads what the service was seeded with and checks export_1 still holds it in meaning, corroborated by export_2.
• Universes. EnvironmentUniverseArtifact.get_file_artifacts() lists <environment_name>/<path> for each file of a collection; a one-file environment is listed at its filename, as before. Exporting a universe that holds a multi-file environment is refused (Unwritable), since a universe's folder holds one file per environment.
• Bundle folders.
  • An environment artifact's folder may hold several files, written as a collection keyed by path from the folder, each file as <id>__<16 hex of its path's sha256>, as a file universe's are. That rule is now one function, store.ids.derive_path_id, which FileArtifactUniverse.put_bundled and put_existing, EnvironmentArtifact.from_toml and collect_artifacts all use; no id changes.
  • A lone file, even inside a subfolder, is still today's one-file environment.
  • description beside several files is refused.
  • Export writes each file at its path. A collection whose one file is kept at a path other than its filename is refused, because a folder's one file reads back kept at its filename.

Readers outside this repo

The hub reads file_artifact_ref from environment documents and calls get_file_artifact(). One-file artifacts are unchanged, so nothing there breaks. A collection in the hub's store would be listed as missing files or fail a universe download, and an agent-env older than this release reads one as malformed. Supporting collections in the hub is a follow-up after CUA OSS v1, since computer envs aren't used through the hub yet.

Not in this PR

• Snapshots keeping several parts, and loading environment artifacts into a computer env (feat(snapshot): snapshot envs without a gateway, keep a file tree's files, and load a computer's snapshot back #423).
• Apps (M1.4, FD-4369).
• A universe folder layout for multi-file environments.
• A CLI flag to put several files; a bundle folder writes one.

Testing

Unit: 12,295 passed, none failed, on macOS.

Also folded in, two test fixes:

• test_the_event_pipe_is_not_left_open_in_the_server (Hermes) counted open file descriptors through /proc/self/fd, which macOS doesn't have, so it failed there every time. It now lists /dev/fd, which macOS and Linux both have (on Linux it links to /proc/self/fd). On macOS it passes, and still fails when the bridge leaves the pipe's write end open (13 descriptors against 12).
• test_a_second_hold_restarts_the_release_timer (protocol computer queue) flaked once under a loaded parallel run, with about 50 ms of slack per tool call. It now holds for 1.5 s with the second hold 0.3 s after the first, and asserts the keys go up no sooner than 1.5 s after the second hold was sent. Being slow can't fail it, and it still fails without the restart (keys up 1.20 s after the second hold); it passed 40 of 40 times in parallel on a loaded CPU.

New and extended tests:

• The model:
  • a one-file put through either argument writes the same document;
  • several files, or one in a folder, are stored by path and read back, while get_file_artifact refuses;
  • a legacy unpinned document reads as its one file;
  • bad paths, and put with neither or both arguments, are refused;
  • one file inside a folder can't be exported.
• Loads:
  • an MCP server env stages each file at its path in a folder of the load's own (folders made) and sends each part with its path, timed by the total size, then removes the folder; a load that fails while staging still removes it;
  • a website env stages several files the same way and removes the folder;
  • a one-file artifact whose filename isn't one plain name is refused before staging;
  • a signed-URL load signs each file and sends each with its path;
  • the pre-v1 path refuses a collection before sending anything;
  • staging paths are refused unless they're plain names joined by /, and nested paths are staged under their folders.
• Staging: a collection goes through stage_files_into_container for a container sandbox, a VM's container and the VM host.
• Folders:
  • several files become a collection with the derived ids, reused by a second run and rewritten when one changes;
  • a lone file in a subfolder is still the one-file environment;
  • description beside several files is refused;
  • export reads back with each file at its path;
  • a universe holding a collection lists each file and refuses export.
• The round-trip validator: a collection is compared export to export only, its files go into the judge's bundle at their paths, and both judge prompts name that folder.

Chaos run (two Linux EC2 boxes plus the laptop, real items MCP server on local Docker; the box harness is not committed):

• Regression check, main vs this PR, one-file loads only: 16 observations identical. Covered: normal load, a load twice, a malformed file, names with spaces, shell characters and unicode, refused names (a/b.json, .., empty), a store failure while staging, and a load cancelled while staging.
• Multi-file loads under faults, all passing:
  • the store failing on the 2nd file leaves the server's data untouched;
  • a malformed file fails data/add with a clear error;
  • a cancelled load and a load into a killed server both raise, the latter within 0.2 s;
  • two loads at once both finish;
  • shell-character and unicode paths load and run nothing;
  • 500 files load in 59 s, a 35 KB document.
  • None leaves its staging folder behind.
• An older agent-env reading a multi-file document gets "malformed EnvironmentArtifact … neither file_artifact_ref nor file_artifact_id present", as described above.
• Bundle folders: .DS_Store and __pycache__ beside one file still give a one-file environment; links inside the bundle are kept, links outside refused.
• Found and fixed (3408872): on a sandbox that is a VM in container mode (the local provider), the load folder's mkdir and rm ran on the sandbox's host, not in the server, so the folder was never removed. They now go through docker exec into the server's container, as staging's writes do. Regression test: test_artifact_staging.py::test_a_server_whose_vm_sandbox_is_its_container_stages_and_removes_a_loads_folder_through_docker.
• Not a regression, worth knowing: files are staged one at a time, about 0.12 s each on a box (500 files in 59 s). Loading many files in parallel is FD-4894, at the end of v1.

Integration, laptop (local Docker):

• A real MCP server env first loads a one-file environment whose file is named inbox, then a two-file environment (inbox/a.json, top.json) in one add_data: the second load isn't blocked by the first one's file, and the state holds both files' items, timed by their total size.
• A local agent gets a two-file environment staged at its paths, found by verify_sandbox and collected back by collect_artifacts.
• The environment artifact and universe integration tests pass unchanged: 41.

🤖 Generated with Claude Code