fix(cleanup): scope the #1581 orphan agent-volume sweep to this stack (#3214)

[dolho]

Summary

The #1581 orphan agent-volume sweep decided "nobody owns this volume" from one stack's database while listing the whole Docker daemon's volumes. A second stack on a shared daemon therefore force-removed every other stack's unattached agent volumes, up to 100 per cycle, and logged it at INFO as a routine reclaim. On one developer host that was 207 of 238 volumes, per the 09-24 learning.

• One identity source. instance_identity.get_instance_id() returns the full installation_id. This is the same durable, write-once id the alert label already abbreviates, so no second notion of identity is introduced. Any failure returns None, and every caller then fails closed.
• Labelled at creation. docker_utils.agent_volume_labels(base, platform) is the one label builder. It adds trinity.instance=<id>, and all six creation sites use it: crud.py ×3, lifecycle.py ×2 and deploy.py. A guard test fails if any site builds agent-volume labels by hand.
• Scoped sweep. list_agent_data_volumes(instance_id) filters on the label key and value at the daemon, so a foreign volume is never a candidate. If this stack's id can't be resolved, the sweep skips the cycle.
• Purge path closed too. is_reclaimable_agent_volume refuses a volume labelled for another instance, and refuses a labelled volume when our own id is unknown. Two stacks can both have an agent called alpha, so the retention purge of alpha could hit the other stack's volumes. This is in blast radius and not named in the issue.
• Legacy policy (AC 3): fail closed. Docker labels can only be set at creation, so the issue's "backfill" can't stamp existing volumes. A backfill would need a new DB table of claimed volume names, which buys almost nothing, because this stack's own volumes are already removed at retention purge from its ownership rows. So the sweep never reclaims an unlabelled volume. Unlabelled volumes with no owner and no container are named in one WARNING, repeated only when that set changes, for a human to clean up. This is stated in the sweep's docstring.
• Logs (AC 4). The sweep's summary and each per-volume removal now log at WARNING, say "unrecoverable" and name the volumes.
• Docs. reliability.md (the sweep's section) and the 09-24 learning, which now points here (AC 6).

Changes

• services/instance_identity.py: get_instance_id
• services/docker_utils.py: AGENT_VOLUME_INSTANCE_LABEL, agent_volume_labels, the scoped list_agent_data_volumes(instance_id), report-only list_all_agent_data_volumes, and instance checks in the guard and remove_agent_volumes
• services/cleanup_service.py: the scoped sweep, _report_unlabelled_orphan_volumes, and WARNING logs
• services/agent_service/{crud,lifecycle,deploy}.py: the label builder at every creation site
• Tests: tests/unit/test_3214_volume_instance_scope.py (18 new). Two existing #1581/#1664 fakes were updated for the new instance_id argument.

Test Plan

• test_3214_volume_instance_scope.py: 18 pass, and all 18 fail without the fix. They cover:
  • a foreign volume, unattached and 2h old, surviving 6 cycles (AC 2);
  • this stack's own orphan still being reclaimed (AC 5);
  • a legacy volume never being reclaimed, and the WARNING firing once (AC 3);
  • the purge path refusing a foreign volume;
  • the WARNING level and "unrecoverable" wording (AC 4);
  • every creation site using the builder (AC 1).
• The related suites (33 files: volume reclaim, renames, cleanup, identity, create/deploy/lifecycle) pass: 626, including under -p randomly --randomly-seed=12345.
• Manual: on a shared daemon with two stacks, a foreign volume survives the 15-minute window, and pre-existing volumes appear in the "unlabelled" WARNING instead of being removed.

Fixes #3214

🤖 Generated with Claude Code

[github-actions]

⚠️ Nightly unit-suite found regressions when this PR is merged into dev (3 of 3 seeds).

Regression details (head_sha: `bed902aee04683fd47ffe85ab3c09ea8fa2b71cf`)

Seed 12345

Backend unit-suite regression diff

Per-XML totals

Side	Path	Total	Pass	Fail	Error	Skip
base	junit-base-pr3231-12345.xml	21379	21341	0	0	38
head	junit-head-pr3231-12345.xml	21397	21353	1	5	38

❌ New failures introduced by HEAD (6)

Tests failing under HEAD that did not fail under BASE in any seed:

• [F] test_2669_minting_accessor_callers::test_every_minting_accessor_use_is_a_classified_writer
• [E] test_agent_readiness_probe::test_non_200_response_keeps_polling
• [E] test_agent_readiness_probe::test_returns_false_on_timeout
• [E] test_agent_readiness_probe::test_returns_true_after_initial_connect_errors
• [E] test_agent_readiness_probe::test_returns_true_when_agent_ready_immediately
• [E] test_agent_readiness_probe::test_unexpected_exception_swallowed

Legend: [F] = assertion failure, [E] = collection or fixture error.
Identity = (classname, name, kind); union taken across all input XMLs.

---

Seed 67890

Backend unit-suite regression diff

Per-XML totals

Side	Path	Total	Pass	Fail	Error	Skip
base	junit-base-pr3231-67890.xml	21379	21341	0	0	38
head	junit-head-pr3231-67890.xml	21397	21353	1	5	38

❌ New failures introduced by HEAD (6)

Tests failing under HEAD that did not fail under BASE in any seed:

• [F] test_2669_minting_accessor_callers::test_every_minting_accessor_use_is_a_classified_writer
• [E] test_agent_readiness_probe::test_non_200_response_keeps_polling
• [E] test_agent_readiness_probe::test_returns_false_on_timeout
• [E] test_agent_readiness_probe::test_returns_true_after_initial_connect_errors
• [E] test_agent_readiness_probe::test_returns_true_when_agent_ready_immediately
• [E] test_agent_readiness_probe::test_unexpected_exception_swallowed

Legend: [F] = assertion failure, [E] = collection or fixture error.
Identity = (classname, name, kind); union taken across all input XMLs.

---

Seed 99999

Backend unit-suite regression diff

Per-XML totals

Side	Path	Total	Pass	Fail	Error	Skip
base	junit-base-pr3231-99999.xml	21379	21341	0	0	38
head	junit-head-pr3231-99999.xml	21397	21352	1	5	39

❌ New failures introduced by HEAD (6)

Tests failing under HEAD that did not fail under BASE in any seed:

• [F] test_2669_minting_accessor_callers::test_every_minting_accessor_use_is_a_classified_writer
• [E] test_agent_readiness_probe::test_non_200_response_keeps_polling
• [E] test_agent_readiness_probe::test_returns_false_on_timeout
• [E] test_agent_readiness_probe::test_returns_true_after_initial_connect_errors
• [E] test_agent_readiness_probe::test_returns_true_when_agent_ready_immediately
• [E] test_agent_readiness_probe::test_unexpected_exception_swallowed

Legend: [F] = assertion failure, [E] = collection or fixture error.
Identity = (classname, name, kind); union taken across all input XMLs.

Reproduce locally: git merge dev && ( cd tests && python -m pytest unit/ -m "not slow" -p randomly --randomly-seed=12345 )

[vybe]

merge-train (2026-10-05, evening run): on the train (#3251). Nothing was pushed to this branch.

Ruling on legacy volumes (#3214 asked for one): fail-closed is accepted. Unlabelled volumes are never swept, so the sweep only applies to volumes created after the upgrade.

Follow-ups from validation, none blocking, for a later PR:

1. The sweep can still remove an unlabelled volume, against the docstring at cleanup_service.py:1838. Candidates are filtered per volume, but remove_agent_volumes(volume_base, instance_id) (:1938) walks all three suffixes of the base, and the guard at docker_utils.py:425-427 passes an unlabelled one. A base with a labelled orphan -public and an unlabelled -workspace loses both, and the second never went through the attached, age or strike triage. It is narrower than dev was. Removing only the listed names, or a strict flag on the sweep's call, would close it; no test composes the sweep with the real removal.
2. get_instance_id mints from read paths. The No OSS guard stops a fourth get_or_create_installation_id call from a read path — the guard lives in the private repo, the accessor lives in OSS #2669 allowlist entry justifies the mint by the create path, but the 5-minute sweep (cleanup_service.py:1854) and the purge (docker_utils.py:454) call it too, so every install mints installation_id within one cleanup cycle of boot. The non-minting get_installation_id() is enough there: with no id, no volume can carry it.
3. Stale "the orphan sweep will retry" comments at cleanup_service.py:1739-1741, the log at :1774, and in deploy.py. For unlabelled or guard-refused volumes that is no longer true.
4. deploy.py:527 and :582 force-remove a workspace volume without the instance check; outside bug: the #1581 orphan agent-volume sweep is instance-blind — a second stack on the same Docker daemon force-removes other stacks' agent volumes and logs it at INFO as a successful reclaim #3214's scope.
5. _report_unlabelled_orphan_volumes adds a second daemon-wide list_attached_volume_names() per cycle; compute once and pass it in.

[vybe]

merge-train: batch validated on train/20261005-1731 (#3251)