feat: split mode across a VM boundary — a host ledger service over a socket

CHANGELOG.md

@@ -6,8 +6,38 @@ and the project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Added
+
+- **Split mode across a VM boundary: a ledger service on the host** (#169). A runner
+  configured with `ledger_socket` (instead of `ledger_home`) keeps no ledger: it reads and
+  writes the host's through a unix socket served by `tdd ledger serve`, which owns the SQLite
+  files and is their only writer. Each run records the source of the socket it arrived on.
+  The service refuses a guest another source's runs, cycles and contracts (`foreign_run`),
+  writes to its own complete or abandoned runs (`run_closed`, except `tdd note`), host-only
+  methods (`method_not_allowed`) and malformed repository paths (`bad_repo`). An unreachable
+  socket fails the verb with `reason: "ledger_unreachable"`; nothing falls back to a local
+  ledger. `tdd doctor` in a guest names the source it reaches.
+- **Operator verbs, `tdd ledger …`** (#169): `serve`, `add-source`, `remove-source`, `bind`
+  (the executor a source's runs record, as `executor_source: "operator"`; an unbound guest's
+  `operator` claim is recorded as `claimed`), `sources`, and host-wide `metrics`, `fleet` and
+  `log` across every source. `tdd ledger import <file> --source <name>` brings an existing
+  ledger in as one source. `tdd runner import` on a socket-mode runner is refused with
+  `reason: "ledger_remote"`.
+- **Host-wide worker leases** (#169). A socket-mode runner takes its worker lease from the
+  ledger service, so one core budget covers every guest; a lease ends when its connection
+  closes, and `tdd fleet` in a guest reports the host's count.
+- **`per_file_collects` in the `baseline_captured` heartbeat** (#168): the number of files that
+  fell to the per-file loop, one runner start each, next to `collect_s`. Reported by serial and
+  `--baseline-jobs` probes alike.
+
 ### Changed
 
+- **Ledger schema v13: every run has a source** (#169). `run` and `plan_contract` gain a
+  `source` column, and the baseline/advance claims and the baseline cache are rebuilt keyed
+  by source, so two guests at the same path neither block each other nor share a cached
+  baseline. Existing rows migrate as `local`. Every ledger access now goes through named
+  `Ledger` methods; `fleet` reads through `ledger.open_readonly`. `tdd metrics` reports each
+  run's `source`.
 - **A successful batch listing is authoritative for its suite's files** (#168). `collect()`
   started the runner once more for every `test_paths` file a successful whole-suite listing did
   not mention — another project's test file on a shared root, a fixture, a helper — and
@@ -19,12 +49,6 @@ and the project adheres to [Semantic Versioning](https://semver.org/).
   `(command, env)` pairs keeps the old loop, and can opt in with a third element naming the
   suite (`None` for the default, else the override).
 
-### Added
-
-- **`per_file_collects` in the `baseline_captured` heartbeat** (#168): the number of files that
-  fell to the per-file loop, one runner start each, next to `collect_s`. Reported by serial and
-  `--baseline-jobs` probes alike.
-
 ## [0.14.1] - 2026-09-25
 
 ### Fixed

README.md

@@ -689,7 +689,11 @@ cannot become:
 - executor identity is `operator` when the runner's config assigns the calling account a
   model, and `claimed` otherwise;
 - `tdd runner import <ledger>` brings an existing ledger under the runner, marked as
-  pre-split history that `tdd metrics` reports.
+  pre-split history that `tdd metrics` reports;
+- across a VM boundary, the runner in each guest keeps no ledger at all: it reads and writes
+  the host's through a socket served by `tdd ledger serve`, which records every run under the
+  source of the socket it came in on, refuses a guest anything but its own open runs, and
+  gives the operator one history and one worker budget for every guest.
 
 `tdd docs split` ([docs/split-runner.md](docs/split-runner.md)) has the setup, the
 verification checklist, and what split mode does not protect against.

docs/split-runner.md

@@ -183,5 +183,164 @@ Afterwards, remove the agent's copy so that nothing reads it by mistake.
 - `tdd fleet`, `tdd metrics` and `tdd log render` are forwarded like everything else
   and read the runner's ledger.
 - A runner invoked directly, with no sudo in between, acts for nobody: every verb
-  except `runner import` is refused with `reason: "no_agent"`. It never falls back to
-  running suites as itself.
+  except `runner import` and `ledger` is refused with `reason: "no_agent"`. It never
+  falls back to running suites as itself.
+
+## Across a VM boundary
+
+Everything above is one machine. Once agents run in disposable VMs or containers, the
+runner has to be in the guest, because that is where the suites run. Its ledger must not
+be: it would be deleted with the guest, two guests' files could never be merged, and an
+agent with root in the guest (a `docker` group membership is enough) could rewrite it
+whatever the file modes say.
+
+So the runner in the guest keeps **no ledger**. It reads and writes the host's through a
+unix socket, served by a **ledger service** on the host. The service owns the SQLite
+files and is their only writer. The runner still runs every suite, gate and git command
+as the agent, in the agent's worktree, exactly as above.
+
+    host                                         guest (one per run)
+    ┌──────────────────────────────┐             ┌──────────────────────────────┐
+    │ tdd ledger serve             │             │ agent ─sudo─▶ runner          │
+    │   admin.sock   (operator)    │             │                ledger_socket │
+    │   vm-1.sock ───forwarded──────────────────▶│ /run/tdd-cli/ledger.sock     │
+    │   <home>/<repo>.sqlite3      │             │ (no ledger file here)        │
+    └──────────────────────────────┘             └──────────────────────────────┘
+
+A run's **source** is the socket it arrived on. The operator names it when they create
+the socket, and the protocol has no field in which a guest could name it instead.
+
+### The service
+
+`/etc/tdd-cli/ledger.toml` (or `TDD_LEDGER_SERVICE_CONFIG`), owned by root or the
+service account and not group- or world-writable, the same trust rule as `runner.toml`:
+
+    [service]
+    admin_socket = "/run/tdd-cli/admin.sock"
+    # home       = "/var/lib/tdd-cli"     # default: the service account's ~/.local/share/tdd-cli
+
+Run `tdd ledger serve` under a dedicated account that owns `home`. It runs until SIGTERM
+and exits 0. A systemd unit:
+
+    [Unit]
+    Description=tdd-cli ledger service
+
+    [Service]
+    User=tdd-ledger
+    Group=tdd-guests
+    RuntimeDirectory=tdd-cli
+    ExecStart=/opt/tdd-cli/bin/tdd ledger serve
+    KillSignal=SIGTERM
+
+    [Install]
+    WantedBy=multi-user.target
+
+Host ledgers are `<home>/<slug>.sqlite3`, one per repository, where the slug is the
+guest's repository path with `/` replaced by `-`, as on a single machine. Every guest
+that checks a repository out at the same path shares that file, kept apart by source.
+
+### One source per guest
+
+The operator adds a source for each guest, from the host:
+
+    tdd ledger add-source vm-1 --socket /run/tdd-cli/vm-1.sock --executor claude-fable-5-1
+    tdd ledger bind vm-1 --executor claude-opus-5-5     # the model for its next runs
+    tdd ledger sources                                   # every source, socket and binding
+    tdd ledger remove-source vm-1                        # closes the socket; its runs stay
+
+Sources are kept in `<home>/sources.json`, so a restarted service listens on all of them
+again. The admin socket is mode 600, the service account's alone. A guest's socket is
+mode 660: give its group to whatever forwards it into the guest, and nothing else.
+
+`tdd ledger` verbs are always answered on the machine you run them on, never forwarded to
+a runner. With no service config they fail with `reason: "no_service"`.
+
+### Forwarding the socket into the guest
+
+The forwarding itself is the operator's, and tdd-cli does not set it up:
+
+- **ssh**: `ssh -R /run/tdd-cli/ledger.sock:/run/tdd-cli/vm-1.sock guest`, with
+  `StreamLocalBindUnlink yes` in the guest's `sshd_config` so a reconnect can replace a
+  stale socket, and `StreamLocalBindMask 0117` so the forwarded socket is not world-open.
+- **containers**: bind-mount the source's socket, `-v /run/tdd-cli/vm-1.sock:/run/tdd-cli/ledger.sock`.
+  Mount the one socket, never the host's ledger directory.
+- **vsock**: forward the source's socket to a vsock port with a relay such as `socat`,
+  and relay it back to a unix socket in the guest.
+
+### The guest
+
+The guest's `runner.toml` names the forwarded socket instead of a ledger home:
+
+    [runner]
+    user          = "tdd-runner"
+    command       = "/opt/tdd-cli/bin/tdd"
+    ledger_socket = "/run/tdd-cli/ledger.sock"   # instead of ledger_home
+
+Naming both `ledger_socket` and `ledger_home` is refused. In the guest, `tdd doctor`'s
+`ledger reachable` check pings the service and names the source it reached:
+`socket /run/tdd-cli/ledger.sock (source vm-1)`.
+
+**It fails closed.** If the socket is missing or the service hangs up, every verb that
+needs the ledger fails with `reason: "ledger_unreachable"`. Nothing in the guest ever
+falls back to a ledger of its own.
+
+### What it does and does not protect
+
+The service enforces this for every request, so a modified client cannot get around it:
+
+- **A guest reads and writes only its own source's runs.** A run, cycle or plan contract
+  of another source is refused with `refusal: "foreign_run"`, and so is one that does not
+  exist, so a guest cannot probe for other sources' ids. Claims and the baseline cache
+  are per source: one guest can neither block another's `run start` at the same path nor
+  hand it a forged cached baseline.
+- **A closed run stays closed.** Writes to a `complete` or `abandoned` run are refused with
+  `refusal: "run_closed"`. `tdd note` is the one exception, so a closing narrative can still
+  be added. A `blocked` run is not closed: `resume --unblock` still works.
+- **A guest reaches only named domain methods.** The wire carries a method name and its
+  arguments, never SQL. Methods meant for the host alone (`set_meta`, the generic row
+  helpers) are refused with `refusal: "method_not_allowed"`, and a repository path that is
+  not absolute is refused with `refusal: "bad_repo"`.
+
+Refusals reach the guest's envelope as `reason: "ledger_refused"` with the `refusal` code.
+
+What a compromised guest can still do is record runs under its own source, and lie in
+them, which is no worse than a forged test report on a single machine.
+
+**Executor identity is set on the host.** A guest's `[executor]` table is guest root's to
+edit, so it is a claim. When the operator binds an executor to a source, the service
+records that model with `executor_source: "operator"` on every run the source starts or
+abandons, whatever the guest sent. With no binding, a guest that says `operator` is
+recorded as `claimed`.
+
+### One history per host
+
+On the host, the operator reads every source together through the admin socket:
+
+    tdd ledger metrics [--repo <slug>]              # `tdd metrics` over every source, per repository
+    tdd ledger fleet                                # active runs of every source, and the host's suites
+    tdd ledger log --repo <slug> --run <id> [--out FILE]
+
+Every run in them carries its `source`. Inside a guest, `tdd metrics`, `tdd fleet` and
+`tdd log render` still work, and see only that guest's source.
+
+**Worker leases are the host's.** A socket-mode runner takes each suite's worker lease
+from the service, which splits the host's cores across every guest's running suites. A
+lease ends when its connection closes, which is also what a dead VM does, and one older
+than the stale limit is not counted. Named leases stay in the guest: they guard
+resources the guest's suites run against.
+
+### Existing ledgers
+
+`tdd runner import` is refused on a socket-mode runner (`reason: "ledger_remote"`): it has
+no ledger to import into. Bring an existing ledger in on the host instead, as one source:
+
+    tdd ledger import /home/agent1/.local/share/tdd-cli/<repo>.sqlite3 --source legacy
+
+It copies the file into the service's home under the same name, migrates it, retags every
+run and plan contract it recorded to the named source, drops its claims, and marks it as
+`pre_split_import`, which `tdd ledger metrics` reports. It refuses a file the host already
+holds (`reason: "ledger_exists"`). There is no merge.
+
+Ledgers from before schema v13 gain a source on their first open: every existing run,
+contract, claim and cached baseline becomes `local`, the source of single-user mode and of
+a split runner on one machine.

src/tddcli/advance.py

@@ -12,7 +12,6 @@
 from . import config as config_mod
 from .adapters.base import FAILED, NOT_COLLECTED, NOT_FOUND, PASSED
 from .envelope import Envelope, NextAction, Verb
-from .ledger import now
 from .machine import (
     AWAITING_IMPL,
     AWAITING_PIN,
@@ -28,22 +27,15 @@
 def _note_nudge(engine: Engine, cycle) -> str:
     if cycle is None:
         return ""
-    events = engine.ledger.all(
-        "SELECT kind FROM integrity_event WHERE cycle_id = ? AND kind IN ({})".format(
-            ",".join("?" * len(NUDGE_KINDS))
-        ),
-        (cycle["id"], *NUDGE_KINDS),
-    )
-    if not events:
+    if not engine.ledger.cycle_event_kinds(cycle["id"], sorted(NUDGE_KINDS)):
         return ""
-    existing_note = engine.ledger.one("SELECT id FROM note WHERE cycle_id = ?", (cycle["id"],))
-    if existing_note is not None:
+    if engine.ledger.cycle_has_note(cycle["id"]):
         return ""
     return ' An integrity event was recorded on this cycle — consider `tdd note "<why>"` while the reason is fresh.'
 
 
 def _reply(engine: Engine, cycle, verb: Verb, detail: str, **result) -> Envelope:
-    fresh = engine.ledger.one("SELECT * FROM cycle WHERE id = ?", (cycle["id"],)) if cycle else None
+    fresh = engine.ledger.cycle(cycle["id"]) if cycle else None
     nudge = _note_nudge(engine, fresh or cycle)
     return Envelope(
         run=engine.run_state(fresh or cycle),
@@ -67,11 +59,7 @@ def _adopt_target(engine: Engine, cycle, missing: list[str]) -> tuple[list[str],
     at_start = engine.ledger.collection(engine.run["id"])
     known = {t for tests in at_start.values() for t in tests}
     run_targets = {
-        t
-        for row in engine.ledger.all(
-            "SELECT target_tests FROM cycle WHERE run_id = ?", (engine.run["id"],)
-        )
-        for t in json.loads(row["target_tests"])
+        t for targets in engine.ledger.run_target_tests(engine.run["id"]) for t in targets
     }
 
     new_tests: list[str] = []
@@ -86,23 +74,11 @@ def _adopt_target(engine: Engine, cycle, missing: list[str]) -> tuple[list[str],
 
 
 def _stub_directive_issued(engine: Engine, cycle) -> bool:
-    return (
-        engine.ledger.one(
-            "SELECT id FROM integrity_event WHERE cycle_id = ? AND kind = 'stub_directive_issued'",
-            (cycle["id"],),
-        )
-        is not None
-    )
+    return engine.ledger.cycle_has_event(cycle["id"], "stub_directive_issued")
 
 
 def _last_outside_emitted(engine: Engine, cycle) -> str | None:
-    row = engine.ledger.one(
-        "SELECT detail FROM integrity_event"
-        " WHERE cycle_id = ? AND kind = 'undeclared_file_touched'"
-        " ORDER BY id DESC LIMIT 1",
-        (cycle["id"],),
-    )
-    return row["detail"] if row else None
+    return engine.ledger.last_cycle_event_detail(cycle["id"], "undeclared_file_touched")
 
 
 def _sanctioned_stubs(engine: Engine, cycle, implementation: list[str]) -> list[str]:
@@ -253,8 +229,8 @@ def _adopt(
             declared=missing,
         )
     engine.ledger.event(engine.run["id"], cycle["id"], "declared_test_mismatch", json.dumps(detail))
-    engine.ledger.update("cycle", cycle["id"], target_tests=json.dumps(kept + [adopted]))
-    cycle = engine.ledger.one("SELECT * FROM cycle WHERE id = ?", (cycle["id"],))
+    engine.ledger.set_targets(cycle["id"], kept + [adopted])
+    cycle = engine.ledger.cycle(cycle["id"])
     if outcome is None:
         return _reply(
             engine,
@@ -599,19 +575,17 @@ def _handle_refactor(engine: Engine, cycle, retried: bool) -> Envelope:
             commit=sha,
         )
 
-    nxt = engine.close_cycle(cycle)
+    # The last cycle settles the run's outcome before ending it: a run is never ended
+    # complete and then reopened as blocked.
+    last = not any(c.ordinal > cycle["ordinal"] for c in engine.declared)
+    blocking = engine.close_undeclared_gate(cycle) if last else []
+    if blocking:
+        engine.ledger.add_blocker(
+            engine.run["id"], cycle["id"], "undeclared_file_uncommitted", json.dumps(blocking)
+        )
+    nxt = engine.close_cycle(cycle, outcome="blocked" if blocking else "complete")
     if nxt is None:
-        blocking = engine.close_undeclared_gate(cycle)
         if blocking:
-            engine.ledger.insert(
-                "blocker",
-                run_id=engine.run["id"],
-                cycle_id=cycle["id"],
-                kind="undeclared_file_uncommitted",
-                detail=json.dumps(blocking),
-                at=now(),
-            )
-            engine.ledger.update("run", engine.run["id"], outcome="blocked")
             return Envelope(
                 run={
                     "id": engine.run["id"],
@@ -684,12 +658,7 @@ def _check_config_drift(engine: Engine, cycle) -> None:
     current = config_mod.config_sha(engine.worktree)
     if current == pinned:
         return
-    already = engine.ledger.one(
-        "SELECT id FROM integrity_event WHERE run_id = ? AND kind = 'config_changed'"
-        " AND detail = ?",
-        (engine.run["id"], current),
-    )
-    if already is None:
+    if not engine.ledger.run_has_event(engine.run["id"], "config_changed", current):
         engine.ledger.event(
             engine.run["id"],
             cycle["id"],