The warm pool module: one budget, a keep rule that fits, and an off switch #368

Part of #359.

Scope

After this PR warmth has one owner: the warm pool module. A released device that fits a waiting request stays warm and is granted to it (closes #350). A released device stays warm when there is room, and is shut down when there is not. warmPool.enabled: false keeps nothing warm. At startup the pool shuts down whatever is over the running limit. Targets and the reserve come in later tasks.

In short

A release ends as today for the caller. In the background the purge commits the driver's result, and the warm pool then runs one pass: it shuts down idle devices over the budget, oldest first, and boots back a shut-down device that serves a waiting request or was released a moment ago and has room. The budget is the running limit minus every slot a leased, reclaiming, quarantined or reserved device holds.

flowchart LR
  A[device.reclaimed on the bus] --> B[warm pool pass]
  B --> C{over budget?}
  C -->|yes| D[shut down idle devices, LRU,<br/>never one that serves a waiting request]
  C -->|no| E{shut-down device serves<br/>a waiting request, or room?}
  E -->|yes| F[boot it back, kick the queue]
  E -->|no| G[leave it shut down]

waiting request: a lease request still queued for a device (state new or queued), not one already booting or creating its own. serves: a device serves a waiting request when it fits the request's model or class, OS and image tag, and its pool mode is the mode the request resolved to. Example: a released iPhone 15 on 26.0 serves a waiting --class phone request; a slim one does not serve --class phone --mode full.

Before, at the running cap with --class phone waiting:

$ simlock release lse_15     # iPhone 15
# iPhone 15 shut down, iPhone 17 booted for the waiter (30 s)

After:

$ simlock release lse_15
# iPhone 15 purged and granted to the waiter

If this goes wrong: a released device is shut down while a request it fits waits, or a device is kept and the machine runs over its limit.

Technical spec

File and line references are to main at 4a52f6e, after #373.

Modules touched

flowchart LR
  classDef changed fill:#f3d9a4,stroke:#8a6a2a
  classDef new fill:#cfe3f7,stroke:#2f5f8f
  BUS((event bus)) -.-> CONV
  subgraph WP[src/core/warm-pool/]
    IDX[index.ts]:::new
    POL[policy.ts]:::new
    CONV[converger.ts]:::new
    CFG[config.ts]:::new
  end
  CONV --> POL
  CONV --> LIFE[managed-device-lifecycle.ts]
  CONV --> CAP[capacity/index.ts]
  CONV --> CLAIMS[device-operation-claims.ts]
  CONV --> REG[registry snapshot]
  CONV --> ACQ[lease-acquisition-coordinator.ts<br/>waitingRequests, kick]:::changed
  POL --> DOM[domain.ts fits, specMode]
  POL --> IDLE[idle-order.ts]
  ENG[lease-engine.ts]:::changed --> IDX
  CONFIG[config.ts]:::changed --> CFG

• New src/core/warm-pool/index.ts exporting WarmPool (construct, start, dispose, pass for tests), warmPoolConfigValidator, defaultWarmPoolConfig, and the WarmPoolConfig type. Nothing outside the directory imports another file in it.
• src/core/warm-pool/policy.ts: evaluate(view): readonly WarmProposal[], pure. view is { devices, leases, waiting, capacity: RunningCapacity, claims: isClaimed, config, now }. A proposal is { action: "shutdown" | "boot", deviceId, reason }. Order: shutdowns over budget by compareLeastRecentlyUsed, skipping any device that serves a waiting request; then boots for a shutdown device that serves the first request in the line (the queue head, whether or not it is in flight; a request behind the head gets no boot, because strict FIFO cannot grant it and idle shutdown would undo the boot; A request waits only behind requests for the same platform #387 later splits the line per platform), then, most recently released first, for shutdown devices released less than idle.shutdownAfterMs ago while the budget has room. With enabled: false every unleased ready device is a shutdown proposal and nothing is a boot.
• Budget per platform: room = maxRunning − reserveSetting − running − reserved from RunningCapacity (running counts ready, leased, reclaiming and quarantined devices, as src/core/capacity/limits.ts does; reserved is its count of in-flight provisioning and running reservations, which is how a device being created is counted, once). room < 0 means over budget by that many devices. A boot proposal fits when room ≥ 1 for the platform and globally. reserveSetting is 0 here and arrives in A reserve of running slots, and a request waits for a device on its way #369.
• serves(device, waiting): fits(waiting.requirement, device.spec, classOf) and specMode(device.spec) === waiting.mode. The pool reads waiters through a new core-side read port on LeaseAcquisitionCoordinator, waitingDemand(): readonly WaitingDemand[], one entry per waiter in state new or queued (a processing waiter has device work in flight, a provision, boot or eviction, and is being served, not waiting), with { platform, requirement, mode, classOf } from the waiter's resolved spec and requirement (AcquisitionWaiter). It is separate from the wire-shaped waitingRequests() on LeaseEngine, which does not change.
• src/core/warm-pool/converger.ts: subscribes to daemon.started, device.reclaimed, device.quarantine-recovered, lease.granted, lease.released, device.shutdown, device.deleted, cleanup.executed, capacity.changed, plus a tick every 30 s (a module constant, not a config key). One pass at a time; a trigger during a pass requests one more. For each proposal, inside decisions.run: revalidate state, lease and claim, then lifecycle.shutdown(device, "warm-pool", "cleanup") or, for a boot, capacity.tryReserveRewarm then lifecycle.boot under a boot claim (a new lifecycle method beside bootForLease that releases its claim on success and commits ready with readyTransitionUpdate), releasing the reservation after the commit. A failed step logs one warn line with deviceId, step, error and leaves the device in the state the lifecycle left it. After every action, succeeded or failed, kick(), so a request waiting on a pool boot re-plans either way.
• src/core/warm-pool/config.ts: enabled: boolean (default true). Validator composed into src/core/config.ts:135,657 beside quarantine.
• src/core/lease-engine.ts: constructs and starts the pool after the startup converger, so its first pass follows daemon.started; disposes it on stop.
• src/core/startup-converger.ts: no change beyond Reclaim is its own coordinator and commits what the driver returns #373; the first pass is what converges a lowered maxRunning.
• Docs: docs/CONFIGURATION.md gains warmPool.enabled; docs/internal/ARCHITECTURE.md "Device state machine" and "Running capacity" paragraphs describe the pass; docs/internal/COMPONENTS.md gains the warm pool row; docs/internal/IDEAS.md unchanged (already points at the ADR).

Contract and event changes

• device.shutdown gains the initiator value warm-pool; both EVENTS.md rows list it.
• No new event. No protocol change.

Rules in play

• architecture.md rules 5 (the pool observes the bus and acts by direct calls), 10 (the budget is enforced here and nowhere else), 14, 15, 16.
• safety.md rule 2: a pass never touches a leased or claimed device, revalidated inside the decision section.
• events.md rule 6: warm-pool is an additive initiator value.

Tests

• policy.ts: a released iPhone 15 that serves a waiting class request is proposed for boot, not shutdown, at the running cap.
• policy.ts: a slim device is not proposed for boot for a waiting full request of its class, and a full one not for a slim request.
• policy.ts: over budget, the least recently used idle device is proposed for shutdown and one that serves a waiting request never is.
• policy.ts: with enabled: false, every unleased ready device is proposed for shutdown and no boot is proposed.
• policy.ts: a shut-down device released longer ago than idle.shutdownAfterMs is not booted back without a waiting request.
• policy.ts: a shut-down device that serves only a request behind the queue head is not booted, including while the head is in flight.
• policy.ts: with room for one boot and two recently released shut-down devices, the more recently released one is proposed.
• converger.ts: a boot that fails still kicks acquisition.
• converger.ts: a device leased between the proposal and the action is left alone.
• converger.ts: a boot holds a rewarm reservation until its commit, and releases it when the boot fails.
• converger.ts: two triggers during one pass run exactly one more pass.
• converger.ts: a failed shutdown logs one line and the device stays ready.
• lease engine: after daemon.started with three ready devices and maxRunning: 2, one is shut down with initiator warm-pool.
• lease engine: release at the cap with a class request waiting grants the released device and provisions nothing (the A released device that fits a waiting class request is shut down and another booted #350 case).
• boundary: no file outside src/core/warm-pool/ imports a file inside it other than index.ts.

Done when

• At the running cap with simlock lease --class phone waiting, simlock release of a fitting iPhone grants that device to the waiter and simlock events shows no device.provisioned for it.
• With warmPool.enabled: false, a released fake iOS device is shutdown and a released fake Android device is shutdown after its device.reclaimed with strategy snapshot.
• With maxRunning lowered below the number of ready devices and the daemon restarted, the excess devices are shutdown with initiator warm-pool after daemon.started.
• simlock config shows warmPool.enabled: true by default.

Out of scope

• reserveRunning and waiting for a device on its way (A reserve of running slots, and a request waits for a device on its way #369).
• Targets (Warm targets: devices booted ahead of demand, exempt from idle shutdown, pre-created under fresh identity #370). Status, doctor and the event (The warm pool in status, doctor, one event, and the console #371).

Depends on

• Reclaim is its own coordinator and commits what the driver returns #373

Approval

• Approved for delivery

Written by an agent.