The warm pool is its own module: targets, a reserve, and an off switch #359

Request: none

Related: #350 (closed by this feature), #76 (its warm-target half is replaced by this feature; the host activation handshake stays there).

Problem

A lease is fast only when a clean device of the right kind is already running. Today that happens by accident: a device is warm only because someone leased it and released it, and capacity happened to allow it to keep running. The first lease of the day, and the first lease of each new model, always waits for a full boot. On a fresh identity machine every lease waits for one.

The operator has no say in it. There is no way to ask Simlock to keep two iPhone 17 ready before the agents arrive, no way to keep a slot free for a model nobody predicted, and no way to turn warmth off on a machine that needs its RAM back after every lease.

The rule that decides whether a released device stays warm is also wrong in one case. At the running cap it keeps the device only when it is exactly the model the oldest waiting request would create. A released iPhone 15 that fits a waiting --class phone request is shut down and an iPhone 17 booted for it (#350).

Who it is for

The operator of a worker, on a CI runner or a shared developer Mac, who knows which devices the agents on it ask for and wants the first lease to be as fast as the tenth. Also the agent whose request is waiting while a device that would do is being shut down.

Outcome

• The operator can name targets: a kind of device and a count that Simlock keeps booted, clean and unleased, ahead of demand.
• The operator can keep a number of running slots free of warm devices, so a request for an unpredicted model boots at once instead of shutting a warm device down first.
• The operator can turn warmth off, so no idle device ever runs.
• A released device that fits a waiting request stays warm and is granted to it.
• A request for a device the pool is already booting waits for that boot instead of starting another.
• simlock status shows each target, how many are ready and booting, and why it is short. simlock doctor reports a target that can never be met as configured.

How it works for the user

flowchart LR
  A[daemon starts] --> B[pool pass: boot targets, one at a time]
  B --> C{lease request}
  C -->|a warm device fits| D[granted in under a second]
  C -->|one is booting for a target| E[waits, granted when the boot ends]
  C -->|nothing fits| F[boots or creates its own, as today]
  D --> G[pool below target: refill in background]
  H[lease released] --> I[device purged]
  I -->|fits a waiting request, or room under the reserve| J[kept warm]
  I -->|no room, or pool off| K[shut down]

Before, on a machine that has just started:

$ simlock lease --model "iPhone 17" --os 26.0
{"state":"booting","estimatedReadyMs":30000}
... 30 s ...
{"leaseId":"lse_1","device":{"model":"iPhone 17","mode":"full"}}

After, with a target of two iPhone 17:

$ simlock config set warmPool.targets '[{"platform":"ios","model":"iPhone 17","osVersion":"26.0","count":2}]'
$ simlock daemon restart
$ simlock lease --model "iPhone 17" --os 26.0
{"leaseId":"lse_1","device":{"model":"iPhone 17","mode":"full"}}
$ simlock status
warm pool: enabled, reserve ios 0 android 0
  iPhone 17 / 26.0 / full   wanted 2  ready 1  booting 1

A target the machine cannot fill:

$ simlock status
warm pool: enabled, reserve ios 1 android 0
  iPhone 17 / 26.0 / full   wanted 4  ready 2  booting 0  short: reserve
$ simlock doctor
warm-pool-target-unreachable  iPhone 17 / 26.0 / full wants 4; the iOS running limit is 3 and the reserve is 1

A target for a runtime that is not installed:

$ simlock doctor
warm-pool-target-unreachable  iPhone 17 / 27.0 / full: iOS 27.0 is not installed; run simlock component install ios 27.0

Turning warmth off:

$ simlock config set warmPool.enabled false
$ simlock daemon restart
$ simlock release lse_1
$ simlock status
warm pool: off
devices: sim_a  iPhone 17  shutdown

Examples

Config keys. Today every key under warmPool except quarantine.* is unknown: a warning at start, then ignored.

What happens to a released device, at the running cap, with one request waiting.

What a target does when the pool is short.

What could go wrong

• A target holds RAM and disk on a quiet machine for as long as it is configured. The operator sees it in status and can lower the count.
• A target whose device keeps failing to boot is retried with a growing delay, up to ten minutes between tries, and shows as short with the reason while it waits.
• A target that does not fit the limits looks like a bug to someone who does not read status. Doctor names the limit that stops it.
• A restart with several targets boots them one after another, so the pool takes a couple of minutes to fill. Requests in that window are served as today.
• A request for a device the pool is booting waits up to a boot's length. It would have booted its own for the same time.
• With the pool off, every lease pays a boot, including on Android. That is what off means.

Words used

• warm device: a device that is running, clean, and not leased. A request it fits is granted in under a second.
• target: a standing order to keep a number of warm devices of one kind, e.g. two iPhone 17 on iOS 26.0 in the default mode. A floor, not a cap, and never filled by evicting anything.
• kept device: a released device left running because a request it fits is waiting or there is room under the reserve. Kept devices follow the idle timers; targeted devices do not.
• reserve: running slots per platform that warm devices may not take, so a request for an unpredicted model boots at once. Default none.
• mode: slim or full. A slim device has simulator daemons disabled. A full request never gets a slim device and a slim request never gets a full one.
• pool pass: one run of the warm pool over the machine, after a start, a grant, a release or a cleanup. It shuts down what is over, boots what is under, one boot at a time.
• short: a target with fewer ready devices than wanted at the end of a pass that could do nothing for it, with one reason, the first that applies: pool disabled, no driver for the platform, runtime not installed, unknown model, request could not be resolved, last boot failed (waiting to retry), device limit, running limit, reserve, RAM budget. A target that is still filling, or waiting behind a request, is not short.
• purge: cleaning a released device. iOS erases it, which leaves it shut down. Android restores its golden snapshot, which leaves it running. Purge is not warmth: with the pool off, Android is restored and then shut down.
• fresh identity: lease.identity: fresh. Every lease gets a device that never served another and is deleted after. A target then pre-creates never-leased devices.

Non-goals

• Targets on a gateway, or any fleet-wide warmth. A gateway keeps routing to the worker that has a warm device.
• Changing targets while the daemon runs. Config is read at start.
• A target naming a class instead of a model.
• Evicting a leased or kept device to make room for a target.
• Downloading a runtime a target names.
• The host activation handshake and the removing/removed identity status of Host activation handshake and removal status for warm identities #76.
• Any change to the capacity limits, the RAM budget, or the idle timers for devices that are not targeted.

Completion conditions

• With warmPool.targets naming two iPhone 17 and no leases, simlock status shows the target with ready 2 within two boots of daemon start, and the first simlock lease for that model is granted without a booting progress line.
• With a target of two and both granted, status shows ready 0 booting 1, and after the boot ready 1.
• With reserveRunning.ios: 1 on a running limit of 3 and two warm iPhones, a request for an iPhone 15 is granted after one boot and no warm device is shut down for it.
• At the running cap with a --class phone request waiting, releasing an iPhone 15 that fits grants it that device; no other device boots. This is the A released device that fits a waiting class request is shut down and another booted #350 case and closes it.
• A request for a model the pool is booting for a target is granted when that boot ends, and device.provisioned does not fire for it.
• A targeted warm device idle for longer than idle.shutdownAfterMs is still running; a kept device of another model is shut down at that time.
• With lease.identity.ios: fresh and a target of one, status shows ready 1 before any lease, a lease is granted without booting, the released device is deleted, and status returns to ready 1 after one boot.
• With warmPool.enabled: false, a released iOS device is shutdown after its purge, a released Android device is shutdown after its snapshot restore, and the next lease of each boots.
• A target whose count exceeds the running limit minus the reserve shows short with the limit named in status, and simlock doctor reports it; nothing is shut down or evicted for it.
• A target naming a runtime that is not installed shows short with the runtime named, doctor reports it with the install command, and no download starts under any downloads.policy.
• With maxConcurrentBoots at its default and three targets of one each, simlock events shows each device.provisioned after the previous device's device.ready.
• A count of 0, or a target with a class key, fails config load with an error naming the key.
• On a Mac with an iOS runtime and an Android system image: the slow lane (e2e/slow-warm-pool.test.ts) proves a target boots a real simulator and a real emulator ahead of a lease, and that off shuts both down after a release.

Open questions

none

Decisions

• ADR 0017 — The warm pool is a module beside the lease transaction, not a step in it (docs(adr): 0017 — the warm pool is a module beside the lease transaction, not a step in it #361, Accepted — not yet implemented): where warmth lives and how it relates to the lease path. It fixes the two components (reclaim and warm pool), the module's surface and pure policy, the dependency direction, the derived budget with the reserve, what a pass does, the boot-claim wait, and the status, doctor and event surface.

Tasks

flowchart LR
  T373[#373 reclaim coordinator] --> T368[#368 warm pool module]
  T368 --> T369[#369 reserve and wait]
  T369 --> T370[#370 targets]
  T369 --> T371[#371 status, doctor, event]
  T370 --> T371
  T371 --> T372[#372 slow lane]

• Reclaim is its own coordinator and commits what the driver returns #373 Reclaim is its own coordinator and commits what the driver returns — nothing new for the user yet; until The warm pool module: one budget, a keep rule that fits, and an off switch #368 every iOS release costs the next lease a boot. Risk: a release leaves a device stuck or does not wake a waiting request.
• The warm pool module: one budget, a keep rule that fits, and an off switch #368 The warm pool module: one budget, a keep rule that fits, and an off switch — a released device that fits a waiting request is granted to it; warmPool.enabled: false keeps nothing warm. Risk: a device shut down while a request it fits waits, or kept over the limit.
• A reserve of running slots, and a request waits for a device on its way #369 A reserve of running slots, and a request waits for a device on its way — a request for an unpredicted model boots at once; a request for a device the pool is booting waits for it. Risk: a request waits on a boot that fails until its deadline.
• Warm targets: devices booted ahead of demand, exempt from idle shutdown, pre-created under fresh identity #370 Warm targets — the first lease of a targeted kind is granted without a boot, including under fresh identity; targeted devices stay warm. Risk: boots nobody asked for, or a boot loop on a broken runtime.
• The warm pool in status, doctor, one event, and the console #371 The warm pool in status, doctor, one event, and the console — the operator sees each target and why it is short. Risk: status says short while a target is filling.
• Slow lane: a target boots a real simulator and emulator, and off shuts both down #372 Slow lane — real devices prove a target boots ahead of a lease and off shuts down after a release. Risk: none for the user; a red lane means a bug in an earlier task.

Written by an agent.