Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
38fd066
test: usage.get and simlock stats: the figures from the event history…
V3RON Oct 5, 2026
9debfbd
feat(bus): the history carries the step in force at the start of a wi…
V3RON Oct 5, 2026
79aa144
feat(usage): computeUsage reads the figures out of a window's events
V3RON Oct 5, 2026
eec7f49
feat(usage): usage.get answers on a worker and a gateway, and simlock…
V3RON Oct 5, 2026
4462bf7
docs: simlock stats, the client's usage call, and the usage reader's …
V3RON Oct 5, 2026
6b92741
test: type the usage figures the e2e reads
V3RON Oct 5, 2026
d37aef5
test: kill the usage mutants that survived
V3RON Oct 5, 2026
500285e
refactor: type the relayed answers so the usage reader has fewer dead…
V3RON Oct 5, 2026
8f9c4d8
test: a requester without a token label carries no label key
V3RON Oct 5, 2026
bf7330c
merge origin/main into task/347
V3RON Oct 5, 2026
c73df7a
fix: round the usage window down at both ends, count a rejection by i…
V3RON Oct 5, 2026
30a5280
test: the stats e2e waits for the window to close over the newest event
V3RON Oct 5, 2026
a98bfc0
feat(usage): a fleet request joins its requester's first relayed answ…
V3RON Oct 5, 2026
3858185
test: kill the carry and join mutants; a dispatch names only its worker
V3RON Oct 5, 2026
cb0fef0
test: the stats e2e tests wait out a bucket, so they get the time to
V3RON Oct 5, 2026
821dac3
refactor: core/usage's index exports only what its callers import
V3RON Oct 5, 2026
9d507be
Merge origin/main into task/347 (conflict: usage reader behind core/i…
V3RON Oct 8, 2026
0fcc591
test: the stats e2e names its window's end instead of waiting out a b…
V3RON Oct 8, 2026
5571536
Merge remote-tracking branch 'origin/main' into task/347
V3RON Oct 9, 2026
b0ddcfc
docs: leave ADR 0016 §6 as written; ADR 0021 replaced its join (#347)
V3RON Oct 9, 2026
d608467
WIP: tests for the fleet join by request id
V3RON Oct 9, 2026
e7dcd61
test: the gateway usage handler reads a fleet grant from request.granted
V3RON Oct 9, 2026
52533dd
refactor(usage): a decline is counted wherever a worker is named
V3RON Oct 9, 2026
b9f0f70
refactor(usage): count a requester's request in flat terms
V3RON Oct 9, 2026
066d3e9
test: kill the mutants the fleet join left alive
V3RON Oct 9, 2026
866849d
fix(usage): join a reused lease ID to its own lease, run the e2e with…
V3RON Oct 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .oxlintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -163,11 +163,13 @@
"LeaseRequestFailure",
"LeaseRequestLimits",
"SerializedDecision",
"UsageReader",
"findCatalogModel",
"fits",
"modelClass",
"newLeaseRequestId",
"pairedRuntimes"
"pairedRuntimes",
"tokenLabelMap"
],
"message": "the gateway may import only the names its allow-list gives"
},
Expand Down
135 changes: 134 additions & 1 deletion docs/CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Part of the user manual: every command the simlock CLI is expected to
implement. Results are JSON on **stdout**; progress/diagnostics are JSON
lines on **stderr** — this is the default output, not an opt-in, because
agents are the primary audience. `status`, `catalog`, and
agents are the primary audience. `status`, `catalog`, `stats`, and
`daemon <start|stop|status|logs>` are the exception: they default to a
human-oriented view for interactive/operator use and accept `--json` to
switch to the structured form. Every other command's output is already
Expand Down Expand Up @@ -80,6 +80,7 @@ command starts it again) to bring the platform up.
| 12 | `INSUFFICIENT_DISK_SPACE` | not enough free disk space to install a component |
| 12 | `LICENSE_NOT_ACCEPTED` | a required license (e.g. an Android SDK license) is not accepted |
| 12 | `UNKNOWN_WORKER` | `worker drain`/`undrain` or `component install --worker` naming a worker the gateway does not know |
| 12 | `HISTORY_NOT_KEPT` | `stats` for a window that ends before the oldest event the history holds |
| 12 | `DOWNLOADS_DISABLED` | `component install` on a machine whose `downloads.policy` is `"never"` |
| 12 | `COMPONENT_NOT_OWNED` | `component remove` of a component Simlock did not install, or one that changed on disk since |
| 12 | `COMPONENT_IN_USE` | `component remove` of a component a device uses, Simlock's or your own |
Expand Down Expand Up @@ -1934,6 +1935,138 @@ gateway's order. A replay (`simlock events`, `--since`) prints events by
millisecond. `--follow` prints each live push as it arrives, so a relayed event
from a worker whose clock is behind prints after a later gateway event.

## `simlock stats [--since <duration> | --from <ISO> [--to <ISO>]] [--json]`

The usage figures for a window: how many requests there were, how long they
waited and held their devices, how full the host was, and what went wrong. They
are worked out when you ask, from the same event history `simlock events`
prints, so they agree with it for the same window and they survive a daemon
restart. Nothing is counted separately and nothing is kept beyond the history,
so a window reaches back only as far as `eventLog.retention` and
`eventLog.maxBytes` keep events (see [CONFIGURATION.md](CONFIGURATION.md)).

- `--since 6h` is the last six hours, up to now. Durations take `ms`, `s`, `m`,
`h` and `d` units, as for `simlock events`.
- `--from 2026-10-04T00:00:00Z --to 2026-10-05T00:00:00Z` names the window. With
`--from` alone the window runs to now.
- With none of them, the window is the last 24 hours.
- `--since` with `--from`, `--to` without `--from`, and a `--from` that is not
earlier than `--to` are usage errors (exit 2). A window can be at most 90
days long.
- `--json` prints the daemon's answer unchanged, as one JSON object; without it
you get a table.

The table opens with the window, then the totals, then a row for each platform,
each worker and each requester:

```
Usage from 2026-10-04T10:00:00.000Z to 2026-10-05T10:00:00.000Z

Totals
Requests: 12 (10 granted, 2 rejected)
Granted: warm 6, booted 3, provisioned 1
Rejected: no-wait 1, timeout 1
Declined: 2
Wait: p50 1.2s, p95 4s, max 9.1s (11 samples)
Held: p50 5m 0s, p95 30m 0s, max 1h 2m (9 samples)
Turnaround: p50 5m 10s, p95 31m 40s, max 1h 3m (9 samples)
Provisioning: p50 1m 30s, p95 1m 30s, max 1m 30s (1 sample)
Boot: p50 20s, p95 40s, max 40s (2 samples)
Slots: peak 3 of 4, mean 1.5
Queue: peak depth 2, mean 0.4
Incidents: 0 quarantined, 1 recovered after a crash, 0 recovered from quarantine, 0 lost

Platforms
ios 12 requests, 10 granted, 2 rejected, wait p50 1.2s, held p50 5m 0s, slots peak 3
android 0 requests, 0 granted, 0 rejected, wait p50 -, held p50 -, slots peak -

Workers
mac-mini-1 (wrk_1) 12 requests, 10 granted, 2 rejected, wait p50 1.2s, held p50 5m 0s, slots peak 3

Requesters
ci-bot (tok_a) 3 requests, 2 granted, 1 rejected, held 12m 5s
```

What the figures count:

- **Requests** are the lease requests made in the window. A request belongs to
the window it was made in: its grant, rejection and end are joined from
events up to the end of the window, so a lease still held at the end of the
window counts as a request and a grant and gives no held or turnaround time.
A request made before the window that is granted or rejected inside it is in
no count.
- **Granted** is the grants of those requests, by how the device came to be
ready: `warm` was already ready, `booted` was started from shutdown,
`provisioned` was created for the request.
- **Rejected** are the requests that ended without a device, by reason
(`timeout`, `no-wait`, `cancelled`, ...), plus the requests refused before
they were stored (a requester that already holds a lease, one that asked
for a lease ID that is taken, or one that came while admission was closed
for `nuke` or maintenance). Those are in the window their rejection falls
in and are not requests, so granted plus rejected can be more than requests.
- **Wait** is the time from the request to its grant or rejection. **Held** is
the time from the grant to the release or expiry of the lease. **Turnaround**
is the time from the request to that end. **Provisioning** and **Boot** are the
durations of creating and starting devices in the window. Each shows its median
(`p50`), `p95` and longest, and how many samples it has; `-` or "no samples"
where there are none.
- **Slots** is how many devices were running or reserved against the most the
host allows: the highest value and the average over time. **RAM** appears when
the host keeps a RAM budget. **Queue** is how many requests were waiting. Where
the history has nothing before the first record in the window, that stretch of
time is left out of the figures rather than counted as zero.
- **Incidents** count devices quarantined, devices recovered after a crash,
devices recovered from quarantine, and devices lost (a recovery that failed, or
a quarantined device given up on). **Failures** counts the failure events in
the window by name: `device.purge-failed`, `device.recovery-failed`,
`component.install-failed`, and any other event whose name ends in
`-failed`. The row is left out when there were none.
- **Declined** counts the times a worker refused a request that a gateway sent
it, by the time of the refusal. It is a count of refusals, not of requests: a
gateway tries another worker after a refusal, so one request can be declined
by several workers. The row is left out when there were none.
- A requester that is a token shows the token's label beside its id.

The window is rounded down to whole steps of the time series `--json` carries,
which is 1 minute for a window of a few hours and grows to 1 day for 90 days.
Neither end moves later than the time you asked for; the window in the answer is
the rounded one.

Against a **gateway** the totals are the fleet's and there is a row for each
worker. Requests, waits and rejections come from the gateway's own record of
each request: it waits from the request until the gateway grants or rejects it,
and a request the gateway was still holding when it stopped counts as rejected
with the reason `daemon-restarted`. How the device came to be ready and how long
the lease was held come from the worker's record of the lease; a grant whose
worker's record the gateway never received counts as `unknown` and has no held
time. A worker's row has the requests granted on it or that failed on it
(`worker-failed`) and its device figures; a request that ended any other way,
or is still waiting, is in the totals and the platform rows only. The queue figure is the fleet queue's.

A worker's own `simlock stats` covers that worker only: its row is itself. A
request that a gateway sent it is not one of the worker's requests: it is counted
under **Probes**, not under requests, and has no wait or turnaround of its own.
Its grant still counts as a grant, with its source and held time, under the
requester id the gateway gave it.

If the history does not reach back to the start of the window (a new daemon, or
older events deleted by retention), the figures cover what it does hold and say
so:

```
Usage from 2026-09-05T12:00:00.000Z to 2026-10-05T12:00:00.000Z
Figures cover from 2026-10-05T11:59:00.000Z: the history does not reach back to the start of the window.
```

In `--json` that is `partial: true` and `coversFrom`. If the window ends before
the oldest event the history holds, there is nothing to show: the command fails
with `HISTORY_NOT_KEPT` (exit 12), and the message names the time the history
reaches back to. A daemon with no history at all has nothing to refuse and
answers with zeros.

`stats` needs the daemon, and the admin credential (see
[Admin credential resolution](#admin-credential-resolution)).

## `simlock daemon <start|stop|status|logs [--follow]>`

Manage the daemon explicitly. Other commands auto-start it on demand; `daemon`
Expand Down
33 changes: 33 additions & 0 deletions docs/CLIENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { connectSimlockAdmin } from "simlock/admin"; // agent + admin role
`connectSimlockAdmin` returns a superset of `connectSimlock`'s client — every
agent-role method plus the admin-role ones (`list`, `runCleanup`, `runNuke`,
`getConfig`, `stopDaemon`, `replayEvents`/`subscribeEvents` (each event carries an `id`),
`usage`,
`createToken`/`listTokens`/`revokeToken`, `installComponent`, `removeComponent`). The split exists so
`simlock/client` doesn't even show admin methods in a caller's editor; the
daemon's own role check is what actually stops an agent-role session from
Expand Down Expand Up @@ -571,6 +572,38 @@ const { waiting = [] } = await client.getStatus();
`list({ kind: "requests" })` lists both, each worker's entries with their
`workerId`, the same list as `GET /v1/lease-requests`.

## How much was used: `usage`

`simlock/admin` only. `usage({ from, to })` returns the usage figures for a
window, the same answer `simlock stats --json` prints (see
[CLI.md](CLI.md), under `simlock stats`,
for what each figure counts). `from` and `to` are epoch milliseconds, `from`
before `to`, at most 90 days apart; the client refuses anything else before it
sends a frame.

```ts
const usage = await admin.usage({ from: Date.now() - 6 * 3_600_000, to: Date.now() });

usage.totals.requests; // lease requests made in the window
usage.totals.wait.p95; // milliseconds, or null when nothing waited
usage.totals.failures.byEvent; // a count per failure event, e.g. "device.purge-failed"
usage.workers[0]?.label; // one entry for each worker; a worker lists itself
usage.series; // one point per bucket, for a chart; `waiting` counts every
// request open at the bucket's end, leaving out a gateway's probes
```

The daemon computes the figures from its event history, so they cover only what
the history holds: `partial` is `true` and `coversFrom` says where they start when
it does not reach the start of the window. `window` in the answer is the window
asked for, rounded down to a whole number of `bucketMs` at both ends, and the series never has more
than 200 points. A window that ends before the oldest event the history holds
rejects with `HISTORY_NOT_KEPT`; its `details.oldestTs` is the oldest time the
history reaches. Against a gateway the totals are the fleet's and `workers` has
one entry for each worker; `bySource.unknown` counts grants whose source the
gateway never learned, and there is no `probes`. Against a worker, `probes`
counts the requests a gateway sent it, which `requests` leaves out. `declined`
counts the refusals of such requests.

## One connection, no reconnect, no retry

This is the one thing to internalize before building anything on top of this
Expand Down
23 changes: 23 additions & 0 deletions docs/internal/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2113,6 +2113,29 @@ The CLI reads the file itself only for `simlock events --since` when no
daemon answers; `--follow` subscribes first, replays, and drops replayed
pushes, so the join neither loses nor repeats an event.

`usage.get` (ADR 0016) is the third reader of that history, and the only one that
turns it into numbers. `computeUsage` (`src/core/usage/`) is pure: it takes
the envelopes of a window, with the latest `capacity.changed`, `queue.changed` and
`daemon.started` at or before the window's start and the latest `lease.requested`
of each requester, that `EventHistory.read`'s `carry` adds, and the ids of the
requests made before it (`requestedBefore`, so a rejection of one is in no count)
and of those answered by then (`answeredBefore`, so a request still waiting when
the window opens counts in the series' `waiting`), and returns the figures.
`readEvents` reads them into one fact per request, joined by `requestId` and
`leaseId`, one per device event and one per `lease.declined`; `compute-usage.ts`
adds them up by platform, by worker and by requester. On a gateway (`fleet`)
request facts come from its own events and device facts from the events its
workers relayed (ADR 0021 §5): a request's outcome is the gateway's own
`request.granted` or `lease.rejected` for its request id, else its next own
`daemon.started` ends it as rejected `daemon-restarted`, else it is open; the
grant's source and held time are the relayed `lease.granted` of the worker and
lease it names. On a worker a request carrying `fleetRequestId` is a probe,
counted apart from its requests. That rule is in `readEvents` and nowhere else. One
`UsageReader` serves both dispatchers: it rounds the window down to the series
bucket at both ends, keeps its last answer by that window and the newest event id, and joins
token labels, so the daemon's two handlers differ only in `fleet`. `simlock stats`
prints what the operation returns and computes nothing.

## Device requests

A request names a device in one of three forms (ADR 0015 §1): an exact
Expand Down
1 change: 1 addition & 0 deletions docs/internal/COMPONENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ pool reads from acquisition.
| `Registry` | `src/core/registry.ts` | Devices, leases, lease requests and component records, written through one commit to `state.json`; the device transition function's only caller; emits the post-commit device and lease facts. | Decide a transition: callers do, inside a decision section. |
| `SerializedDecision` | `src/core/serialized-decision.ts` | Serialising short read-decide-commit sections. | Hold driver work or other long I/O. |
| `DeviceOperationClaims` | `src/core/device-operation-claims.ts` | Exclusive per-device operation claims: boot, eviction, cleanup, nuke, reclaim. | Any lifecycle, cleanup or leasing policy. |
| `UsageReader`, `computeUsage` | `src/core/usage/` (surface: `index.ts`) | `usage.get`'s one answerer for a worker and a gateway: reads the event history for a window rounded down to the series bucket, turns it into figures (`computeUsage`, `readEvents`, the timelines), joins token labels, keeps the last answer by window and newest event. | Write anything, or read the registry, capacity or lifecycle engine: events in, numbers out. |
| `DeviceProvisioner` | `src/core/device-provisioner.ts` | Creating a device: the component-removal gate, the registry record, the driver's `provision`, readiness for the lease handoff. | Decide whether to provision. |
| `ManagedDeviceLifecycle` | `src/core/managed-device-lifecycle.ts` | Registry-owned device operations with a claim and a revalidation each: boot for a lease, boot back to warm, shutdown, destroy, dispose, recover a leased device. Every driver verb on an existing device goes through here. | Decide when to run them. |
| `ReclaimCoordinator` | `src/core/reclaim-coordinator.ts` | After a release: the driver's reclaim, committing the state it returns (`shutdown` on iOS, `ready` on Android) and waking the queue, handing a failed purge to quarantine, deleting a spent `fresh` device, recovering an interrupted reclaim at startup (a shutdown; for a device whose wipe a start put off, the full reclaim started in the background under a claim, and `settle` for a graceful stop). | Decide whether a device stays warm, boot anything, or read capacity or the queue. The warm pool (`src/core/warm-pool/`, ADR 0017) does. |
Expand Down
Loading
Loading