From 75355608268694c88724149b97d3667f622069e0 Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Thu, 8 Oct 2026 21:46:19 +0200 Subject: [PATCH 1/8] docs(adr): a gateway dispatch is a probe, and the worker's lease events name the fleet request (ADR 0021) --- ...-derived-on-read-from-the-event-history.md | 4 +- ...-is-a-probe-and-names-its-fleet-request.md | 153 ++++++++++++++++++ docs/internal/adr/README.md | 3 +- 3 files changed, 158 insertions(+), 2 deletions(-) create mode 100644 docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md diff --git a/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md b/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md index b16332ce..ef792972 100644 --- a/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md +++ b/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md @@ -1,6 +1,8 @@ # 0016. Usage figures are derived on read from the event history -- **Status:** Accepted — not yet implemented +- **Status:** Accepted — not yet implemented. §6's join rule (its second + paragraph) is superseded by [ADR + 0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md). - **Date:** 2026-10-05 - **Issue:** [#329](https://github.com/callstackincubator/simlock/issues/329) - **Supersedes:** nothing. Extends [ADR diff --git a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md new file mode 100644 index 00000000..4326017b --- /dev/null +++ b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md @@ -0,0 +1,153 @@ +# 0021. A gateway dispatch is a probe, and the worker's lease events name the fleet request + +- **Status:** Accepted — not yet implemented +- **Date:** 2026-10-08 +- **Issue:** [#329](https://github.com/callstackincubator/simlock/issues/329) +- **Supersedes:** [ADR + 0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) + §6's join rule (its second paragraph). The rest of §6 stands. +- **Depends on:** [ADR 0005](0005-gateway-and-worker-modes.md) requirements + 11, 12 and 27a, [ADR 0009](0009-gateway-routing-is-a-list-of-stages.md) + §5, [ADR 0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) + for `workerId` on a relayed event. + +## Context + +A gateway sends every request to a worker as `lease.request` with `noWait: +true` (ADR 0005 requirement 12). Some refusals don't end the request; the +gateway tries again elsewhere: + +- an immediate `NO_CAPACITY`, which is a stale view (requirement 11); +- `UNKNOWN_MODEL`, `RUNTIME_MISSING` or `NO_DRIVER` before any progress + push (ADR 0009 §5). + +In each of these cases the request stays in the fleet queue. + +The worker does not know that. It handles the request like any `--no-wait` +caller's: it emits `lease.requested`, then `lease.rejected` with reason +`no-wait` or `unknown-model`. Both are relayed to the gateway. The history +then records a final rejection for a request that another worker later +granted, or that is still waiting. + +ADR 0016 §6 joins a fleet request to its outcome by position: the first +relayed answer for the namespaced requester after `request.dispatched`. +That fails two ways: + +- a relayed refusal from a worker the gateway moved past is taken as the + outcome; +- `request.dispatched` is emitted when the grant or the first progress push + reaches the gateway, so a warm grant's relayed `lease.granted` is older + than the dispatch it should follow. + +The gateway never tells the worker which fleet request a dispatch serves, +so no id joins the two records. The same refusals also inflate a worker's +own figures: every stale-view refusal counts as a rejected request that no +caller ever saw. + +```mermaid +sequenceDiagram + participant G as Gateway + participant A as Worker A + participant B as Worker B + G->>A: lease.request (noWait) + A-->>G: NO_CAPACITY + Note over A: today: lease.rejected no-wait
after: lease.declined + G->>B: lease.request (noWait) + B-->>G: grant + Note over B: lease.granted + Note over G: request.dispatched +``` + +## Decision + +### 1. A gateway dispatch is a probe and carries the fleet request id + +`lease.request` gains an optional `fleetRequestId`. Only the gateway's own +uplink session may set it, as with `owner` (requirement 27a). Any other +session that sets it is refused with `FORBIDDEN`. + +The gateway sets it on every dispatch, to its own request id. A request +that carries it is a **probe**. A probe never waits; the gateway keeps +sending `noWait: true`. Its RPC answer is unchanged, so the gateway's walk +(requirement 11, ADR 0009 §5) works as it does today. + +### 2. A refused probe is declined, not rejected + +A probe that the worker refuses with one of the codes the gateway retries +on emits `lease.declined` instead of `lease.rejected`. Those codes are +`NO_CAPACITY`, `UNKNOWN_MODEL`, `RUNTIME_MISSING` and `NO_DRIVER`, before +the first progress push. The payload is `{ requestId, fleetRequestId, +requester, reason }`, with the same reason values `lease.rejected` uses. + +Any other refusal of a probe is still `lease.rejected`: `already-leased`, +`lease-id-taken`, a failure after a progress push, a boot timeout. The +gateway treats each of these as the request's terminal failure. + +The set of retried codes is defined once, in the contract. The gateway's +walk and the worker's choice of event both read it, so the two cannot +disagree about which refusals end a request. + +### 3. Every lease event of a probe names the fleet request + +The worker adds `fleetRequestId` to every lease event it emits for a probe: + +- `lease.requested`, +- `lease.granted`, +- `lease.rejected`, +- `lease.declined`. + +A probe is never queued, so there is no `lease.queued`. Later events of the +lease (`lease.renewed`, `lease.released`, `lease.expired`) are joined by +`leaseId` as today. The field is additive on the existing events (events +rule 6). + +### 4. Usage joins a fleet request by id + +This replaces ADR 0016 §6's join rule. On a gateway, a fleet request's +outcome is the relayed `lease.granted` or `lease.rejected` whose +`fleetRequestId` is the gateway's request id. Order and timestamps play no +part, and `request.dispatched` is not needed for the join. A relayed +`lease.declined` is never an outcome. A request with no such event is open. +Refusals the gateway makes itself (its own `no-wait`, `timeout`, +`cancelled`) are its own `lease.rejected` events, as today. + +On a worker, a request that ends in `lease.declined` is counted under a +`declined` figure. It is not counted under requests or rejections, and it +gives no wait sample. + +### 5. Protocol +1, and an older worker is incompatible + +A worker without `fleetRequestId` would refuse the field, and its events +would not carry it. The protocol therefore moves up by one, and a worker on +the previous version is `incompatible` with the gateway, as with every +other change to what the gateway sends a worker (`src/contract/protocol.ts`). +There is no shim. + +## Consequences + +- A worker in a fleet no longer emits `lease.rejected` for a stale-view or + cannot-serve refusal from the gateway. A consumer that counted those + rejections sees `lease.declined` instead. The emission changes, not the + payload. Both `EVENTS.md` files say so. +- `simlock events` on a worker shows each probe that missed as + `lease.declined`, so an operator can tell gateway traffic that went + elsewhere from a refusal a caller saw. +- The fleet join in `usage.get` is one id lookup. The rule about the first + answer after a dispatch is gone. +- `usage.get` gains a `declined` count, per platform and per worker. +- A gateway and its workers must upgrade together. + +## Alternatives considered + +- **A public `--if-possible` flag on `simlock lease`.** Rejected: a user's + `--no-wait` refusal is a real rejection that the caller sees. Only the + gateway's probe is not. +- **Keep `lease.rejected` and add `fleetRequestId` only.** Rejected: the + join would be fixed, but a worker's own history would still record + refusals that no caller saw as rejections. +- **Tighten the positional join instead.** Rejected: the dispatch event + follows a warm grant, so position cannot tell a stale refusal from the + outcome. It also leaves the worker's figures wrong. +- **A shim for older workers.** Rejected: no other gateway-to-worker change + kept one, and a fleet that mixes versions gets figures that are wrong + without saying so. diff --git a/docs/internal/adr/README.md b/docs/internal/adr/README.md index b90e73f5..5ed39677 100644 --- a/docs/internal/adr/README.md +++ b/docs/internal/adr/README.md @@ -54,8 +54,9 @@ the status is stale. | [0013](0013-the-console-reads-routes-and-follows-the-event-stream.md) | The console reads the routes and follows the event stream | Accepted — not yet implemented | | [0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) | An event has one id, minted where the fact happened | Accepted | | [0015](0015-a-lease-request-is-a-set-of-constraints.md) | A lease request is a set of constraints, and the catalog says which class each model is | Accepted — not yet implemented | -| [0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) | Usage figures are derived on read from the event history | Accepted — not yet implemented | +| [0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) | Usage figures are derived on read from the event history | Accepted — not yet implemented; §6's join superseded by 0021 | | [0017](0017-the-warm-pool-is-a-module-beside-the-lease-transaction.md) | The warm pool is a module beside the lease transaction, not a step in it | Accepted — not yet implemented | | [0018](0018-leasing-is-one-module-and-every-module-is-entered-through-its-index.md) | Leasing is one module, and every module is entered through its index | Accepted — not yet implemented | | [0019](0019-startup-ends-every-lease-whose-device-is-not-running.md) | Startup ends every lease whose device is not running | Accepted — not yet implemented | | [0020](0020-a-requester-may-choose-its-lease-id.md) | A requester may choose its lease id, and the gateway passes it through bare | Accepted — not yet implemented | +| [0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md) | A gateway dispatch is a probe, and the worker's lease events name the fleet request | Accepted — not yet implemented | From 4960ac94fc7a81ba5caecf01f540d9ad7d207835 Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Thu, 8 Oct 2026 21:56:27 +0200 Subject: [PATCH 2/8] =?UTF-8?q?docs(adr):=200021=20=E2=80=94=20the=20worke?= =?UTF-8?q?r=20declines=20every=20probe=20refusal,=20the=20gateway=20recor?= =?UTF-8?q?ds=20every=20fleet=20failure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...-derived-on-read-from-the-event-history.md | 4 +- ...-is-a-probe-and-names-its-fleet-request.md | 211 +++++++++++------- docs/internal/adr/README.md | 2 +- 3 files changed, 137 insertions(+), 80 deletions(-) diff --git a/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md b/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md index ef792972..7cfb5da5 100644 --- a/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md +++ b/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md @@ -1,7 +1,7 @@ # 0016. Usage figures are derived on read from the event history -- **Status:** Accepted — not yet implemented. §6's join rule (its second - paragraph) is superseded by [ADR +- **Status:** Accepted — not yet implemented. §6's rules on relayed + rejections and the fleet join are superseded by [ADR 0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md). - **Date:** 2026-10-05 - **Issue:** [#329](https://github.com/callstackincubator/simlock/issues/329) diff --git a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md index 4326017b..ecc26f18 100644 --- a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md +++ b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md @@ -5,117 +5,166 @@ - **Issue:** [#329](https://github.com/callstackincubator/simlock/issues/329) - **Supersedes:** [ADR 0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) - §6's join rule (its second paragraph). The rest of §6 stands. + §6's rules on which relayed `lease.rejected` counts, on how a fleet request + is joined to its outcome, and its line "The gateway emits no rejection of + its own for a request that failed on its worker". The rest of §6 stands: + request facts come from the gateway's own events, device facts from + relayed ones. - **Depends on:** [ADR 0005](0005-gateway-and-worker-modes.md) requirements 11, 12 and 27a, [ADR 0009](0009-gateway-routing-is-a-list-of-stages.md) §5, [ADR 0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) - for `workerId` on a relayed event. + for `workerId` on a relayed event, [ADR + 0020](0020-a-requester-may-choose-its-lease-id.md) for a forwarded + `leaseId`. ## Context A gateway sends every request to a worker as `lease.request` with `noWait: -true` (ADR 0005 requirement 12). Some refusals don't end the request; the -gateway tries again elsewhere: +true` (ADR 0005 requirement 12). Some refusals do not end the request; the +gateway tries another worker: - an immediate `NO_CAPACITY`, which is a stale view (requirement 11); - `UNKNOWN_MODEL`, `RUNTIME_MISSING` or `NO_DRIVER` before any progress push (ADR 0009 §5). -In each of these cases the request stays in the fleet queue. +The request then fails only when no worker is left. A `noWait` caller gets +one more walk. -The worker does not know that. It handles the request like any `--no-wait` -caller's: it emits `lease.requested`, then `lease.rejected` with reason -`no-wait` or `unknown-model`. Both are relayed to the gateway. The history -then records a final rejection for a request that another worker later -granted, or that is still waiting. +The worker does not know any of that. It handles the request like a local +`--no-wait` caller's, and emits `lease.requested` followed by `lease.rejected` +with reason `no-wait` or `unresolvable-spec`. Both are relayed to the +gateway. The history then records a final rejection for a request that +another worker granted, or that is still waiting. + +Who records the end of a fleet request also differs by path: + +- When the gateway ends it itself (`timeout`, `cancelled`, `no-wait`, + `no-worker`), the gateway emits `lease.rejected`. +- When it ends on a worker's failure, the gateway emits nothing and relies + on the worker's relayed `lease.rejected`. This covers a terminal refusal, + a failure after progress, the last cannot-serve refusal, + `WORKER_UNREACHABLE` and `INTERNAL`. For the last two the worker may have + emitted nothing at all. ADR 0016 §6 joins a fleet request to its outcome by position: the first relayed answer for the namespaced requester after `request.dispatched`. -That fails two ways: +That fails in three ways: -- a relayed refusal from a worker the gateway moved past is taken as the +- a refusal relayed from a worker the gateway moved past counts as the outcome; -- `request.dispatched` is emitted when the grant or the first progress push - reaches the gateway, so a warm grant's relayed `lease.granted` is older - than the dispatch it should follow. +- a warm grant's relayed `lease.granted` is older than the + `request.dispatched` it should follow, because the gateway emits + `request.dispatched` when the grant reaches it; +- a request that failed before any progress has no `request.dispatched` at + all. The gateway never tells the worker which fleet request a dispatch serves, -so no id joins the two records. The same refusals also inflate a worker's -own figures: every stale-view refusal counts as a rejected request that no -caller ever saw. +so no id joins the two records. The same refusals inflate a worker's own +figures: every stale-view refusal counts as a rejected request that no +caller saw. ```mermaid sequenceDiagram participant G as Gateway participant A as Worker A participant B as Worker B - G->>A: lease.request (noWait) + G->>A: lease.request (probe, fleetRequestId) A-->>G: NO_CAPACITY - Note over A: today: lease.rejected no-wait
after: lease.declined - G->>B: lease.request (noWait) + Note over A: today: lease.rejected
after: lease.declined + G->>B: lease.request (probe, fleetRequestId) B-->>G: grant - Note over B: lease.granted - Note over G: request.dispatched + Note over B: lease.granted (fleetRequestId) ``` ## Decision ### 1. A gateway dispatch is a probe and carries the fleet request id -`lease.request` gains an optional `fleetRequestId`. Only the gateway's own -uplink session may set it, as with `owner` (requirement 27a). Any other -session that sets it is refused with `FORBIDDEN`. +`lease.request` gains an optional `fleetRequestId`: a string of at most 200 +characters, the same bound as `idempotencyKey`. Only the gateway's own +uplink session may set it, and any other session that sets it is refused +with `FORBIDDEN`. Requirement 27a lets any `admin` session set `owner`, but +the worker already accepts `owner` only on the uplink session; this field +follows that narrower rule from the start. The field is not part of the +MCP lease tool's input or the HTTP lease body, so neither offers it. -The gateway sets it on every dispatch, to its own request id. A request -that carries it is a **probe**. A probe never waits; the gateway keeps -sending `noWait: true`. Its RPC answer is unchanged, so the gateway's walk -(requirement 11, ADR 0009 §5) works as it does today. +The gateway sets `fleetRequestId` to its own request id on every dispatch. +A request that carries it is a **probe**. A probe must also carry `noWait: +true`; one without it is refused with `BAD_REQUEST` before it is stored. The +RPC answer to a probe is unchanged, so the gateway's walk (requirement 11, +ADR 0009 §5) works as it does today. -### 2. A refused probe is declined, not rejected +### 2. A worker declines a probe, and never rejects one -A probe that the worker refuses with one of the codes the gateway retries -on emits `lease.declined` instead of `lease.rejected`. Those codes are -`NO_CAPACITY`, `UNKNOWN_MODEL`, `RUNTIME_MISSING` and `NO_DRIVER`, before -the first progress push. The payload is `{ requestId, fleetRequestId, -requester, reason }`, with the same reason values `lease.rejected` uses. +A worker never owns the outcome of a probe; the gateway does. So every +refusal or failure of a probe on a worker is recorded as `lease.declined`, +never `lease.rejected`, whatever the reason and however far the work got. +That covers: -Any other refusal of a probe is still `lease.rejected`: `already-leased`, -`lease-id-taken`, a failure after a progress push, a boot timeout. The -gateway treats each of these as the request's terminal failure. +- `no-wait`, `unresolvable-spec`, `already-leased`, `lease-id-taken`; +- `boot-timeout`, `killed`, and `daemon-restarted` at the next start. -The set of retried codes is defined once, in the contract. The gateway's -walk and the worker's choice of event both read it, so the two cannot -disagree about which refusals end a request. +The payload is `{ requestId, fleetRequestId, requester, reason }`, with the +reasons `lease.rejected` uses. The worker makes no judgement about whether +the gateway will retry: the event depends only on whether the request is a +probe. A local request is rejected exactly as today. + +The stored request record keeps `fleetRequestId`, so a probe settled at the +next start still names it. ### 3. Every lease event of a probe names the fleet request -The worker adds `fleetRequestId` to every lease event it emits for a probe: +The worker adds `fleetRequestId` to each lease event it emits for a probe: +`lease.requested`, `lease.granted` and `lease.declined`. A probe is never +queued, so there is no `lease.queued`. Later events of the lease +(`lease.renewed`, `lease.released`, `lease.expired`) are joined by `leaseId` +as today. On the existing events the field is additive (events rule 6). + +### 4. The gateway records every fleet request that ends without a grant + +Whenever a fleet request ends without a grant, the gateway emits its own +`lease.rejected` for it, exactly once: + +- each case it covers today (`timeout`, `cancelled`, `no-wait`, + `no-worker`, `lease-id-taken`); +- the last cannot-serve refusal, as `unresolvable-spec`, whether or not the + request had been queued; +- every other failure on the worker it went to, as the new reason + `worker-failed`. That covers a terminal refusal, a failure after progress, + `WORKER_UNREACHABLE`, `INTERNAL` and a dispatch timeout. + +A `worker-failed` rejection carries `code`, the error code the caller got. +The worker that failed is told by the relayed `lease.declined` with the same +`fleetRequestId`. + +### 5. Usage joins a fleet request by id -- `lease.requested`, -- `lease.granted`, -- `lease.rejected`, -- `lease.declined`. +This replaces ADR 0016 §6's join. On a gateway, a fleet request ends in one +of two ways: -A probe is never queued, so there is no `lease.queued`. Later events of the -lease (`lease.renewed`, `lease.released`, `lease.expired`) are joined by -`leaseId` as today. The field is additive on the existing events (events -rule 6). +- **Rejected:** the gateway's own `lease.rejected` for its request id. It + wins over any relayed grant, so a grant that arrives after the gateway gave + up is not that request's outcome. +- **Granted:** the relayed `lease.granted` whose `fleetRequestId` is the + gateway's request id. -### 4. Usage joins a fleet request by id +A gateway can give back a grant and try again. It does this when the +worker's lease id is not the `leaseId` the caller chose (ADR 0020), so two +grants then carry the same `fleetRequestId`. The outcome is the grant whose +`leaseId` is the chosen one. To make that possible, the gateway's +`lease.requested` gains `leaseId` when the caller chose one. Without a +chosen id there is no retry after a grant, so at most one grant carries the +`fleetRequestId`. -This replaces ADR 0016 §6's join rule. On a gateway, a fleet request's -outcome is the relayed `lease.granted` or `lease.rejected` whose -`fleetRequestId` is the gateway's request id. Order and timestamps play no -part, and `request.dispatched` is not needed for the join. A relayed -`lease.declined` is never an outcome. A request with no such event is open. -Refusals the gateway makes itself (its own `no-wait`, `timeout`, -`cancelled`) are its own `lease.rejected` events, as today. +A request with neither is open. Order, timestamps and `request.dispatched` +play no part in the join. Relayed `lease.declined` events are never an +outcome. They count under a `declined` figure for the worker that emitted +them. -On a worker, a request that ends in `lease.declined` is counted under a -`declined` figure. It is not counted under requests or rejections, and it -gives no wait sample. +On a worker, a request that ended in `lease.declined` counts under +`declined`, not under requests or rejections, and gives no wait sample. -### 5. Protocol +1, and an older worker is incompatible +### 6. Protocol +1, and an older worker is incompatible A worker without `fleetRequestId` would refuse the field, and its events would not carry it. The protocol therefore moves up by one, and a worker on @@ -125,16 +174,19 @@ There is no shim. ## Consequences -- A worker in a fleet no longer emits `lease.rejected` for a stale-view or - cannot-serve refusal from the gateway. A consumer that counted those - rejections sees `lease.declined` instead. The emission changes, not the - payload. Both `EVENTS.md` files say so. +- A worker in a fleet no longer emits `lease.rejected` for gateway traffic. + A consumer that counted those rejections sees `lease.declined` instead. + The emission changes, not the payload. Both `EVENTS.md` files say so. - `simlock events` on a worker shows each probe that missed as `lease.declined`, so an operator can tell gateway traffic that went - elsewhere from a refusal a caller saw. -- The fleet join in `usage.get` is one id lookup. The rule about the first - answer after a dispatch is gone. -- `usage.get` gains a `declined` count, per platform and per worker. + elsewhere from a refusal a local caller saw. +- A gateway's `simlock events` shows one `lease.rejected` for every fleet + request that failed, including failures on a worker. `lease.rejected` + gains the reason `worker-failed` and the field `code`. The gateway's + `lease.requested` gains `leaseId`. +- The fleet join in `usage.get` is an id lookup. The rule about the first + answer after a dispatch is gone. `usage.get` gains a `declined` count, + per platform and per worker. - A gateway and its workers must upgrade together. ## Alternatives considered @@ -142,12 +194,17 @@ There is no shim. - **A public `--if-possible` flag on `simlock lease`.** Rejected: a user's `--no-wait` refusal is a real rejection that the caller sees. Only the gateway's probe is not. +- **The worker declines only the refusals the gateway retries.** Rejected: + the worker would have to repeat the gateway's retry rule, including the + moment of the first progress push. A push can be dropped on the way, and + then the two sides disagree. The worker would also need to map errors to + codes, which only the daemon does today. And a terminal failure on a + worker would still leave the gateway with no event of its own. - **Keep `lease.rejected` and add `fleetRequestId` only.** Rejected: the join would be fixed, but a worker's own history would still record - refusals that no caller saw as rejections. -- **Tighten the positional join instead.** Rejected: the dispatch event - follows a warm grant, so position cannot tell a stale refusal from the - outcome. It also leaves the worker's figures wrong. + refusals no caller saw as rejections. +- **Tighten the positional join.** Rejected: a warm grant comes before the + dispatch event, and a failure before progress has no dispatch event. - **A shim for older workers.** Rejected: no other gateway-to-worker change - kept one, and a fleet that mixes versions gets figures that are wrong - without saying so. + kept one, and a fleet that mixes versions would get wrong figures without + saying so. diff --git a/docs/internal/adr/README.md b/docs/internal/adr/README.md index 5ed39677..df6a46b6 100644 --- a/docs/internal/adr/README.md +++ b/docs/internal/adr/README.md @@ -54,7 +54,7 @@ the status is stale. | [0013](0013-the-console-reads-routes-and-follows-the-event-stream.md) | The console reads the routes and follows the event stream | Accepted — not yet implemented | | [0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) | An event has one id, minted where the fact happened | Accepted | | [0015](0015-a-lease-request-is-a-set-of-constraints.md) | A lease request is a set of constraints, and the catalog says which class each model is | Accepted — not yet implemented | -| [0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) | Usage figures are derived on read from the event history | Accepted — not yet implemented; §6's join superseded by 0021 | +| [0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) | Usage figures are derived on read from the event history | Accepted — not yet implemented; §6's fleet join superseded by 0021 | | [0017](0017-the-warm-pool-is-a-module-beside-the-lease-transaction.md) | The warm pool is a module beside the lease transaction, not a step in it | Accepted — not yet implemented | | [0018](0018-leasing-is-one-module-and-every-module-is-entered-through-its-index.md) | Leasing is one module, and every module is entered through its index | Accepted — not yet implemented | | [0019](0019-startup-ends-every-lease-whose-device-is-not-running.md) | Startup ends every lease whose device is not running | Accepted — not yet implemented | From 011b3f8a35a7817bc341f165fee4eb262750d45b Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Thu, 8 Oct 2026 22:04:30 +0200 Subject: [PATCH 3/8] =?UTF-8?q?docs(adr):=200021=20=E2=80=94=20the=20gatew?= =?UTF-8?q?ay=20records=20every=20fleet=20request's=20outcome?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...-is-a-probe-and-names-its-fleet-request.md | 272 ++++++++++-------- docs/internal/adr/README.md | 2 +- 2 files changed, 150 insertions(+), 124 deletions(-) diff --git a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md index ecc26f18..45d237f5 100644 --- a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md +++ b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md @@ -1,19 +1,24 @@ -# 0021. A gateway dispatch is a probe, and the worker's lease events name the fleet request +# 0021. A gateway dispatch is a probe, and the gateway records every fleet request's outcome - **Status:** Accepted — not yet implemented - **Date:** 2026-10-08 - **Issue:** [#329](https://github.com/callstackincubator/simlock/issues/329) - **Supersedes:** [ADR 0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) - §6's rules on which relayed `lease.rejected` counts, on how a fleet request - is joined to its outcome, and its line "The gateway emits no rejection of - its own for a request that failed on its worker". The rest of §6 stands: - request facts come from the gateway's own events, device facts from - relayed ones. + §6's rules on: + - which relayed `lease.rejected` counts; + - how a fleet request is joined to its outcome; + - where a fleet grant is counted from; + - the line "The gateway emits no rejection of its own for a request that + failed on its worker". + + The rest of §6 stands: request facts come from the gateway's own events, + device facts from relayed ones. - **Depends on:** [ADR 0005](0005-gateway-and-worker-modes.md) requirements - 11, 12 and 27a, [ADR 0009](0009-gateway-routing-is-a-list-of-stages.md) - §5, [ADR 0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) - for `workerId` on a relayed event, [ADR + 11, 12, 27a and 30, [ADR + 0009](0009-gateway-routing-is-a-list-of-stages.md) §5, [ADR + 0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) for + `workerId` on a relayed event, [ADR 0020](0020-a-requester-may-choose-its-lease-id.md) for a forwarded `leaseId`. @@ -27,53 +32,49 @@ gateway tries another worker: - `UNKNOWN_MODEL`, `RUNTIME_MISSING` or `NO_DRIVER` before any progress push (ADR 0009 §5). -The request then fails only when no worker is left. A `noWait` caller gets -one more walk. +The request fails only when no worker is left. A `noWait` caller gets one +more walk. The worker does not know any of that. It handles the request like a local -`--no-wait` caller's, and emits `lease.requested` followed by `lease.rejected` -with reason `no-wait` or `unresolvable-spec`. Both are relayed to the -gateway. The history then records a final rejection for a request that -another worker granted, or that is still waiting. +`--no-wait` caller's: `lease.requested`, then `lease.rejected` with reason +`no-wait` or `unresolvable-spec`. Both are relayed to the gateway. The +history then records a final rejection for a request that another worker +granted, or that is still waiting. -Who records the end of a fleet request also differs by path: +The gateway's own record of a fleet request is also incomplete: -- When the gateway ends it itself (`timeout`, `cancelled`, `no-wait`, - `no-worker`), the gateway emits `lease.rejected`. -- When it ends on a worker's failure, the gateway emits nothing and relies - on the worker's relayed `lease.rejected`. This covers a terminal refusal, - a failure after progress, the last cannot-serve refusal, - `WORKER_UNREACHABLE` and `INTERNAL`. For the last two the worker may have - emitted nothing at all. +- It emits `lease.rejected` only when it ends the request itself + (`timeout`, `cancelled`, `no-wait`, `no-worker`). +- When the request fails on a worker, it emits nothing and relies on the + worker's relayed event. For `WORKER_UNREACHABLE` there may be none. +- It emits no event when it grants. The only grant fact is the worker's + relayed `lease.granted`. That event is stamped by the worker's clock, and + it is missing when the worker's event subscription is down. ADR 0016 §6 joins a fleet request to its outcome by position: the first -relayed answer for the namespaced requester after `request.dispatched`. -That fails in three ways: +relayed answer for the namespaced requester after `request.dispatched`. That +fails in three ways: - a refusal relayed from a worker the gateway moved past counts as the outcome; -- a warm grant's relayed `lease.granted` is older than the - `request.dispatched` it should follow, because the gateway emits - `request.dispatched` when the grant reaches it; -- a request that failed before any progress has no `request.dispatched` at - all. +- a warm grant comes before the `request.dispatched` it should follow; +- a request that failed before any progress has no `request.dispatched`. -The gateway never tells the worker which fleet request a dispatch serves, -so no id joins the two records. The same refusals inflate a worker's own -figures: every stale-view refusal counts as a rejected request that no -caller saw. +The same refusals inflate a worker's own figures: every stale-view refusal +counts as a rejected request that no caller saw. ```mermaid sequenceDiagram participant G as Gateway participant A as Worker A participant B as Worker B - G->>A: lease.request (probe, fleetRequestId) + G->>A: lease.request (probe) A-->>G: NO_CAPACITY - Note over A: today: lease.rejected
after: lease.declined - G->>B: lease.request (probe, fleetRequestId) + Note over A: lease.declined + G->>B: lease.request (probe) B-->>G: grant - Note over B: lease.granted (fleetRequestId) + Note over B: lease.granted + Note over G: request.granted (new) ``` ## Decision @@ -84,93 +85,113 @@ sequenceDiagram characters, the same bound as `idempotencyKey`. Only the gateway's own uplink session may set it, and any other session that sets it is refused with `FORBIDDEN`. Requirement 27a lets any `admin` session set `owner`, but -the worker already accepts `owner` only on the uplink session; this field -follows that narrower rule from the start. The field is not part of the -MCP lease tool's input or the HTTP lease body, so neither offers it. +the worker already accepts `owner` only on the uplink session. This field +follows that narrower rule from the start. The field is not part of the MCP +lease tool's input or the HTTP lease body. -The gateway sets `fleetRequestId` to its own request id on every dispatch. -A request that carries it is a **probe**. A probe must also carry `noWait: -true`; one without it is refused with `BAD_REQUEST` before it is stored. The -RPC answer to a probe is unchanged, so the gateway's walk (requirement 11, -ADR 0009 §5) works as it does today. +The gateway sets it to its own request id on every dispatch. A request that +carries it is a **probe**. A probe must carry `noWait: true`, and one +without it is refused with `BAD_REQUEST` before it is stored. A probe is +never queued on the worker: where the worker would queue a request (after a +second failed provision, for example), it declines a probe instead. + +The RPC answer to a probe is unchanged, so the gateway's walk works as +today (requirement 11, ADR 0009 §5). ### 2. A worker declines a probe, and never rejects one -A worker never owns the outcome of a probe; the gateway does. So every -refusal or failure of a probe on a worker is recorded as `lease.declined`, -never `lease.rejected`, whatever the reason and however far the work got. -That covers: +A worker never owns the outcome of a probe; the gateway does. Every refusal +or failure of a probe on a worker is therefore `lease.declined`, never +`lease.rejected`, whatever the reason and however far the work got. That +covers: - `no-wait`, `unresolvable-spec`, `already-leased`, `lease-id-taken`; - `boot-timeout`, `killed`, and `daemon-restarted` at the next start. -The payload is `{ requestId, fleetRequestId, requester, reason }`, with the -reasons `lease.rejected` uses. The worker makes no judgement about whether -the gateway will retry: the event depends only on whether the request is a -probe. A local request is rejected exactly as today. - -The stored request record keeps `fleetRequestId`, so a probe settled at the -next start still names it. - -### 3. Every lease event of a probe names the fleet request - -The worker adds `fleetRequestId` to each lease event it emits for a probe: -`lease.requested`, `lease.granted` and `lease.declined`. A probe is never -queued, so there is no `lease.queued`. Later events of the lease -(`lease.renewed`, `lease.released`, `lease.expired`) are joined by `leaseId` -as today. On the existing events the field is additive (events rule 6). - -### 4. The gateway records every fleet request that ends without a grant - -Whenever a fleet request ends without a grant, the gateway emits its own -`lease.rejected` for it, exactly once: - -- each case it covers today (`timeout`, `cancelled`, `no-wait`, - `no-worker`, `lease-id-taken`); -- the last cannot-serve refusal, as `unresolvable-spec`, whether or not the - request had been queued; -- every other failure on the worker it went to, as the new reason - `worker-failed`. That covers a terminal refusal, a failure after progress, - `WORKER_UNREACHABLE`, `INTERNAL` and a dispatch timeout. - -A `worker-failed` rejection carries `code`, the error code the caller got. -The worker that failed is told by the relayed `lease.declined` with the same -`fleetRequestId`. - -### 5. Usage joins a fleet request by id - -This replaces ADR 0016 §6's join. On a gateway, a fleet request ends in one -of two ways: - -- **Rejected:** the gateway's own `lease.rejected` for its request id. It - wins over any relayed grant, so a grant that arrives after the gateway gave - up is not that request's outcome. -- **Granted:** the relayed `lease.granted` whose `fleetRequestId` is the - gateway's request id. - -A gateway can give back a grant and try again. It does this when the -worker's lease id is not the `leaseId` the caller chose (ADR 0020), so two -grants then carry the same `fleetRequestId`. The outcome is the grant whose -`leaseId` is the chosen one. To make that possible, the gateway's -`lease.requested` gains `leaseId` when the caller chose one. Without a -chosen id there is no retry after a grant, so at most one grant carries the -`fleetRequestId`. - -A request with neither is open. Order, timestamps and `request.dispatched` -play no part in the join. Relayed `lease.declined` events are never an -outcome. They count under a `declined` figure for the worker that emitted -them. +The payload is `{ requestId, fleetRequestId, requester, requestSpec, reason +}`, with `lease.rejected`'s reasons. `requestSpec` is there for the same +reason as on `lease.rejected`: a decline at admission has no +`lease.requested`. + +The worker makes no judgement about whether the gateway will retry: the +event depends only on whether the request is a probe. Every worker site +that emits `lease.rejected` applies that one condition. A local request is +rejected exactly as today. The stored request record keeps `fleetRequestId`, +so a probe settled at the next start still names it. + +### 3. A probe's lease events name the fleet request + +The worker adds `fleetRequestId` to `lease.requested`, `lease.granted` and +`lease.declined` for a probe. Later events of the lease are joined by +`leaseId` as today. On the existing events the field is additive (events +rule 6). Usage does not need it, because §4 records the outcome on the +gateway. It lets an operator reading a worker's events see which fleet +request a probe served. + +### 4. The gateway records every fleet request's outcome + +A fleet request ends in exactly one event of the gateway's own, by the +gateway's clock: + +- **`request.granted`** `{ requestId, workerId, leaseId, workerLeaseId }`, + emitted when the gateway settles a grant into its lease index. `leaseId` + is the gateway lease id and `workerLeaseId` the worker's. A grant the + gateway gave back (ADR 0020's mismatched id) is never settled and emits + nothing. +- **`lease.rejected`**, for every other ending: + - the cases it covers today (`timeout`, `cancelled`, `no-wait`, + `no-worker`, `lease-id-taken`); + - the last cannot-serve refusal, as `unresolvable-spec`, whether or not + the request had been queued; + - every other failure on the worker it went to, as the new reason + `worker-failed`, with `code` (the error code the caller got) and + `workerId`. That covers a terminal refusal, a failure after progress, + `WORKER_UNREACHABLE`, `INTERNAL` and a dispatch timeout. + +A gateway that stops or crashes loses its open requests without an event, +because its queue lives in memory (requirement 30). Usage closes them at the +gateway's next `daemon.started` (§5). + +### 5. Usage on a gateway reads the outcome from the gateway's own events + +This replaces ADR 0016 §6's join and its source for fleet grants. + +**Outcome.** A fleet request's outcome is its gateway `request.granted` or +`lease.rejected`, by request id. A request with neither, followed by a +later `daemon.started` of the gateway, ended at that start and counts as +rejected `daemon-restarted`. Any other request with neither is open. + +**Wait.** A fleet request's wait runs from its `lease.requested` to its +outcome, both by the gateway's clock. + +**Device facts.** These come from the relayed `lease.granted` with that +`workerId` and `workerLeaseId`: the grant source, and held time to that +lease's relayed release or expiry, all by the worker's clock. + +- If the relayed grant is missing, the grant still counts, with source + `unknown`, and gives no held sample. +- Turnaround is wait plus held, so it never subtracts one host's clock from + another's. + +**Per worker.** A grant counts for the `workerId` of its `request.granted`, +and a `worker-failed` rejection for its own `workerId`. Relayed +`lease.declined` events count per worker and per platform (from +`requestSpec`) under `declined`. + +`declined` counts decline events, not requests: one fleet request may be +declined by several workers, or by one worker more than once. Relayed +`lease.rejected`, `lease.declined` and `request.dispatched` never decide an +outcome. On a worker, a request that ended in `lease.declined` counts under `declined`, not under requests or rejections, and gives no wait sample. ### 6. Protocol +1, and an older worker is incompatible -A worker without `fleetRequestId` would refuse the field, and its events -would not carry it. The protocol therefore moves up by one, and a worker on -the previous version is `incompatible` with the gateway, as with every -other change to what the gateway sends a worker (`src/contract/protocol.ts`). -There is no shim. +A worker without `fleetRequestId` would refuse the field. The protocol +therefore moves up by one, and a worker on the previous version is +`incompatible` with the gateway, as with every other change to what the +gateway sends a worker (`src/contract/protocol.ts`). There is no shim. ## Consequences @@ -180,13 +201,16 @@ There is no shim. - `simlock events` on a worker shows each probe that missed as `lease.declined`, so an operator can tell gateway traffic that went elsewhere from a refusal a local caller saw. -- A gateway's `simlock events` shows one `lease.rejected` for every fleet - request that failed, including failures on a worker. `lease.rejected` - gains the reason `worker-failed` and the field `code`. The gateway's - `lease.requested` gains `leaseId`. -- The fleet join in `usage.get` is an id lookup. The rule about the first - answer after a dispatch is gone. `usage.get` gains a `declined` count, - per platform and per worker. +- A gateway's `simlock events` shows one `request.granted` or one + `lease.rejected` for every fleet request that ended while it ran. + `lease.rejected` gains the reason `worker-failed` and the fields `code` + and `workerId`. The `workerId` field sits in the payload, as it does on + `request.dispatched`; it is not the envelope's relay marker. +- Fleet figures no longer depend on the worker's event subscription for + counts and waits, or on the two hosts' clocks agreeing. Only the grant + source and held time need the relayed events. +- `usage.get` gains a `declined` count, per platform and per worker, and an + `unknown` grant source. - A gateway and its workers must upgrade together. ## Alternatives considered @@ -198,11 +222,13 @@ There is no shim. the worker would have to repeat the gateway's retry rule, including the moment of the first progress push. A push can be dropped on the way, and then the two sides disagree. The worker would also need to map errors to - codes, which only the daemon does today. And a terminal failure on a - worker would still leave the gateway with no event of its own. -- **Keep `lease.rejected` and add `fleetRequestId` only.** Rejected: the - join would be fixed, but a worker's own history would still record - refusals no caller saw as rejections. + codes, which only the daemon does today. +- **Join a fleet grant through the relayed `lease.granted` by + `fleetRequestId`.** Rejected: a grant would vanish from the figures when + the worker's subscription is down, and the wait would subtract one host's + clock from another's. +- **The gateway emits `lease.rejected` at stop.** Rejected: it would not + cover a crash. A rule applied when the figures are read covers both. - **Tighten the positional join.** Rejected: a warm grant comes before the dispatch event, and a failure before progress has no dispatch event. - **A shim for older workers.** Rejected: no other gateway-to-worker change diff --git a/docs/internal/adr/README.md b/docs/internal/adr/README.md index df6a46b6..236bc736 100644 --- a/docs/internal/adr/README.md +++ b/docs/internal/adr/README.md @@ -59,4 +59,4 @@ the status is stale. | [0018](0018-leasing-is-one-module-and-every-module-is-entered-through-its-index.md) | Leasing is one module, and every module is entered through its index | Accepted — not yet implemented | | [0019](0019-startup-ends-every-lease-whose-device-is-not-running.md) | Startup ends every lease whose device is not running | Accepted — not yet implemented | | [0020](0020-a-requester-may-choose-its-lease-id.md) | A requester may choose its lease id, and the gateway passes it through bare | Accepted — not yet implemented | -| [0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md) | A gateway dispatch is a probe, and the worker's lease events name the fleet request | Accepted — not yet implemented | +| [0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md) | A gateway dispatch is a probe, and the gateway records every fleet request's outcome | Accepted — not yet implemented | From 89804194933de70ab41880f70b4d46fde73e12e8 Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Thu, 8 Oct 2026 22:12:36 +0200 Subject: [PATCH 4/8] =?UTF-8?q?docs(adr):=200021=20=E2=80=94=20grant=20emi?= =?UTF-8?q?tted=20only=20when=20handed=20to=20a=20caller,=20worker=20field?= =?UTF-8?q?=20not=20the=20relay=20mark,=20probes=20counted=20apart?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...-is-a-probe-and-names-its-fleet-request.md | 65 ++++++++++++------- 1 file changed, 42 insertions(+), 23 deletions(-) diff --git a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md index 45d237f5..de5aecf0 100644 --- a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md +++ b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md @@ -84,7 +84,8 @@ sequenceDiagram `lease.request` gains an optional `fleetRequestId`: a string of at most 200 characters, the same bound as `idempotencyKey`. Only the gateway's own uplink session may set it, and any other session that sets it is refused -with `FORBIDDEN`. Requirement 27a lets any `admin` session set `owner`, but +with `FORBIDDEN`. A gateway's own `lease.request` handler refuses it from +every session, because gateways do not chain. Requirement 27a lets any `admin` session set `owner`, but the worker already accepts `owner` only on the uplink session. This field follows that narrower rule from the start. The field is not part of the MCP lease tool's input or the HTTP lease body. @@ -92,8 +93,9 @@ lease tool's input or the HTTP lease body. The gateway sets it to its own request id on every dispatch. A request that carries it is a **probe**. A probe must carry `noWait: true`, and one without it is refused with `BAD_REQUEST` before it is stored. A probe is -never queued on the worker: where the worker would queue a request (after a -second failed provision, for example), it declines a probe instead. +never queued on the worker. Where the worker would queue a request (after a +second failed provision, for example), it declines a probe instead, with +reason `no-wait`, and answers `NO_CAPACITY`. The RPC answer to a probe is unchanged, so the gateway's walk works as today (requirement 11, ADR 0009 §5). @@ -133,11 +135,16 @@ request a probe served. A fleet request ends in exactly one event of the gateway's own, by the gateway's clock: -- **`request.granted`** `{ requestId, workerId, leaseId, workerLeaseId }`, - emitted when the gateway settles a grant into its lease index. `leaseId` - is the gateway lease id and `workerLeaseId` the worker's. A grant the - gateway gave back (ADR 0020's mismatched id) is never settled and emits - nothing. +- **`request.granted`** `{ requestId, worker, leaseId, workerLeaseId }`, + emitted when the gateway hands the grant to its caller: after the lease + index accepted it, and only if the waiter was still open. `leaseId` is the + gateway lease id and `workerLeaseId` the worker's. + + A grant that hands nothing to a caller emits nothing: + - one given back for ADR 0020's mismatched id; + - one the lease index refused, which ends in `lease-id-taken`; + - one that lands after the waiter was settled (timeout, cancel, or the + gateway stopping). - **`lease.rejected`**, for every other ending: - the cases it covers today (`timeout`, `cancelled`, `no-wait`, `no-worker`, `lease-id-taken`); @@ -145,9 +152,15 @@ gateway's clock: the request had been queued; - every other failure on the worker it went to, as the new reason `worker-failed`, with `code` (the error code the caller got) and - `workerId`. That covers a terminal refusal, a failure after progress, + `worker`. That covers a terminal refusal, a failure after progress, `WORKER_UNREACHABLE`, `INTERNAL` and a dispatch timeout. +The worker field on these two events is named `worker`, not `workerId`. +`payload.workerId` is the only mark of a relayed event (ADR 0014 §6), so a +gateway's own event must not carry it. The worker link refuses a +`request.granted` pushed by a worker, as it refuses the gateway's other own +event names. + A gateway that stops or crashes loses its open requests without an event, because its queue lives in memory (requirement 30). Usage closes them at the gateway's next `daemon.started` (§5). @@ -158,8 +171,8 @@ This replaces ADR 0016 §6's join and its source for fleet grants. **Outcome.** A fleet request's outcome is its gateway `request.granted` or `lease.rejected`, by request id. A request with neither, followed by a -later `daemon.started` of the gateway, ended at that start and counts as -rejected `daemon-restarted`. Any other request with neither is open. +later `daemon.started` of the gateway itself (not a relayed one), ended at +that start and counts as rejected `daemon-restarted`. Any other request with neither is open. **Wait.** A fleet request's wait runs from its `lease.requested` to its outcome, both by the gateway's clock. @@ -173,18 +186,25 @@ lease's relayed release or expiry, all by the worker's clock. - Turnaround is wait plus held, so it never subtracts one host's clock from another's. -**Per worker.** A grant counts for the `workerId` of its `request.granted`, -and a `worker-failed` rejection for its own `workerId`. Relayed +**Per worker.** A grant counts for the `worker` of its `request.granted`, +and a `worker-failed` rejection for its own `worker`. Relayed `lease.declined` events count per worker and per platform (from `requestSpec`) under `declined`. `declined` counts decline events, not requests: one fleet request may be -declined by several workers, or by one worker more than once. Relayed -`lease.rejected`, `lease.declined` and `request.dispatched` never decide an -outcome. - -On a worker, a request that ended in `lease.declined` counts under -`declined`, not under requests or rejections, and gives no wait sample. +declined by several workers, or by one worker more than once. A decline +belongs to the window its `lease.declined` falls in. Relayed +`lease.granted`, `lease.rejected`, `lease.declined`, `request.dispatched` +and `daemon.started` never decide a fleet request's outcome. + +On a worker, the figures separate the two kinds of request: +- **`requests`** counts only local requests, those whose `lease.requested` + has no `fleetRequestId`. +- **`probes`** counts the ones that do. +- **Grants, source and held time** count both kinds, since both use the + worker's devices. +- **`declined`** counts the worker's `lease.declined` events. A probe never + gives a wait sample on the worker; its wait is the gateway's. ### 6. Protocol +1, and an older worker is incompatible @@ -204,13 +224,12 @@ gateway sends a worker (`src/contract/protocol.ts`). There is no shim. - A gateway's `simlock events` shows one `request.granted` or one `lease.rejected` for every fleet request that ended while it ran. `lease.rejected` gains the reason `worker-failed` and the fields `code` - and `workerId`. The `workerId` field sits in the payload, as it does on - `request.dispatched`; it is not the envelope's relay marker. + and `worker`. - Fleet figures no longer depend on the worker's event subscription for counts and waits, or on the two hosts' clocks agreeing. Only the grant source and held time need the relayed events. -- `usage.get` gains a `declined` count, per platform and per worker, and an - `unknown` grant source. +- `usage.get` gains a `declined` count, per platform and per worker, a + `probes` count on a worker, and an `unknown` grant source. - A gateway and its workers must upgrade together. ## Alternatives considered From e5ef6ddd032c5c0ed3921a593ec1ea92808dd13a Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Thu, 8 Oct 2026 22:13:06 +0200 Subject: [PATCH 5/8] =?UTF-8?q?docs(adr):=200021=20=E2=80=94=20name=20the?= =?UTF-8?q?=20device-fact=20join=20fields?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...eway-dispatch-is-a-probe-and-names-its-fleet-request.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md index de5aecf0..eddc2e22 100644 --- a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md +++ b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md @@ -177,9 +177,10 @@ that start and counts as rejected `daemon-restarted`. Any other request with nei **Wait.** A fleet request's wait runs from its `lease.requested` to its outcome, both by the gateway's clock. -**Device facts.** These come from the relayed `lease.granted` with that -`workerId` and `workerLeaseId`: the grant source, and held time to that -lease's relayed release or expiry, all by the worker's clock. +**Device facts.** These come from the relayed `lease.granted` whose +`workerId` is the `request.granted`'s `worker` and whose `leaseId` is its +`workerLeaseId`. They are the grant source, and held time to that lease's +relayed release or expiry, all by the worker's clock. - If the relayed grant is missing, the grant still counts, with source `unknown`, and gives no held sample. From c77a2cf7093333cd1a74ea31825422c362b77129 Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Thu, 8 Oct 2026 22:20:50 +0200 Subject: [PATCH 6/8] =?UTF-8?q?docs(adr):=200021=20=E2=80=94=20check=20ord?= =?UTF-8?q?er,=20probe=20figures=20on=20a=20worker,=20per-worker=20fleet?= =?UTF-8?q?=20figures,=20narrows=20ADR=200014=20=C2=A76?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...-is-a-probe-and-names-its-fleet-request.md | 37 +++++++++++++++---- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md index eddc2e22..40f2c0a0 100644 --- a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md +++ b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md @@ -13,7 +13,12 @@ failed on its worker". The rest of §6 stands: request facts come from the gateway's own events, - device facts from relayed ones. + device facts from relayed ones. It also narrows [ADR + 0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) §6's + "A worker's own events carry no marker ... its `simlock events` shows + what it always showed". A probe's events on a worker carry + `fleetRequestId`, and a refused probe shows as `lease.declined`. A + worker's events for its local requests are unchanged. - **Depends on:** [ADR 0005](0005-gateway-and-worker-modes.md) requirements 11, 12, 27a and 30, [ADR 0009](0009-gateway-routing-is-a-list-of-stages.md) §5, [ADR @@ -91,14 +96,19 @@ follows that narrower rule from the start. The field is not part of the MCP lease tool's input or the HTTP lease body. The gateway sets it to its own request id on every dispatch. A request that -carries it is a **probe**. A probe must carry `noWait: true`, and one -without it is refused with `BAD_REQUEST` before it is stored. A probe is +carries it is a **probe**. A probe must carry `noWait: true`. The checks run +in the handler, in this order, before anything is stored: +1. the session check, which answers `FORBIDDEN`; +2. the `noWait` check, which answers `BAD_REQUEST`. + +So a caller that may not send the field always gets `FORBIDDEN`. A probe is never queued on the worker. Where the worker would queue a request (after a second failed provision, for example), it declines a probe instead, with reason `no-wait`, and answers `NO_CAPACITY`. -The RPC answer to a probe is unchanged, so the gateway's walk works as -today (requirement 11, ADR 0009 §5). +Apart from that second-failure case, which today queues and answers +nothing, the RPC answer to a probe is unchanged. So the gateway's walk works +as today (requirement 11, ADR 0009 §5). ### 2. A worker declines a probe, and never rejects one @@ -204,8 +214,21 @@ On a worker, the figures separate the two kinds of request: - **`probes`** counts the ones that do. - **Grants, source and held time** count both kinds, since both use the worker's devices. -- **`declined`** counts the worker's `lease.declined` events. A probe never - gives a wait sample on the worker; its wait is the gateway's. +- **`declined`** counts the worker's `lease.declined` events. +- **Wait and turnaround.** A probe gives no sample of either on the worker: + both belong to the gateway's request. +- **Requester rows.** A probe counts under its namespaced `gw:` requester in + `granted` and held time, never in `requests` or `rejected`. + +**On a gateway, per worker.** A worker's entry has its device facts: +provisioning, boots, capacity and incidents. It also covers the fleet +requests that ended on it, with their waits and turnarounds: +- those its `request.granted` names; +- its `worker-failed` rejections. + +It has its `declined` events too. A fleet request that ended any other way +(a gateway-reason rejection, `daemon-restarted`, or still open) counts in +the totals and per platform, under no worker. ### 6. Protocol +1, and an older worker is incompatible From 25f5fe6bf3b32073b68e8462420b124295b33582 Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Fri, 9 Oct 2026 21:32:05 +0200 Subject: [PATCH 7/8] =?UTF-8?q?docs(adr):=200021=20=E2=80=94=20check=20ord?= =?UTF-8?q?er=20wording,=20worker-failed=20for=20a=20worker's=20own=20refu?= =?UTF-8?q?sals,=20pointers=20from=200014=20and=200016?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...s-one-id-minted-where-the-fact-happened.md | 3 ++- ...-derived-on-read-from-the-event-history.md | 3 ++- ...-is-a-probe-and-names-its-fleet-request.md | 22 ++++++++++++++----- docs/internal/adr/README.md | 4 ++-- 4 files changed, 22 insertions(+), 10 deletions(-) diff --git a/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md b/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md index a116034f..bce3a024 100644 --- a/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md +++ b/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md @@ -1,6 +1,7 @@ # 0014. An event has one id, minted where the fact happened -- **Status:** Accepted +- **Status:** Accepted §6's "a worker's own events carry no marker" is narrowed by [ADR + 0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md) for a gateway's probes. - **Date:** 2026-10-04 - **Issue:** [#314](https://github.com/callstackincubator/simlock/issues/314) - **Supersedes:** nothing. Narrows [ADR diff --git a/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md b/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md index 7cfb5da5..35cb5fdb 100644 --- a/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md +++ b/docs/internal/adr/0016-usage-figures-are-derived-on-read-from-the-event-history.md @@ -1,7 +1,8 @@ # 0016. Usage figures are derived on read from the event history - **Status:** Accepted — not yet implemented. §6's rules on relayed - rejections and the fleet join are superseded by [ADR + rejections and the fleet join, and the Consequence on gateway requests + counted on a worker, are superseded by [ADR 0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md). - **Date:** 2026-10-05 - **Issue:** [#329](https://github.com/callstackincubator/simlock/issues/329) diff --git a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md index 40f2c0a0..b6ae809c 100644 --- a/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md +++ b/docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md @@ -18,7 +18,10 @@ "A worker's own events carry no marker ... its `simlock events` shows what it always showed". A probe's events on a worker carry `fleetRequestId`, and a refused probe shows as `lease.declined`. A - worker's events for its local requests are unchanged. + worker's events for its local requests are unchanged. And it replaces + ADR 0016's Consequence that a worker in a fleet counts gateway requests + as its own requests, with their wait as the boot or creation time: on a + worker they are probes (§5). - **Depends on:** [ADR 0005](0005-gateway-and-worker-modes.md) requirements 11, 12, 27a and 30, [ADR 0009](0009-gateway-routing-is-a-list-of-stages.md) §5, [ADR @@ -101,7 +104,9 @@ in the handler, in this order, before anything is stored: 1. the session check, which answers `FORBIDDEN`; 2. the `noWait` check, which answers `BAD_REQUEST`. -So a caller that may not send the field always gets `FORBIDDEN`. A probe is +So a caller that may not send the field gets `FORBIDDEN`, whatever `noWait` +says. Input the schema refuses (a `fleetRequestId` out of bounds, a TTL over +the cap) is still `BAD_REQUEST` first, as for any request. A probe is never queued on the worker. Where the worker would queue a request (after a second failed provision, for example), it declines a probe instead, with reason `no-wait`, and answers `NO_CAPACITY`. @@ -157,13 +162,16 @@ gateway's clock: gateway stopping). - **`lease.rejected`**, for every other ending: - the cases it covers today (`timeout`, `cancelled`, `no-wait`, - `no-worker`, `lease-id-taken`); + `no-worker`, and `lease-id-taken` when the gateway's own lease index + refuses the id); - the last cannot-serve refusal, as `unresolvable-spec`, whether or not the request had been queued; - every other failure on the worker it went to, as the new reason `worker-failed`, with `code` (the error code the caller got) and `worker`. That covers a terminal refusal, a failure after progress, - `WORKER_UNREACHABLE`, `INTERNAL` and a dispatch timeout. + `WORKER_UNREACHABLE`, `INTERNAL` and a dispatch timeout. A worker's own + `ALREADY_LEASED` or `LEASE_ID_TAKEN` is `worker-failed` too, with that + `code`. The worker field on these two events is named `worker`, not `workerId`. `payload.workerId` is the only mark of a relayed event (ADR 0014 §6), so a @@ -216,7 +224,8 @@ On a worker, the figures separate the two kinds of request: worker's devices. - **`declined`** counts the worker's `lease.declined` events. - **Wait and turnaround.** A probe gives no sample of either on the worker: - both belong to the gateway's request. + both belong to the gateway's request. A probe is not counted in the + worker's `waiting` series either. - **Requester rows.** A probe counts under its namespaced `gw:` requester in `granted` and held time, never in `requests` or `rejected`. @@ -226,7 +235,8 @@ requests that ended on it, with their waits and turnarounds: - those its `request.granted` names; - its `worker-failed` rejections. -It has its `declined` events too. A fleet request that ended any other way +It has its `declined` events too. `probes` is a worker's own figure: a +gateway's answer has none, in totals or in a worker's entry. A fleet request that ended any other way (a gateway-reason rejection, `daemon-restarted`, or still open) counts in the totals and per platform, under no worker. diff --git a/docs/internal/adr/README.md b/docs/internal/adr/README.md index 236bc736..cd0737d1 100644 --- a/docs/internal/adr/README.md +++ b/docs/internal/adr/README.md @@ -52,9 +52,9 @@ the status is stale. | [0011](0011-the-console-is-a-built-app-the-daemon-serves.md) | The console is a built app the daemon serves at `/` | Accepted — not yet implemented | | [0012](0012-a-worker-answers-the-fleet-operations-as-a-fleet-of-one.md) | A worker answers the fleet operations as a fleet of one | Accepted — not yet implemented | | [0013](0013-the-console-reads-routes-and-follows-the-event-stream.md) | The console reads the routes and follows the event stream | Accepted — not yet implemented | -| [0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) | An event has one id, minted where the fact happened | Accepted | +| [0014](0014-an-event-has-one-id-minted-where-the-fact-happened.md) | An event has one id, minted where the fact happened | Accepted; §6 narrowed by 0021 | | [0015](0015-a-lease-request-is-a-set-of-constraints.md) | A lease request is a set of constraints, and the catalog says which class each model is | Accepted — not yet implemented | -| [0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) | Usage figures are derived on read from the event history | Accepted — not yet implemented; §6's fleet join superseded by 0021 | +| [0016](0016-usage-figures-are-derived-on-read-from-the-event-history.md) | Usage figures are derived on read from the event history | Accepted — not yet implemented; §6's fleet join and a Consequence superseded by 0021 | | [0017](0017-the-warm-pool-is-a-module-beside-the-lease-transaction.md) | The warm pool is a module beside the lease transaction, not a step in it | Accepted — not yet implemented | | [0018](0018-leasing-is-one-module-and-every-module-is-entered-through-its-index.md) | Leasing is one module, and every module is entered through its index | Accepted — not yet implemented | | [0019](0019-startup-ends-every-lease-whose-device-is-not-running.md) | Startup ends every lease whose device is not running | Accepted — not yet implemented | From a53634c95dd035a028aca5d94efbf7e02656afe9 Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Fri, 9 Oct 2026 21:32:42 +0200 Subject: [PATCH 8/8] docs(adr): 0014 status punctuation --- .../0014-an-event-has-one-id-minted-where-the-fact-happened.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md b/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md index bce3a024..c4292aed 100644 --- a/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md +++ b/docs/internal/adr/0014-an-event-has-one-id-minted-where-the-fact-happened.md @@ -1,6 +1,6 @@ # 0014. An event has one id, minted where the fact happened -- **Status:** Accepted §6's "a worker's own events carry no marker" is narrowed by [ADR +- **Status:** Accepted. §6's "a worker's own events carry no marker" is narrowed by [ADR 0021](0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md) for a gateway's probes. - **Date:** 2026-10-04 - **Issue:** [#314](https://github.com/callstackincubator/simlock/issues/314)