diff --git a/docs/adr/0007-remote-device-leases.md b/docs/adr/0007-remote-device-leases.md index 4302946f89..b126b7e0c8 100644 --- a/docs/adr/0007-remote-device-leases.md +++ b/docs/adr/0007-remote-device-leases.md @@ -4,7 +4,7 @@ Accepted -Decision history: 2026-10-21, a `macos-app` lease session takes its app device's claim instead of no claim, so exclusion against whole-Mac and same-app sessions also holds across daemons (ADR 0035, #3323). +Decision history: 2026-10-21, a `macos-app` lease session takes its app device's claim instead of no claim, so exclusion against whole-Mac and same-app sessions also holds across daemons (ADR 0035, #3323). 2026-11-02, the Revyl provider runtime serves its lease lifecycle from a shared lease table instead of a table of its own, and its runtime-owner instance widens to the shared 32-byte digest, which Revyl keeps in-process only (#4102). ## Rules at a glance @@ -13,6 +13,7 @@ Decision history: 2026-10-21, a `macos-app` lease session takes its app device's - Do not mutate lease ownership before the lock. Scope resolution may happen before the lock. - Generated connection profiles must strip daemon and Metro bearer tokens. - A session cannot outlive its lease. Lease-end teardown clears the session's device claim even when the device no longer answers. +- A cloud provider runtime forgets a rented device only after the provider confirmed its stop, and it refuses to allocate a lease that it is ending. - Report each lease that a request ended once, at response level. A client never sends a release after such a request. - Admitted work that is still wanted preserves its lease. If the client hung up, the request neither defers expiry past that cancellation nor renews the lease. - Before admission, send lease heartbeats for as long as a client phase runs, for example an artifact upload. Renew under the window that the caller named, or else under the lease's existing window. @@ -161,6 +162,21 @@ The lease cannot expire during that allocation. If a successful allocation is still wanted by its requester, the inactivity TTL starts from the moment allocation completed. A requester that hung up preserves nothing, and its allocation is released as before. +The Revyl provider runtime serves allocation, release, expired-lease recovery, and shutdown from a shared lease table, `@agent-device/provision-kit/provider-lease-table`. +The provider supplies its start timeout, an optional stop timeout, start, stop, find-by-lease, a key that identifies a rental, and the errors that it reports. +The table owns the rules that decide whether a billed device is still rented: + +- One lease id has at most one start. Overlapping allocations join it, and a ready session is reused. +- Each caller waits until the earlier of its own deadline and the start timeout. A caller whose budget is already spent sends no start. A caller that gives up stops nothing. The start runs while any caller still waits, for at most the start timeout: no single caller's signal or deadline reaches it, and the table tells the provider to abandon it once no caller waits or the lease starts ending. A ready device goes to the callers still waiting within their budgets, and it is stopped when no caller waits. Each caller's budget is checked when the session is handed over, on reuse as well. +- A start can publish its rental before it answers. A teardown waits for the answer for at most the start timeout, then stops the published rental and keeps the lease pending until the answer arrives. +- Release, recovery, a failed start, and shutdown each end a lease through one teardown that all of them join. The teardown fences the lease in the same step: from then on, an allocation of the lease is refused, including an allocation that was already waiting, and the device inventory and interactor no longer serve the lease. Revyl reports a refusal for a lease that a release, a recovery, or a failed start is ending as `provider-lease-releasing`, and a refusal after shutdown began as `runtime-shutting-down`. +- A teardown forgets a lease only after the provider confirmed the stop of every rental that it knows. It stops them one at a time, in the order they were named, and a rental named while a teardown settles gets a teardown of its own. A failed or late stop keeps the lease for the next teardown, which joins the stop still in flight. Shutdown is a fence: from then on, an allocation, or a recovery of a lease that has no entry, is refused with `runtime-shutting-down`, and a release only joins an entry that already exists, so one drain pass ends every entry. It reports what stays rented with `provider-cleanup-pending`. +- If a start fails without an answer, or a recovery finds no entry, the teardown asks find-by-lease. If the provider cannot name the rental, the lease stays pending, and the next teardown asks again. + +A table in each provider was rejected: each provider derived the same rules again, and most review findings on the second provider (#4092) were about that bookkeeping. +Contracts was rejected as the home, because the table is lifecycle mechanics and not wire vocabulary. Capture-kit was rejected, because renting a device prepares it for execution, which ADR 0019 gives to provision-kit. +Open gap: Limrun keeps its own lease ownership until it moves onto the table (#4102). + ## Client-side work that precedes admission The protection of admitted work covers nothing that happens before a request is admitted. diff --git a/docs/adr/0019-request-bound-platform-runtime.md b/docs/adr/0019-request-bound-platform-runtime.md index ad9700e226..042f8b7ffd 100644 --- a/docs/adr/0019-request-bound-platform-runtime.md +++ b/docs/adr/0019-request-bound-platform-runtime.md @@ -212,6 +212,7 @@ Its workspace compliance includes `check:affected` selection, R11/R13 package en > - Host toolchain readiness probes. > - Device boot-failure classification. > - App-resolution caches. +> - The lease table that cloud device provider runtimes rent devices through (ADR 0007). > Platform packages may import provision-kit. > Provision-kit may not import a platform package. > Capture-kit may not import provision-kit. diff --git a/packages/contracts/src/platform-runtime.test.ts b/packages/contracts/src/platform-runtime.test.ts index f8baab3ba7..90ca51a54c 100644 --- a/packages/contracts/src/platform-runtime.test.ts +++ b/packages/contracts/src/platform-runtime.test.ts @@ -4,6 +4,8 @@ import { AppError } from '@agent-device/kernel/errors'; import { localRuntimeOwner, narrowDeviceBinding, + providerRuntimeInstance, + type ProviderRuntimePrincipalField, providerRuntimeOwner, runtimeOwnerKey, sameRuntimeOwner, @@ -213,6 +215,75 @@ function testBinding( }; } +const LIMRUN_PRINCIPAL: readonly ProviderRuntimePrincipalField[] = [ + ['region', 'eu'], + ['apiKey', 'secret-key'], + ['attachedIos', undefined], + ['attachedAndroid', undefined], +]; + +/** + * The digest is durable: a provider descriptor written before this release names the instance its + * runtime derived, so the pin is the contract. A change here orphans every live descriptor. + */ +test('providerRuntimeInstance keeps the digest durable descriptors carry', () => { + assert.equal( + providerRuntimeInstance('limrun', LIMRUN_PRINCIPAL), + 'principal-1426b5849137d86f197cdb796882e8f619e8be77eda1b424144502e1da55af29', + ); + // The salt carries the provider, so one set of credentials is a different account elsewhere. + assert.notEqual( + providerRuntimeInstance('other-provider', LIMRUN_PRINCIPAL), + providerRuntimeInstance('limrun', LIMRUN_PRINCIPAL), + ); + // The provider is normalized as `providerRuntimeOwner` normalizes it, so one account cannot + // arrive as two owners through spelling. + assert.equal( + providerRuntimeInstance(' limrun ', LIMRUN_PRINCIPAL), + providerRuntimeInstance('limrun', LIMRUN_PRINCIPAL), + ); + // An unset fact is the same account as an absent one, never a second instance. + assert.equal( + providerRuntimeInstance('limrun', LIMRUN_PRINCIPAL.slice(0, 2)), + providerRuntimeInstance('limrun', LIMRUN_PRINCIPAL), + ); + // No secret is readable from what travels into diagnostics and durable state. + assert.ok(!providerRuntimeInstance('limrun', LIMRUN_PRINCIPAL).includes('secret-key')); +}); + +test('providerRuntimeInstance refuses a principal that cannot name one account', () => { + assert.throws(() => providerRuntimeInstance(' ', LIMRUN_PRINCIPAL), /non-empty provider/); + assert.throws( + () => providerRuntimeInstance('limrun', [['apiKey', undefined]]), + /at least one fact/, + ); + assert.throws( + () => + providerRuntimeInstance('limrun', [ + ['apiKey', 'a'], + ['apiKey', 'b'], + ]), + /empty or repeated/, + ); + assert.throws(() => providerRuntimeInstance('limrun', [['provider', 'x']]), /empty or repeated/); + // A repeated name is refused even when one of the two is unset. + assert.throws( + () => + providerRuntimeInstance('limrun', [ + ['apiKey', undefined], + ['apiKey', 'b'], + ]), + /empty or repeated/, + ); + // JSON would move an integer-like name ahead of the declared order. + assert.throws(() => providerRuntimeInstance('limrun', [['1', 'x']]), /must not be an integer/); + // `__proto__` is an ordinary fact, never a prototype that drops it from the digest. + assert.notEqual( + providerRuntimeInstance('limrun', [['__proto__', 'a']]), + providerRuntimeInstance('limrun', [['__proto__', 'b']]), + ); +}); + test('unavailableFact freezes the cell and omits hint when none is given', () => { const bare = unavailableFact('unsupported-device-kind'); const hinted = unavailableFact('unsupported-provider-mode', 'provider-owned'); diff --git a/packages/contracts/src/platform-runtime.ts b/packages/contracts/src/platform-runtime.ts index 9af2035f4f..dc18513ebd 100644 --- a/packages/contracts/src/platform-runtime.ts +++ b/packages/contracts/src/platform-runtime.ts @@ -1,3 +1,4 @@ +import { scryptSync } from 'node:crypto'; import { isPlatform, type AppleOS, @@ -76,6 +77,77 @@ export function providerRuntimeOwner( }); } +// The key length and the scrypt cost, block size and parallelism all feed the digest: changing any of +// them re-identifies every derived instance (`maxmem` is only a cap). An instance a composition names +// outright, such as Limrun's `runtimeInstance` option, is not derived here at all. For a provider that +// writes its instance into durable state, such as Limrun's app-log descriptors, a change orphans every +// such resource; Revyl keeps its instance in-process only. Each provider pins its own digest through +// the wrapper that builds its principal. +const RUNTIME_INSTANCE_KEY_LENGTH_BYTES = 32; +const RUNTIME_INSTANCE_SCRYPT_COST = 16_384; +const RUNTIME_INSTANCE_SCRYPT_BLOCK_SIZE = 8; +const RUNTIME_INSTANCE_SCRYPT_PARALLELISM = 1; +const RUNTIME_INSTANCE_SCRYPT_MAX_MEMORY = 64 * 1024 * 1024; + +/** + * One fact of a provider principal: its name, and its value, or `undefined` when this account has + * none. An unset fact is left out of the digest, exactly as an absent one. + */ +export type ProviderRuntimePrincipalField = readonly [name: string, value: string | undefined]; + +/** + * The runtime-owner instance for one provider account, for `providerRuntimeOwner`. The principal is + * the ordered list of facts that decide which devices one runtime rents (credentials, region, attached + * endpoints): two runtimes with the same principal share one owner. It is hashed, not carried, + * because the instance travels into diagnostics and durable envelopes. + * + * The order of the facts is part of the digest, so a provider declares them in one place and pins the + * result. `provider` is normalized as `providerRuntimeOwner` normalizes it, and the salt derives from + * it, so one set of credentials is a different instance under another provider. + * + * The digest is a synchronous, memory-hard scrypt (about 16 MiB, tens of milliseconds): derive it once + * when a runtime is built, never on a request path. + */ +export function providerRuntimeInstance( + provider: string, + principal: readonly ProviderRuntimePrincipalField[], +): string { + const normalizedProvider = provider.trim(); + if (normalizedProvider.length === 0) { + throw new TypeError('Provider runtime instance requires a non-empty provider'); + } + // No prototype, so a field named `__proto__` is an own key like any other. + const facts: Record = Object.create(null) as Record; + facts.provider = normalizedProvider; + const names = new Set(['provider']); + for (const [name, value] of principal) { + if (name.length === 0 || names.has(name)) { + throw new TypeError(`Provider runtime principal field '${name}' is empty or repeated`); + } + // JSON orders integer-like keys first, which would break the declared field order. + if (/^\d+$/.test(name)) { + throw new TypeError(`Provider runtime principal field '${name}' must not be an integer`); + } + names.add(name); + if (value !== undefined) facts[name] = value; + } + if (Object.keys(facts).length === 1) { + throw new TypeError('Provider runtime principal needs at least one fact with a value'); + } + const fingerprint = scryptSync( + JSON.stringify(facts), + `agent-device:${normalizedProvider}-runtime-owner:v1`, + RUNTIME_INSTANCE_KEY_LENGTH_BYTES, + { + N: RUNTIME_INSTANCE_SCRYPT_COST, + r: RUNTIME_INSTANCE_SCRYPT_BLOCK_SIZE, + p: RUNTIME_INSTANCE_SCRYPT_PARALLELISM, + maxmem: RUNTIME_INSTANCE_SCRYPT_MAX_MEMORY, + }, + ); + return `principal-${fingerprint.toString('hex')}`; +} + export function runtimeOwnerKey(owner: RuntimeOwnerRef): string { switch (owner.kind) { case 'local-family': diff --git a/packages/provider-limrun/src/runtime-instance.test.ts b/packages/provider-limrun/src/runtime-instance.test.ts index d30ef3ebb1..96e0aedd37 100644 --- a/packages/provider-limrun/src/runtime-instance.test.ts +++ b/packages/provider-limrun/src/runtime-instance.test.ts @@ -29,3 +29,20 @@ test('derives an identity from attached instances without an API key', () => { resolveLimrunRuntimeInstance({ apiKey: 'k' }), ); }); + +/** + * Durable app-log descriptors name the instance a Limrun runtime derived, so these digests are the + * contract: field names, field order, and region normalization all feed them. The values are the ones + * Limrun derived before the fingerprint moved to `providerRuntimeInstance`. + */ +test('keeps the durable digest of an account and of an attached instance', () => { + expect(resolveLimrunRuntimeInstance({ apiKey: 'secret-key', region: ' EU ' })).toBe( + 'principal-1426b5849137d86f197cdb796882e8f619e8be77eda1b424144502e1da55af29', + ); + expect( + resolveLimrunRuntimeInstance({ + apiKey: 'secret-key', + instances: { ios: { apiUrl: 'https://attached.example/api', token: 't' } }, + }), + ).toBe('principal-5e4bdc0668a05c1c16f984e4655fc1d1a3bcd25b5965e3b9f3a40838121d5110'); +}); diff --git a/packages/provider-limrun/src/runtime-instance.ts b/packages/provider-limrun/src/runtime-instance.ts index 52d10f0224..5d57a62937 100644 --- a/packages/provider-limrun/src/runtime-instance.ts +++ b/packages/provider-limrun/src/runtime-instance.ts @@ -1,11 +1,10 @@ -import { scryptSync } from 'node:crypto'; +import { providerRuntimeInstance } from '@agent-device/contracts/platform-runtime'; import type { LimrunInstanceAccess } from './instance-access.ts'; -const RUNTIME_INSTANCE_KEY_LENGTH = 32; -const RUNTIME_INSTANCE_SCRYPT_COST = 16_384; -const RUNTIME_INSTANCE_SCRYPT_MAX_MEMORY = 64 * 1024 * 1024; -const RUNTIME_INSTANCE_SALT = 'agent-device:limrun-runtime-owner:v1'; - +/** + * The runtime-owner instance for one Limrun account: the explicit identity when the composition + * names one, otherwise the fingerprint of the facts that decide which devices this runtime rents. + */ export function resolveLimrunRuntimeInstance(options: { apiKey?: string; region?: string; @@ -17,18 +16,10 @@ export function resolveLimrunRuntimeInstance(options: { if (!explicit) throw new TypeError('Limrun runtimeInstance must be a non-empty string'); return explicit; } - const principal = JSON.stringify({ - provider: 'limrun', - region: options.region?.trim().toLowerCase() || 'default', - apiKey: options.apiKey, - attachedIos: options.instances?.ios?.apiUrl, - attachedAndroid: options.instances?.android?.apiUrl, - }); - const fingerprint = scryptSync(principal, RUNTIME_INSTANCE_SALT, RUNTIME_INSTANCE_KEY_LENGTH, { - N: RUNTIME_INSTANCE_SCRYPT_COST, - r: 8, - p: 1, - maxmem: RUNTIME_INSTANCE_SCRYPT_MAX_MEMORY, - }); - return `principal-${fingerprint.toString('hex')}`; + return providerRuntimeInstance('limrun', [ + ['region', options.region?.trim().toLowerCase() || 'default'], + ['apiKey', options.apiKey], + ['attachedIos', options.instances?.ios?.apiUrl], + ['attachedAndroid', options.instances?.android?.apiUrl], + ]); } diff --git a/packages/provider-revyl/package.json b/packages/provider-revyl/package.json index 1f4964ea95..0532bdf955 100644 --- a/packages/provider-revyl/package.json +++ b/packages/provider-revyl/package.json @@ -25,6 +25,7 @@ "@agent-device/capture-kit": "workspace:*", "@agent-device/contracts": "workspace:*", "@agent-device/kernel": "workspace:*", + "@agent-device/provision-kit": "workspace:*", "@agent-device/provider-webdriver": "workspace:*", "@agent-device/test-kit": "workspace:*", "agent-device": "workspace:*" diff --git a/packages/provider-revyl/src/index.test.ts b/packages/provider-revyl/src/index.test.ts index d27a14ea7b..db6adbb8a3 100644 --- a/packages/provider-revyl/src/index.test.ts +++ b/packages/provider-revyl/src/index.test.ts @@ -138,6 +138,7 @@ test('one account and origin is one runtime identity whatever the URL spelling', resolveRevylRuntimeInstance({ apiKey: 'k', backendUrl: 'https://staging.revyl.test' }), base, ); - // The identity is a hash, never the key: it travels into diagnostics. - assert.match(base, /^principal-[0-9a-f]{32}$/); + // The identity is a hash, never the key: it travels into diagnostics. The pin keeps a reordered or + // renamed principal field from re-identifying every Revyl owner without a test noticing. + assert.equal(base, 'principal-f22648ac37319e6167fe173ccefb0d9c093c4b80f96c3d757c646468b6f83957'); }); diff --git a/packages/provider-revyl/src/revyl-backend.fixtures.ts b/packages/provider-revyl/src/revyl-backend.fixtures.ts index 518ab3b68d..4ec2fb55d7 100644 --- a/packages/provider-revyl/src/revyl-backend.fixtures.ts +++ b/packages/provider-revyl/src/revyl-backend.fixtures.ts @@ -121,6 +121,14 @@ export async function startRevylFixtureBackend( const raw = Buffer.concat(chunks).toString('utf8'); const url = request.url ?? '/'; const method = request.method ?? 'GET'; + // The fixture listens on an ephemeral localhost port that any process on the host can reach. A + // request with neither the API key nor a path under Revyl's API is another process's probe and is + // not recorded. Every other request is, so a runtime call that drops the key or leaves the API + // still shows up in the assertions. + if (header(request, 'authorization') === undefined && !url.startsWith('/api/')) { + response.writeHead(404).end(); + return; + } const recorded = readJson(raw); requests.push({ method, diff --git a/packages/provider-revyl/src/revyl-client.ts b/packages/provider-revyl/src/revyl-client.ts index 5b75f6b4e7..86b2dd6fae 100644 --- a/packages/provider-revyl/src/revyl-client.ts +++ b/packages/provider-revyl/src/revyl-client.ts @@ -21,7 +21,7 @@ const REVYL_ATTRIBUTION = 'agent-device'; const DEFAULT_REQUEST_TIMEOUT_MS = 30_000; /** How long `start_device` may take to answer: Revyl assigns a worker and boots the device in it. */ -const REVYL_START_DEVICE_TIMEOUT_MS = 120_000; +export const REVYL_START_DEVICE_TIMEOUT_MS = 120_000; const RETRY_DELAYS_MS = [500, 1_500] as const; /** Device-proxy actions Revyl answers to a GET; every other action is a POST. */ diff --git a/packages/provider-revyl/src/revyl-session.ts b/packages/provider-revyl/src/revyl-session.ts index 76792f359e..c158eb3d32 100644 --- a/packages/provider-revyl/src/revyl-session.ts +++ b/packages/provider-revyl/src/revyl-session.ts @@ -3,7 +3,11 @@ import { setTimeout as sleep } from 'node:timers/promises'; import type { DeviceLease } from '@agent-device/contracts/device'; import type { DeviceInfo } from '@agent-device/kernel/device'; import { buildRevylDevice, type RevylPlatform } from './device.ts'; -import { revylFailureRefusedTheRequest, type RevylClient } from './revyl-client.ts'; +import { + REVYL_START_DEVICE_TIMEOUT_MS, + revylFailureRefusedTheRequest, + type RevylClient, +} from './revyl-client.ts'; import { revylSessionIdForLease } from './session-identity.ts'; import type { RevylStartDeviceRequest, @@ -13,6 +17,14 @@ import type { /** How long Revyl may take to assign a worker and connect the device before the start is given up on. */ const READY_TIMEOUT_MS = 240_000; +/** + * Covers the gap between `start_device` answering and readiness starting its clock, plus timer jitter. + * Readiness clamps every poll and sleep to its own deadline, so no poll interval needs room here. + */ +const READY_ANSWER_MARGIN_MS = 10_000; +/** The longest one allocation may take: `start_device`'s own answer budget, then readiness. */ +export const REVYL_ALLOCATION_TIMEOUT_MS = + REVYL_START_DEVICE_TIMEOUT_MS + READY_TIMEOUT_MS + READY_ANSWER_MARGIN_MS; /** Revyl's worker answers `health` with 503 until the device is attached; the CLI polls on that beat. */ const READY_POLL_INTERVAL_MS = 2_000; /** Revyl releases an idle session on its own; this is the lease the runtime asks for. */ @@ -51,26 +63,16 @@ export type RevylStartRequestInput = Readonly<{ signal?: AbortSignal; /** Epoch-ms deadline the caller has for the whole allocation; it outranks the runtime's own. */ deadline?: number; + /** Told the session once `start_device` answered with it, the first moment Revyl holds it. */ + onRented?: (rented: RevylRentedSession) => void; }>; /** * One caller's own patience: the signal it cancels with and the deadline it stops at. Both are optional, - * and a caller with neither waits for whatever its work takes. + * and the lease table cuts the deadline to `REVYL_ALLOCATION_TIMEOUT_MS`. */ export type RevylCallerBudget = Readonly<{ signal?: AbortSignal; deadline?: number }>; -/** - * One caller's wait for work this process is already doing, bounded by that caller's own deadline and - * canceled with its own signal. `settled` says whether this wait is what ended the race, which is what - * lets a caller report its own cancellation rather than the outcome of a shared request it stopped - * watching. `dispose` retires the listener once the work itself is done. - */ -export type RevylCallerWait = Readonly<{ - promise: Promise; - settled: () => boolean; - dispose: () => void; -}>; - /** * The session one start left under rent: the session that holds the device, and the run that routes to * it. `sessionId` is always the id the backend was told, or the one it answered with, so the end always @@ -158,6 +160,7 @@ export async function startRevylSession( ), }; } + input.onRented?.({ runId, sessionId }); // Revyl may have answered a run for a start whose caller has already hung up. The device is rented // either way, so the answer still reports it as what is owed an end. if (input.signal?.aborted) { @@ -214,44 +217,6 @@ async function submitRevylStart( return await client.startDevice(request); } -/** - * One caller's wait for work this process is already doing, bounded by that caller's own deadline and - * canceled with its own signal. `settled` says whether this wait is what ended the race, which is what - * lets a caller report its own cancellation rather than the outcome of a shared request it stopped - * watching. `dispose` retires the listener once the work itself is done. - */ -export function revylCallerWait( - deadline: number, - budget: RevylCallerBudget, - onGiveUp: () => unknown, -): RevylCallerWait | undefined { - if (budget.deadline === undefined && budget.signal === undefined) return undefined; - const parts: AbortSignal[] = []; - if (Number.isFinite(deadline)) - parts.push(AbortSignal.timeout(Math.max(0, deadline - Date.now()))); - if (budget.signal !== undefined) parts.push(budget.signal); - const signal = parts.length === 1 ? parts[0]! : AbortSignal.any(parts); - let rejected = false; - let onAbort: () => void = () => {}; - const reject = (reason: unknown) => { - rejected = true; - return reason; - }; - // A signal that is already aborted never fires its listener, so that wait has to start over the - // edge rather than sit on it. - const promise = signal.aborted - ? Promise.reject(reject(onGiveUp())) - : new Promise((_, rejectPromise) => { - onAbort = () => rejectPromise(reject(onGiveUp())); - signal.addEventListener('abort', onAbort, { once: true }); - }); - return Object.freeze({ - promise, - settled: () => rejected, - dispose: () => signal.removeEventListener('abort', onAbort), - }); -} - function startCanceled(input: RevylStartRequestInput): AppError { return new AppError( 'COMMAND_FAILED', diff --git a/packages/provider-revyl/src/runtime-instance.ts b/packages/provider-revyl/src/runtime-instance.ts index ff0c285188..1706531deb 100644 --- a/packages/provider-revyl/src/runtime-instance.ts +++ b/packages/provider-revyl/src/runtime-instance.ts @@ -1,16 +1,10 @@ -import { scryptSync } from 'node:crypto'; +import { providerRuntimeInstance } from '@agent-device/contracts/platform-runtime'; import { REVYL_PROVIDER } from './device.ts'; import { REVYL_DEFAULT_BACKEND_URL, resolveRevylBackendUrl } from './revyl-client.ts'; -const RUNTIME_INSTANCE_KEY_LENGTH = 16; -const RUNTIME_INSTANCE_SCRYPT_COST = 16_384; -const RUNTIME_INSTANCE_SCRYPT_MAX_MEMORY = 64 * 1024 * 1024; -const RUNTIME_INSTANCE_SALT = 'agent-device:revyl-runtime-owner:v1'; - /** - * The stable identity of one Revyl runtime, and so of one owner. Two runtimes that rent devices for - * the same account against the same origin share it; two accounts never do. The key is hashed rather - * than carried, because the owner's instance travels into diagnostics. + * The stable identity of one Revyl runtime, and so of one owner: one per account and origin. The + * instance travels into diagnostics, so `providerRuntimeInstance` digests the principal, never carries it. */ export function resolveRevylRuntimeInstance( options: Readonly<{ @@ -18,18 +12,10 @@ export function resolveRevylRuntimeInstance( backendUrl?: string; }>, ): string { - const principal = JSON.stringify({ - provider: REVYL_PROVIDER, + return providerRuntimeInstance(REVYL_PROVIDER, [ // The origin the client actually calls: `https://backend.revyl.ai/` and // `https://backend.revyl.ai` are one account, and must not become two owner identities. - backendUrl: resolveRevylBackendUrl(options.backendUrl ?? REVYL_DEFAULT_BACKEND_URL), - apiKey: options.apiKey, - }); - const fingerprint = scryptSync(principal, RUNTIME_INSTANCE_SALT, RUNTIME_INSTANCE_KEY_LENGTH, { - N: RUNTIME_INSTANCE_SCRYPT_COST, - r: 8, - p: 1, - maxmem: RUNTIME_INSTANCE_SCRYPT_MAX_MEMORY, - }); - return `principal-${fingerprint.toString('hex')}`; + ['backendUrl', resolveRevylBackendUrl(options.backendUrl ?? REVYL_DEFAULT_BACKEND_URL)], + ['apiKey', options.apiKey], + ]); } diff --git a/packages/provider-revyl/src/runtime.test.ts b/packages/provider-revyl/src/runtime.test.ts index fb8ce2047f..2fd8e5c329 100644 --- a/packages/provider-revyl/src/runtime.test.ts +++ b/packages/provider-revyl/src/runtime.test.ts @@ -20,6 +20,17 @@ afterEach(async () => { await Promise.all(openBackends.splice(0).map((fixture) => fixture.close())); }); +/** Asserts an allocation failed with this typed reason. */ +async function refusedWith( + allocation: Promise | undefined, + reason: string, +): Promise { + await assert.rejects( + async () => await allocation, + (error: unknown) => error instanceof AppError && error.details?.reason === reason, + ); +} + /** Stops the fixture recorded, which is what every lifecycle assertion reads. */ function stops(fixture: RevylFixtureBackend): string[] { return fixture.paths().filter((path) => path.includes('/stop')); @@ -191,35 +202,40 @@ test('a canceled joiner reports its own cancellation while the shared start runs ]); }); +test('a joiner keeps the device when the caller that started it hangs up mid-readiness', async () => { + const fixture = await backend({ healthDelayMs: 200 }); + const runtime = fixture.createRuntime(); + const controller = new AbortController(); + const first = runtime.leaseLifecycle!.allocate!(revylLease('ios'), { signal: controller.signal }); + const firstCanceled = refusedWith(first, 'request_canceled'); + const joining = runtime.leaseLifecycle!.allocate!(revylLease('ios')); + await waitFor(() => fixture.paths().some((path) => path.endsWith('/health'))); + controller.abort(); + await firstCanceled; + assert.equal((await joining)?.providerSessionId, revylSessionIdForLease('lease-1')); + assert.deepEqual(stops(fixture), []); + await runtime.shutdown(); + assert.deepEqual(stops(fixture), [stopPathFor('lease-1')]); +}); + test('a release that lands while the shared start is still running stops the device it rents', async () => { - // The daemon drops the lease the moment its caller is gone, and a canceled joiner is gone: this release - // arrives while the caller that started the device is still waiting for it. The session is not in the - // table yet, so a release that only read the table would report a device Revyl is about to rent. - const fixture = await backend({ startDelayMs: 400 }); + // A canceled joiner makes the daemon release the lease while the first caller still waits in readiness. + const fixture = await backend({ healthDelayMs: 2_000 }); const runtime = fixture.createRuntime(); const lease = revylLease('ios'); const first = runtime.leaseLifecycle?.allocate?.(lease); - await waitFor(() => fixture.paths().some((path) => path.endsWith('/start_device'))); - + const firstRefused = refusedWith(first, 'provider-lease-releasing'); + await waitFor(() => fixture.paths().some((path) => path.endsWith('/health'))); const controller = new AbortController(); const joining = runtime.leaseLifecycle!.allocate!(lease, { signal: controller.signal }); controller.abort(); - await assert.rejects( - async () => await joining, - (error: unknown) => error instanceof AppError && error.details?.reason === 'request_canceled', - ); - + await refusedWith(joining, 'request_canceled'); const released = await runtime.leaseLifecycle!.release!(lease); assert.equal(released?.providerSessionId, revylSessionIdForLease('lease-1')); - assert.deepEqual(stops(fixture), [ - `POST /api/v1/execution/device/sessions/${revylSessionIdForLease('lease-1')}/stop`, - ]); - - // The caller that rented the device is not made to carry the joiner's release: it still gets its - // answer, and the runtime holds no session a later shutdown would stop twice. - assert.equal((await first)?.providerSessionId, revylSessionIdForLease('lease-1')); + await firstRefused; + assert.ok(!fixture.answeredPaths().some((path) => path.endsWith('/health'))); await runtime.shutdown(); - assert.equal(stops(fixture).length, 1); + assert.deepEqual(stops(fixture), [stopPathFor('lease-1')]); }); test("a joiner whose deadline expires names its own budget, not the other caller's", async () => { @@ -391,6 +407,34 @@ test('shutdown waits for a start already in flight and releases the device it re ); }); +test('a sole caller that hangs up during readiness has its session stopped before health answers', async () => { + const fixture = await backend({ healthDelayMs: 2_000 }); + const runtime = fixture.createRuntime(); + const controller = new AbortController(); + const first = runtime.leaseLifecycle!.allocate!(revylLease('ios'), { signal: controller.signal }); + const canceled = refusedWith(first, 'request_canceled'); + await waitFor(() => fixture.paths().some((path) => path.endsWith('/health'))); + controller.abort(); + await canceled; + await waitFor(() => stops(fixture).length > 0); + assert.deepEqual(stops(fixture), [stopPathFor('lease-1')]); + assert.ok(!fixture.answeredPaths().some((path) => path.endsWith('/health'))); +}); + +test('a caller that gives up while a release is in flight is told the lease is being released', async () => { + const fixture = await backend({ startDelayMs: 400 }); + const runtime = fixture.createRuntime(); + const controller = new AbortController(); + const first = runtime.leaseLifecycle!.allocate!(revylLease('ios'), { signal: controller.signal }); + const refused = refusedWith(first, 'provider-lease-releasing'); + await waitFor(() => fixture.paths().some((path) => path.endsWith('/start_device'))); + const releasing = runtime.leaseLifecycle!.release!(revylLease('ios')); + controller.abort(); + await refused; + await releasing; + assert.deepEqual(stops(fixture), [stopPathFor('lease-1')]); +}); + test('a release that lands mid-start stops the session only once Revyl has made it', async () => { // Revyl creates the session when it answers `start_device`, not when it is asked. A `/stop` sent inside // that gap names a session Revyl never had, and Revyl's 404 is exactly what a stop reads as "already @@ -401,13 +445,15 @@ test('a release that lands mid-start stops the session only once Revyl has made const lease = revylLease('ios'); const allocating = runtime.leaseLifecycle?.allocate?.(lease); + const refused = refusedWith(allocating, 'provider-lease-releasing'); await waitFor(() => fixture.paths().some((path) => path.endsWith('/start_device'))); // The release lands while the start is in flight. It waits for the answer rather than stopping the id it // only has from the lease, and that is what makes the stop below land after the session exists. const released = await runtime.leaseLifecycle!.release!(lease); assert.equal(released?.providerSessionId, revylSessionIdForLease('lease-1')); - assert.equal((await allocating)?.providerSessionId, revylSessionIdForLease('lease-1')); + assert.equal(released?.revylRunId, 'run-1'); + await refused; const answered = fixture.answeredPaths(); const stopAt = answered.findIndex((path) => path.endsWith('/stop')); diff --git a/packages/provider-revyl/src/runtime.ts b/packages/provider-revyl/src/runtime.ts index 476a8a6119..75eabce4e5 100644 --- a/packages/provider-revyl/src/runtime.ts +++ b/packages/provider-revyl/src/runtime.ts @@ -23,22 +23,30 @@ import { } from './device.ts'; import { createRevylClient, isRevylSessionGone, type RevylClient } from './revyl-client.ts'; import { + REVYL_ALLOCATION_TIMEOUT_MS, revylCallerGaveUp, - revylCallerWait, revylCleanupPending, startRevylSession, stopRevylSession, type RevylCallerBudget, type RevylRentedSession, type RevylSession, - type RevylStart, } from './revyl-session.ts'; +import { + createProviderLeaseTable, + type ProviderLeaseCleanupScope, + type ProviderLeaseDriver, + type ProviderLeaseTable, +} from '@agent-device/provision-kit/provider-lease-table'; import { createRevylInteractor } from './interactor.ts'; import { providerRuntimeOwner } from '@agent-device/contracts/platform-runtime'; import { resolveRevylRuntimeInstance } from './runtime-instance.ts'; import { revylSessionIdForLease } from './session-identity.ts'; import type { RevylStartRequest } from './session-allocation.ts'; +/** What one lease's start asks Revyl for, with the platform the lease was admitted for. */ +type RevylLeaseRequest = RevylStartRequest & Readonly<{ platform: RevylPlatform }>; + export type RevylRuntimeOptions = Readonly<{ apiKey: string; /** Overrides the Revyl backend origin; unset uses the public backend. */ @@ -63,62 +71,6 @@ export type RevylRuntime = ProviderDeviceRuntime & { hasLiveSession(device: DeviceInfo): boolean; }; -/** What one lease's start ended with, as the entry that owns its cleanup records it. */ -type RevylStartOutcome = - | Readonly<{ status: 'ready'; session: RevylSession }> - | Readonly<{ status: 'failed'; error: unknown }>; - -/** What ending one lease amounted to: the session Revyl was told to end, and what it left owed. */ -type RevylStopOutcome = Readonly<{ - session: RevylRentedSession; - /** The ended session's own report, so a release answers with the device it let go of. */ - data?: Record; - /** Why Revyl did not confirm the stop; absent once the device is no longer rented. */ - failure?: string; -}>; - -/** - * One lease's place in {@link RevylRuntimeImplementation.leases}: the start that rents its device, the - * session that start produced, and the `/stop` that has not been confirmed yet. - */ -class RevylLeaseEntry { - /** What the start was asked for, which a joining allocation has to ask for too. */ - readonly request: RevylStartRequest; - /** - * Settles once this lease's `start_device` has answered and the stop that a failed start owed has - * settled with it, so a reader that awaits it cannot miss a rented device. - */ - readonly start: Promise; - /** The live device, while this lease has one. */ - session: RevylSession | undefined; - /** The session this start left under rent; absent while the request has not answered yet. */ - stop: RevylRentedSession | undefined; - /** Whether Revyl confirmed `/stop` for {@link stop}. Only this lets the entry leave the map. */ - stopConfirmed: boolean; - /** - * The `/stop` in flight for {@link stop}, shared by every release, recovery and shutdown that arrives - * while it runs, so Revyl sees one stop per attempt. Cleared once it settles, so a failed stop is - * retried by the next caller. - */ - stopping: Promise | undefined; - - /** - * `start` fills this entry in as it goes; it may write the entry's own fields, but it must never wait - * on them, because the assignment below has not happened while its body first runs. - */ - constructor( - request: RevylStartRequest, - start: (entry: RevylLeaseEntry) => Promise, - ) { - this.request = request; - this.session = undefined; - this.stop = undefined; - this.stopConfirmed = false; - this.stopping = undefined; - this.start = start(this); - } -} - /** * Builds the Revyl runtime and the owner the host loads its bindings through. The owner's module is * loaded on first bind, so a session that only rents a device never evaluates the platform-runtime @@ -145,20 +97,9 @@ export function createRevylRuntime(options: RevylRuntimeOptions): RevylRuntimeRe } class RevylRuntimeImplementation implements ProviderDeviceRuntime { - private readonly client: RevylClient; private readonly options: RevylRuntimeOptions; - /** - * A lease entry exists from the moment `start_device` is sent until Revyl confirms `/stop` for every - * session that start produced. Only a confirmed stop removes it. Allocate, join, release, recovery and - * shutdown all read and write this one entry; nothing else holds a cleanup handle. - * - * `start` is how every reader learns what the lease rented, including a start whose caller stopped - * waiting for it: the request outlives that caller to learn what it rented, and until it answers, - * nobody knows whether there is a device to stop. `stop` is what the entry owes, so an entry that is - * still here is an unconfirmed release, and one that rented nothing is not here at all. - */ - private readonly leases = new Map(); - private shuttingDown = false; + /** Every lease this runtime rents a device for, until Revyl confirms that device's `/stop`. */ + private readonly leases: ProviderLeaseTable; readonly provider = REVYL_PROVIDER; readonly leaseLifecycle: LeaseLifecycleProvider = { @@ -168,21 +109,12 @@ class RevylRuntimeImplementation implements ProviderDeviceRuntime { readonly recoverExpiredLease: ProviderExpiredLeaseRecovery = async (lease) => { requireRevylLease(lease, 'recover'); - const entry = this.leases.get(lease.leaseId); - // The daemon restarted and holds a persisted lease with no entry beside it. The id derived from the - // lease is the only handle left, and Revyl answers a session it never had as gone. - const derived = { sessionId: revylSessionIdForLease(lease.leaseId) }; - const outcome = entry - ? await this.endLease(lease.leaseId, entry) - : { session: derived, failure: await this.stopRented(derived) }; - // The daemon forgets a pending release once recovery resolves, so a stop Revyl did not confirm has to - // keep the lease pending rather than report a device Revyl may still be holding. - if (outcome?.failure !== undefined) throw revylReleasePending(lease.leaseId, outcome); + await this.leases.recover(lease); }; readonly deviceInventoryProvider: DeviceInventoryProvider = async (request) => { if (request.leaseProvider !== this.provider || !request.leaseId) return null; - const session = this.leases.get(request.leaseId)?.session; + const session = this.leases.live(request.leaseId); if (!session) return null; // `--platform ios` names the same device as `--platform apple`, so the request is answered with // the platform selector's own rule rather than a field compare that would drop it. @@ -194,11 +126,12 @@ class RevylRuntimeImplementation implements ProviderDeviceRuntime { throw new TypeError('Revyl runtime requires an apiKey'); } this.options = options; - this.client = createRevylClient({ + const client = createRevylClient({ apiKey: options.apiKey, ...(options.backendUrl === undefined ? {} : { backendUrl: options.backendUrl }), clientVersion: options.clientVersion, }); + this.leases = createProviderLeaseTable(createRevylLeaseDriver(client)); } ownsDevice(device: DeviceInfo): boolean { @@ -216,37 +149,10 @@ class RevylRuntimeImplementation implements ProviderDeviceRuntime { return session ? createRevylInteractor(session, runner?.signal) : undefined; } - /** - * Ends every session this runtime holds and reports what it could not confirm. Every entry is awaited - * first, whether or not its start has answered: a start already in flight has a device on its way, and - * a start whose caller gave up is still owed the answer that names the device it rented. A stop that - * failed keeps its entry in the map, so a release that follows, or a daemon that survives this - * shutdown, retries the same session instead of recording a release Revyl never confirmed. - */ async shutdown(): Promise { - this.shuttingDown = true; - // Snapshot first: `allocate` refuses once the flag is set, and it does that before it sends anything, - // so no entry can be added while this runs. - const entries = [...this.leases.entries()]; - const outcomes = await Promise.all( - entries.map(async ([leaseId, entry]) => await this.endLease(leaseId, entry)), - ); - const unconfirmed = outcomes.flatMap((outcome) => - outcome?.failure === undefined ? [] : [{ ...outcome.session, failure: outcome.failure }], - ); - if (unconfirmed.length === 0) return; - throw revylCleanupPending(unconfirmed, { operation: 'shutdown' }); + await this.leases.shutdown(); } - /** - * One allocation of one lease. A lease already in the map is joined rather than started again, so two - * allocations that overlap share the one device the first of them rents; a lease not in the map has its - * entry registered before the request goes out, which is what makes it the only handle for what that - * request rents. - * - * A caller that ran out of its own patience reports that, and leaves the entry alone: the device belongs - * to every caller that joined the start, and the ones still waiting get the answer this one gave up on. - */ private async allocate( lease: DeviceLease, context?: LeaseLifecycleContext, @@ -264,231 +170,136 @@ class RevylRuntimeImplementation implements ProviderDeviceRuntime { // Before the join below: an allocation that reuses a live lease still carries flags of its own, // and Revyl reads none of them — whether it starts a device or joins one that is starting. rejectRefusedRevylProfileFields(context); - const request = resolveRevylStartRequest(context, { - ...(this.options.deviceModel === undefined ? {} : { deviceModel: this.options.deviceModel }), - ...(this.options.osVersion === undefined ? {} : { osVersion: this.options.osVersion }), - }); - // Nothing between here and the entry's registration awaits, so one caller can never miss another's. - if (this.shuttingDown) throw shuttingDownDuring(lease.leaseId, 'allocate'); - const existing = this.leases.get(lease.leaseId); - if (existing) requireSameRevylStartRequest(lease.leaseId, existing.request, request); - // The caller that started the device waits on it the same way a joiner does: one entry, one awaitable, - // each caller bounded by its own patience. - const entry = existing ?? this.rent(lease, platform, request, context); - const outcome = await this.join(lease.leaseId, entry, platform, context); - if (this.shuttingDown) { - // The device this start rented is already an entry shutdown is ending, so this caller only reports - // the refusal rather than handing a session to a runtime that is gone. - throw shuttingDownDuring(lease.leaseId, 'allocate'); - } - if (outcome.status === 'failed') throw outcome.error; - return allocationResult(outcome.session); - } - - /** - * Rents the lease's device and returns the entry that owns what the request rents. The entry is in the - * map before the first `await`, so an allocation that arrives mid-start joins it instead of renting a - * second device, and a release that arrives then has somewhere to find the session. - */ - private rent( - lease: DeviceLease, - platform: RevylPlatform, - request: RevylStartRequest, - context?: LeaseLifecycleContext, - ) { - const entry = new RevylLeaseEntry(request, (target) => - this.runStart(target, lease, platform, context), - ); - this.leases.set(lease.leaseId, entry); - return entry; - } - - /** - * Starts the lease's device, records what it rented in its entry, and ends from that entry a start that - * never became ready — before the failure reaches the caller, so a failed allocate never leaves a device - * running with only a later release to end it. A start Revyl refused outright rented nothing, so its - * entry is closed rather than kept. - */ - private async runStart( - entry: RevylLeaseEntry, - lease: DeviceLease, - platform: RevylPlatform, - context?: LeaseLifecycleContext, - ): Promise { - const leaseId = lease.leaseId; - let started: RevylStart; - try { - started = await startRevylSession(this.client, lease, { - platform, - appUrl: entry.request.appUrl, - deviceModel: entry.request.deviceModel, - osVersion: entry.request.osVersion, - ...revylCallerBudget(context), - }); - } catch (error) { - // The start failed somewhere this module does not model, and the request may already have gone out, - // so the id it carried is the handle the entry has to end. - started = { - status: 'failed', - error, - rented: { sessionId: revylSessionIdForLease(leaseId) }, - }; - } - if (started.status === 'ready') { - entry.session = started.session; - entry.stop = { runId: started.session.runId, sessionId: started.session.sessionId }; - return { status: 'ready', session: started.session }; - } - const rented = started.rented; - // Revyl settled with a refusal: no session came of this start, so the entry owes nothing and is not - // kept for a stop that would only answer 404. - if (rented === undefined) { - entry.stopConfirmed = true; - this.closeLease(leaseId, entry); - return { status: 'failed', error: started.error }; - } - entry.stop = rented; - // The entry ends what a failed start rented rather than waiting for a release to ask: the device is - // billed from the moment Revyl made it, and this is the first moment its id is known. - const failure = await this.stopOwed(entry, rented); - if (failure === undefined) { - entry.stopConfirmed = true; - this.closeLease(leaseId, entry); - return { status: 'failed', error: started.error }; - } - // What is still billed outranks why the start did not finish, so the stop's failure is reported with - // the session id that has to be ended by hand. - return { - status: 'failed', - error: revylCleanupPending([{ ...rented, failure }], {}, started.error), + const request: RevylLeaseRequest = { + ...resolveRevylStartRequest(context, { + ...(this.options.deviceModel === undefined + ? {} + : { deviceModel: this.options.deviceModel }), + ...(this.options.osVersion === undefined ? {} : { osVersion: this.options.osVersion }), + }), + platform, }; + const session = await this.leases.allocate(lease, request, { + ...revylCallerBudget(context), + requireJoinable: (started) => requireSameRevylStartRequest(lease.leaseId, started, request), + }); + return allocationResult(session); } /** - * Joins the start another caller already has running for this lease, within this caller's own budget. - * Hanging up here never stops that start and never closes the entry: this caller took no device, so it - * ends none, and the callers still waiting are owed the same answer. - */ - private async join( - leaseId: string, - entry: RevylLeaseEntry, - platform: RevylPlatform, - context?: LeaseLifecycleContext, - ): Promise { - const live = entry.session; - // A repeat allocation of a session this runtime already holds is that same device, and Revyl is asked - // for nothing. - if (live) return { status: 'ready', session: live }; - const budget = revylCallerBudget(context); - // The number the report names is the wait this caller was granted, so it is measured from the moment - // the join started rather than from the deadline's own epoch instant. - const joinedAt = Date.now(); - const gaveUp = revylCallerWait(budget.deadline ?? Number.POSITIVE_INFINITY, budget, () => - revylCallerGaveUp( - platform, - budget, - Math.max(0, (budget.deadline ?? joinedAt) - joinedAt), - entry.stop?.sessionId ?? revylSessionIdForLease(leaseId), - ), - ); - if (gaveUp === undefined) return await entry.start; - try { - return await Promise.race([entry.start, gaveUp.promise]); - } finally { - gaveUp.dispose(); - } - } - - /** - * Ends one entry: waits for its start, sends the `/stop` it owes, and closes it only when Revyl confirms - * the session is gone. Every path that ends a device comes through here, so an unconfirmed stop stays in - * the map for the next release, recovery, or shutdown to retry on the same session. - * - * The wait is what makes a release that lands mid-start stop the device that start rents. A release - * arrives that way when the daemon drops a lease whose caller is gone while a caller still waits for the - * shared start. - */ - private async endLease( - leaseId: string, - entry: RevylLeaseEntry, - ): Promise { - await entry.start; - const target = entry.stop; - // Nothing owed means nothing rented: the start was refused, or its stop is already confirmed. - if (target === undefined) return undefined; - const session = entry.session; - // A failed start has already ended its own session before this point, and a caller that releases such - // a lease must not make Revyl see a second stop for a session already gone. - const failure = entry.stopConfirmed ? undefined : await this.stopOwed(entry, target); - if (failure === undefined) { - entry.stopConfirmed = true; - this.closeLease(leaseId, entry); - } - return { - session: target, - ...(session === undefined ? {} : { data: allocationResult(session) }), - ...(failure === undefined ? {} : { failure }), - }; - } - - /** Removes an entry once Revyl has confirmed its stop, and only while it is still the map's entry. */ - private closeLease(leaseId: string, entry: RevylLeaseEntry): void { - if (this.leases.get(leaseId) === entry) this.leases.delete(leaseId); - } - - /** - * Ends the lease's session and reports what is still owed, or throws so the daemon keeps the release - * pending and retries it. A device Revyl still holds is never reported as released. + * Ends the lease's session, or throws so the daemon keeps the release pending and retries it. A + * device Revyl still holds is never reported as released. */ private async release(lease: DeviceLease): Promise | undefined> { if (lease.leaseProvider !== this.provider || !platformForRevylLeaseBackend(lease.backend)) { return undefined; } - const entry = this.leases.get(lease.leaseId); - // No entry means nothing is owed: this lease never rented, or a confirmed stop already closed it, and - // Revyl is not asked again for a release it has already confirmed. - if (!entry) return undefined; - const outcome = await this.endLease(lease.leaseId, entry); - if (outcome === undefined) return undefined; - if (outcome.failure !== undefined) throw revylReleasePending(lease.leaseId, outcome); - return outcome.data ?? { providerSessionId: outcome.session.sessionId }; - } - - /** The entry's one in-flight `/stop`, started by the first caller and joined by the rest. */ - private stopOwed( - entry: RevylLeaseEntry, - target: RevylRentedSession, - ): Promise { - entry.stopping ??= this.stopRented(target).finally(() => { - entry.stopping = undefined; - }); - return entry.stopping; - } - - /** - * One `/stop`, with Revyl's own "no such session" read as released: that device is not rented. The - * failure's message is what the caller reports; `undefined` is the confirmation the entry needs. - */ - private async stopRented(target: RevylRentedSession): Promise { - try { - await stopRevylSession(this.client, target.sessionId); - return undefined; - } catch (error) { - return isRevylSessionGone(error) ? undefined : errorMessage(error); - } + const released = await this.leases.release(lease.leaseId); + if (released === undefined) return undefined; + return released.session + ? allocationResult(released.session) + : { revylRunId: released.handle.runId, providerSessionId: released.handle.sessionId }; } private sessionForDevice(device: DeviceInfo): RevylSession | undefined { const parsed = parseRevylDeviceId(device.id); if (!parsed) return undefined; - const session = this.leases.get(parsed.leaseId)?.session; + const session = this.leases.live(parsed.leaseId); return session?.platform === parsed.platform ? session : undefined; } } /** - * The caller's own patience, read from the lease context it arrived with. A caller that names neither - * waits for whatever the work takes. + * What Revyl supplies to the lease table. A session is stopped by the id this runtime issued for its + * lease, so that id is also what a lease this process never started is recovered by. + */ +function createRevylLeaseDriver( + client: RevylClient, +): ProviderLeaseDriver { + return { + startTimeoutMs: REVYL_ALLOCATION_TIMEOUT_MS, + // A session is created when `start_device` answers, not when it is sent, so the issued id is never + // published before that answer: a stop sent earlier reads as "already gone" and leaves the device. + start: async (lease, request, control) => { + const started = await startRevylSession(client, lease, { + platform: request.platform, + appUrl: request.appUrl, + deviceModel: request.deviceModel, + osVersion: request.osVersion, + // The table aborts this once no caller waits or the lease is ending; Revyl bounds `start_device` + // and readiness with its own budgets, which `REVYL_ALLOCATION_TIMEOUT_MS` covers. + signal: control.signal, + onRented: control.rented, + }); + if (started.status === 'failed') return started; + const { runId, sessionId } = started.session; + return { status: 'ready', session: started.session, handle: { runId, sessionId } }; + }, + stop: async (handle) => { + try { + await stopRevylSession(client, handle.sessionId); + } catch (error) { + // Revyl answers a session it no longer has as gone: that device is not rented. + if (!isRevylSessionGone(error)) throw error; + } + }, + handleKey: (handle) => handle.sessionId, + findByLease: async (lease) => ({ sessionId: revylSessionIdForLease(lease.leaseId) }), + shuttingDown: (lease) => shuttingDownDuring(lease.leaseId, 'allocate'), + releasing: (lease) => + new AppError('COMMAND_FAILED', `Revyl lease ${lease.leaseId} is being released.`, { + reason: 'provider-lease-releasing', + leaseId: lease.leaseId, + hint: 'Run open again; it allocates a new lease and rents a fresh device.', + }), + gaveUp: (lease, caller, budgetMs, rented) => + revylCallerGaveUp( + revylLeasePlatform(lease), + caller, + budgetMs, + rented?.sessionId ?? revylSessionIdForLease(lease.leaseId), + ), + cleanupPending: (unconfirmed, scope, cause) => + revylCleanupPending( + unconfirmed.map((stop) => ({ + ...(stop.handle ?? { sessionId: revylSessionIdForLease(stop.leaseId) }), + failure: errorMessage(stop.cause), + })), + revylCleanupDetails(scope), + cause, + ), + }; +} + +/** Which lease or drain a cleanup report belongs to; a failed start's report names its session only. */ +function revylCleanupDetails(scope: ProviderLeaseCleanupScope): Record { + switch (scope.route) { + case 'shutdown': + return { operation: 'shutdown' }; + case 'start': + return {}; + case 'release': + case 'recover': + return { leaseId: scope.leaseId }; + } +} + +/** The platform of a lease the runtime already admitted as Revyl's. */ +function revylLeasePlatform(lease: DeviceLease): RevylPlatform { + const platform = platformForRevylLeaseBackend(lease.backend); + if (platform === undefined) { + throw new AppError('UNSUPPORTED_OPERATION', 'Revyl cannot serve this lease.', { + leaseId: lease.leaseId, + leaseProvider: lease.leaseProvider, + leaseBackend: lease.backend, + }); + } + return platform; +} + +/** + * The caller's own patience, read from the lease context it arrived with. The lease table cuts its + * deadline to `REVYL_ALLOCATION_TIMEOUT_MS`. */ function revylCallerBudget(context?: LeaseLifecycleContext): RevylCallerBudget { return { @@ -497,11 +308,6 @@ function revylCallerBudget(context?: LeaseLifecycleContext): RevylCallerBudget { }; } -/** The failure for a release Revyl did not confirm, with the session an operator has to end by hand. */ -function revylReleasePending(leaseId: string, outcome: RevylStopOutcome): AppError { - return revylCleanupPending([{ ...outcome.session, failure: outcome.failure ?? '' }], { leaseId }); -} - /** Recovery and lifecycle calls arrive for one provider's lease; anything else is a routing bug. */ function requireRevylLease(lease: DeviceLease, operation: 'recover' | 'release'): void { if (lease.leaseProvider === REVYL_PROVIDER && platformForRevylLeaseBackend(lease.backend)) return; diff --git a/packages/provider-revyl/tsconfig.json b/packages/provider-revyl/tsconfig.json index 34fb3ee780..71c7fd8f75 100644 --- a/packages/provider-revyl/tsconfig.json +++ b/packages/provider-revyl/tsconfig.json @@ -27,6 +27,9 @@ { "path": "../provider-webdriver" }, + { + "path": "../provision-kit" + }, { "path": "../test-kit" } diff --git a/packages/provision-kit/package.json b/packages/provision-kit/package.json index 78c9108f09..b2cb443c23 100644 --- a/packages/provision-kit/package.json +++ b/packages/provision-kit/package.json @@ -38,6 +38,10 @@ "types": "./src/install-source-network-transport.ts", "default": "./src/install-source-network-transport.ts" }, + "./provider-lease-table": { + "types": "./src/provider-lease-table.ts", + "default": "./src/provider-lease-table.ts" + }, "./toolchain-probe": { "types": "./src/toolchain-probe.ts", "default": "./src/toolchain-probe.ts" diff --git a/packages/provision-kit/src/provider-lease-table-interleaving.test.ts b/packages/provision-kit/src/provider-lease-table-interleaving.test.ts new file mode 100644 index 0000000000..0ea815f896 --- /dev/null +++ b/packages/provision-kit/src/provider-lease-table-interleaving.test.ts @@ -0,0 +1,194 @@ +import assert from 'node:assert/strict'; +import { test, vi } from 'vitest'; +import type { DeviceLease } from '@agent-device/contracts/device'; +import { createProviderLeaseTable, type ProviderLeaseStart } from './provider-lease-table.ts'; + +type Handle = Readonly<{ id: string }>; +type Session = Readonly<{ id: string }>; +type Pending = { resolve(value: T): void; reject(error: unknown): void }; + +/** A seeded generator, so a failing seed replays the same interleaving. */ +function random(seed: number): () => number { + let state = seed >>> 0; + return () => { + state = (state + 0x6d2b79f5) >>> 0; + let mixed = Math.imul(state ^ (state >>> 15), 1 | state); + mixed ^= mixed + Math.imul(mixed ^ (mixed >>> 7), 61 | mixed); + return ((mixed ^ (mixed >>> 14)) >>> 0) / 4_294_967_296; + }; +} + +const LEASES = ['lease-a', 'lease-b']; +/** A lease whose device a previous daemon left rented; this process owns it once it recovers it. */ +const INHERITED = 'lease-c'; + +function lease(leaseId: string): DeviceLease { + const base = { tenantId: 't', runId: 'r', createdAt: 0, heartbeatAt: 0 }; + return { ...base, leaseId, backend: 'ios-instance', expiresAt: Number.MAX_SAFE_INTEGER }; +} + +async function settle(): Promise { + await new Promise((resolve) => setImmediate(resolve)); +} + +/** + * One seeded run: allocations, aborts, releases, recoveries, one shutdown, and time passing, issued in + * seeded bursts so some land in the same tick, with every start and stop answered in a seeded order. + * The provider side records which devices are rented, and which of them this table took on. + */ +async function run(seed: number): Promise { + const next = random(seed); + const pick = (items: readonly T[]): T | undefined => items[Math.floor(next() * items.length)]; + const rented = new Set([`${INHERITED}#0`]); + const owned = new Set(); + const lastDevice = new Map([[INHERITED, `${INHERITED}#0`]]); + const starts: Array> & { id: string }> = []; + const stops: Array> = []; + const stopsInFlight = new Map(); + const controllers: AbortController[] = []; + let confirmEveryStop = false; + let devices = 0; + // Recorded rather than thrown: a throw inside a provider callback would read as a provider failure. + const violations: string[] = []; + const fail = (message: string): void => { + violations.push(message); + }; + const stillRented = (): string[] => [...owned].filter((id) => rented.has(id)); + const table = createProviderLeaseTable({ + startTimeoutMs: 50, + stopTimeoutMs: 20, + start: async (target, _request, control) => + await new Promise((resolve, reject) => { + const id = `${target.leaseId}#${(devices += 1)}`; + lastDevice.set(target.leaseId, id); + starts.push({ id, resolve, reject }); + // Some starts publish their device before they answer, as a provider that names it early does. + if (next() < 0.3) { + rented.add(id); + owned.add(id); + control.rented({ id }); + } + }), + stop: async ({ id }) => { + const inFlight = (stopsInFlight.get(id) ?? 0) + 1; + if (inFlight > 1) fail(`two stops of ${id} in flight`); + stopsInFlight.set(id, inFlight); + try { + if (!confirmEveryStop) { + await new Promise((resolve, reject) => stops.push({ resolve, reject })); + } + rented.delete(id); + } finally { + stopsInFlight.set(id, inFlight - 1); + } + }, + handleKey: ({ id }) => id, + findByLease: async (target) => { + const id = lastDevice.get(target.leaseId) ?? `${target.leaseId}#0`; + owned.add(id); + return { id }; + }, + shuttingDown: () => new Error('shutting down'), + releasing: () => new Error('releasing'), + gaveUp: () => new Error('gave up'), + cleanupPending: (unconfirmed) => + Object.assign(new Error('pending'), { + ids: unconfirmed.map((stop) => stop.handle?.id), + unnamed: unconfirmed.flatMap((stop) => (stop.handle ? [] : [stop.leaseId])), + }), + }); + + const allocate = (leaseId: string): void => { + const controller = new AbortController(); + controllers.push(controller); + void table.allocate(lease(leaseId), {}, { signal: controller.signal }).then( + (session) => { + // Delivered only while the lease serves it, and only for a device still rented. + if (table.live(leaseId) !== session) fail(`${session.id} handed out after its lease ended`); + if (!rented.has(session.id)) fail(`${session.id} handed out after its device stopped`); + }, + () => undefined, + ); + }; + const answerStart = (): void => { + const start = starts.splice(Math.floor(next() * starts.length), 1)[0]; + if (start === undefined) return; + const kind = next(); + const handle = { id: start.id }; + // A device the start already published is the one the answer names; it is not rented twice. + if (kind < 0.85 && !owned.has(start.id)) { + rented.add(start.id); + owned.add(start.id); + } + if (kind < 0.55) start.resolve({ status: 'ready', session: handle, handle }); + else if (kind < 0.7) start.resolve({ status: 'failed', error: new Error('x'), rented: handle }); + else if (kind < 0.85) start.reject(new Error('lost answer')); + else start.resolve({ status: 'failed', error: new Error('refused') }); + }; + const answerStop = (): void => { + const stop = stops.splice(Math.floor(next() * stops.length), 1)[0]; + if (stop === undefined) return; + if (next() < 0.7) stop.resolve(); + else stop.reject(new Error('stop refused')); + }; + let shutdownCalled = false; + const shutdown = (): void => { + shutdownCalled = true; + void table.shutdown().then( + () => { + if (stillRented().length > 0) fail(`shutdown resolved with ${stillRented().join(', ')}`); + }, + (error: { ids?: string[]; unnamed?: string[] }) => { + // A device is reported by its id, or by its lease when the report could not name it yet. + for (const id of stillRented()) { + const leaseId = id.split('#')[0]!; + if (!error.ids?.includes(id) && !error.unnamed?.includes(leaseId)) { + fail(`shutdown left ${id} rented and unreported`); + } + } + }, + ); + }; + const act = (): void => { + const roll = next(); + const leaseId = pick(LEASES)!; + if (roll < 0.2 && !shutdownCalled) allocate(leaseId); + else if (roll < 0.24) void table.recover(lease(INHERITED)).catch(() => undefined); + else if (roll < 0.3) vi.advanceTimersByTime(Math.floor(next() * 60)); + else if (roll < 0.36) pick(controllers)?.abort(); + else if (roll < 0.43) void table.release(leaseId).catch(() => undefined); + else if (roll < 0.48) void table.recover(lease(leaseId)).catch(() => undefined); + else if (roll < 0.51 && !shutdownCalled) shutdown(); + else if (roll < 0.76) answerStart(); + else answerStop(); + }; + + for (let step = 0; step < 40; step += 1) { + const burst = 1 + Math.floor(next() * 3); + for (let op = 0; op < burst; op += 1) act(); + await settle(); + } + // Every start answers and every stop confirms from here on, so a failed stop must still be retried. + confirmEveryStop = true; + while (starts.length > 0 || stops.length > 0) { + starts + .splice(0) + .forEach((start) => start.resolve({ status: 'failed', error: new Error('late') })); + stops.splice(0).forEach((stop) => stop.resolve()); + vi.advanceTimersByTime(100); + await settle(); + } + await table.shutdown().catch(() => undefined); + await table.shutdown(); + assert.deepEqual(violations, [], `seed ${seed}`); + assert.deepEqual(stillRented(), [], `seed ${seed}: devices left rented after the final drain`); +} + +test('seeded interleavings keep every lease-table invariant', async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date'] }); + try { + for (let seed = 0; seed < 400; seed += 1) await run(seed); + } finally { + vi.useRealTimers(); + } +}); diff --git a/packages/provision-kit/src/provider-lease-table.test.ts b/packages/provision-kit/src/provider-lease-table.test.ts new file mode 100644 index 0000000000..88c6924fab --- /dev/null +++ b/packages/provision-kit/src/provider-lease-table.test.ts @@ -0,0 +1,658 @@ +import assert from 'node:assert/strict'; +import { test, vi } from 'vitest'; +import type { DeviceLease } from '@agent-device/contracts/device'; +import { + createProviderLeaseTable, + type ProviderLeaseCleanupScope, + type ProviderLeaseStart, + type ProviderLeaseStartControl, +} from './provider-lease-table.ts'; + +type Session = Readonly<{ id: string }>; +type Handle = Readonly<{ id: string }>; +type Request = Readonly<{ app?: string }>; + +class TableError extends Error { + readonly details: Readonly>; + constructor(details: Readonly>, cause?: unknown) { + super(String(details.reason), { cause }); + this.details = details; + } +} + +type Deferred = { promise: Promise; resolve(value: T): void; reject(error: unknown): void }; + +function deferred(): Deferred { + let resolve!: (value: T) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((onResolve, onReject) => { + resolve = onResolve; + reject = onReject; + }); + return { promise, resolve, reject }; +} + +/** + * A provider whose starts and stops the test answers by hand. Each start waits on `starts`, each stop on + * the next queued answer when one is queued, and every call is recorded. + */ +function fakeProvider( + options: { startTimeoutMs?: number; stopTimeoutMs?: number; releasingThrows?: boolean } = {}, +) { + const controls: Array> = []; + const stopped: string[] = []; + const found: string[] = []; + const starts: Array>> = []; + const stopAnswers: Array<() => Promise> = []; + let findAnswer: (lease: DeviceLease) => Promise = async (lease) => ({ + id: `found-${lease.leaseId}`, + }); + const table = createProviderLeaseTable({ + startTimeoutMs: options.startTimeoutMs ?? 60_000, + ...(options.stopTimeoutMs === undefined ? {} : { stopTimeoutMs: options.stopTimeoutMs }), + start: async (_lease, _request, control) => { + controls.push(control); + const answer = deferred>(); + starts.push(answer); + return await answer.promise; + }, + stop: async (handle) => { + stopped.push(handle.id); + await stopAnswers.shift()?.(); + }, + handleKey: (handle) => handle.id, + findByLease: async (lease) => { + found.push(lease.leaseId); + return await findAnswer(lease); + }, + shuttingDown: (lease) => + new TableError({ reason: 'runtime-shutting-down', leaseId: lease.leaseId }), + releasing: (lease) => { + if (options.releasingThrows) throw new Error('driver bug'); + return new TableError({ reason: 'lease-releasing', leaseId: lease.leaseId }); + }, + gaveUp: (lease, caller, budgetMs, rented) => + new TableError({ + reason: caller.signal?.aborted ? 'request_canceled' : 'provider-not-ready', + leaseId: lease.leaseId, + budgetMs, + callerDeadline: caller.deadline, + rented: rented?.id, + }), + cleanupPending: (unconfirmed, scope: ProviderLeaseCleanupScope, cause) => + new TableError( + { + reason: 'provider-cleanup-pending', + route: scope.route, + handles: unconfirmed.map((stop) => stop.handle?.id ?? `unnamed-${stop.leaseId}`), + }, + cause, + ), + }); + return { + table, + controls, + stopped, + found, + /** Answers the n-th start, in the order the starts were sent. */ + answerStart(index: number, answer: ProviderLeaseStart) { + starts[index]!.resolve(answer); + }, + failStart(index: number, error: unknown) { + starts[index]!.reject(error); + }, + queueStop(answer: () => Promise) { + stopAnswers.push(answer); + }, + answerFind(answer: (lease: DeviceLease) => Promise) { + findAnswer = answer; + }, + }; +} + +function lease(leaseId = 'lease-1'): DeviceLease { + return { + leaseId, + tenantId: 'tenant', + runId: 'run', + backend: 'ios-instance', + leaseProvider: 'fake-cloud', + createdAt: 0, + heartbeatAt: 0, + expiresAt: Number.MAX_SAFE_INTEGER, + }; +} + +function ready(id: string): ProviderLeaseStart { + return { status: 'ready', session: { id }, handle: { id } }; +} + +function reason(expected: string, route?: string) { + return (error: unknown) => + error instanceof TableError && + error.details.reason === expected && + (route === undefined || error.details.route === route); +} + +function refusedStop() { + return async () => { + throw new Error('backend down'); + }; +} + +async function settle(): Promise { + await new Promise((resolve) => setImmediate(resolve)); +} + +async function allocated(provider: ReturnType, id = 'session-1') { + const allocating = provider.table.allocate(lease(), {}); + await settle(); + provider.answerStart(provider.controls.length - 1, ready(id)); + return await allocating; +} + +test('overlapping allocations of one lease share one start, and a repeat reuses its session', async () => { + const provider = fakeProvider(); + const first = provider.table.allocate(lease(), { app: 'a' }); + const second = provider.table.allocate(lease(), {}, { requireJoinable: () => undefined }); + await assert.rejects( + provider.table.allocate( + lease(), + { app: 'b' }, + { + requireJoinable: (started) => { + throw new Error(`started for ${started.app}`); + }, + }, + ), + /started for a/, + ); + await settle(); + assert.equal(provider.controls.length, 1, 'one lease rents one device'); + + provider.answerStart(0, ready('session-1')); + assert.deepEqual(await first, { id: 'session-1' }); + assert.deepEqual(await second, { id: 'session-1' }); + assert.deepEqual(await provider.table.allocate(lease(), {}), { id: 'session-1' }); + await assert.rejects( + provider.table.allocate( + lease(), + { app: 'b' }, + { + requireJoinable: () => { + throw new Error('different request'); + }, + }, + ), + /different request/, + ); + assert.equal(provider.controls.length, 1); +}); + +test('a failed stop keeps the lease for the retry that confirms it, and serves no allocation', async () => { + const provider = fakeProvider(); + await allocated(provider); + + provider.queueStop(refusedStop()); + await assert.rejects( + provider.table.release('lease-1'), + reason('provider-cleanup-pending', 'release'), + ); + assert.equal(provider.table.live('lease-1'), undefined); + await assert.rejects(provider.table.allocate(lease(), {}), reason('lease-releasing')); + + assert.deepEqual(await provider.table.release('lease-1'), { + handle: { id: 'session-1' }, + session: { id: 'session-1' }, + }); + assert.equal( + await provider.table.release('lease-1'), + undefined, + 'a confirmed stop is not sent again', + ); + assert.deepEqual(provider.stopped, ['session-1', 'session-1']); +}); + +test('a release during a start refuses the waiting allocation and stops the device it rents', async () => { + const provider = fakeProvider(); + const allocating = provider.table.allocate(lease(), {}); + const refused = assert.rejects(allocating, reason('lease-releasing')); + await settle(); + + const releasing = provider.table.release('lease-1'); + await settle(); + assert.deepEqual(provider.stopped, [], 'the stop waits for the answer that names the device'); + assert.ok(provider.controls[0]!.signal.aborted, 'the start is told the lease is ending'); + provider.answerStart(0, ready('session-1')); + assert.deepEqual(await releasing, { + handle: { id: 'session-1' }, + session: { id: 'session-1' }, + }); + await refused; + assert.deepEqual(provider.stopped, ['session-1']); +}); + +test("a ready answer after the only caller's deadline is stopped, even before its timer fires", async () => { + // The clock moves past the deadline without running timers, which is a busy event loop: the caller's + // own give-up has not run yet when the start answers. + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date'] }); + try { + const provider = fakeProvider(); + const allocating = provider.table.allocate(lease(), {}, { deadline: Date.now() + 100 }); + const refused = assert.rejects(allocating, reason('provider-not-ready')); + await settle(); + vi.setSystemTime(Date.now() + 200); + provider.answerStart(0, ready('session-1')); + await refused; + await settle(); + assert.deepEqual(provider.stopped, ['session-1']); + } finally { + vi.useRealTimers(); + } +}); + +test('a caller whose budget is already spent sends no start', async () => { + const provider = fakeProvider(); + await assert.rejects( + provider.table.allocate(lease('lease-1'), {}, { deadline: Date.now() - 1 }), + reason('provider-not-ready'), + ); + await assert.rejects( + provider.table.allocate(lease('lease-2'), {}, { signal: AbortSignal.abort() }), + reason('request_canceled'), + ); + await settle(); + assert.equal(provider.controls.length, 0); +}); + +test('a caller that arrives after the start was abandoned is refused, and a later one starts afresh', async () => { + const provider = fakeProvider(); + const controller = new AbortController(); + const first = provider.table.allocate(lease(), {}, { signal: controller.signal }); + controller.abort(); + await assert.rejects(first, reason('request_canceled')); + assert.ok(provider.controls[0]!.signal.aborted); + + await assert.rejects(provider.table.allocate(lease(), {}), reason('lease-releasing')); + provider.answerStart(0, { + status: 'failed', + error: new Error('abandoned'), + rented: { id: 'r1' }, + }); + await settle(); + await settle(); + assert.deepEqual(provider.stopped, ['r1']); + + const fresh = provider.table.allocate(lease(), {}); + await settle(); + provider.answerStart(1, ready('session-2')); + assert.deepEqual(await fresh, { id: 'session-2' }); +}); + +test('a joiner still waiting keeps the device when the caller that started it hangs up', async () => { + const provider = fakeProvider(); + const controller = new AbortController(); + const first = provider.table.allocate(lease(), {}, { signal: controller.signal }); + const joining = provider.table.allocate(lease(), {}); + controller.abort(); + await assert.rejects(first, reason('request_canceled')); + assert.equal(provider.controls[0]!.signal.aborted, false, 'the start still has a caller'); + + provider.answerStart(0, ready('session-1')); + assert.deepEqual(await joining, { id: 'session-1' }); + assert.deepEqual(provider.stopped, []); +}); + +test("a joiner still waiting keeps the device when the first caller's deadline passes", async () => { + const provider = fakeProvider(); + const first = provider.table.allocate(lease(), {}, { deadline: Date.now() + 10 }); + const joining = provider.table.allocate(lease(), {}); + await assert.rejects(first, reason('provider-not-ready')); + assert.equal(provider.controls[0]!.signal.aborted, false, 'the start still has a caller'); + + provider.answerStart(0, ready('session-1')); + assert.deepEqual(await joining, { id: 'session-1' }); + assert.deepEqual(provider.stopped, []); +}); + +test('allocation is bounded by the earlier of the caller deadline and the provider timeout', async () => { + const provider = fakeProvider({ startTimeoutMs: 40 }); + const before = Date.now(); + const unbounded = provider.table.allocate(lease('lease-1'), {}); + const callerDeadline = Date.now() + 10; + const shorter = provider.table.allocate(lease('lease-2'), {}, { deadline: callerDeadline }); + const after = Date.now(); + await settle(); + for (const control of provider.controls) { + assert.ok(control.deadline >= before + 40 && control.deadline <= after + 40); + } + + await assert.rejects(shorter, (error: unknown) => { + assert.ok(reason('provider-not-ready')(error)); + assert.equal((error as TableError).details.callerDeadline, callerDeadline); + return true; + }); + await assert.rejects(unbounded, (error: unknown) => { + assert.ok(reason('provider-not-ready')(error)); + assert.equal( + (error as TableError).details.callerDeadline, + undefined, + 'a caller that named no deadline is not reported one', + ); + return true; + }); + + assert.ok(provider.controls.every((control) => control.signal.aborted)); + provider.answerStart(0, ready('session-1')); + provider.answerStart(1, ready('session-2')); + await settle(); + assert.deepEqual(provider.stopped.sort(), ['session-1', 'session-2']); +}); + +test('a failed start is stopped before its failure reaches the caller', async () => { + const provider = fakeProvider(); + const refused = provider.table.allocate(lease('lease-1'), {}); + const rented = provider.table.allocate(lease('lease-2'), {}); + const unstoppable = provider.table.allocate(lease('lease-3'), {}); + await settle(); + provider.answerStart(0, { status: 'failed', error: new Error('refused') }); + await assert.rejects(refused, /refused/); + assert.deepEqual(provider.stopped, [], 'a refused start rented nothing'); + + provider.answerStart(1, { + status: 'failed', + error: new Error('not ready'), + rented: { id: 'r2' }, + }); + await assert.rejects(rented, /not ready/); + assert.deepEqual(provider.stopped, ['r2']); + + provider.queueStop(refusedStop()); + provider.answerStart(2, { status: 'failed', error: new Error('late'), rented: { id: 'r3' } }); + await assert.rejects(unstoppable, (error: unknown) => { + assert.ok(reason('provider-cleanup-pending', 'start')(error)); + assert.deepEqual((error as TableError).details.handles, ['r3']); + assert.match(String((error as TableError).cause), /late/); + return true; + }); + // The stop that failed stays owed: the lease serves no allocation, and a release retries it. + await assert.rejects(provider.table.allocate(lease('lease-3'), {}), reason('lease-releasing')); + assert.deepEqual(await provider.table.release('lease-3'), { + handle: { id: 'r3' }, + session: undefined, + }); +}); + +test('a start that fails answers its caller at once, stops what it rented, and frees the lease', async () => { + const provider = fakeProvider({ startTimeoutMs: 60_000 }); + const allocating = provider.table.allocate(lease(), {}); + await settle(); + // With a 60 s start timeout, a failure held to that timeout would never settle within the test. + setImmediate(() => provider.failStart(0, new Error('lost answer'))); + await assert.rejects(allocating, /lost answer/); + assert.deepEqual(provider.found, ['lease-1']); + assert.deepEqual(provider.stopped, ['found-lease-1']); + const fresh = provider.table.allocate(lease(), {}); + await settle(); + provider.answerStart(1, ready('session-2')); + assert.deepEqual(await fresh, { id: 'session-2' }); +}); + +test("a driver whose releasing() throws still has a late answer's device stopped", async () => { + const provider = fakeProvider({ startTimeoutMs: 30, releasingThrows: true }); + void provider.table.allocate(lease(), {}).catch(() => undefined); + await settle(); + await assert.rejects(provider.table.release('lease-1'), reason('provider-cleanup-pending')); + provider.answerStart(0, ready('late')); + await settle(); + await settle(); + assert.deepEqual(provider.stopped, ['late']); +}); + +test('a lost start answer the provider cannot find keeps the lease until a retry finds it', async () => { + const provider = fakeProvider(); + provider.answerFind(async () => { + throw new Error('search failed'); + }); + const allocating = provider.table.allocate(lease(), {}); + await settle(); + provider.failStart(0, new Error('lost answer')); + await assert.rejects(allocating, (error: unknown) => { + assert.ok(reason('provider-cleanup-pending', 'start')(error)); + assert.deepEqual((error as TableError).details.handles, ['unnamed-lease-1']); + return true; + }); + await assert.rejects(provider.table.allocate(lease(), {}), reason('lease-releasing')); + await assert.rejects( + provider.table.release('lease-1'), + reason('provider-cleanup-pending', 'release'), + ); + await assert.rejects(provider.table.shutdown(), reason('provider-cleanup-pending', 'shutdown')); + + provider.answerFind(async () => ({ id: 'found-late' })); + assert.deepEqual(await provider.table.release('lease-1'), { + handle: { id: 'found-late' }, + session: undefined, + }); + assert.deepEqual(provider.stopped, ['found-late']); + assert.equal(provider.controls.length, 1); +}); + +test('a published rental is stopped when its start stalls, and the late answer ends the lease', async () => { + const provider = fakeProvider({ startTimeoutMs: 30 }); + const allocating = provider.table.allocate(lease(), {}); + // The release is in flight when the caller's deadline passes, so the refusal names the release. + const gaveUp = assert.rejects(allocating, reason('lease-releasing')); + await settle(); + provider.controls[0]!.rented({ id: 'instance-1' }); + + await assert.rejects(provider.table.release('lease-1'), (error: unknown) => { + assert.ok(reason('provider-cleanup-pending', 'release')(error)); + assert.deepEqual((error as TableError).details.handles, ['instance-1']); + return true; + }); + await gaveUp; + assert.deepEqual(provider.stopped, ['instance-1'], 'the stalled rental is stopped by its handle'); + await assert.rejects(provider.table.allocate(lease(), {}), reason('lease-releasing')); + assert.equal(provider.controls.length, 1); + + provider.answerStart(0, ready('instance-1')); + await settle(); + assert.equal(await provider.table.release('lease-1'), undefined); + assert.deepEqual(provider.stopped, ['instance-1']); +}); + +test('a rental named while an earlier stop is still in flight waits for that stop to confirm', async () => { + const provider = fakeProvider({ startTimeoutMs: 30, stopTimeoutMs: 20 }); + const allocating = provider.table.allocate(lease(), {}); + const gaveUp = assert.rejects(allocating, reason('lease-releasing')); + await settle(); + provider.controls[0]!.rented({ id: 'A' }); + const stopA = deferred(); + provider.queueStop(async () => await stopA.promise); + + // A's stop outlives its budget, so the release reports A pending. + await assert.rejects(provider.table.release('lease-1'), (error: unknown) => { + assert.deepEqual((error as TableError).details.handles, ['A']); + return true; + }); + await gaveUp; + // The start then answers with another rental. Its stop waits for A's, so B can never be the stop that + // lets the lease go while A is still unconfirmed. + provider.answerStart(0, ready('B')); + await settle(); + assert.deepEqual(provider.stopped, ['A']); + await assert.rejects(provider.table.release('lease-1'), (error: unknown) => { + assert.deepEqual((error as TableError).details.handles, ['A', 'B']); + return true; + }); + assert.deepEqual(provider.stopped, ['A']); + + // A's stop settles unconfirmed. That no longer holds B back, and A stays owed. + stopA.reject(new Error('backend down')); + await settle(); + provider.queueStop(refusedStop()); + await assert.rejects(provider.table.shutdown(), (error: unknown) => { + assert.deepEqual((error as TableError).details.handles, ['A']); + return true; + }); + assert.deepEqual(provider.stopped, ['A', 'A', 'B']); + assert.deepEqual(await provider.table.release('lease-1'), { + handle: { id: 'B' }, + session: { id: 'B' }, + }); + assert.deepEqual(provider.stopped, ['A', 'A', 'B', 'A']); +}); + +test('every rental a teardown could not stop is reported, and none is left unstopped', async () => { + const provider = fakeProvider(); + const allocating = provider.table.allocate(lease(), {}); + await settle(); + provider.controls[0]!.rented({ id: 'A' }); + provider.queueStop(refusedStop()); + provider.queueStop(refusedStop()); + provider.answerStart(0, { status: 'failed', error: new Error('not ready'), rented: { id: 'B' } }); + await assert.rejects(allocating, (error: unknown) => { + assert.deepEqual((error as TableError).details.handles, ['A', 'B']); + return true; + }); + provider.queueStop(refusedStop()); + provider.queueStop(refusedStop()); + await assert.rejects(provider.table.shutdown(), (error: unknown) => { + assert.deepEqual((error as TableError).details.handles, ['A', 'B']); + return true; + }); + assert.deepEqual(provider.stopped, ['A', 'B', 'A', 'B']); +}); + +test('a late answer that lands while a teardown is settling is still stopped', async () => { + // The answer lands at every microtask offset after the teardown's wait for it ran out, including the + // one between the teardown deciding `pending` and the teardown being cleared. + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date'] }); + try { + for (let hops = 0; hops < 12; hops += 1) { + const provider = fakeProvider({ startTimeoutMs: 50 }); + const allocating = provider.table.allocate(lease(), {}).catch(() => undefined); + await settle(); + const releasing = provider.table.release('lease-1').catch(() => undefined); + await settle(); + vi.advanceTimersByTime(50); + for (let hop = 0; hop < hops; hop += 1) await Promise.resolve(); + provider.answerStart(0, ready('session-1')); + await releasing; + await allocating; + await settle(); + await settle(); + assert.deepEqual(provider.stopped, ['session-1'], `an answer ${hops} microtasks late`); + } + } finally { + vi.useRealTimers(); + } +}); + +test('a session goes only to callers still within their own budget', async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date'] }); + try { + const provider = fakeProvider(); + const short = provider.table.allocate(lease(), {}, { deadline: Date.now() + 100 }); + const long = provider.table.allocate(lease(), {}); + const shortRefused = assert.rejects(short, reason('provider-not-ready')); + await settle(); + // The short caller's deadline passes before its timer could run, and then the start answers. + vi.setSystemTime(Date.now() + 200); + provider.answerStart(0, ready('session-1')); + await shortRefused; + assert.deepEqual(await long, { id: 'session-1' }); + + // Reuse checks the same budget: a caller that already hung up is refused, and the session stays. + await assert.rejects( + provider.table.allocate(lease(), {}, { signal: AbortSignal.abort() }), + reason('request_canceled'), + ); + assert.deepEqual(provider.table.live('lease-1'), { id: 'session-1' }); + assert.deepEqual(provider.stopped, []); + } finally { + vi.useRealTimers(); + } +}); + +test('shutdown waits for an in-flight allocation, stops its device, and refuses new ones', async () => { + const provider = fakeProvider(); + const allocating = provider.table.allocate(lease(), {}); + const refusal = assert.rejects(allocating, reason('runtime-shutting-down')); + await settle(); + + const shutdown = provider.table.shutdown(); + await settle(); + assert.deepEqual(provider.stopped, [], 'the drain waits for the start to name its device'); + provider.answerStart(0, ready('session-1')); + await shutdown; + await refusal; + assert.deepEqual(provider.stopped, ['session-1']); + + await assert.rejects( + provider.table.allocate(lease('lease-2'), {}), + reason('runtime-shutting-down'), + ); + assert.equal(provider.controls.length, 1); +}); + +test('shutdown fences recovery: a lease it holds nothing for is left to the next daemon', async () => { + const provider = fakeProvider(); + await allocated(provider); + const stop = deferred(); + provider.queueStop(async () => await stop.promise); + + const shutdown = provider.table.shutdown(); + await assert.rejects(provider.table.recover(lease('lease-2')), reason('runtime-shutting-down')); + stop.resolve(); + await shutdown; + assert.deepEqual(provider.found, [], 'a refused recovery searches for nothing'); + assert.deepEqual(provider.stopped, ['session-1']); +}); + +test('recovery stops what the provider finds, and keeps that handle when the stop fails', async () => { + const provider = fakeProvider(); + provider.queueStop(refusedStop()); + await assert.rejects( + provider.table.recover(lease()), + reason('provider-cleanup-pending', 'recover'), + ); + await provider.table.recover(lease()); + assert.deepEqual(provider.found, ['lease-1'], 'the handle found once is the one retried'); + assert.deepEqual(provider.stopped, ['found-lease-1', 'found-lease-1']); +}); + +test('recovery that finds nothing rejects, so the lease stays pending', async () => { + const provider = fakeProvider(); + const notFound = new Error('cannot search'); + provider.answerFind(async () => { + throw notFound; + }); + await assert.rejects(provider.table.recover(lease()), (error: unknown) => { + assert.ok(reason('provider-cleanup-pending', 'recover')(error)); + assert.equal((error as TableError).cause, undefined); + return true; + }); + assert.deepEqual(provider.stopped, []); +}); + +test('recoveries, a release, and a shutdown of an entryless lease share one lookup and one stop', async () => { + const provider = fakeProvider(); + const stop = deferred(); + provider.queueStop(async () => await stop.promise); + + const ending = Promise.all([ + provider.table.recover(lease()), + provider.table.recover(lease()), + provider.table.release('lease-1'), + provider.table.shutdown(), + ]); + await settle(); + await assert.rejects(provider.table.allocate(lease(), {}), reason('runtime-shutting-down')); + stop.resolve(); + const [, , released] = await ending; + assert.equal(released?.handle.id, 'found-lease-1'); + assert.deepEqual(provider.found, ['lease-1']); + assert.deepEqual(provider.stopped, ['found-lease-1']); +}); diff --git a/packages/provision-kit/src/provider-lease-table.ts b/packages/provision-kit/src/provider-lease-table.ts new file mode 100644 index 0000000000..534568d524 --- /dev/null +++ b/packages/provision-kit/src/provider-lease-table.ts @@ -0,0 +1,775 @@ +import type { DeviceLease } from '@agent-device/contracts/device'; + +/** + * The lease table one cloud device provider runtime serves its lease lifecycle from. + * + * A provider rents a billed device per lease. This table owns the rules that decide whether that device + * is still rented (ADR 0007); the provider supplies start, stop, find-by-lease, and its own errors. + * + * Keep this module free of value imports: provider runtimes load it on their construction path. + */ + +/** One caller's patience: the signal it cancels with, and the epoch-ms instant it stops waiting. */ +export type ProviderLeaseBudget = Readonly<{ signal?: AbortSignal; deadline?: number }>; + +export type ProviderLeaseAllocateOptions = ProviderLeaseBudget & + Readonly<{ + /** Throws to refuse joining a start made for another request; called before anything is joined. */ + requireJoinable?(started: Request): void; + }>; + +export type ProviderLeaseStartControl = Readonly<{ + /** + * Aborts when no caller still waits for the start before it answers, or the lease starts ending. + * No single caller's signal or deadline reaches the start: it runs for every caller that joins it. + */ + signal: AbortSignal; + /** Epoch-ms by which the start should answer: `startTimeoutMs` after it was sent. */ + deadline: number; + /** + * Publishes a rental the provider can already stop, before the start answers; the table stops every + * rental it is told of. Never publish a handle whose stop could arrive before its session exists. + */ + rented(handle: Handle): void; +}>; + +/** + * What one start amounted to: a ready session and its handle, or a failure naming in `rented` what it + * left rented beyond what it published. A rejection may have rented; the table stops what is published + * or what `findByLease` names. + */ +export type ProviderLeaseStart = + | Readonly<{ status: 'ready'; session: Session; handle: Handle }> + | Readonly<{ status: 'failed'; error: unknown; rented?: Handle }>; + +/** Which route found a rental still rented, so the provider reports it with the matching details. */ +export type ProviderLeaseCleanupScope = + | Readonly<{ route: 'start' | 'release' | 'recover'; leaseId: string }> + | Readonly<{ route: 'shutdown' }>; + +/** A rental that stays rented, and why; `handle` is absent when nobody could name it yet. */ +export type ProviderUnconfirmedStop = Readonly<{ + leaseId: string; + handle: Handle | undefined; + cause: unknown; +}>; + +export type ProviderLeaseDriver = Readonly<{ + /** + * The longest one start may take to answer. It cuts each caller's wait, and it is how long a teardown + * waits for a start whose caller already gave up, since only that answer may name the rental. + */ + startTimeoutMs: number; + /** + * How long a teardown waits for one stop; a later teardown joins a stop that outlived it. Unset waits. + */ + stopTimeoutMs?: number; + start( + lease: DeviceLease, + request: Request, + control: ProviderLeaseStartControl, + ): Promise>; + /** Stops one rental. Resolves only once the provider confirmed the device is no longer rented. */ + stop(handle: Handle): Promise; + /** Identifies a rental, so one rental is stopped once whichever route named it. */ + handleKey(handle: Handle): string; + /** Names what a lease rented after a restart or a rejected start; rejecting keeps the lease pending. */ + // The table does not bound it, so each provider bounds it with its own request timeout. + findByLease(lease: DeviceLease): Promise; + /** The error for an allocation that arrives or finishes while the runtime shuts down. */ + shuttingDown(lease: DeviceLease): Error; + /** The error for an allocation of a lease that a release, a recovery, or a failed start is ending. */ + releasing(lease: DeviceLease): Error; + /** + * The error for a caller that stopped waiting: its signal aborted, or its wait ran out. `caller` is the + * budget it asked for, and `budgetMs` the wait it was granted after `startTimeoutMs` cut it. + */ + gaveUp( + lease: DeviceLease, + caller: ProviderLeaseBudget, + budgetMs: number, + rented: Handle | undefined, + ): Error; + /** The error for rentals that stay rented because their stop was not confirmed. */ + cleanupPending( + unconfirmed: readonly ProviderUnconfirmedStop[], + scope: ProviderLeaseCleanupScope, + cause?: unknown, + ): Error; +}>; + +/** What a release ended: the stopped handle, and the session it served when the start became ready. */ +export type ProviderLeaseReleased = Readonly<{ + handle: Handle; + session: Session | undefined; +}>; + +export type ProviderLeaseTable = Readonly<{ + /** The ready session a lease holds in this process, while nothing is ending it. */ + live(leaseId: string): Session | undefined; + /** + * Rents one device per lease id. A lease that is starting is joined, and one that is ready is reused. + * Each caller waits within its own budget. A ready device is stopped instead of handed out when no + * caller is still waiting for it. A lease that something is ending is refused with `releasing`. + */ + allocate( + lease: DeviceLease, + request: Request, + options?: ProviderLeaseAllocateOptions, + ): Promise; + /** + * Ends the lease and forgets it once every rental is confirmed stopped. Resolves `undefined` when this + * process holds nothing for the lease. Rejects, keeping the lease for a retry, when a rental stays. + */ + release(leaseId: string): Promise | undefined>; + /** Ends an expired lease, from its entry when this process has one and from `findByLease` otherwise. */ + recover(lease: DeviceLease): Promise; + /** Refuses new allocations, then ends every lease. Rejects with what stays rented. */ + shutdown(): Promise; +}>; + +type Outcome = + | Readonly<{ status: 'ready'; session: Session }> + | Readonly<{ status: 'failed'; error: unknown }>; + +/** One rental an entry knows of, and the state of its stop. */ +type RentalRecord = { + readonly handle: Handle; + stopped: boolean; + /** + * The stop in flight, which outlives a teardown that stopped waiting for it. It is also the fence that + * keeps a later rental's stop from overtaking this one, so it is cleared only when the stop settles. + */ + stopping: Promise | undefined; +}; + +type StopFailure = Readonly<{ cause: unknown }>; + +/** One caller waiting for a start, with the budget that decides whether it still wants the device. */ +type Waiter = Readonly<{ deadline: number; signal: AbortSignal | undefined }>; + +type StartAnswer = + | ProviderLeaseStart + | Readonly<{ status: 'rejected'; error: unknown }>; + +type Ended = + | Readonly<{ status: 'stopped'; handle: Handle; session: Session | undefined }> + | Readonly<{ status: 'nothing-rented' }> + | Readonly<{ status: 'pending'; owed: readonly Owed[] }>; + +/** A rental still rented after a teardown, and why; `handle` is absent when nobody could name it. */ +type Owed = Readonly<{ handle: Handle | undefined; cause: unknown }>; + +/** + * One lease's place in the table, from the moment its start is sent, or its recovery begins, until every + * rental it knows of is confirmed stopped. + */ +class LeaseEntry { + readonly lease: DeviceLease; + /** Which entry of its lease id this is; a later entry of the same lease id has a higher one. */ + readonly generation: number; + /** + * `starting` until the start answers, `ready` once a waiting caller took its session, `ending` once + * anything ends the lease. Only `starting` → `ready`, and any phase → `ending`, are allowed. + */ + phase: 'starting' | 'ready' | 'ending' = 'starting'; + /** What the start was asked for; absent for an entry a recovery opened. */ + readonly request: Request | undefined; + /** When the start should have answered, and when a teardown stops waiting: `startTimeoutMs` after it was sent. */ + readonly answerBy: number; + /** Aborted once no caller waits for the start before it answers. */ + readonly abandon = new AbortController(); + /** Settles once the start answered and, for a start that failed, once its teardown settled. */ + start: Promise> = Promise.resolve({ status: 'failed', error: undefined }); + answered = false; + readonly answer: Promise; + markAnswered: () => void = () => undefined; + session: Session | undefined = undefined; + /** Every rental this lease named, by `handleKey`, in the order named; each stays until stopped. */ + readonly rentals = new Map>(); + /** The rental named last, which a report names for the lease. */ + latest: Handle | undefined = undefined; + /** Whether the lease may hold a rental that nobody could name yet. */ + unnamed = false; + /** Callers still waiting for the start; one leaves the moment it gives up. */ + readonly waiters = new Set(); + /** Whether a teardown is already queued behind the one in flight. */ + followUpQueued = false; + /** The one teardown in flight, joined by every release, recovery, and drain. */ + ending: Promise> | undefined = undefined; + + constructor( + lease: DeviceLease, + generation: number, + request: Request | undefined, + answerBy: number, + ) { + this.lease = lease; + this.generation = generation; + this.request = request; + this.answerBy = answerBy; + this.answer = new Promise((resolve) => { + this.markAnswered = resolve; + }); + } +} + +export function createProviderLeaseTable( + driver: ProviderLeaseDriver, +): ProviderLeaseTable { + const table = new LeaseTable(driver); + return Object.freeze({ + live: (leaseId: string) => table.live(leaseId), + allocate: async ( + lease: DeviceLease, + request: Request, + options?: ProviderLeaseAllocateOptions, + ) => await table.allocate(lease, request, options), + release: async (leaseId: string) => await table.release(leaseId), + recover: async (lease: DeviceLease) => await table.recover(lease), + shutdown: async () => await table.shutdown(), + }); +} + +class LeaseTable { + /** An entry leaves only once nothing it rented is still rented. */ + private readonly entries = new Map>(); + private shuttingDown = false; + private generations = 0; + private readonly driver: ProviderLeaseDriver; + + constructor(driver: ProviderLeaseDriver) { + this.driver = driver; + } + + live(leaseId: string): Session | undefined { + const entry = this.entries.get(leaseId); + return entry?.phase === 'ready' ? entry.session : undefined; + } + + /** Whether this entry is still its lease id's entry; checked after every await before acting. */ + private isCurrent(entry: LeaseEntry): boolean { + return this.entries.get(entry.lease.leaseId)?.generation === entry.generation; + } + + /** Every refusal names shutdown first, then a lease that is ending or replaced, else `fallback`. */ + private refusal( + lease: DeviceLease, + entry: LeaseEntry | undefined, + fallback: () => unknown, + ): unknown { + if (this.shuttingDown) return this.driver.shuttingDown(lease); + if (entry && (entry.phase === 'ending' || !this.isCurrent(entry))) { + return this.driver.releasing(lease); + } + return fallback(); + } + + async allocate( + lease: DeviceLease, + request: Request, + options: ProviderLeaseAllocateOptions = {}, + ): Promise { + // Nothing between these checks and the entry's registration awaits, so no teardown is missed. + const existing = this.entries.get(lease.leaseId); + if (this.shuttingDown || existing?.phase === 'ending') + throw this.refusal(lease, existing, () => 0); + if (existing?.request !== undefined) options.requireJoinable?.(existing.request); + const caller = callerBudget(options); + const timeoutAt = Date.now() + this.driver.startTimeoutMs; + const deadline = Math.min(caller.deadline ?? timeoutAt, timeoutAt); + // A caller whose budget is already spent never sends a start that nobody would wait for. + if (!existing && budgetSpent(caller, deadline)) { + throw this.driver.gaveUp(lease, caller, 0, undefined); + } + const entry = existing ?? this.rent(lease, request, caller, timeoutAt); + const budgetMs = Math.max(0, deadline - Date.now()); + const outcome = await this.join(entry, caller, deadline); + return this.deliver(entry, outcome, caller, deadline, budgetMs); + } + + /** Hands the session to one caller, or says why this caller does not get it. */ + private deliver( + entry: LeaseEntry, + outcome: Outcome, + caller: ProviderLeaseBudget, + deadline: number, + budgetMs: number, + ): Session { + // A start that failed on its own reports its own error; only shutdown outranks it. + if (outcome.status === 'failed') + throw this.refusal(entry.lease, undefined, () => outcome.error); + // A release that started while this caller waited owns the device now. The budget is checked at + // delivery, on a fresh start and on reuse alike: the session stays for callers in budget. + const refused = this.refusal(entry.lease, entry, () => + budgetSpent(caller, deadline) + ? this.driver.gaveUp(entry.lease, caller, budgetMs, entry.latest) + : undefined, + ); + if (refused !== undefined) throw refused; + return outcome.session; + } + + async release(leaseId: string): Promise | undefined> { + const entry = this.entries.get(leaseId); + if (!entry) return undefined; + const ended = await this.teardown(entry); + if (ended.status === 'pending') throw this.pending(entry, ended, { route: 'release', leaseId }); + return ended.status === 'stopped' + ? { handle: ended.handle, session: ended.session } + : undefined; + } + + async recover(lease: DeviceLease): Promise { + const leaseId = lease.leaseId; + // The daemon forgets a recovered lease once this resolves, so a rental still rented has to reject. + let entry = this.entries.get(leaseId); + if (entry === undefined) { + // A drain that already started would not see a new entry, and the next daemon recovers the lease. + if (this.shuttingDown) throw this.driver.shuttingDown(lease); + entry = this.recovering(lease); + } + const ended = await this.teardown(entry); + if (ended.status === 'pending') throw this.pending(entry, ended, { route: 'recover', leaseId }); + } + + async shutdown(): Promise { + // The fence: from here on, no allocation or recovery registers an entry, so this drain sees all. + this.shuttingDown = true; + const unconfirmed: ProviderUnconfirmedStop[] = []; + const entries = [...this.entries.values()]; + const ended = await Promise.all(entries.map(async (entry) => await this.teardown(entry))); + ended.forEach((outcome, index) => { + if (outcome.status !== 'pending') return; + const leaseId = entries[index]!.lease.leaseId; + for (const owed of outcome.owed) unconfirmed.push({ leaseId, ...owed }); + }); + if (unconfirmed.length > 0) { + throw this.driver.cleanupPending(unconfirmed, { route: 'shutdown' }); + } + } + + /** Registers the entry before the first await, so a concurrent allocation or release finds it. */ + private rent( + lease: DeviceLease, + request: Request, + caller: ProviderLeaseBudget, + answerBy: number, + ): LeaseEntry { + const entry = new LeaseEntry( + lease, + (this.generations += 1), + request, + answerBy, + ); + entry.start = this.runStart(entry, request, caller); + this.entries.set(lease.leaseId, entry); + return entry; + } + + /** Registers an entry for a lease this process holds nothing for, so every route joins its teardown. */ + private recovering(lease: DeviceLease): LeaseEntry { + const entry = new LeaseEntry( + lease, + (this.generations += 1), + undefined, + Date.now(), + ); + entry.answered = true; + entry.markAnswered(); + entry.unnamed = true; + entry.phase = 'ending'; + this.entries.set(lease.leaseId, entry); + return entry; + } + + /** + * Runs the start and records what it rented. A ready device goes to the callers still waiting; one + * nobody waits for, or one a release already owns, is stopped instead. A failed start is torn down + * before its failure reaches any caller. + */ + private async runStart( + entry: LeaseEntry, + request: Request, + caller: ProviderLeaseBudget, + ): Promise> { + const startedAt = Date.now(); + const answer = await this.callStart(entry, request); + recordAnswer(entry, answer, this.driver.handleKey); + if (answer.status === 'ready') { + // Recorded even when nobody takes it, so the teardown reports the device it stopped. + entry.session = answer.session; + // The answer may belong to an entry that ended, or was replaced, while the start ran. + if (entry.phase === 'starting' && this.isCurrent(entry) && stillWanted(entry)) { + entry.phase = 'ready'; + return { status: 'ready', session: answer.session }; + } + } + let failure: unknown; + try { + failure = this.startFailure(entry, answer, caller, startedAt); + } catch (error) { + // A driver whose error factory throws must not keep a rented device from being stopped. + failure = error; + } + return await this.endFailedStart(entry, failure); + } + + /** The start's answer, with a rejection read as an answer that may have rented. */ + private async callStart( + entry: LeaseEntry, + request: Request, + ): Promise> { + try { + return await this.driver.start(entry.lease, request, { + signal: entry.abandon.signal, + deadline: entry.answerBy, + rented: (handle) => { + if (entry.answered) return; + holdRental(entry, handle, this.driver.handleKey(handle)); + if (entry.phase === 'ending') this.followUp(entry); + }, + }); + } catch (error) { + return { status: 'rejected', error }; + } + } + + /** Why a start that hands out no session failed, for the callers still waiting on it. */ + private startFailure( + entry: LeaseEntry, + answer: StartAnswer, + caller: ProviderLeaseBudget, + startedAt: number, + ): unknown { + // A lease that is ending is why the callers get nothing, whatever the aborted start answered. + const budgetMs = Math.max(0, entry.answerBy - startedAt); + return this.refusal(entry.lease, entry, () => + answer.status === 'ready' + ? this.driver.gaveUp(entry.lease, caller, budgetMs, answer.handle) + : answer.error, + ); + } + + /** + * Tears down a start that hands out no session. A release, recovery, or drain that already owns the + * teardown waits on this answer, so the start only nudges it. + */ + private async endFailedStart( + entry: LeaseEntry, + error: unknown, + ): Promise> { + if (entry.phase === 'ending') { + this.followUp(entry); + return { status: 'failed', error }; + } + const ended = await this.teardown(entry); + if (ended.status !== 'pending') return { status: 'failed', error }; + // What is still billed outranks why the start failed, so the report names the rental to end. + const leaseId = entry.lease.leaseId; + return { + status: 'failed', + error: this.driver.cleanupPending( + ended.owed.map((owed) => ({ leaseId, ...owed })), + { route: 'start', leaseId }, + error, + ), + }; + } + + /** + * Waits for the entry's start within this caller's budget. A caller that gives up stops nothing: the + * device belongs to every caller still waiting. + */ + private async join( + entry: LeaseEntry, + caller: ProviderLeaseBudget, + deadline: number, + ): Promise> { + if (entry.phase === 'ready') return { status: 'ready', session: entry.session! }; + const budgetMs = Math.max(0, deadline - Date.now()); + // Guarded, so a throwing driver factory still settles this caller. + const giveUp = (): unknown => { + try { + return this.refusal(entry.lease, entry, () => + this.driver.gaveUp(entry.lease, caller, budgetMs, entry.latest), + ); + } catch (error) { + return error; + } + }; + const timeout = new AbortController(); + const timer = setTimeout(() => timeout.abort(), budgetMs); + // Listened to directly rather than through `AbortSignal.any`, so a long-lived caller signal keeps no + // link to this entry once the wait is over. + const signals = caller.signal ? [caller.signal, timeout.signal] : [timeout.signal]; + const waiter: Waiter = { deadline, signal: caller.signal }; + let onAbort = (): void => undefined; + try { + entry.waiters.add(waiter); + if (signals.some((signal) => signal.aborted)) throw giveUp(); + const gaveUp = new Promise((_, reject) => { + onAbort = () => { + // Built before this caller leaves: its own departure must not read as someone else's release. + const refusal = giveUp(); + entry.waiters.delete(waiter); + this.abandonIfUnwanted(entry); + reject(refusal); + }; + for (const signal of signals) signal.addEventListener('abort', onAbort, { once: true }); + }); + return await Promise.race([entry.start, gaveUp]); + } finally { + entry.waiters.delete(waiter); + this.abandonIfUnwanted(entry); + clearTimeout(timer); + for (const signal of signals) signal.removeEventListener('abort', onAbort); + } + } + + /** + * Abandons a start that no caller still waits for. The lease is fenced and torn down in the same step, + * so a caller that arrives later is refused rather than joining a start that is being given up. + */ + private abandonIfUnwanted(entry: LeaseEntry): void { + if (entry.answered || entry.phase === 'ending' || stillWanted(entry)) return; + void this.teardown(entry).catch(() => undefined); + } + + /** + * The one teardown of an entry, joined by every route that ends it. It fences the entry against + * allocations in the same step, and it is the only place a rental is stopped or an entry forgotten. + */ + private teardown(entry: LeaseEntry): Promise> { + entry.phase = 'ending'; + // No caller can take this start's device any more, so the provider may stop working on it. + if (!entry.answered) entry.abandon.abort(); + if (entry.ending) return entry.ending; + const ending = this.runTeardown(entry).finally(() => { + if (entry.ending === ending) entry.ending = undefined; + }); + entry.ending = ending; + return ending; + } + + private async runTeardown( + entry: LeaseEntry, + ): Promise> { + // Each rental is tried at most once per teardown; a later teardown retries what stays owed. + const tried = new Map, Owed>(); + for (;;) { + await this.untilAnswered(entry); + const unnamed = await this.nameRental(entry); + if (unnamed !== undefined) return unnamed; + const owed = await this.stopRentals(entry, tried); + if (owed === 'progress') continue; + if (owed.length > 0) return { status: 'pending', owed }; + // The start may have recorded a rental while the stops above were awaited; look again before + // forgetting the lease. + if (entry.unnamed || [...entry.rentals.values()].some((rental) => !rental.stopped)) continue; + return this.finishTeardown(entry); + } + } + + /** + * Stops the entry's rentals in the order they were named and collects every one still rented. A stop + * still in flight is never overtaken: a later rental waits for it. A stop that settled unconfirmed does + * not hold the rest back. Returns `progress` after a confirmed stop, so the caller looks again at a + * start that may have answered or named another rental meanwhile. + */ + private async stopRentals( + entry: LeaseEntry, + tried: Map, Owed>, + ): Promise[] | 'progress'> { + for (const rental of entry.rentals.values()) { + if (rental.stopped) continue; + const earlier = tried.get(rental); + if (earlier !== undefined) { + if (rental.stopping !== undefined) break; + continue; + } + const failure = await this.stop(rental); + if (failure === undefined) return 'progress'; + tried.set(rental, { handle: rental.handle, cause: failure.cause }); + if (rental.stopping !== undefined) break; + } + // Every rental still rented is reported, including one that waits behind a stop still in flight. + return [...entry.rentals.values()] + .filter((rental) => !rental.stopped) + .map( + (rental) => + tried.get(rental) ?? { + handle: rental.handle, + cause: new Error( + 'The stop waits for an earlier stop of this lease that is still in flight.', + ), + }, + ); + } + + /** Asks the provider for a rental this process may hold but cannot name. */ + private async nameRental( + entry: LeaseEntry, + ): Promise | undefined> { + if (!entry.unnamed) return undefined; + try { + const handle = await this.driver.findByLease(entry.lease); + holdRental(entry, handle, this.driver.handleKey(handle)); + entry.unnamed = false; + return undefined; + } catch (error) { + return { status: 'pending', owed: [{ handle: undefined, cause: error }] }; + } + } + + /** Forgets an entry whose rentals are all stopped, unless its start may still rent something. */ + private finishTeardown(entry: LeaseEntry): Ended { + const handle = entry.latest; + if (!entry.answered) { + // The entry stays for the teardown that the start's answer starts. + const cause = new Error('The provider start did not answer within its start timeout.'); + return { status: 'pending', owed: [{ handle, cause }] }; + } + if (this.isCurrent(entry)) this.entries.delete(entry.lease.leaseId); + return handle === undefined + ? { status: 'nothing-rented' } + : { status: 'stopped', handle, session: entry.session }; + } + + /** + * Runs a teardown that looks at what the start just recorded. A teardown already in flight may have + * decided before the record, so another one follows it rather than joining it. + */ + private followUp(entry: LeaseEntry): void { + const current = entry.ending; + if (current === undefined) { + void this.teardown(entry).catch(() => undefined); + return; + } + // One queued teardown sees everything recorded before it runs, so more are never chained. + if (entry.followUpQueued) return; + entry.followUpQueued = true; + void current + .then(async () => { + entry.followUpQueued = false; + return await this.teardown(entry); + }) + .catch(() => undefined); + } + + /** Waits for the start's answer, at most `startTimeoutMs` after it was sent. */ + private async untilAnswered(entry: LeaseEntry): Promise { + if (entry.answered) return; + let timer: ReturnType | undefined; + const expired = new Promise((resolve) => { + timer = setTimeout(resolve, Math.max(0, entry.answerBy - Date.now())); + }); + try { + await Promise.race([entry.answer, expired]); + } finally { + clearTimeout(timer); + } + } + + /** Joins this rental's stop in flight, or sends it, within `stopTimeoutMs`. */ + private async stop(rental: RentalRecord): Promise { + rental.stopping ??= this.stopHandle(rental.handle).then((failure) => { + if (failure === undefined) rental.stopped = true; + rental.stopping = undefined; + return failure; + }); + const stopping = rental.stopping; + const limitMs = this.driver.stopTimeoutMs; + if (limitMs === undefined) return await stopping; + let timer: ReturnType | undefined; + const expired = new Promise((resolve) => { + timer = setTimeout( + () => + resolve({ cause: new Error(`The provider stop did not answer within ${limitMs} ms.`) }), + limitMs, + ); + }); + try { + return await Promise.race([stopping, expired]); + } finally { + clearTimeout(timer); + } + } + + private async stopHandle(handle: Handle): Promise { + try { + await this.driver.stop(handle); + return undefined; + } catch (error) { + return { cause: error }; + } + } + + private pending( + entry: LeaseEntry, + ended: Extract, { status: 'pending' }>, + scope: ProviderLeaseCleanupScope, + ): Error { + return this.driver.cleanupPending( + ended.owed.map((owed) => ({ leaseId: entry.lease.leaseId, ...owed })), + scope, + ); + } +} + +/** Records what the start's answer says it rented, and wakes any teardown waiting on it. */ +function recordAnswer( + entry: LeaseEntry, + answer: StartAnswer, + handleKey: (handle: Handle) => string, +): void { + switch (answer.status) { + case 'ready': + holdRental(entry, answer.handle, handleKey(answer.handle)); + break; + case 'failed': + if (answer.rented !== undefined) holdRental(entry, answer.rented, handleKey(answer.rented)); + break; + case 'rejected': + if (entry.rentals.size === 0) entry.unnamed = true; + break; + } + entry.answered = true; + entry.markAnswered(); +} + +/** Adds a rental the entry owes a stop, unless the same rental is already known. */ +function holdRental( + entry: LeaseEntry, + handle: Handle, + key: string, +): void { + if (!entry.rentals.has(key)) + entry.rentals.set(key, { handle, stopped: false, stopping: undefined }); + entry.latest = handle; +} + +/** The budget a caller named, without the fields it left unset. */ +function callerBudget(options: ProviderLeaseBudget): ProviderLeaseBudget { + return { + ...(options.signal === undefined ? {} : { signal: options.signal }), + ...(options.deadline === undefined ? {} : { deadline: options.deadline }), + }; +} + +/** Whether a caller hung up or ran past its deadline. */ +function budgetSpent(caller: ProviderLeaseBudget, deadline: number): boolean { + return caller.signal?.aborted === true || Date.now() >= deadline; +} + +/** Whether any caller still waits for the start within its own budget. */ +function stillWanted( + entry: LeaseEntry, +): boolean { + if (entry.phase === 'ending') return false; + const now = Date.now(); + for (const waiter of entry.waiters) { + if (!waiter.signal?.aborted && now < waiter.deadline) return true; + } + return false; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index cb9f320f5b..dc7af47ae0 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -878,6 +878,9 @@ importers: '@agent-device/provider-webdriver': specifier: workspace:* version: link:../provider-webdriver + '@agent-device/provision-kit': + specifier: workspace:* + version: link:../provision-kit '@agent-device/test-kit': specifier: workspace:* version: link:../test-kit diff --git a/scripts/layering/package-boundaries.test.ts b/scripts/layering/package-boundaries.test.ts index 84987fd44a..29fd68b23e 100644 --- a/scripts/layering/package-boundaries.test.ts +++ b/scripts/layering/package-boundaries.test.ts @@ -578,6 +578,7 @@ test('the real tree parses, declares, and passes R11', () => { '@agent-device/provision-kit/install-source-config', '@agent-device/provision-kit/install-source-network', '@agent-device/provision-kit/install-source-network-transport', + '@agent-device/provision-kit/provider-lease-table', '@agent-device/provision-kit/toolchain-probe', ]); assert.deepEqual([...provisionKitPackage.workspaceDependencies].sort(), [ diff --git a/website/docs/docs/revyl.md b/website/docs/docs/revyl.md index 89c4ce0d44..b42400f7d0 100644 --- a/website/docs/docs/revyl.md +++ b/website/docs/docs/revyl.md @@ -173,3 +173,6 @@ takes when the daemon loads the plugin from the plugin store. returns nothing. The plugin retries the read; if it still fails, run `snapshot` again. - A device that Revyl released while your session was open reports `provider-session-gone`. Run `close` and `open` again to rent a fresh one. +- An `open` still waiting for its device when its lease is released reports `provider-lease-releasing`. + That lease is ending; run `open` again, which allocates a new lease and rents a fresh device. An + `open` that names no deadline stops waiting for a device after 370 s.