diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index 441db611b4b..c3f824567bd 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -298,7 +298,10 @@ Ranges are both a request and a filter. A subscriber reads only the groups its own ranges cover, resolved again whenever it replaces them, even when another subscriber of the same track asked for more. Widening them, including lowering the start, also serves the cached groups they newly cover, never one already -served. A group they cover below the live feed that the cache lacks (evicted +served. A reader that has read every group its ranges cover waits rather than +ends, since the track goes on: a moq-lite-07 publisher ends a bounded +subscription once it covered it, and the copy stays cached for the same ranges +until an update asks for groups it did not cover, wider or shifted. A group they cover below the live feed that the cache lacks (evicted before the subscriber first looked for it, skipped, or below where the feed starts) is asked of the track's dynamic handler as a fill, which a relay serves by fetching upstream one group at a time, skipping @@ -312,7 +315,8 @@ track, and over moq-lite-07, which carries a relative or time start upstream and reports the live edge and media time, so a copy resolves the ends itself. A Rust moq-transport copy resolves a relative anchor from the Largest Object its live join reports, accepts time anchors its cache can already resolve, and sends group bounds -upstream; it refuses any other time anchor. Older +upstream; it refuses any other time anchor. A JavaScript one refuses every time +anchor. Older moq-lite versions accept absolute anchors but refuse time anchors and relative anchors other than the live and default starts. An untimed track refuses a time anchor. At the live edge, a time end diff --git a/doc/lib/js/net.md b/doc/lib/js/net.md index 038d93ef9f1..307f5bffa71 100644 --- a/doc/lib/js/net.md +++ b/doc/lib/js/net.md @@ -52,6 +52,7 @@ publication.announce(); - **Timing is per track, with no default.** A track that declares a `timescale` carries a `timestamp` on every frame; one without is [untimed](/concept/moq-lite#subscriptions) and its frames carry none. Omitting it means untimed, as in Rust. Nothing stamps a frame for you: the publisher passes the timestamp, and the `Timed` type (`{ value, at }`) carries one into the `@moq/json` producers. - **A subscription's `delay` is a `Track.Delay { min, max }`, as in Rust.** Staleness is `delay.max`: it counts from a group's successor, by media time and by wall clock, whichever reaches it first. `delay.min` is the target latency, carried on moq-lite 07; a peer that cannot carry it leaves it `null` (not stated). Passing the old `maxAge` or `maxDelay` key throws a `TypeError` naming `delay.max`. Publisher retention is the separate `Track.Info.maxAge`. - **Subscription defaults.** `Track.Subscription.default()` returns the default subscription, whose zero `delay.max` skips a superseded group at once; calling `subscribe()` with no options means the same. An omitted `delay` field means the same zero; `max: null` sets no subscriber limit. +- **A subscription asks for `ranges`.** Each [range](/concept/moq-lite#subscriptions) starts and ends at an `absolute` group, one `relative` to the latest, or the group containing a `time` (a `Timestamp`), and a reader reads only its own ranges. A read whose ranges all end finishes once every group in them is handed out, dropped, or skipped, as in Rust. A time anchor needs a timed track and `moq-lite-07-wip` over a session. Passing the old `groups` key throws a `TypeError` naming `ranges`; `withRange` only narrows reads locally. - **Hidden paths** stay out of discovery unless the announce request opts in. See [hidden broadcasts](/concept/moq-lite#hidden-broadcasts). - **Authorization is in band.** On moq-lite 07 (opt in with `versions: ["moq-lite-07-wip", "moq-lite-06"]`, listing every fallback you still want) and on moq-transport draft-18+ with the [MoQ Auth extension](/draft/moq-auth), `auth.grant` watches what the relay lets this side publish and subscribe to, and `auth.add(token)` presents another token without reconnecting. Publishing outside the grant closes the connection with `SessionCode.Unauthorized`, naming the path. On moq-lite, a subscription or track listing the grant stops covering resets with `StreamCode.Unauthorized` and the session stays up. On moq-transport, `auth.add(token)` keeps the token even without the extension, and a broadcast announced with the `Auth.Credential` it returns (`publication.announce({ auth })`) carries it on that announcement's `PUBLISH_NAMESPACE`; announcing again with another renews the request. A `Connection`'s `auth` handle (a getter on a pooled connection, following its URL) outlives its sessions: a token added there is presented on every reconnect and keeps riding the announcements that name it, with no re-add or re-announce. A one-shot session's `auth` ends with it, unless `Connection.connect` was given a `new Auth.Handle()` as `auth`. - **A graceful close waits.** `await connection.close()` withdraws announcements and drains for up to one second. `moq-lite-05` and newer wait for the subscriber to receive finished tails; older Lite versions wait only for transport ACKs. IETF media streams are not drained. `abort()` ends immediately. diff --git a/js/net/MIGRATE.md b/js/net/MIGRATE.md index b9d95ca28e4..8346353ad3b 100644 --- a/js/net/MIGRATE.md +++ b/js/net/MIGRATE.md @@ -11,6 +11,17 @@ Replace `maxDelay: d` with `delay: { max: d }`. `max` keeps `maxDelay`'s meaning `min` is the subscriber's target latency; `null` means not stated, which only a peer on moq-lite before 07 or moq-transport sends. moq-lite-07 SUBSCRIBE and SUBSCRIBE\_UPDATE carry it, and a track's aggregate takes the smallest stated one. `FetchGroupOptions.maxDelay` is unchanged. +### `Subscription.groups` is `ranges` ([#6949](https://github.com/moq-dev/moq/pull/6949)) + +`Track.Subscription` asks for `ranges: Track.Range[]` in place of `groups`, which is now refused at runtime. Each `Track.Range` starts and ends at a `Track.Anchor`: an `absolute` group, one `relative` to the latest, or the group containing a `time`. +Replace `groups: { start: { included: a }, end: { excluded: b } }` with `ranges: [{ start: { absolute: a }, end: { absolute: b } }]`. An inclusive end `b` becomes `absolute: b + 1`, and an excluded start `a` becomes `absolute: a + 1`. +A reader reads only its own ranges, whatever another subscriber widens the request to. `withRange` only narrows them; widen with `update({ ranges })`. + +### Bounded reads end ([#6949](https://github.com/moq-dev/moq/pull/6949)) + +A `recvGroup` or `nextGroup` whose ranges all end now returns `undefined` once every group in them was handed out, dropped, skipped as stale, or will never arrive, as Rust's reads do since [#7545](https://github.com/moq-dev/moq/pull/7545). The track goes on: replacing the ranges with `update` reads on from the same subscriber. A `withRange` cap holding a group back keeps the read open, since the cap may rise. +A loop that read a bounded range until a timeout now ends on its own; one that waited for more groups past the end of its ranges must widen them first. + ### Ordered subscriptions wait for owed groups An ordered reader on a subscribed track now waits at a missing group while the feed may still deliver it, up to `delay.max` by media time or wall clock. Local tracks still skip gaps, and the zero default waits for nothing. diff --git a/js/net/api/index.api.md b/js/net/api/index.api.md index 3644e7fb118..322146e3cc4 100644 --- a/js/net/api/index.api.md +++ b/js/net/api/index.api.md @@ -575,6 +575,19 @@ export interface Timed { } export namespace Track { + export type Anchor = { + absolute: number; + relative?: never; + time?: never; + } | { + absolute?: never; + relative: number; + time?: never; + } | { + absolute?: never; + relative?: never; + time: Time.Timestamp; + }; export type Bound = { excluded?: never; included: number; @@ -665,6 +678,10 @@ export namespace Track { get subscription(): Signals.Getter; writeFrame(frame: Group.Frame): void; } + export interface Range { + end?: Track.Anchor; + start?: Track.Anchor; + } export class Request { accept(info?: Partial): Track.Producer; demand(): Track.Demand; @@ -693,9 +710,9 @@ export namespace Track { }; export interface Subscription { delay?: Track.Delay; - groups?: Track.Groups; order?: Track.Order; priority?: number; + ranges?: Track.Range[]; } export function infoDefaults(info?: Partial): Track.Info; } diff --git a/js/net/bench/replay.ts b/js/net/bench/replay.ts index b79a787699d..065e807625c 100644 --- a/js/net/bench/replay.ts +++ b/js/net/bench/replay.ts @@ -10,7 +10,7 @@ const PAYLOAD = 8; const APPENDS = 64; const REPS = 5; // Every retained group, however old: replay is the work being measured. -const SUBSCRIPTION = { groups: { start: { included: 0 } }, delay: { max: null } }; +const SUBSCRIPTION = { ranges: [{ start: { absolute: 0 } }], delay: { max: null } }; // The arrival order of sequences 0..count-1: ascending, descending, or a seeded shuffle. function arrivals(count: number, order: Order): number[] { diff --git a/js/net/src/broadcast.test.ts b/js/net/src/broadcast.test.ts index e62532d6d9b..4c471872ca3 100644 --- a/js/net/src/broadcast.test.ts +++ b/js/net/src/broadcast.test.ts @@ -319,12 +319,12 @@ test("a request exposes the aggregate subscription options", async () => { consumer.track("video").subscribe({ priority: 3, delay: { max: Milli(100) }, - groups: { start: { included: 10 }, end: { excluded: 20 } }, + ranges: [{ start: { absolute: 10 }, end: { absolute: 20 } }], }); consumer.track("video").subscribe({ priority: 7, delay: { max: Milli(250) }, - groups: { start: { included: 0 }, end: { excluded: 30 } }, + ranges: [{ start: { absolute: 0 }, end: { absolute: 30 } }], }); const request = await pendingRequest(consumer); @@ -332,7 +332,7 @@ test("a request exposes the aggregate subscription options", async () => { priority: 7, delay: { min: Milli.zero, max: Milli(250) }, order: "descending", - groups: { start: { included: 0 }, end: { excluded: 30 } }, + ranges: [{ start: { absolute: 0 }, end: { absolute: 30 } }], }); expect(request?.priority).toBe(7); }); diff --git a/js/net/src/delivered.ts b/js/net/src/delivered.ts index c39c8966267..b7de5241268 100644 --- a/js/net/src/delivered.ts +++ b/js/net/src/delivered.ts @@ -66,6 +66,14 @@ export class Delivered { this.#runs.splice(first, last - first + 1, merged); } + /** A record of what either this or `other` handed out. */ + union(other: Delivered): Delivered { + const union = new Delivered(); + union.#runs = this.#runs.map(([first, last]) => [first, last]); + for (const [first, last] of other.#runs) union.addRun(first, last + 1); + return union; + } + /** The parts of `[start, end)` not handed out, in order. */ gaps(start: number, end: number): { start: number; end: number }[] { const gaps: { start: number; end: number }[] = []; diff --git a/js/net/src/ietf/publisher.test.ts b/js/net/src/ietf/publisher.test.ts index 86fb980a622..9bc2e781775 100644 --- a/js/net/src/ietf/publisher.test.ts +++ b/js/net/src/ietf/publisher.test.ts @@ -2387,7 +2387,7 @@ test("draft-22: an absolute filter trims the range it serves", async () => { // The request forwarded upstream carries the model's exclusive end, one past the // filter's inclusive last group. expect(track.subscription.peek()).toMatchObject({ - groups: { start: { included: 1 }, end: { excluded: 3 } }, + ranges: [{ start: { absolute: 1 }, end: { absolute: 3 } }], }); const first = await nextUni(fx.uni); @@ -3441,7 +3441,7 @@ test("REQUEST_UPDATE applies priority and preserves it when omitted", async () = expect(track.subscription.peek()?.priority).toBe(117); // Only the priority changes; retention and the group range survive. expect(track.subscription.peek()?.delay?.max).toBe(before?.delay?.max); - expect(track.subscription.peek()?.groups).toEqual(before?.groups); + expect(track.subscription.peek()?.ranges).toEqual(before?.ranges); await client.writer.write(new Uint8Array([0x02, 0, 2, 4, 0])); expect(await client.reader.u53()).toBe(RequestOk.id); await RequestOk.decode(client.reader, fx.version); @@ -3945,7 +3945,7 @@ test("IETF draft-18: range updates preserve the shared subscription", async () = await new SubscribeUpdate({ requestId: id, parameters }).encode(client.writer, fx.version); expect(await client.reader.u53()).toBe(RequestOk.id); await RequestOk.decode(client.reader, fx.version); - expect(track.subscription.peek()?.groups).toEqual({ start: { included: 6 }, end: { excluded: 8 } }); + expect(track.subscription.peek()?.ranges).toEqual([{ start: { absolute: 6 }, end: { absolute: 8 } }]); } } finally { client.close(); diff --git a/js/net/src/ietf/publisher.ts b/js/net/src/ietf/publisher.ts index fe3482f8413..1e8ad0f2b1c 100644 --- a/js/net/src/ietf/publisher.ts +++ b/js/net/src/ietf/publisher.ts @@ -29,7 +29,7 @@ import { sendPayload } from "../payload.ts"; import { Priority, sendOrder } from "../priority.ts"; import { type Stream, Writer } from "../stream.ts"; import { Timescale } from "../time.ts"; -import type { Groups, Info, Position as TrackPosition, Subscriber as TrackSubscriber } from "../track.ts"; +import type { Groups, Info, Range, Position as TrackPosition, Subscriber as TrackSubscriber } from "../track.ts"; import { untilAborted } from "../util/abort.ts"; import { nextChange } from "../util/change.ts"; import { Deadline, REQUEST_TIMEOUT_MS } from "../util/deadline.ts"; @@ -497,14 +497,17 @@ export class Publisher { // A range that resolves without the live edge has a fixed start, which is no join: // bound the subscription before subscribing so the refresh ratio never judges it. const fixed = subscribeRange(msg, {}, version); - return source.track(msg.trackName).subscribe({ - priority: fromWire(msg.subscriberPriority), - order: msg.groupOrder, - // An absent budget preserves everything retained by the producer. - // moq-transport has no field for a minimum. - delay: { max: Milli(msg.maxAge ?? Varint.MAX_U53), min: null }, - groups: fixed.start ? rangeGroups(fixed) : undefined, - }); + // Parked past a bounded window's end: the peer reads this stream's end as the track's. + return hooks.parked( + source.track(msg.trackName).subscribe({ + priority: fromWire(msg.subscriberPriority), + order: msg.groupOrder, + // An absent budget preserves everything retained by the producer. + // moq-transport has no field for a minimum. + delay: { max: Milli(msg.maxAge ?? Varint.MAX_U53), min: null }, + ranges: fixed.start ? serveRanges(fixed) : undefined, + }), + ); }; let track = subscribe(broadcast); @@ -710,7 +713,7 @@ export class Publisher { order, // moq-transport has no field for a minimum. delay: { max: Milli(msg.maxAge ?? Varint.MAX_U53), min: null }, - groups: rangeGroups(range), + ranges: serveRanges(range), }); const startGroup = range.start ? Number(range.start.group) : hooks.latestSequence(track); if (startGroup !== undefined) track.withRange({ ...rangeGroups(range), start: { included: startGroup } }); @@ -864,7 +867,7 @@ export class Publisher { range.start = tail.start; range.end = tail.end; bounds.set(requested, true); - track.update({ ...track.subscription.peek(), groups: rangeGroups(range) }); + track.update({ ...track.subscription.peek(), ranges: serveRanges(range) }); // Below its floor the live cursor already passed every group or never owned it, // so lowering it for a fill-only update would hand those groups out a second time. let start = range.start ? Number(range.start.group) : 0; @@ -2849,6 +2852,14 @@ async function servingGroup(track: TrackSubscriber): Promise { - if (subscription?.groups?.start?.included === 1) resolveProducer?.(producer); + if (subscription?.ranges?.[0]?.start?.absolute === 1) resolveProducer?.(producer); }; check(); producer.subscription.subscribe(check); @@ -208,7 +208,7 @@ test("integration: lite subscription options and updates reach the publisher", a const subscriber = remote.track("video").subscribe({ priority: 3, delay: { max: Milli(250) }, - groups: { start: { included: 1 }, end: { excluded: 9 } }, + ranges: [{ start: { absolute: 1 }, end: { absolute: 9 } }], }); const producer = await accepted; expect(producer.subscription.peek()).toEqual({ @@ -216,20 +216,20 @@ test("integration: lite subscription options and updates reach the publisher", a // The negotiated lite-06 cannot state a minimum. delay: { min: null, max: Milli(250) }, order: "descending", - groups: { start: { included: 1 }, end: { excluded: 9 } }, + ranges: [{ start: { absolute: 1 }, end: { absolute: 9 } }], }); const updated = producer.subscription.changed(); subscriber.update({ priority: 8, delay: { max: Milli(500) }, - groups: { start: { included: 2 }, end: { excluded: 12 } }, + ranges: [{ start: { absolute: 2 }, end: { absolute: 12 } }], }); expect(await updated).toEqual({ priority: 8, delay: { min: null, max: Milli(500) }, order: "descending", - groups: { start: { included: 2 }, end: { excluded: 12 } }, + ranges: [{ start: { absolute: 2 }, end: { absolute: 12 } }], }); subscriber.close(); @@ -295,7 +295,7 @@ test("integration: lite carries a fractional delay as a whole millisecond", asyn const producer = request.accept({ timescale: Timescale.MILLI }); // The TRACK lookup opens the request and the SUBSCRIBE joins it, so its options land after. const check = (subscription = producer.subscription.peek()) => { - if (subscription?.groups?.start?.included === 1) resolveProducer?.(producer); + if (subscription?.ranges?.[0]?.start?.absolute === 1) resolveProducer?.(producer); }; check(); producer.subscription.subscribe(check); @@ -307,7 +307,7 @@ test("integration: lite carries a fractional delay as a whole millisecond", asyn // nothing resubscribes. The publisher must see the budget rounded up instead. const subscriber = remote .track("video") - .subscribe({ delay: { max: Milli(38.75) }, groups: { start: { included: 1 } } }); + .subscribe({ delay: { max: Milli(38.75) }, ranges: [{ start: { absolute: 1 } }] }); // A failed subscribe never reaches the publisher, so race its closure to report the // encode error rather than block until the suite times out. @@ -319,7 +319,7 @@ test("integration: lite carries a fractional delay as a whole millisecond", asyn expect(producer.subscription.peek()?.delay?.max).toBe(Milli(39)); const updated = producer.subscription.changed(); - subscriber.update({ delay: { max: Milli(500.25) }, groups: { start: { included: 1 } } }); + subscriber.update({ delay: { max: Milli(500.25) }, ranges: [{ start: { absolute: 1 } }] }); expect((await Promise.race([updated, failed]))?.delay?.max).toBe(Milli(501)); subscriber.close(); @@ -357,27 +357,35 @@ test("integration: lite applies initial and updated group bounds", async () => { .track("video") .subscribe({ delay: { max: Milli(REPLAY_LATENCY_MS) }, - groups: { start: { included: INITIAL_START_GROUP }, end: { excluded: INITIAL_END_GROUP } }, + ranges: [{ start: { absolute: INITIAL_START_GROUP }, end: { absolute: INITIAL_END_GROUP } }], }) .ordered(); try { expect((await subscriber.nextGroup())?.sequence).toBe(INITIAL_START_GROUP); expect((await subscriber.nextGroup())?.sequence).toBe(INITIAL_END_GROUP - 1); - const pending = subscriber.nextGroup(); - expect(await Promise.race([pending, sleepMs(PENDING_ASSERT_MS).then(() => "pending")])).toBe("pending"); + // A bounded read ends once its window is read, as in Rust. + expect(await subscriber.nextGroup()).toBeUndefined(); + // Replacing the ranges reads on from the same reader, the update going upstream. subscriber.update({ delay: { max: Milli(REPLAY_LATENCY_MS) }, - groups: { start: { included: UPDATED_GROUP }, end: { excluded: UPDATED_END_GROUP } }, + ranges: [{ start: { absolute: UPDATED_GROUP }, end: { absolute: UPDATED_END_GROUP } }], }); - expect( - (await withTimeout(pending, Nano.fromMilli(Milli(UPDATE_TIMEOUT_MS)), "updated group bound timed out")) - ?.sequence, - ).toBe(UPDATED_GROUP); + const updated = (pending: Promise) => + withTimeout(pending, Nano.fromMilli(Milli(UPDATE_TIMEOUT_MS)), "updated group bound timed out"); + expect(((await updated(subscriber.nextGroup())) as { sequence: number } | undefined)?.sequence).toBe( + UPDATED_GROUP, + ); + expect(await updated(subscriber.nextGroup())).toBeUndefined(); - const capped = subscriber.nextGroup(); - expect(await Promise.race([capped, sleepMs(PENDING_ASSERT_MS).then(() => "pending")])).toBe("pending"); + // An open range parks at the live edge instead. + subscriber.update({ + delay: { max: Milli(REPLAY_LATENCY_MS) }, + ranges: [{ start: { absolute: GROUP_COUNT } }], + }); + const live = subscriber.nextGroup(); + expect(await Promise.race([live, sleepMs(PENDING_ASSERT_MS).then(() => "pending")])).toBe("pending"); } finally { subscriber.close(); remote.close(); @@ -406,31 +414,42 @@ test("integration: lite refuses an empty requested range on open and on update", const remote = wireOf(client).consume(Path.from("test")); const video = remote.track("video"); try { - // Bounds that meet cannot go on the wire: the nearest encoding inverts the range. + // Bounds that meet cannot go on the wire: the nearest encoding inverts the range. The read + // of an empty window ends at once, and the copy is refused. const empty = video.subscribe({ delay: { max: Milli(REPLAY_LATENCY_MS) }, - groups: { start: { included: 2 }, end: { excluded: 2 } }, + ranges: [{ start: { absolute: 2 }, end: { absolute: 2 } }], }); - await expect( - withTimeout(empty.recvGroup(), Nano.fromMilli(Milli(TIMEOUT_MS)), "empty open never settled"), - ).rejects.toThrow("empty subscription range cannot be encoded"); + expect(await empty.recvGroup()).toBeUndefined(); + expect( + await withTimeout( + Promise.resolve(empty.closed), + Nano.fromMilli(Milli(TIMEOUT_MS)), + "empty open never settled", + ), + ).toHaveProperty("message", "empty subscription range cannot be encoded"); empty.close(); // A live subscription whose demand later collapses to nothing fails the same way. const live = video .subscribe({ delay: { max: Milli(REPLAY_LATENCY_MS) }, - groups: { start: { included: 1 }, end: { excluded: 2 } }, + ranges: [{ start: { absolute: 1 }, end: { absolute: 2 } }], }) .ordered(); expect((await live.nextGroup())?.sequence).toBe(1); live.update({ delay: { max: Milli(REPLAY_LATENCY_MS) }, - groups: { start: { included: 3 }, end: { excluded: 3 } }, + ranges: [{ start: { absolute: 3 }, end: { absolute: 3 } }], }); - await expect( - withTimeout(live.nextGroup(), Nano.fromMilli(Milli(TIMEOUT_MS)), "empty update never settled"), - ).rejects.toThrow("empty subscription range cannot be encoded"); + expect(await live.nextGroup()).toBeUndefined(); + expect( + await withTimeout( + Promise.resolve(live.closed), + Nano.fromMilli(Milli(TIMEOUT_MS)), + "empty update never settled", + ), + ).toHaveProperty("message", "empty subscription range cannot be encoded"); live.close(); } finally { remote.close(); @@ -440,6 +459,145 @@ test("integration: lite refuses an empty requested range on open and on update", } }); +// Lite-07 carries every range, so a sparse request counting back from the latest group delivers +// just its groups, and the publisher ends it once they are accounted for. +test("integration: lite-07 serves sparse and relative ranges", async () => { + const pair = createMockTransportPair(Lite.ALPN_07_WIP); + const origin = new OriginProducer(); + const [client, server] = await Promise.all([ + connect(url, { transport: pair.client }), + accept(pair.server, url, { publish: origin.consume() }), + ]); + + const broadcast = publish(origin, Path.from("test")); + const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); + for (let sequence = 0; sequence < 10; sequence++) producer.appendGroup().finish(); + + const remote = wireOf(client).consume(Path.from("test")); + try { + const reader = remote + .track("video") + .subscribe({ + delay: { max: null }, + order: "ascending", + ranges: [ + { start: { absolute: 1 }, end: { absolute: 3 } }, + { start: { relative: 2 }, end: { absolute: 9 } }, + ], + }) + .ordered(); + // Covering the ranges ends the subscription upstream, and the read with its window. + const got: number[] = []; + for (let i = 0; i < 4; i++) got.push((await reader.nextGroup())?.sequence ?? -1); + expect(got).toEqual([1, 2, 7, 8]); + expect(await reader.nextGroup()).toBeUndefined(); + reader.close(); + } finally { + remote.close(); + broadcast.close(); + client.abort(); + server.abort(); + } +}); + +// A lite-07 publisher ends a bounded subscription once it covered the range, releasing its +// demand, and the subscriber keeps the copy: the read ends with its window, the same range reads +// again from the cache, and a wider one subscribes again for the rest. Mirrors the Rust +// `a_covered_bounded_subscription_ends_on_lite07`; reads end as Rust's do since moq-dev/moq#7545. +test("integration: lite-07 keeps a covered bounded subscription's copy", async () => { + const pair = createMockTransportPair(Lite.ALPN_07_WIP); + const origin = new OriginProducer(); + const [client, server] = await Promise.all([ + connect(url, { transport: pair.client }), + accept(pair.server, url, { publish: origin.consume() }), + ]); + + const broadcast = publish(origin, Path.from("test")); + const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); + for (const sequence of [0, 1, 2, 4, 5]) producer.createGroup(sequence).finish(); + + const remote = wireOf(client).consume(Path.from("test")); + const bounded = (end: number) => + remote.track("video").subscribe({ + delay: { max: Milli(60_000) }, + order: "ascending", + ranges: [{ start: { absolute: 1 }, end: { absolute: end } }], + }); + // Arrival reads, as the Rust test's: an ordered one would wait out the missing group 3. + const read = async (reader: ReturnType, count: number) => { + const got: number[] = []; + for (let i = 0; i < count; i++) got.push((await reader.recvGroup())?.sequence ?? -1); + return got; + }; + try { + const reader = bounded(3); + expect(await read(reader, 2)).toEqual([1, 2]); + // The publisher's demand ends with the covered subscription, and the read with its window. + while (producer.subscription.peek() !== undefined) await producer.subscription.changed(); + expect(await reader.recvGroup()).toBeUndefined(); + + const again = bounded(3); + expect(await read(again, 2)).toEqual([1, 2]); + expect(producer.subscription.peek()).toBeUndefined(); + + const wider = bounded(6); + expect(await read(wider, 4)).toEqual([1, 2, 4, 5]); + reader.close(); + again.close(); + wider.close(); + } finally { + remote.close(); + broadcast.close(); + client.abort(); + server.abort(); + } +}); + +// A time range goes upstream in the track's timescale, the publisher resolves its start from +// where its groups start, and the copy's reader resolves the end from the groups it receives. +test("integration: lite-07 serves a time range", async () => { + const pair = createMockTransportPair(Lite.ALPN_07_WIP); + const origin = new OriginProducer(); + const [client, server] = await Promise.all([ + connect(url, { transport: pair.client }), + accept(pair.server, url, { publish: origin.consume() }), + ]); + + const broadcast = publish(origin, Path.from("test")); + const producer = broadcast.createTrack("video", { timescale: Timescale.MICRO }); + for (let sequence = 0; sequence < 10; sequence++) { + const group = producer.appendGroup(); + group.writeFrame(textFrame(`${sequence}`, Timestamp.fromMillis(sequence * 1000))); + group.finish(); + } + + const remote = wireOf(client).consume(Path.from("test")); + try { + const reader = remote + .track("video") + .subscribe({ + delay: { max: null }, + order: "ascending", + ranges: [{ start: { time: Timestamp.fromMillis(2500) }, end: { time: Timestamp.fromMillis(5000) } }], + }) + .ordered(); + const got: number[] = []; + for (let i = 0; i < 4; i++) got.push((await reader.nextGroup())?.sequence ?? -1); + expect(got).toEqual([2, 3, 4, 5]); + reader.close(); + + // An untimed track refuses it. + broadcast.createTrack("untimed", {}); + const untimed = remote.track("untimed").subscribe({ ranges: [{ start: { time: Timestamp.fromMillis(0) } }] }); + await expect(untimed.recvGroup()).rejects.toThrow(TimestampMismatch); + } finally { + remote.close(); + broadcast.close(); + client.abort(); + server.abort(); + } +}); + test("integration: lite draft-06", async () => { // Exercises announce ids: every active assigns an ordinal on the wire. await runPublishSubscribeFlow(Lite.ALPN_06); diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 44563205c21..c2b21129ee1 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -19,6 +19,7 @@ import * as Path from "./path.ts"; import type { Timestamp } from "./time.ts"; import type { Delay, Demand, Groups, Info, Position, Producer, Request, Subscriber, Subscription } from "./track.ts"; import * as Varint from "./varint.ts"; +import type { Window } from "./window.ts"; import type { Advertised, Advertisements } from "./wire.ts"; /** What a null {@link Delay.max} goes out as: the largest value every peer can read, as Rust sends for `None`. */ @@ -347,6 +348,12 @@ export interface ResumeSource { * copy that still holds a group for a later group read is kept. */ tryDatagram(delivered: Delivered): RecvDatagram; + /** + * Whether the reader's bounded `window` has nothing left to deliver, judged by the serving + * copy against what it and the logical reader (`delivered`) handed out. A copy that may still + * deliver keeps it open. Mirrors the Rust `Reader::exhausted`. + */ + exhausted(delivered: Delivered, window: Window): boolean; /** Wake once a read may go differently; fires at most once. */ changed(fn: () => void): Dispose; /** Forget the gaps in `delivered` below each group past the budget, judged on the newest copy. */ @@ -357,6 +364,15 @@ export interface ResumeSource { start(): number | "pending" | undefined; /** The logical track's declared exclusive end, as the first copy to know it declared it. */ end(): Position | undefined; + /** Where each range starts on the serving copy once delivery starts at `from`, undefined without one. */ + starts(from: number): (number | undefined)[] | undefined; + /** The runs of `[start, end)` the serving copy reads, undefined without one. */ + runs(start: number, end: number): Run[] | undefined; + /** + * The exclusive end past every group the serving copy reads: undefined while one of its + * ranges stays open or without a serving copy. + */ + servingEnd(): number | undefined; /** The error the logical track's end carries, if it does not end cleanly. */ endError(): Error | undefined; /** Resolve with the logical track's exclusive end once known. */ @@ -469,6 +485,12 @@ export const hooks: { * waits on its START, as the Rust `request_start`. */ startTrack: (producer: Producer, served: { start?: number | null; requested?: number }) => void; + /** + * Mark a track as a copy received over a wire without ranges, which cannot carry a relative + * anchor other than the live start, so its readers refuse one rather than guess from a + * partial copy. Mirrors the Rust `Holdings::drops_anchors`; assigned by `track.ts`. + */ + dropAnchors: (producer: Producer) => void; /** Mint a track {@link Request}; assigned by `track.ts`. */ makeRequest: (options: TrackRequestOptions) => Request; /** @@ -519,6 +541,21 @@ export const hooks: { * kept for it, so a resumed reader takes groups the plain way and drops from here. */ tryDrop: (subscriber: Subscriber) => (Recv & { kind: "drop" }) | undefined; + /** + * The runs of `[start, end)` a subscriber reads: inside its ranges and its local bounds. + * Assigned by `track.ts`. + */ + servingRuns: (subscriber: Subscriber, start: number, end: number) => Run[]; + /** + * The exclusive end past every group a subscriber reads, or undefined while one of its ranges + * stays open. Assigned by `track.ts`. + */ + servingEnd: (subscriber: Subscriber) => number | undefined; + /** + * Where each of a subscriber's ranges starts once delivery starts at `from`, undefined for one + * with no group there: what lite-07 SUBSCRIBE_START answers. Assigned by `track.ts`. + */ + servingStarts: (subscriber: Subscriber, from: number) => (number | undefined)[]; /** The lowest buffered candidate, which an ordered take may wait below; `floor` passes sequences taken on other copies. */ peekGroup: (subscriber: Subscriber, ordered: boolean, floor?: number) => number | undefined; /** Take the next group in sequence order without waiting, as {@link tryRecvGroup} does in arrival order. */ @@ -534,6 +571,17 @@ export const hooks: { * cursor, and no group in `[start, end)` that `delivered` lacks. */ spent: (subscriber: Subscriber, start: number, end: number | undefined, delivered: Delivered) => boolean; + /** + * Mark a wire publisher's reader to park past the end of a bounded window rather than end the + * read there, since its peer reads the end of a stream as the track's. Mirrors the Rust + * crate-private `track::Subscriber::parked`. Returns the subscriber. + */ + parked: (subscriber: Subscriber) => Subscriber; + /** + * Whether a route copy's bounded window has nothing left for the logical reader that handed + * out `delivered`, counting what the copy handed out itself. Mirrors the Rust `exhausted_for`. + */ + exhaustedFor: (subscriber: Subscriber, delivered: Delivered) => boolean; /** How many runs of handed-out groups a subscriber remembers: one per gap, plus the floor. Tests only. */ deliveredRuns: (subscriber: Subscriber) => number; /** Whether the subscriber already handed out group `sequence`, or no longer expects it. */ @@ -694,6 +742,18 @@ export const hooks: { startTrack: () => { throw new Error("track.ts not loaded"); }, + dropAnchors: () => { + throw new Error("track.ts not loaded"); + }, + servingRuns: () => { + throw new Error("track.ts not loaded"); + }, + servingEnd: () => { + throw new Error("track.ts not loaded"); + }, + servingStarts: () => { + throw new Error("track.ts not loaded"); + }, makeRequest: () => { throw new Error("track.ts not loaded"); }, @@ -766,6 +826,12 @@ export const hooks: { spent: () => { throw new Error("track.ts not loaded"); }, + parked: () => { + throw new Error("track.ts not loaded"); + }, + exhaustedFor: () => { + throw new Error("track.ts not loaded"); + }, staleAt: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/ledger.ts b/js/net/src/ledger.ts index 3590f22c318..168c3597ac5 100644 --- a/js/net/src/ledger.ts +++ b/js/net/src/ledger.ts @@ -15,6 +15,12 @@ const BTree = (btree.default as unknown as { default?: typeof btree.default }).d */ export const GRACE = Milli(1000); +/** + * How long idle cached content is kept, matching Rust `cache::DEFAULT_EXPIRY`: a group older + * than this is evicted, so nothing waits for one longer. + */ +export const CACHE_EXPIRY = Milli(30_000); + /** The grace for a subscription with `delay.max`: how long it was willing to wait for a late group anyway, or {@link GRACE} without one. */ export function grace(maxDelay: Milli): Milli { return maxDelay > 0 ? maxDelay : GRACE; @@ -113,6 +119,12 @@ export class Ledger { return this.#runs.minKey(); } + /** The exclusive end of the run accounting for `sequence`, undefined when none does. */ + runEnd(sequence: number): number | undefined { + const run = this.#runs.getPairOrNextLower(sequence)?.[1]; + return run !== undefined && sequence < run.end ? run.end : undefined; + } + /** Whether every sequence in `[start, end)` is accounted for. */ covers(start: number, end: number): boolean { if (start >= end) return true; diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 23426bf536d..ed80f6d2fec 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -697,6 +697,8 @@ async function servedSubscription( // Answer the subscription with a front's broadcast, as an application forwarding a // routed one does, so the track reads through the front's serving route. routed?: boolean; + // Groups the track holds before the SUBSCRIBE arrives, stamped by sequence. + prefill?: number[]; // Leave the track unresolved until `accept`, as an on-demand track the SUBSCRIBE beats. pending?: boolean; } = {}, @@ -725,6 +727,12 @@ async function servedSubscription( } const info = { timescale: Timescale.MILLI, maxAge: options.maxAge }; let track = options.pending ? undefined : broadcast.createTrack("video", info); + for (const sequence of options.prefill ?? []) { + if (!track) throw new Error("a pending track holds nothing to prefill"); + const group = track.createGroup(sequence); + for (const frame of frames) group.writeFrame(textFrame(frame, Timestamp.fromMillis(sequence))); + group.finish(); + } // Pulled before the SUBSCRIBE arrives: a broadcast nobody pulls refuses an unknown track. const request = options.pending ? wireOf(broadcast).requested() : undefined; @@ -853,28 +861,70 @@ async function servedSubscription( }; } -// Refuse unsupported anchors on either control message before promising a delivery start. -for (const kind of [2, 3]) { - for (const end of [false, true]) { - for (const update of [false, true]) { - test(`lite-07 refuses anchor ${kind} at ${end ? "end" : "start"} in ${update ? "UPDATE" : "SUBSCRIBE"}`, async () => { - const version = Version.DRAFT_07; - const anchor = end ? [0, kind, 0] : [kind, 0, 0]; - // Priority, order, max delay, an unstated min delay, one range, then the wanted group. - const bytes = new Uint8Array([9, 128, 0, 0, 0, 1, ...anchor, 0]); - const decoded = await SubscribeUpdate.decode(new Reader(undefined, bytes, version), version); - const sub = await servedSubscription({ version, ranges: update ? undefined : decoded.ranges }); - try { - if (update) await sub.client.writer.write(bytes); - await sub.serving; - await expect(decodeSubscribeResponse(sub.client.reader, version)).rejects.toThrow(); - } finally { - await sub.close(); - } - }); - } +// SUBSCRIBE_START tells the subscriber where each range's start landed, which a relative or time +// start leaves it no way to work out from the groups it receives. Mirrors the Rust +// `lite07_start_names_each_resolved_range_start`. +test("lite-07 SUBSCRIBE_START names each resolved range start", async () => { + const version = Version.DRAFT_07; + const sub = await servedSubscription({ + version, + prefill: [0, 1, 2, 3, 4], + maxAge: Milli(3_600_000), + ranges: [ + { start: { relative: 1 } }, + // Group 1 starts at 1 ms. + { start: { time: 1 }, end: { absolute: 2 } }, + // Empty, so it starts nowhere. + { start: { absolute: 3 }, end: { absolute: 3 } }, + // Empty only once its start resolves to group 4. + { start: { relative: 0 }, end: { absolute: 2 } }, + ], + }); + try { + const resp = await decodeSubscribeResponse(sub.client.reader, version); + if (!("start" in resp)) throw new Error("expected SUBSCRIBE_START"); + expect(resp.start.starts).toEqual([3, 1, undefined, undefined]); + } finally { + await sub.close(); } -} +}); + +// A bounded window that resolves empty has no group to answer it, so the publisher answers it +// at once and ends the subscription as covered, as a local read of it ends at once. +test("lite-07 answers and ends a bounded window that resolves empty", async () => { + const version = Version.DRAFT_07; + const sub = await servedSubscription({ + version, + prefill: [0, 1, 2, 3, 4, 5], + maxAge: Milli(3_600_000), + ranges: [{ start: { relative: 0 }, end: { absolute: 3 } }], + }); + try { + const resp = await decodeSubscribeResponse(sub.client.reader, version); + if (!("start" in resp)) throw new Error("expected SUBSCRIBE_START"); + expect(resp.start.starts).toEqual([undefined]); + expect(await decodeSubscribeResponseMaybe(sub.client.reader, version)).toBeUndefined(); + } finally { + await sub.close(); + } +}); + +// A time anchor waits for the track's timescale. On an on-demand track that lookup is the +// subscription's own request, so the track it answers is the one served. +test("lite-07 serves a time range from the on-demand track its lookup resolved", async () => { + const sub = await servedSubscription({ + version: Version.DRAFT_07, + pending: true, + ranges: [{ start: { time: 0 } }], + }); + try { + await sub.accept(); + sub.serve(0); + expect(await sub.servedSequence()).toBe(0); + } finally { + await sub.close(); + } +}); // A relay can ingest back-to-back groups micro-reordered (the upstream leg sends // newest-first). The older group is cached and in demand, so serving must still @@ -1042,8 +1092,8 @@ test.each([Version.DRAFT_06, Version.DRAFT_07])("lite %s: datagrams honor group // The publisher applies an update's datagram bounds before forwarding its group range, and // each update here moves the range's end. const applied = async (end: number) => { - let groups = sub.track.subscription.peek()?.groups; - while (groups?.end?.excluded !== end) groups = (await sub.track.subscription.changed())?.groups; + let ranges = sub.track.subscription.peek()?.ranges; + while (ranges?.[0]?.end?.absolute !== end) ranges = (await sub.track.subscription.changed())?.ranges; }; try { send([0, 2, 1]); @@ -1418,7 +1468,7 @@ test("lite draft-05: a group popped before a cap update is still served", async expect(await sub.servedSequence()).toBe(1); while (ranges.mock.calls.length === 0) await flush(); // Dropping the named floor re-pins to the latest group, as an unfloored SUBSCRIBE does. - expect(lastGroups(ranges)).toEqual({ start: { included: 1 }, end: { included: 0 } }); + expect(lastGroups(ranges)).toEqual({ start: { included: 1 } }); } finally { ranges.mockRestore(); sub.release(); @@ -1596,7 +1646,7 @@ test("lite draft-06: a burst of updates coalesces before the next group pop", as expect(await sub.servedGroup()).toEqual({ sequence: 1, frameStart: 0, payloads: ["a"] }); expect(ranges).toHaveBeenCalledTimes(1); // Draft-06 encodes a floor of group 0 as no floor, which assigns 0. - expect(lastGroups(ranges)).toEqual({ start: { included: 0 }, end: { included: 1 } }); + expect(lastGroups(ranges)).toEqual({ start: { included: 0 } }); } finally { ranges.mockRestore(); } @@ -1670,14 +1720,14 @@ test("lite draft-06: scheduling updates apply while SUBSCRIBE_START is blocked", // Lite-06 cannot state a minimum. delay: { min: null, max: TEST_MAX_DELAY_MS }, order: "descending", - groups: { start: undefined, end: { excluded: 6 } }, + ranges: [{ end: { absolute: 6 } }], }); expect(ranges).not.toHaveBeenCalled(); sub.release(); expect(await sub.servedSequence()).toBe(0); while (ranges.mock.calls.length === 0) await flush(); - expect(lastGroups(ranges)).toEqual({ start: { included: 0 }, end: { included: 5 } }); + expect(lastGroups(ranges)).toEqual({ start: { included: 0 } }); } finally { ranges.mockRestore(); sub.release(); diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 56ba1626d6f..bf59ef19506 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -40,8 +40,8 @@ import { Group as GroupMessage } from "./group.ts"; import { Probe } from "./probe.ts"; import { encodeSubscribeResponse, - exclusiveGroupEnd, - type RangeGroups, + hasWireTime, + requestedRanges, type Subscribe, SubscribeDrop, SubscribeEnd, @@ -61,6 +61,7 @@ import { hasInstance, hasLargest, hasProbeRtt, + hasRanges, hasRouteCost, hasSynced, hasTrackStream, @@ -199,7 +200,7 @@ type FrameBounds = { * The frames of `sequence` a subscription asked for, as a start index and an inclusive end. * * The frame bounds qualify the start and end group only; every other group is served whole. - * Which groups are served at all is the subscriber's read cursor (`withRange`), + * Which groups are served at all is the serving subscriber's ranges and floor, * applied when a group is popped rather than re-checked here. * * The serving loop calls this synchronously after the pop, before any SUBSCRIBE_UPDATE can @@ -356,15 +357,10 @@ class SubscriptionControls { } } -// Where each range starts once delivery starts at `from`, `undefined` for one with no group -// there. Before lite-07 there is one range, which starts there. -function rangeStarts(ranges: (RangeGroups | undefined)[] | undefined, from: number): (number | undefined)[] { - if (!ranges) return [from]; - return ranges.map((range) => { - if (!range) return undefined; - const start = Math.max(range.start, from); - return range.end === undefined || range.end > start ? start : undefined; - }); +// Where each range of `track` starts once delivery starts at `from`, undefined for one with no +// group there. Before lite-07 there is one range, which starts there. +function rangeStarts(version: Version, track: track.Subscriber, from: number): (number | undefined)[] { + return hasRanges(version) ? hooks.servingStarts(track, from) : [from]; } // The later of two locations, either of which may be unknown. @@ -400,11 +396,11 @@ function waitForSubscription( * * A version without the field decodes as `0`, which is indistinguishable from a peer * genuinely asking for the live edge. Serving that as real time would discard backlog - * a legacy subscriber never declined, so fall back to a window wide enough not to drop + * a legacy subscriber never declined, so fall back to the publisher's max age alone * and leave enforcement to the receiver, as the IETF path does for the same reason. */ -function servingMaxDelay(version: Version, requested: number | undefined): number { - return carriesMaxDelay(version) ? (requested ?? 0) : Number.MAX_SAFE_INTEGER; +function servingMaxDelay(version: Version, requested: number): Milli | null { + return carriesMaxDelay(version) ? Milli(requested) : null; } /** Whether this version's SUBSCRIBE carries Subscriber Max Age at all. */ @@ -872,11 +868,6 @@ export class Publisher { return; } - if (msg.ranges?.some((range) => range?.unresolved)) { - stream.abort(new Error("relative and time subscription anchors are not supported")); - return; - } - // The requester leaving while the broadcast and track resolve ends the request, as // Rust polls the requester throughout, so a cancelled subscription gives back its // charge instead of holding it until the track is answered, which may be never. A @@ -904,26 +895,48 @@ export class Publisher { return; } - const endGroup = exclusiveGroupEnd(msg.endGroup); - const track = front.track(msg.track).subscribe({ + const consumer = front.track(msg.track); + const subscription = (ranges: track.Range[]): track.Subscription => ({ priority: msg.priority, order: msg.order, delay: { - max: Milli(servingMaxDelay(this.version, msg.maxDelay)), + max: servingMaxDelay(this.version, msg.maxDelay), min: msg.minDelay === undefined ? null : Milli(msg.minDelay), }, - groups: { - start: msg.startGroup === undefined ? undefined : { included: msg.startGroup }, - end: endGroup === undefined ? undefined : { excluded: endGroup }, - }, + ranges, }); + // A time anchor counts in the timescale this peer was told, so it waits for the info. + // Meanwhile the subscription asks for nothing but holds the track it resolves, so an + // on-demand track's request answers it, rather than one released once its info lands. + const timed = hasWireTime(msg.ranges); + let ranges: track.Range[] = []; + try { + if (!timed) ranges = requestedRanges(msg, undefined); + } catch (err: unknown) { + stream.writer.reset(error(err)); + return; + } + // Parked past a bounded window's end: the peer reads this stream's end as the track's. + const track = hooks.parked(consumer.subscribe(subscription(ranges))); serving(track); + if (timed) { + try { + const info = await untilAborted(track.info(), left.signal); + track.update(subscription(requestedRanges(msg, info.timescale ?? undefined))); + } catch (err: unknown) { + track.close(); + stream.writer.reset(error(err)); + return; + } + } + if (watch.revoked()) { + track.close(); + return; + } const latest = hooks.latestSequence(track); let floor = cursorFloor(this.version, msg.startGroup, latest); - track.withRange({ - start: { included: floor }, - end: endGroup === undefined ? undefined : { excluded: endGroup }, - }); + // The ranges cap what is served; the floor only pins where an older wire starts. + track.withRange({ start: { included: floor } }); // The best-effort datagram loop, started once serving begins. It parks when the // track finishes (recvDatagram returns undefined), so #runTrack alone ends the @@ -960,10 +973,7 @@ export class Publisher { const resolved = cursorFloor(this.version, msg.startGroup, hooks.latestSequence(track)); if (latest === undefined && resolved !== floor) { floor = resolved; - track.withRange({ - start: { included: floor }, - end: endGroup === undefined ? undefined : { excluded: endGroup }, - }); + track.withRange({ start: { included: floor } }); } // Older drafts carry no timestamps. let stamp: Stamp | undefined; @@ -1004,8 +1014,6 @@ export class Publisher { writer: stream.writer, version: this.version, apply: (update) => { - if (update.ranges?.some((range) => range?.unresolved)) - throw new Error("relative and time subscription anchors are not supported"); Object.assign(datagramBounds, { startGroup: update.startGroup, startFrame: update.startFrame, @@ -1017,18 +1025,14 @@ export class Publisher { const largest = hooks.peekLargest(track).largest?.group; if (update.wantedGroup !== undefined && (largest === undefined || update.wantedGroup > largest)) hooks.joinSubscriber(track); - const end = exclusiveGroupEnd(update.endGroup); track.update({ priority: update.priority, order: update.order, delay: { - max: Milli(servingMaxDelay(this.version, update.maxDelay)), + max: servingMaxDelay(this.version, update.maxDelay), min: update.minDelay === undefined ? null : Milli(update.minDelay), }, - groups: { - start: update.startGroup === undefined ? undefined : { included: update.startGroup }, - end: end === undefined ? undefined : { excluded: end }, - }, + ranges: requestedRanges(update, info.timescale ?? undefined), }); }, }); @@ -1041,7 +1045,6 @@ export class Publisher { served, floor, bounds, - ranges: msg.ranges, }); console.debug(`publish done: broadcast=${msg.broadcast} track=${track.name}`); @@ -1207,19 +1210,12 @@ export class Publisher { // The read cursor's floor; see {@link cursorFloor}. floor: number; bounds: FrameBounds; - // Each lite-07 range, which SUBSCRIBE_START answers for. - ranges: (RangeGroups | undefined)[] | undefined; }, ): Promise { const { sub, broadcast, stamp, publisher, compress, served, bounds } = serving; - // The read cursor's floor, assigned with the cap in {@link bounds} on every move. + // The read cursor's floor; the subscription's ranges cap it. let floor = serving.floor; - let ranges = serving.ranges; - const moveCursor = () => - track.withRange({ - start: { included: floor }, - end: bounds.endGroup === undefined ? undefined : { included: bounds.endGroup }, - }); + const moveCursor = () => track.withRange({ start: { included: floor } }); // Raise the floor to `start`, keeping any higher floor already set. const raiseFloor = (start: number) => { if (start <= floor) return; @@ -1276,6 +1272,15 @@ export class Publisher { ); }; + // The FIN tells the subscriber every group is accounted for, so it waits until each group + // stream finished or reset. False once the subscriber left instead, which cancels whatever + // is still queued. `race`, since this repeats per batch of drops against the subscription's end. + const drained = async (): Promise => { + const done = Symbol("drained"); + const end = await race([Promise.all(groups).then(() => done), controls.ended]); + if (end instanceof Error) throw end; + return end === done; + }; // Before lite-07 an errored end is only the reset at that end: a clean SUBSCRIBE_END sent // early could let the subscriber settle before the reset arrives. A subscription capped // short of the end is owed no error, so its boundary goes out. @@ -1361,7 +1366,6 @@ export class Publisher { bounds.startFrame = update.startFrame; bounds.endGroup = update.endGroup; bounds.endFrame = update.endFrame; - ranges = update.ranges; moveCursor(); // A microtask is too short: decoding one framed update crosses several // awaits, each of which can requeue behind the serving continuation. A @@ -1410,7 +1414,25 @@ export class Publisher { startSent = true; startSequence = bounds.startGroup; raiseFloor(bounds.startGroup); - const start = new SubscribeStart(rangeStarts(ranges, bounds.startGroup), largest); + const start = new SubscribeStart( + rangeStarts(this.version, track, bounds.startGroup), + largest, + ); + if ( + !(await controls.response(encodeSubscribeResponse(stream, { start }, this.version))) + ) + return false; + continue; + } + } + // A lite-07 window whose bounded ranges all resolved empty owes nothing: say so at + // once, and the subscription ends as covered below. No group would ever answer it. + if (emitRange && !startSent && hasRanges(this.version)) { + const end = hooks.servingEnd(track); + if (end !== undefined && hooks.servingRuns(track, 0, end).length === 0) { + startSent = true; + startSequence = end; + const start = new SubscribeStart(rangeStarts(this.version, track, end), largest); if ( !(await controls.response(encodeSubscribeResponse(stream, { start }, this.version))) ) @@ -1423,6 +1445,23 @@ export class Publisher { if (!(await sendEnd())) return false; continue; } + // Lite-07 ends a subscription whose every range has an end once it accounted for + // each group in them, without SUBSCRIBE_END: the FIN ends the subscription, not the + // track. What lies below SUBSCRIBE_START was accounted for by it. + if (hasRanges(this.version) && served && startSequence !== undefined) { + const end = hooks.servingEnd(track); + const covered = + end !== undefined && + hooks + .servingRuns(track, startSequence, end) + .every((run) => runGaps(served, run.start, run.end).length === 0); + if (covered) { + if (!(await drained())) return false; + if (pendingDrops.length > 0) continue; + finished = true; + return true; + } + } await waitForSubscription(controls, track, drops); continue; case "boundary": @@ -1445,23 +1484,17 @@ export class Publisher { // Name every sequence from the start to the end this subscription never // got. A cap below the end bounds what the subscriber is owed, as on its side. if (startSequence !== undefined) { - const end = - bounds.endGroup === undefined ? boundary() : Math.min(boundary(), bounds.endGroup + 1); const unserved = new StreamError(StreamCode.NotFound, { message: "group not served" }); - if (!(await drop(startSequence, end, unserved))) return false; + // Only the groups inside the ranges are owed. + for (const run of hooks.servingRuns(track, startSequence, boundary())) { + if (!(await drop(run.start, run.end, unserved))) return false; + } } if (!endSent) { if (!(await sendEnd())) return false; continue; } - // The FIN tells the subscriber every group is accounted for, so it waits - // until each group stream finished or reset. The subscriber leaving - // instead cancels whatever is still queued. - const drained = Symbol("drained"); - // `race`, since this repeats per batch of drops against the subscription's end. - const end = await race([Promise.all(groups).then(() => drained), controls.ended]); - if (end instanceof Error) throw end; - if (end !== drained) return false; + if (!(await drained())) return false; if (pendingDrops.length > 0) continue; finished = true; return true; @@ -1500,7 +1533,7 @@ export class Publisher { !(await controls.response( encodeSubscribeResponse( stream, - { start: new SubscribeStart(rangeStarts(ranges, startSequence), largest) }, + { start: new SubscribeStart(rangeStarts(this.version, track, startSequence), largest) }, this.version, ), )) diff --git a/js/net/src/lite/subscribe.test.ts b/js/net/src/lite/subscribe.test.ts index 2fcaa9c010c..b327b3571b3 100644 --- a/js/net/src/lite/subscribe.test.ts +++ b/js/net/src/lite/subscribe.test.ts @@ -1,13 +1,13 @@ import { expect, test } from "bun:test"; -import { ProtocolViolation } from "../error.ts"; +import { ProtocolViolation, TimestampMismatch } from "../error.ts"; import * as Path from "../path.ts"; import { Reader, Writer } from "../stream.ts"; +import { Timescale, Timestamp } from "../time.ts"; import { decodeSubscribeResponse, - emptyRange, encodeSubscribeResponse, - exclusiveGroupEnd, inclusiveGroupEnd, + requestedRanges, Subscribe, SubscribeDrop, SubscribeEnd, @@ -15,6 +15,7 @@ import { type SubscribeResponse, SubscribeStart, SubscribeUpdate, + toWireRanges, } from "./subscribe.ts"; import { Version } from "./version.ts"; @@ -207,28 +208,68 @@ test("a lite-07 list decodes as the span covering it", async () => { // Groups 2 through 4 and 7 through frame 1 of group 9. const sparse = await update([2, 1, 2, 0, 1, 5, 0, 1, 7, 0, 1, 9, 2]); expect([sparse.startGroup, sparse.startFrame, sparse.endGroup, sparse.endFrame]).toEqual([2, 0, 9, 1]); + // The ranges stay whole groups, so an end past frame 0 includes its group. expect(sparse.ranges).toEqual([ - { start: 2, end: 5 }, - { start: 7, end: 10 }, + { start: { absolute: 2 }, end: { absolute: 5 } }, + { start: { absolute: 7 }, end: { absolute: 10 } }, ]); - // Preserve that a relative anchor needs resolution before the publisher can serve it. + // The span covering a relative start has no floor; the ranges keep it for the serving reader. const relative = await update([2, 2, 3, 1, 5, 0, 1, 7, 0, 1, 9, 0]); expect([relative.startGroup, relative.endGroup, relative.endFrame]).toEqual([undefined, 8, undefined]); expect(relative.ranges).toEqual([ - { start: 0, end: 5, unresolved: true }, - { start: 7, end: 9 }, + { start: { relative: 3 }, end: { absolute: 5 } }, + { start: { absolute: 7 }, end: { absolute: 9 } }, ]); + // A time anchor carries ticks of the track's timescale, which only the serving track knows. + const timed = await update([1, 3, 40, 3, 90]); + expect(timed.ranges).toEqual([{ start: { time: 40 }, end: { time: 90 } }]); + expect([timed.startGroup, timed.endGroup]).toEqual([undefined, undefined]); + expect(requestedRanges(timed, Timescale.MILLI)).toEqual([ + { start: { time: Timestamp.fromMillis(40) }, end: { time: Timestamp.fromMillis(90) } }, + ]); + expect(() => requestedRanges(timed, undefined)).toThrow(TimestampMismatch); - // No start reaches back to group 0; an empty range has no groups. + // No start reaches back to group 0; one empty partway through group 5 covers no group. const oldest = await update([2, 0, 1, 3, 0, 1, 5, 2, 1, 5, 2]); - expect(oldest.ranges).toEqual([{ start: 0, end: 3 }, undefined]); + expect(oldest.ranges).toEqual([{ end: { absolute: 3 } }, { start: { absolute: 5 }, end: { absolute: 5 } }]); await expect(update([0])).rejects.toThrow(ProtocolViolation); await expect(update([1, 4, 0, 0])).rejects.toThrow(ProtocolViolation); await expect(update([1, 0, 1, 0, 0])).rejects.toThrow(ProtocolViolation); }); +test("lite-07 carries every range, relative and time anchors included, and older wires the span alone", async () => { + const ranges = toWireRanges( + [ + { start: { absolute: 2 }, end: { absolute: 5 } }, + { start: { relative: 1 } }, + { end: { relative: 0 } }, + // In the track's timescale, rounded down. + { start: { time: Timestamp.fromMicros(1_500_900) }, end: { time: Timestamp.fromMillis(3_000) } }, + ], + Timescale.MILLI, + ); + expect(ranges[3]).toEqual({ start: { time: 1_500 }, end: { time: 3_000 } }); + expect(() => toWireRanges([{ start: { time: Timestamp.fromMillis(1) } }], undefined)).toThrow(TimestampMismatch); + const message = new Subscribe({ + id: 1n, + broadcast: Path.from("test"), + track: "video", + priority: 0, + startGroup: 0, + ranges, + }); + const decode = async (version: Version) => + Subscribe.decode(new Reader(undefined, await encodeMessage(version, message), version), version); + expect((await decode(Version.DRAFT_07)).ranges).toEqual(ranges); + const older = await decode(Version.DRAFT_05); + expect([older.ranges, older.startGroup, older.endGroup]).toEqual([undefined, 0, undefined]); + await expect(encodeMessage(Version.DRAFT_07, new Subscribe({ ...message, ranges: [] }))).rejects.toThrow( + "empty subscription range cannot be encoded", + ); +}); + test("SubscribeStart round-trips on draft-05", async () => { const got = await responseRoundtrip(Version.DRAFT_05, { start: new SubscribeStart([42]) }); expect("start" in got).toBe(true); @@ -326,25 +367,13 @@ test("frame bounds without their group bounds are rejected before encoding", asy ); }); -test("model and wire group ends convert without an off-by-one", () => { - expect(exclusiveGroupEnd(undefined)).toBeUndefined(); - expect(exclusiveGroupEnd(0)).toBe(1); - expect(exclusiveGroupEnd(9)).toBe(10); +test("model group ends convert to the wire without an off-by-one", () => { expect(inclusiveGroupEnd(undefined)).toBeUndefined(); expect(inclusiveGroupEnd(1)).toBe(0); expect(inclusiveGroupEnd(10)).toBe(9); expect(() => inclusiveGroupEnd(0)).toThrow("empty subscription range cannot be encoded"); }); -test("a requested range is empty when its bounds meet anywhere", () => { - expect(emptyRange({})).toBe(false); - expect(emptyRange({ endGroup: 0 })).toBe(true); - expect(emptyRange({ startGroup: 5, endGroup: 5 })).toBe(true); - expect(emptyRange({ startGroup: 6, endGroup: 5 })).toBe(true); - expect(emptyRange({ startGroup: 5, endGroup: 6 })).toBe(false); - expect(emptyRange({ startGroup: 5 })).toBe(false); -}); - // Lite-07 carries `Group Order` right after the priority, on SUBSCRIBE and SUBSCRIBE_UPDATE // alike, and refuses a value it does not define. test.each(["ascending", "descending"] as const)("lite-07 round-trips a %s group order", async (order) => { diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index 199e7578c9c..5dcf14ddd32 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -1,8 +1,16 @@ -import { fromTransport, ProtocolViolation, StreamCode, StreamError, toStreamCode } from "../error.ts"; +import { + fromTransport, + ProtocolViolation, + StreamCode, + StreamError, + TimestampMismatch, + toStreamCode, +} from "../error.ts"; import type * as Instance from "../instance.ts"; import * as Path from "../path.ts"; import type { Reader, Writer } from "../stream.ts"; -import type { Order, Position } from "../track.ts"; +import { type Timescale, Timestamp } from "../time.ts"; +import type { Anchor, Order, Position, Range } from "../track.ts"; import { decodePin, encodePin } from "./instance.ts"; import * as Message from "./message.ts"; import { fromWire, toWire } from "./priority.ts"; @@ -119,26 +127,80 @@ interface Span { startFrame: number; endGroup?: number; endFrame?: number; - /** See {@link Subscribe.ranges}. */ - ranges?: (RangeGroups | undefined)[]; } /** - * The groups of one lite-07 range: from `start` up to the exclusive `end`, or through the - * live edge without one. An unresolved anchor must be refused by the publisher. + * One end of a lite-07 range as the wire carries it: a model {@link Anchor}, except that a time + * counts in the track's `Timescale` as the peer was told it. */ -export interface RangeGroups { - start: number; - end?: number; - /** Whether a relative or time anchor still needs resolution this publisher cannot do. */ - unresolved?: true; +export type WireAnchor = + | { absolute: number; relative?: never; time?: never } + | { relative: number; absolute?: never; time?: never } + | { time: number; absolute?: never; relative?: never }; + +/** A lite-07 range as the wire carries it; see {@link WireAnchor}. */ +export interface WireRange { + start?: WireAnchor; + end?: WireAnchor; +} + +/** + * The wire form of `ranges`, with time anchors in `timescale`, the track's as the publisher + * declared it. Throws {@link TimestampMismatch} for a time anchor on an untimed track. + */ +export function toWireRanges(ranges: readonly Range[], timescale: Timescale | undefined): WireRange[] { + const anchor = (anchor: Anchor | undefined): WireAnchor | undefined => { + if (anchor?.time === undefined) return anchor as WireAnchor | undefined; + if (timescale === undefined) throw new TimestampMismatch(); + return { time: Math.floor(anchor.time.as(timescale)) }; + }; + return ranges.map((range) => { + const out: WireRange = {}; + const start = anchor(range.start); + const end = anchor(range.end); + if (start) out.start = start; + if (end) out.end = end; + return out; + }); +} + +/** Whether any range starts or ends at a presentation time, which needs the track's timescale. */ +export function hasWireTime(ranges: readonly WireRange[] | undefined): boolean { + return ranges?.some((range) => range.start?.time !== undefined || range.end?.time !== undefined) ?? false; +} + +/** Encode one lite-07 range end, `undefined` for none: an absolute end names a group's first frame. */ +async function encodeAnchor(w: Writer, anchor: WireAnchor | undefined) { + if (anchor?.absolute !== undefined) { + await w.u53(1); + await w.u53(anchor.absolute); + await w.u53(0); + } else if (anchor?.relative !== undefined) { + await w.u53(2); + await w.u53(anchor.relative); + } else if (anchor?.time !== undefined) { + await w.u53(3); + await w.u53(anchor.time); + } else { + await w.u53(0); + } } /** * Encode the span after the max delay. Before lite-07 that is `Group Start`, `Group End`, - * and the frame bounds; lite-07 carries a list of ranges, and this span is its only one. + * and the frame bounds; lite-07 carries a list of ranges, which is `ranges` when given and + * otherwise this span as its only one. */ -async function encodeSpan(w: Writer, version: Version, span: Span) { +async function encodeSpan(w: Writer, version: Version, span: Span, ranges?: readonly WireRange[]) { + if (ranges !== undefined && hasRanges(version)) { + if (ranges.length === 0) throw new Error(EMPTY_RANGE); + await w.u53(ranges.length); + for (const range of ranges) { + await encodeAnchor(w, range.start); + await encodeAnchor(w, range.end); + } + return; + } if (!hasRanges(version)) { await encodeStartGroup(w, version, span.startGroup); await w.u53(span.endGroup !== undefined ? span.endGroup + 1 : 0); @@ -169,6 +231,33 @@ async function encodeSpan(w: Writer, version: Version, span: Span) { } } +/** + * The model ranges a decoded SUBSCRIBE or SUBSCRIBE_UPDATE asks for: its lite-07 list, or the one + * span an older wire carries, whole groups either way. A time anchor counts in `timescale`, the + * track's as the peer was told it, and is refused with {@link TimestampMismatch} without one. + */ +export function requestedRanges(msg: Span & { ranges?: WireRange[] }, timescale: Timescale | undefined): Range[] { + if (msg.ranges) { + const anchor = (anchor: WireAnchor | undefined): Anchor | undefined => { + if (anchor?.time === undefined) return anchor as Anchor | undefined; + if (timescale === undefined) throw new TimestampMismatch(); + return { time: new Timestamp(anchor.time, timescale) }; + }; + return msg.ranges.map((range) => { + const out: Range = {}; + const start = anchor(range.start); + const end = anchor(range.end); + if (start) out.start = start; + if (end) out.end = end; + return out; + }); + } + const range: Range = {}; + if (msg.startGroup !== undefined) range.start = { absolute: msg.startGroup }; + if (msg.endGroup !== undefined) range.end = { absolute: msg.endGroup + 1 }; + return [range]; +} + /** A position as a comparable pair, group then frame. */ type Point = [group: number, frame: number]; @@ -177,30 +266,32 @@ function before(a: Point, b: Point): boolean { } /** - * Decode one lite-07 range end: its position, `null` for none, or `undefined` for a relative - * or time anchor, which the publisher refuses rather than widening it. + * Decode one lite-07 range end: its absolute position, if any, and the wire anchor, `undefined` + * for none. An absolute end past frame 0 includes its group. */ -async function decodeAnchor(r: Reader): Promise { +async function decodeAnchor(r: Reader, end: boolean): Promise<{ point?: Point; anchor?: WireAnchor }> { const kind = await r.u53(); switch (kind) { case 0: - return null; - case 1: - return [await r.u53(), await r.u53()]; + return {}; + case 1: { + const point: Point = [await r.u53(), await r.u53()]; + return { point, anchor: { absolute: end && point[1] > 0 ? point[0] + 1 : point[0] } }; + } case 2: + return { anchor: { relative: await r.u53() } }; case 3: - await r.u53(); - return undefined; + return { anchor: { time: await r.u53() } }; default: throw new ProtocolViolation(`unknown range anchor: ${kind}`); } } /** - * The inverse of {@link encodeSpan}. A lite-07 list decodes as the one span covering every - * range in it, which the subscriber narrows again. + * The inverse of {@link encodeSpan}. A lite-07 list decodes as its ranges and as the one span + * covering every range in it, which the subscriber narrows again. */ -async function decodeSpan(r: Reader, version: Version): Promise { +async function decodeSpan(r: Reader, version: Version): Promise { if (!hasRanges(version)) { const startGroup = await decodeStartGroup(r, version); const endGroup = (await r.u53()) || undefined; @@ -215,26 +306,21 @@ async function decodeSpan(r: Reader, version: Version): Promise { let end: Point | undefined; let floor = true; let live = false; - // Not preallocated: each range costs at least two bytes of the bounded message. - const ranges: (RangeGroups | undefined)[] = []; + const ranges: WireRange[] = []; for (let i = 0; i < count; i++) { - const from = await decodeAnchor(r); - const to = await decodeAnchor(r); - // An end partway through a group still covers it. - const empty = from && to && !before(from, to); - ranges.push( - empty - ? undefined - : { - start: from?.[0] ?? 0, - end: to ? to[0] + (to[1] > 0 ? 1 : 0) : undefined, - ...(from === undefined || to === undefined ? { unresolved: true as const } : {}), - }, - ); - if (!from) floor = false; - else if (start === undefined || before(from, start)) start = from; - if (!to) live = true; - else if (end === undefined || before(end, to)) end = to; + const from = await decodeAnchor(r, false); + const to = await decodeAnchor(r, true); + const range: WireRange = {}; + if (from.anchor) range.start = from.anchor; + if (to.anchor) range.end = to.anchor; + // Empty partway through a group, which whole groups can't say: keep its place, + // covering no group, as the Rust publisher does. + if (from.point && to.point && !before(from.point, to.point)) range.end = { absolute: from.point[0] }; + ranges.push(range); + if (from.point === undefined) floor = false; + else if (start === undefined || before(from.point, start)) start = from.point; + if (to.point === undefined) live = true; + else if (end === undefined || before(end, to.point)) end = to.point; } if (!floor) start = undefined; if (live) end = undefined; @@ -243,11 +329,11 @@ async function decodeSpan(r: Reader, version: Version): Promise { const startFrame = start?.[1] ?? 0; return { + ranges, startGroup: canonicalStartGroup(version, start?.[0], startFrame), startFrame, endGroup: end === undefined ? undefined : end[1] === 0 ? end[0] - 1 : end[0], endFrame: end === undefined || end[1] === 0 ? undefined : end[1] - 1, - ranges, }; } @@ -293,24 +379,14 @@ async function encodeOrder(w: Writer, version: Version, order: Order) { await w.u8(order === "ascending" ? 1 : 0); } -/** - * Exclusive model/local cap from a decoded inclusive last group. - * - * The wire's `Group End` is inclusive once decoded; the track model and `withRange` are - * exclusive. `undefined` stays unbounded. - */ -export function exclusiveGroupEnd(inclusive?: number): number | undefined { - return inclusive === undefined ? undefined : inclusive + 1; -} - -/** The error for a requested range the wire cannot carry; see {@link emptyRange}. */ +/** The error for a requested range the wire cannot carry: one asking for nothing. */ export const EMPTY_RANGE = "empty subscription range cannot be encoded"; /** * Inclusive last group a Subscribe message carries, from an exclusive model end. * * Empty (`0`) cannot be encoded: the wire's 0 means unbounded. Callers refuse an empty - * requested range with {@link emptyRange} before reaching here. + * requested range before reaching here. */ export function inclusiveGroupEnd(exclusive?: number): number | undefined { if (exclusive === undefined) return undefined; @@ -318,18 +394,6 @@ export function inclusiveGroupEnd(exclusive?: number): number | undefined { return exclusive - 1; } -/** - * Whether a requested range asks for nothing. - * - * The wire has no encoding for one: its bounds are inclusive, so flooring the end would - * either hand back the group the caller excluded (0 means unbounded) or invert the - * range once the two bounds meet. An absent start is the live edge, so it only empties - * the range when the end is 0. - */ -export function emptyRange({ startGroup, endGroup }: { startGroup?: number; endGroup?: number }): boolean { - return endGroup !== undefined && (startGroup ?? 0) >= endGroup; -} - /** Encode the `Subscriber Min Age`: milliseconds plus one, zero meaning not stated. Dropped * on a wire without the field. */ async function encodeMinDelay(w: Writer, version: Version, minDelay: number | undefined) { @@ -364,9 +428,8 @@ export class SubscribeUpdate { startFrame: number; /** See {@link Subscribe.endFrame}. */ endFrame?: number; - /** See {@link Subscribe.ranges}. */ - ranges?: (RangeGroups | undefined)[]; + ranges?: WireRange[]; constructor(props: { priority: number; @@ -378,7 +441,7 @@ export class SubscribeUpdate { endGroup?: number; startFrame?: number; endFrame?: number; - ranges?: (RangeGroups | undefined)[]; + ranges?: WireRange[]; }) { if (props.wantedGroup !== undefined && (!Number.isSafeInteger(props.wantedGroup) || props.wantedGroup < 0)) throw new RangeError("wanted group must be a safe non-negative integer"); @@ -405,7 +468,7 @@ export class SubscribeUpdate { await encodeOrder(w, version, this.order); await w.u53(this.maxDelay); await encodeMinDelay(w, version, this.minDelay); - await encodeSpan(w, version, this); + await encodeSpan(w, version, this, this.ranges); if (hasRefresh(version)) await w.u62(this.wantedGroup === undefined ? 0n : BigInt(this.wantedGroup) + 1n); break; @@ -490,10 +553,10 @@ export class Subscribe { endFrame?: number; /** - * The groups of each lite-07 range in turn, `undefined` for an empty one. Decoded only, - * so SUBSCRIBE_START can answer for every range. + * Every range. Lite-07 encodes these in place of the span when given, and older versions + * the span alone; decoding a lite-07 list fills them alongside the span covering them all. */ - ranges?: (RangeGroups | undefined)[]; + ranges?: WireRange[]; constructor(props: { id: bigint; @@ -508,7 +571,7 @@ export class Subscribe { endGroup?: number; startFrame?: number; endFrame?: number; - ranges?: (RangeGroups | undefined)[]; + ranges?: WireRange[]; }) { this.id = props.id; this.broadcast = props.broadcast; @@ -540,7 +603,7 @@ export class Subscribe { await encodeOrder(w, version, this.order); await w.u53(this.maxDelay); await encodeMinDelay(w, version, this.minDelay); - await encodeSpan(w, version, this); + await encodeSpan(w, version, this, this.ranges); break; } } diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index c1c0acba951..fbcd0c55a8d 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -23,7 +23,7 @@ import { GRACE } from "../ledger.ts"; import * as Path from "../path.ts"; import { Reader, Writer } from "../stream.ts"; import { Timescale } from "../time.ts"; -import type { Producer } from "../track.ts"; +import type { Producer, Subscriber as TrackSubscriber } from "../track.ts"; import { type AnnounceBroadcast, AnnounceInit, AnnounceOk, encodeAnnounceBroadcast } from "./announce.ts"; import { Datagram as DatagramMessage } from "./datagram.ts"; import { Group } from "./group.ts"; @@ -2476,7 +2476,7 @@ test("an empty START remains resolved when its range widens", async () => { const { quic, streams, opened } = fakeSession(); const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); const broadcast = subscriber.consume(Path.from("room")); - const track = broadcast.track("video").subscribe({ groups: { start: { included: 0 }, end: { excluded: 1 } } }); + const track = broadcast.track("video").subscribe({ ranges: [{ start: { absolute: 0 }, end: { absolute: 1 } }] }); try { await opened(0); await answerTrackInfo(streams[0], version); @@ -2485,7 +2485,7 @@ test("an empty START remains resolved when its range widens", async () => { expect(hooks.sourceStart(track)).toBe("pending"); await respond(request, version, [{ start: new SubscribeStart([undefined]) }]); await drainReactions(); - track.update({ groups: { start: { included: 0 }, end: { excluded: 2 } } }); + track.update({ ranges: [{ start: { absolute: 0 }, end: { absolute: 2 } }] }); await drainReactions(); expect(hooks.sourceStart(track)).not.toBe("pending"); } finally { @@ -2496,14 +2496,15 @@ test("an empty START remains resolved when its range widens", async () => { }); // A lite-07 publisher FINs a bounded subscription without SUBSCRIBE_END once it covered the range, -// which ends the subscription cleanly rather than violating the protocol. Covering a range is -// not ending the track, so no end is declared. -test("a lite-07 publisher FIN ends a covered bounded subscription", async () => { +// which is no protocol violation. Covering a range is not ending the track, so the copy stays for +// its readers, as the Rust copy does: they park, and widening the range asks again on the same +// track. Decided 2026-10-31 (maintainer). +test("a lite-07 publisher FIN keeps a covered bounded subscription's copy", async () => { const version = Version.DRAFT_07; const { quic, streams, opened, closes } = fakeSession(); const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); const broadcast = subscriber.consume(Path.from("room")); - const track = broadcast.track("video").subscribe({ groups: { start: { included: 0 }, end: { excluded: 2 } } }); + const track = broadcast.track("video").subscribe({ ranges: [{ start: { absolute: 0 }, end: { absolute: 2 } }] }); try { await opened(0); await answerTrackInfo(streams[0], version); @@ -2522,9 +2523,18 @@ test("a lite-07 publisher FIN ends a covered bounded subscription", async () => ); for (const chunk of chunks) stream.inbound.enqueue(chunk); stream.inbound.close(); - expect(await track.closed).toBeNull(); - await expect(track.finished()).rejects.toThrow("track closed before its end was known"); await drainReactions(); + expect(track.closed.peek()).toBeUndefined(); + expect(streams).toHaveLength(2); + + track.update({ ranges: [{ start: { absolute: 0 }, end: { absolute: 3 } }] }); + const reopened = await opened(2); + await reopened.reading; + const reader = new Reader(undefined, new Uint8Array(Buffer.concat(reopened.written)), version); + expect(await reader.u53()).toBe(StreamId.Subscribe); + const msg = await Subscribe.decode(reader, version); + expect(msg.ranges).toEqual([{ start: { absolute: 0 }, end: { absolute: 3 } }]); + expect(track.closed.peek()).toBeUndefined(); expect(closes).toEqual([]); } finally { track.close(); @@ -2533,6 +2543,50 @@ test("a lite-07 publisher FIN ends a covered bounded subscription", async () => } }); +// A widening reopens a covered copy, and its readers leaving while the publisher never takes the +// new SUBSCRIBE gives the setup up: its stream is reset and the copy released, not held forever. +test("a reopened lite-07 subscription ends once its readers leave", async () => { + const version = Version.DRAFT_07; + const { quic, streams, opened } = fakeSession([], [2]); + const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); + const broadcast = subscriber.consume(Path.from("room")); + const track = broadcast.track("video").subscribe({ ranges: [{ start: { absolute: 0 }, end: { absolute: 2 } }] }); + let next: TrackSubscriber | undefined; + try { + await opened(0); + await answerTrackInfo(streams[0], version); + const stream = await opened(1); + await stream.reading; + const chunks: Uint8Array[] = []; + const writer = new Writer( + new WritableStream({ write: (chunk) => void chunks.push(new Uint8Array(chunk)) }), + version, + ); + await encodeSubscribeResponse(writer, { start: new SubscribeStart([0]) }, version); + await encodeSubscribeResponse( + writer, + { drop: new SubscribeDrop({ start: 0, end: 1, error: StreamCode.NotFound }) }, + version, + ); + for (const chunk of chunks) stream.inbound.enqueue(chunk); + stream.inbound.close(); + await drainReactions(); + + track.update({ ranges: [{ start: { absolute: 0 }, end: { absolute: 3 } }] }); + await (await opened(2)).writing; + track.close(); + await drainReactions(); + // A reader arriving now asks afresh, rather than join a copy held by the stalled setup. + next = broadcast.track("video").subscribe(); + await opened(3); + } finally { + next?.close(); + track.close(); + broadcast.close(); + subscriber.close(); + } +}); + // Before lite-05 the publisher FINs the subscribe stream only once the track is over, so that FIN // finishes the track once the tail settles, rather than releasing it for another route. test.each( @@ -2621,23 +2675,22 @@ test.each([ } }); -// The FIN may answer the bounded range before the publisher read an update that widened it, -// so the track fails rather than ending clean short of the demand. +// The FIN may answer the bounded range before the publisher read an update that widened it, so +// the copy asks again for the demand rather than end clean short of it or fail its readers. test.each([ - ["an open widening after SUBSCRIBE_START", { start: { included: 0 } }, true, false], - ["a bounded widening without SUBSCRIBE_START", { start: { included: 0 }, end: { excluded: 5 } }, false, false], - ["a widening the publisher stopped reading", { start: { included: 0 }, end: { excluded: 5 } }, false, true], -] as const)("a lite-07 FIN short of %s fails the track", async (_, groups, start, stopped) => { + ["an open widening after SUBSCRIBE_START", { start: { absolute: 0 } }, true, false], + ["a bounded widening without SUBSCRIBE_START", { start: { absolute: 0 }, end: { absolute: 5 } }, false, false], + ["a widening the publisher stopped reading", { start: { absolute: 0 }, end: { absolute: 5 } }, false, true], +] as const)("a lite-07 FIN short of %s subscribes again on the same track", async (_, range, start, stopped) => { const clock = new Manual(); const installed = clock.install(); const version = Version.DRAFT_07; const { quic, streams, opened, closes } = fakeSession(); const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); const broadcast = subscriber.consume(Path.from("room")); - const track = broadcast.track("video").subscribe({ - delay: { max: Milli.zero }, - groups: { start: { included: 0 }, end: { excluded: 1 } }, - }); + const track = broadcast + .track("video") + .subscribe({ delay: { max: Milli.zero }, ranges: [{ start: { absolute: 0 }, end: { absolute: 1 } }] }); try { await opened(0); await answerTrackInfo(streams[0], version); @@ -2651,7 +2704,7 @@ test.each([ }), ) : undefined; - track.update({ delay: { max: Milli.zero }, groups }); + track.update({ delay: { max: Milli.zero }, ranges: [range] }); await refused; await drainReactions(); @@ -2673,7 +2726,14 @@ test.each([ await drainReactions(); await clock.advance(Nano.fromMilli(GRACE)); - expect(await track.closed).toBeInstanceOf(Error); + const reopened = await opened(2); + await reopened.reading; + const reader = new Reader(undefined, new Uint8Array(Buffer.concat(reopened.written)), version); + expect(await reader.u53()).toBe(StreamId.Subscribe); + const msg = await Subscribe.decode(reader, version); + expect(msg.id).toBe(1n); + expect(msg.ranges).toEqual([range]); + expect(track.closed.peek()).toBeUndefined(); expect(closes).toEqual([]); } finally { track.close(); @@ -2747,7 +2807,7 @@ test("an update whose bound the wire can't carry fails the track", async () => { await opened(0); await answerTrackInfo(streams[0], Version.DRAFT_05); await (await opened(1)).reading; - track.update({ groups: { start: { included: Number.MAX_SAFE_INTEGER } } }); + track.update({ ranges: [{ start: { absolute: Number.MAX_SAFE_INTEGER } }] }); expect(await track.closed).toBeInstanceOf(RangeError); } finally { track.close(); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 77a699ffb6a..4ff89d4421d 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -22,16 +22,7 @@ import { import * as netGroup from "../group.ts"; import { Cost, type Hop, MAX_HOPS, type Route, routesEqual, UNKNOWN_HOP } from "../hop.ts"; import * as Instance from "../instance.ts"; -import { - groupBounds, - hiddenBelow, - hooks, - maxDelayOf, - minDelayOf, - scopeCaptures, - scopeHead, - scopeOverlaps, -} from "../internal.ts"; +import { hiddenBelow, hooks, maxDelayOf, minDelayOf, scopeCaptures, scopeHead, scopeOverlaps } from "../internal.ts"; import type { Lister } from "../listing.ts"; import * as Path from "../path.ts"; import { receiveFrame } from "../payload.ts"; @@ -46,6 +37,7 @@ import { Deadline, REQUEST_TIMEOUT_MS, requestOptions } from "../util/deadline.t import { abandoned, isAbandoned, unwanted } from "../util/demand.ts"; import { LIMITS, type Limits, SlotMap, Slots } from "../util/limits.ts"; import { Budget } from "../util/yield.ts"; +import { bounded, hasTime, rangesStart, Window, wireRanges, wireSpan } from "../window.ts"; import { overrideBroadcastWire, wireOf } from "../wire.ts"; import { AnnounceHistory, @@ -65,11 +57,10 @@ import { decodeSubscribeResponse, decodeSubscribeResponseMaybe, EMPTY_RANGE, - emptyRange, - exclusiveGroupEnd, inclusiveGroupEnd, Subscribe, SubscribeUpdate, + toWireRanges, } from "./subscribe.ts"; import { TrackInfo, Track as TrackMessage } from "./track.ts"; import { decodeTracksEventMaybe, Tracks as TracksMessage } from "./tracks.ts"; @@ -139,6 +130,8 @@ interface SubscribeEntry { compress: boolean; // Per-frame timestamp scale (0 = none), resolved before this entry is registered. timescale: number; + // The track's timescale as its model declares it, which a time anchor counts in. + scale?: Time.Timescale; // The group streams received, so the subscription can wait for the ones still owed // after the publisher ends it. tail: Tail; @@ -148,6 +141,11 @@ interface SubscribeEntry { // (SUBSCRIBE_END), once it declares them. start?: number; end?: number; + // Where SUBSCRIBE_START says each of `asked` starts on lite-07, undefined for one with + // nothing to deliver. It answers `asked` unless `updated`. + starts?: (number | undefined)[]; + // Whether a SUBSCRIBE_UPDATE went out, which the publisher may have read before answering. + updated?: boolean; // Set by the subscribe stream's FIN before lite-05, the track's only end there. finished?: boolean; // Settles once the subscribe stream's responses are read to its FIN or reset, which @@ -156,9 +154,9 @@ interface SubscribeEntry { // Whether the stream ever asked for a bounded range, which a lite-07 publisher FINs without // SUBSCRIBE_END once it covers it. bounded: boolean; - // The groups the SUBSCRIBE asked for, which that FIN covers unless the demand moved since - // in an update the publisher may not have read. - asked: { start: number; end?: number }; + // The ranges the SUBSCRIBE asked for, as `wireRanges` gives them, which that FIN covers + // unless the demand moved since in an update the publisher may not have read. + asked: track.Range[]; // The TRACK stream that answered (lite-05+), held open as interest in the track until // the SUBSCRIBE has its first response, so the publisher's demand never lapses between // the two. @@ -951,9 +949,7 @@ export class Subscriber { async #runSubscribe(broadcast: Path.Valid, consumed: Consumed, request: track.Request) { const id = this.#subscribeNext++; - const subscription = request.subscription; - const initialBounds = groupBounds(subscription.groups); - if (emptyRange({ startGroup: initialBounds.start, endGroup: initialBounds.end })) { + if (wireRanges(request.subscription.ranges ?? [{}]).length === 0) { request.reject(new Error(EMPTY_RANGE)); return; } @@ -1003,39 +999,21 @@ export class Subscriber { request: track.Request, state: SubscribeSetup, ) { - const { instance, valid, answering } = consumed; - const subscription = request.subscription; + const { valid } = consumed; + const ranges = wireRanges(request.subscription.ranges ?? [{}]); + const msg = this.#subscribeMessage(id, broadcast, consumed, request.name, request.subscription, ranges); console.debug(`subscribe start: id=${id} broadcast=${broadcast} track=${request.name}`); - const bounds = groupBounds(subscription.groups); - - const msg = new Subscribe({ - id, - broadcast, - instance, - track: request.name, - priority: subscription.priority ?? 0, - order: subscription.order, - maxDelay: maxDelayOf(subscription), - minDelay: minDelayOf(subscription), - startGroup: subscription.groups?.start === undefined ? undefined : bounds.start, - endGroup: inclusiveGroupEnd(bounds.end), - }); // Open the stream under a timeout. The stream handles flow back via `state` // so the timeout path can close them if they finish opening after the deadline, // and `cancel` ends a setup still running once the deadline passed, resetting // its TRACK stream so a peer that never answers can't hold one per attempt. - const retirer = (signal?: AbortSignal) => () => { - state.cancel.abort(signal?.reason); - state.producer?.abort(error(signal?.reason)); - state.stream?.abort(error(signal?.reason)); - }; - const retire = retirer(valid); + const retire = this.#retirer(state, valid); valid?.addEventListener("abort", retire); if (valid?.aborted) retire(); const deadline = new Deadline(); - const setup = this.#openSubscribe(state, msg, request, id, deadline); + const setup = this.#openSubscribe(state, msg, ranges, request, id, deadline); let opened: { stream: Stream; entry: SubscribeEntry }; try { @@ -1063,18 +1041,134 @@ export class Subscriber { setup.then(leave, leave); return; } - - const { stream, entry } = opened; valid?.removeEventListener("abort", retire); + + let { stream, entry } = opened; + let current = { id, msg }; + // A lite-07 publisher that FINs a covered bounded subscription leaves the copy to its + // readers. If the demand moved past what it covered, ask again on the same track, as the + // Rust copy does, rather than fail the readers still waiting on it. + while (await this.#serveStream(current.id, broadcast, consumed, state, stream, entry, current.msg)) { + const producer = entry.track; + const subscription = producer.subscription.peek(); + const ranges = wireRanges(subscription?.ranges ?? [{}]); + if (!subscription || ranges.length === 0) { + hooks.releaseTrack(producer); + return; + } + const id = this.#subscribeNext++; + const msg = this.#subscribeMessage(id, broadcast, consumed, producer.name, subscription, ranges); + console.debug(`subscribe reopen: id=${id} broadcast=${broadcast} track=${producer.name}`); + // Bounded like the first setup, and given up once nobody wants the track, so a peer + // that stops reading holds neither the stream nor its entry. + // Watched only while the setup runs, so a reopen leaves no watcher behind. + const left = new AbortController(); + const demand = producer.demand(); + let unwatch: Dispose | undefined; + const watch = () => { + if (isAbandoned(demand)) left.abort(new StreamError(StreamCode.Cancel, { message: "cancel" })); + else unwatch = Signal.race(demand.wanted, hooks.demandReleased(demand)).changed(watch); + }; + watch(); + let sent: Promise | undefined; + valid?.addEventListener("abort", retire); + try { + if (valid?.aborted) retire(); + state.cancel.signal.throwIfAborted(); + const framing = { compress: entry.compress, timescale: entry.timescale, scale: entry.scale }; + entry = this.#register( + id, + this.#withRanges(msg, ranges, entry.scale), + ranges, + producer, + framing, + state, + ); + const deadline = new Deadline(); + sent = this.#sendSubscribe(state, msg, deadline); + stream = await deadline.wait(untilAborted(untilAborted(sent, valid), left.signal)); + current = { id, msg }; + } catch (err) { + const e = + state.revoked ?? (left.signal.aborted ? left.signal.reason : await sessionCause(this.#quic, err)); + state.cancel.abort(e); + this.#subscribes.delete(id); + // A stream that opens late is reset too, as the first setup's are. + const close = () => state.stream?.abort(e); + close(); + sent?.then(close, close); + if (left.signal.aborted && !state.revoked) { + hooks.releaseTrack(producer); + return; + } + producer.abort(e); + console.warn( + `subscribe error: id=${id} broadcast=${broadcast} track=${producer.name} error=${reason(e)}`, + ); + return; + } finally { + unwatch?.(); + valid?.removeEventListener("abort", retire); + } + } + } + + // The SUBSCRIBE for `subscription`, asking for `ranges`, its wire ranges in the model. Its own + // wire ranges wait for the track's timescale when they name a time (see `#withRanges`). + #subscribeMessage( + id: bigint, + broadcast: Path.Valid, + consumed: Consumed, + name: string, + subscription: Readonly, + ranges: readonly track.Range[], + ): Subscribe { + const span = wireSpan(ranges); + return new Subscribe({ + id, + broadcast, + instance: consumed.instance, + track: name, + priority: subscription.priority ?? 0, + order: subscription.order, + maxDelay: maxDelayOf(subscription), + minDelay: minDelayOf(subscription), + startGroup: span.start, + endGroup: inclusiveGroupEnd(span.end), + }); + } + + // Ends a subscription's setup once `signal` aborts: its streams, and the track it accepted. + #retirer(state: SubscribeSetup, signal?: AbortSignal): () => void { + return () => { + state.cancel.abort(signal?.reason); + state.producer?.abort(error(signal?.reason)); + state.stream?.abort(error(signal?.reason)); + }; + } + + // Serve one SUBSCRIBE stream into its track until it ends. Resolves true when a lite-07 + // publisher covered the bounded ranges it read but an update since moved the demand past + // them, leaving the track open for the caller to ask again. + async #serveStream( + id: bigint, + broadcast: Path.Valid, + consumed: Consumed, + state: SubscribeSetup, + stream: Stream, + entry: SubscribeEntry, + msg: Subscribe, + ): Promise { + const { answering } = consumed; + const producer = entry.track; // Sent but unanswered (lite-05+; older drafts already waited for SUBSCRIBE_OK), only a // replacement refuses it, since that instance may be the one answering. - const replaced = retirer(answering); + const replaced = this.#retirer(state, answering); if (hasTrackStream(this.version) && answering) { answering.addEventListener("abort", replaced); entry.answered = () => answering.removeEventListener("abort", replaced); if (answering.aborted) replaced(); } - const producer = entry.track; try { // Watch for subscription changes and send SUBSCRIBE_UPDATE. Lite01/Lite02 // don't carry SUBSCRIBE_UPDATE on the wire, so skip the watcher there @@ -1087,9 +1181,9 @@ export class Subscriber { entry.responses = responses.catch(() => {}); let tailSettled = false; // Set when the publisher ended the subscription short of the demand. - let short: Error | undefined; + let short = false; const closed = responses.then(async () => { - if (!(await this.#settleTail(entry))) short = new Error("subscription ended before covering its range"); + short = !(await this.#settleTail(entry)); tailSettled = true; }); // A reset that lands after the race below settled is moot; the race observes one before. @@ -1114,10 +1208,11 @@ export class Subscriber { // down (e.g. a quickly unmuted tile) resumes on the same subscription. const idle = Symbol("idle"); const demand = producer.demand(); + let reason_: unknown; for (;;) { - const reason = await race([done, unwanted(demand).then(() => idle)]); + reason_ = await race([done, unwanted(demand).then(() => idle)]); if ( - reason === idle && + reason_ === idle && hooks.demandReleased(demand).peek() === undefined && demand.wanted.peek() !== undefined ) @@ -1125,26 +1220,32 @@ export class Subscriber { break; } - // A publisher that covered the range before reading an update widening it sent - // fewer groups than the track has, so it fails rather than ending clean. Otherwise - // only a received SUBSCRIBE_END, or before lite-05 a FIN, declares the track's end: - // a range covered or a subscription nobody wants anymore is a released copy, not - // finished content. - if (short) producer.abort(short); - else { - if (tailSettled && entry.finished) producer.finish(); - hooks.releaseTrack(producer); - } // A settled lite-05+ tail acknowledges completion with FIN, without cancelling // the publisher's already-finished receive half. if (tailSettled && waitsForSubscriberFin(this.version)) stream.writer.close(); else stream.close(); - console.debug(`subscribe close: id=${id} broadcast=${broadcast} track=${request.name}`); + console.debug(`subscribe close: id=${id} broadcast=${broadcast} track=${producer.name}`); + + // A publisher that covered the range before reading an update widening it sent fewer + // groups than the demand asks for, so the caller asks again for the rest. A lite-07 + // FIN without SUBSCRIBE_END that covered the demand leaves the copy to its readers + // until the demand moves past it. Otherwise only a received SUBSCRIBE_END, or before + // lite-05 a FIN, declares the track's end: a subscription nobody wants anymore is a + // released copy, not finished content. + const live = reason_ !== idle && hooks.trackReleased(producer).peek() === undefined; + const covered = live && tailSettled && entry.end === undefined && hasRanges(this.version); + const reopen = live && (short || (covered && (await this.#parkCovered(entry)))); + if (!reopen) { + if (tailSettled && entry.finished) producer.finish(); + hooks.releaseTrack(producer); + } + return reopen; } catch (err) { const e = await sessionCause(this.#quic, err); producer.abort(e); - console.warn(`subscribe error: id=${id} broadcast=${broadcast} track=${request.name} error=${reason(e)}`); + console.warn(`subscribe error: id=${id} broadcast=${broadcast} track=${producer.name} error=${reason(e)}`); stream.abort(e); + return false; } finally { answering?.removeEventListener("abort", replaced); entry.held?.close(); @@ -1152,6 +1253,21 @@ export class Subscriber { } } + // Keep a copy whose lite-07 publisher covered its bounded ranges, as the Rust copy does: its + // readers park rather than end, the cache answers the same ranges, and an update can still + // widen them. Resolves true once the demand asks for groups it lacks, to ask again on the + // same track, and false once nobody wants the track or it closed. + async #parkCovered(entry: SubscribeEntry): Promise { + const producer = entry.track; + const demand = producer.demand(); + const released = hooks.trackReleased(producer); + for (;;) { + if (released.peek() !== undefined || isAbandoned(demand)) return false; + if (!this.#demandCovered(entry)) return true; + await race([Signal.race(producer.subscription, demand.wanted, hooks.demandReleased(demand)), released]); + } + } + // Open the subscribe stream and accept the request (so the application's track.Subscriber // resolves and incoming groups have a producer to write into) with the track's immutable // properties. `state.stream` is populated as soon as it opens, and `state.track` once @@ -1160,11 +1276,14 @@ export class Subscriber { // On lite-05+ the properties come from a TRACK stream, held open until the SUBSCRIBE is // answered, and the SUBSCRIBE goes out alongside it rather than a round trip after: it is // accepted implicitly (no SUBSCRIBE_OK), and needs the info only to read what it receives. - // A group stream that beats the info waits on `#pending`, unread. Older drafts carry no - // per-track properties, so they resolve to defaults and just drain SUBSCRIBE_OK. + // A time anchor counts in the track's timescale, so a SUBSCRIBE naming one waits for the + // info, as the Rust `pipeline` does. A group stream that beats the info waits on `#pending`, + // unread. Older drafts carry no per-track properties, so they resolve to defaults and just + // drain SUBSCRIBE_OK. async #openSubscribe( state: SubscribeSetup, msg: Subscribe, + ranges: readonly track.Range[], request: track.Request, id: bigint, deadline: Deadline, @@ -1173,7 +1292,8 @@ export class Subscriber { if (!hasTrackStream(this.version)) { // Older drafts negotiate nothing per-track: verbatim frames with no timeline. - const entry = this.#register(id, request.accept(), { compress: false, timescale: 0 }, msg, request, state); + const framing = { compress: false, timescale: 0, scale: undefined }; + const entry = this.#register(id, msg, ranges, request.accept(), framing, state); const stream = await this.#sendSubscribe(state, msg, deadline); // The first response MUST be a SUBSCRIBE_OK (older drafts only). const resp = await decodeSubscribeResponse(stream.reader, this.version); @@ -1201,7 +1321,14 @@ export class Subscriber { settle = resolve; }), ); - const sending = this.#sendSubscribe(state, msg, deadline); + const timed = hasRanges(this.version) && hasTime(ranges); + const sending = timed + ? info.then((answered) => { + // Refused on an untimed track before anything is accepted. + msg.ranges = toWireRanges(ranges, this.#toModelInfo(answered.info).timescale ?? undefined); + return this.#sendSubscribe(state, msg, deadline); + }) + : this.#sendSubscribe(state, this.#withRanges(msg, ranges, undefined), deadline); try { // Either failing fails the subscription: a reset TRACK rejects the request, so the // origin can try another route, and the caller resets the SUBSCRIBE. @@ -1209,15 +1336,15 @@ export class Subscriber { // The deadline passed as TRACK_INFO landed: the request is already rejected, so don't // accept it again. state.cancel.signal.throwIfAborted(); - const producer = request.accept(this.#toModelInfo(answered.info)); - const entry = this.#register( - id, - producer, - { compress: answered.info.compress, timescale: answered.info.timescale ?? 0 }, - msg, - request, - state, - ); + const model = this.#toModelInfo(answered.info); + const producer = request.accept(model); + const framing = { + compress: answered.info.compress, + timescale: answered.info.timescale ?? 0, + scale: model.timescale ?? undefined, + }; + const entry = this.#register(id, msg, ranges, producer, framing, state); + entry.held = state.track; settle(entry); return { stream, entry }; } catch (err) { @@ -1237,20 +1364,28 @@ export class Subscriber { } } - // Register a subscription's accepted copy, so incoming groups and datagrams find it. + // `msg` asking for `ranges` on a wire that carries them, its time anchors counted in `scale`, + // the timescale the track declares: refused on an untimed one. + #withRanges(msg: Subscribe, ranges: readonly track.Range[], scale: Time.Timescale | undefined): Subscribe { + if (hasRanges(this.version)) msg.ranges = toWireRanges(ranges, scale); + return msg; + } + + // Register a subscription's accepted copy, asking for `ranges`, so incoming groups and + // datagrams find it. #register( id: bigint, - producer: track.Producer, - framing: { compress: boolean; timescale: number }, msg: Subscribe, - request: track.Request, + ranges: readonly track.Range[], + producer: track.Producer, + framing: { compress: boolean; timescale: number; scale: Time.Timescale | undefined }, state: SubscribeSetup, ): SubscribeEntry { hooks.receiveTrack(producer); + if (!hasRanges(this.version)) hooks.dropAnchors(producer); // Lite-06+ resolves where the feed starts only with SUBSCRIBE_START, so a relay serving // this copy holds its first group until the answer arrives. if (resolvesStart(this.version)) hooks.startTrack(producer, { requested: msg.startGroup }); - const entry: SubscribeEntry = { track: producer, ...framing, @@ -1261,9 +1396,8 @@ export class Subscriber { grace: () => hooks.tailGrace(producer), }), requested: msg.startGroup, - bounded: msg.endGroup !== undefined, - asked: groupBounds(request.subscription.groups), - held: state.track, + bounded: hasRanges(this.version) && bounded(ranges), + asked: [...ranges], }; this.#subscribes.set(id, entry); state.producer = producer; @@ -1639,9 +1773,13 @@ export class Subscriber { if ("ok" in resp) throw new ProtocolViolation("duplicate SUBSCRIBE_OK"); if ("start" in resp) { - // Every SUBSCRIBE and update we send carries one range, so one start answers it. - if (hasRanges(this.version) && resp.start.starts.length !== 1) - throw new ProtocolViolation(`SUBSCRIBE_START resolves ${resp.start.starts.length} ranges, not 1`); + // One start per range the SUBSCRIBE carried, unless an update the publisher read + // first replaced them. + if (hasRanges(this.version) && !entry.updated && resp.start.starts.length !== entry.asked.length) + throw new ProtocolViolation( + `SUBSCRIBE_START resolves ${resp.start.starts.length} ranges, not ${entry.asked.length}`, + ); + if (hasRanges(this.version)) entry.starts = resp.start.starts; // The lowest range start is the first group served; none leaves nothing owed. entry.start = resp.start.starts.reduce( (lowest, start) => @@ -1691,36 +1829,56 @@ export class Subscriber { // demand uncovered: it covered the bounded range it read, and the demand has moved since. async #settleTail(entry: SubscribeEntry): Promise { const { tail, track } = entry; - - const complete = () => { - const groups = track.subscription.peek()?.groups; - const bounds = groupBounds(groups ?? {}); - // Without SUBSCRIBE_END (older drafts) nothing says which groups are owed, unless a - // lite-07 publisher covered the bounded range. - const end = - entry.end === undefined - ? hasRanges(this.version) - ? bounds.end - : undefined - : Math.min(entry.end, bounds.end ?? entry.end); - if (end === undefined) return false; - // Without SUBSCRIBE_START the publisher served no group at all. - if (entry.start === undefined) return true; - // Owed from the floor the demand last asked for, which an update can move either - // way, or where SUBSCRIBE_START resolved a live-edge one. The groups the SUBSCRIBE - // asked for below its SUBSCRIBE_START were accounted for when it arrived. - const start = groups?.start === undefined ? entry.start : bounds.start; - return tail.covers(start, end); - }; - + const complete = () => this.#covered(entry); const released = hooks.trackReleased(track); await tail.settle(complete, released); if (entry.end !== undefined || !hasRanges(this.version) || released.peek() !== undefined) return true; - const demand = groupBounds(track.subscription.peek()?.groups); - if (demand.start === entry.asked.start && demand.end === entry.asked.end) return true; + return this.#demandCovered(entry); + } + + // Whether a lite-07 FIN without SUBSCRIBE_END covered the demand: the ranges it asked for, or + // the groups that arrived account for the demand since. + #demandCovered(entry: SubscribeEntry): boolean { + const demand = wireRanges(entry.track.subscription.peek()?.ranges ?? [{}]); + if (JSON.stringify(demand) === JSON.stringify(entry.asked)) return true; // Without SUBSCRIBE_START nothing says which ranges the publisher read, so only the // groups that arrived prove the demand covered. - return entry.start !== undefined && complete(); + return entry.start !== undefined && this.#covered(entry); + } + + // Whether the groups received and dropped account for every group the demand is owed once + // the publisher ended the subscription. + #covered(entry: SubscribeEntry): boolean { + const ranges = wireRanges(entry.track.subscription.peek()?.ranges ?? [{}]); + const span = wireSpan(ranges); + // Without SUBSCRIBE_END (older drafts) nothing says which groups are owed, unless a + // lite-07 publisher covered the bounded ranges. + const end = + entry.end === undefined + ? hasRanges(this.version) + ? span.end + : undefined + : Math.min(entry.end, span.end ?? entry.end); + if (end === undefined) return false; + // Without SUBSCRIBE_START the publisher served no group at all. + if (entry.start === undefined) return true; + // Owed from the floor the demand last asked for, which an update can move either + // way, or where SUBSCRIBE_START resolved a live-edge one. The groups the SUBSCRIBE + // asked for below its SUBSCRIBE_START were accounted for when it arrived. Only the + // groups inside the ranges are owed: the publisher accounts for nothing between them. + // While SUBSCRIBE_START answers these ranges, each is owed from the start it named, which + // accounts for a skipped prefix and resolves a relative or time start. Otherwise a time + // start resolved at the publisher, at the lowest start it named. + const starts = !entry.updated && JSON.stringify(ranges) === JSON.stringify(entry.asked) && entry.starts; + const start = starts ? entry.start : (rangesStart(ranges) ?? entry.start); + const owed = starts + ? ranges.flatMap((range, index) => { + const at = starts[index]; + return at === undefined ? [] : [{ ...range, start: { absolute: at } }]; + }) + : ranges.map((range) => (range.start?.time ? { ...range, start: { absolute: start } } : range)); + const window = new Window(owed, entry.track.largest()?.group); + return window.runs(start, end).every((run) => entry.tail.covers(run.start, run.end)); } /** @@ -1751,10 +1909,7 @@ export class Subscriber { priority: msg.priority, order: msg.order, delay: { max: Milli(msg.maxDelay), min: msg.minDelay === undefined ? null : Milli(msg.minDelay) }, - groups: { - start: msg.startGroup === undefined ? undefined : { included: msg.startGroup }, - end: msg.endGroup === undefined ? undefined : { excluded: exclusiveGroupEnd(msg.endGroup) ?? 0 }, - }, + ranges: entry.asked, }; // The wanted group is subscription state: an update announces each new one past our @@ -1784,13 +1939,15 @@ export class Subscriber { // Demand collapsing to nothing is refused the same way an initial empty // request is: the error closes the track, so every local subscriber sees it. - const bounds = groupBounds(current.groups); - if (emptyRange({ startGroup: bounds.start, endGroup: bounds.end })) throw new Error(EMPTY_RANGE); + const ranges = wireRanges(current.ranges ?? [{}]); + if (ranges.length === 0) throw new Error(EMPTY_RANGE); + const span = wireSpan(ranges); // A lowered floor owes groups nobody asked for until now. - if (current.groups?.start !== undefined) { - const floor = lastSent.groups?.start === undefined ? entry.start : groupBounds(lastSent.groups).start; - entry.tail.demand(bounds.start, floor ?? Number.POSITIVE_INFINITY); + const start = rangesStart(ranges); + if (start !== undefined) { + const floor = rangesStart(wireRanges(lastSent.ranges ?? [{}])) ?? entry.start; + entry.tail.demand(start, floor ?? Number.POSITIVE_INFINITY); } // Round-trip the other Subscribe parameters so the publisher doesn't @@ -1801,8 +1958,9 @@ export class Subscriber { order: current.order, maxDelay: maxDelayOf(current), minDelay: minDelayOf(current), - startGroup: current.groups?.start === undefined ? undefined : bounds.start, - endGroup: inclusiveGroupEnd(bounds.end), + startGroup: span.start, + endGroup: inclusiveGroupEnd(span.end), + ranges: hasRanges(this.version) ? toWireRanges(ranges, entry.scale) : undefined, }); // Encoded apart from the write, so a bound the wire can't carry still fails the track. const encoded = await Writer.collect(this.version, (w) => update.encode(w, this.version)); @@ -1814,19 +1972,28 @@ export class Subscriber { return; } if (refresh !== undefined) sentWanted = refresh; - entry.bounded ||= update.endGroup !== undefined; + entry.bounded ||= bounded(ranges); + entry.updated = true; lastSent = { ...current }; console.debug(`subscribe update: id=${id} broadcast=${broadcast} track=${track.name}`); } } #sameSubscription(a: track.Subscription, b: track.Subscription): boolean { - const ag = groupBounds(a.groups); - const bg = groupBounds(b.groups); - // `groupBounds` reads an omitted start as 0. A pre-06 wire tells them apart: omitted - // joins at the publisher's start, and 0 is group 0. Lite-06 encodes both as 0. - const sameStart = - resolvesStart(this.version) || (a.groups?.start === undefined) === (b.groups?.start === undefined); + const ar = wireRanges(a.ranges ?? [{}]); + const br = wireRanges(b.ranges ?? [{}]); + let sameRanges: boolean; + if (hasRanges(this.version)) sameRanges = JSON.stringify(ar) === JSON.stringify(br); + else { + const as = wireSpan(ar); + const bs = wireSpan(br); + // A lite-06 `Group Start` of 0 is no floor, so the two spell the same; a pre-06 wire tells + // them apart: omitted joins at the publisher's start, and 0 is group 0. + const start = resolvesStart(this.version) + ? (span: typeof as) => span.start ?? 0 + : (span: typeof as) => span.start; + sameRanges = start(as) === start(bs) && as.end === bs.end; + } // A wire without `Group Order` cannot carry a change to it. const sameOrder = !hasOrder(this.version) || (a.order ?? "descending") === (b.order ?? "descending"); return ( @@ -1835,9 +2002,7 @@ export class Subscriber { maxDelayOf(a) === maxDelayOf(b) && // A wire without `Subscriber Min Age` cannot carry a change to it. (!hasMinDelay(this.version) || minDelayOf(a) === minDelayOf(b)) && - sameStart && - ag.start === bg.start && - ag.end === bg.end + sameRanges ); } diff --git a/js/net/src/lite/tail.test.ts b/js/net/src/lite/tail.test.ts index 0a1771e91f1..7c2affdd1e2 100644 --- a/js/net/src/lite/tail.test.ts +++ b/js/net/src/lite/tail.test.ts @@ -8,7 +8,7 @@ import { createMockTransportPair } from "../mock.ts"; import * as Path from "../path.ts"; import { Reader, Stream } from "../stream.ts"; import { Timescale } from "../time.ts"; -import type { Groups } from "../track.ts"; +import type { Range } from "../track.ts"; import { Group as GroupMessage } from "./group.ts"; import { StreamId } from "./stream.ts"; import { @@ -54,13 +54,13 @@ function groupStream(subscriber: Subscriber, sequence: number) { * it answers TRACK_INFO, then writes whatever responses the test asks for on the subscribe * stream and FINs it when told. */ -async function subscribed(version: Version, maxDelay = GRACE, groups?: Groups) { +async function subscribed(version: Version, maxDelay = GRACE, range?: Range) { const pair = createMockTransportPair(ALPN_05); const subscriber = new Subscriber(pair.client, version, randomHop()); const reader = subscriber .consume(Path.from("room")) .track("video") - .subscribe({ delay: { max: maxDelay }, groups }); + .subscribe({ delay: { max: maxDelay }, ranges: range && [range] }); const info = await Stream.accept(pair.server, version); if (!info) throw new Error("the subscriber never asked for TRACK_INFO"); @@ -111,7 +111,7 @@ async function settlesWithin(promise: Promise, ms: number): Promise { test("the source start waits for SUBSCRIBE_START where it resolves the start", async () => { - const { reader, respond } = await subscribed(version, GRACE, { start: { included: 2 } }); + const { reader, respond } = await subscribed(version, GRACE, { start: { absolute: 2 } }); expect(hooks.sourceStart(reader)).toBe(resolvesStart(version) ? "pending" : undefined); await respond({ start: new SubscribeStart([3]) }); await settle(); @@ -335,7 +335,7 @@ describe.each([Version.DRAFT_05, Version.DRAFT_06, Version.DRAFT_07])("%s", (ver test("a floor below SUBSCRIBE_START owes nothing below it", async () => { const { subscriber, reader, respond, fin } = await subscribed(version, Milli(60_000), { - start: { included: 1 }, + start: { absolute: 1 }, }); await respond({ start: new SubscribeStart([3]) }); const group = groupStream(subscriber, 3); @@ -351,7 +351,7 @@ describe.each([Version.DRAFT_05, Version.DRAFT_06, Version.DRAFT_07])("%s", (ver test("a lowered floor owes the groups it newly asked for", async () => { const maxDelay = Milli(60_000); - const { subscriber, respond, fin } = await subscribed(version, maxDelay, { start: { included: 3 } }); + const { subscriber, respond, fin } = await subscribed(version, maxDelay, { start: { absolute: 3 } }); await respond({ start: new SubscribeStart([3]) }); const group = groupStream(subscriber, 3); group.write("3.0"); @@ -362,7 +362,7 @@ describe.each([Version.DRAFT_05, Version.DRAFT_06, Version.DRAFT_07])("%s", (ver const lower = subscriber .consume(Path.from("room")) .track("video") - .subscribe({ delay: { max: maxDelay }, groups: { start: { included: 1 } } }); + .subscribe({ delay: { max: maxDelay }, ranges: [{ start: { absolute: 1 } }] }); await respond({ end: new SubscribeEnd(4) }); await fin(); // Long enough for a subscription that owed nothing more to have ended. diff --git a/js/net/src/order.test.ts b/js/net/src/order.test.ts index 6c494a7eb29..120e89622f6 100644 --- a/js/net/src/order.test.ts +++ b/js/net/src/order.test.ts @@ -129,12 +129,12 @@ async function backlog(protocol: string, order: Order, update?: Order) { const subscriber = remote.track("video").subscribe({ delay: { max: MAX_DELAY }, order, - groups: { start: { included: 0 } }, + ranges: [{ start: { absolute: 0 } }], }); await receive(subscriber, groups.length); if (update !== undefined) { - subscriber.update({ delay: { max: MAX_DELAY }, order: update, groups: { start: { included: 0 } } }); + subscriber.update({ delay: { max: MAX_DELAY }, order: update, ranges: [{ start: { absolute: 0 } }] }); await aggregate(video, (upstream) => upstream === update); } diff --git a/js/net/src/resume.model.test.ts b/js/net/src/resume.model.test.ts index dbc1d598f38..77a0d8b70e4 100644 --- a/js/net/src/resume.model.test.ts +++ b/js/net/src/resume.model.test.ts @@ -256,11 +256,9 @@ interface Reader { gone: boolean; /** The subscription's group range. */ ranges: Range; - /** - * The one group cursor reads honor. Unlike the Rust reader's two filters, a JS subscriber's - * subscription range only seeds it: an update resets it to that range, and local bounds - * replace it in either direction. - */ + /** The local bounds `withRange` last set, which narrow the subscription's range. */ + local: Range; + /** The groups reads honor: the subscription's range within the local bounds. */ cursor: Range; /** Every group handed out, in order. */ handed: number[]; @@ -290,6 +288,11 @@ function capped(reader: Reader): boolean { return (reader.cursor?.[1] ?? GROUPS) < GROUPS; } +function intersect(a: Range, b: Range): Range { + if (!a || !b) return a ?? b; + return [Math.max(a[0], b[0]), Math.min(a[1], b[1])]; +} + function within(range: Range, group: number): boolean { return range === undefined || (range[0] <= group && group < range[1]); } @@ -310,7 +313,10 @@ function groups(range: Range): track.Groups { } function subscription(budget: Budget, range: Range): track.Subscription { - return { delay: { max: budget === "hour" ? HOUR : SHORT }, groups: range ? groups(range) : undefined }; + return { + delay: { max: budget === "hour" ? HOUR : SHORT }, + ranges: range ? [{ start: { absolute: range[0] }, end: { absolute: range[1] } }] : undefined, + }; } function max(values: Iterable): number | undefined { @@ -413,15 +419,21 @@ class Model { const reader = this.readers[op.reader]; if (!reader) return; reader.ranges = op.range; - reader.cursor = op.range; + reader.cursor = intersect(op.range, reader.local); reader.sub.update(subscription(reader.spec.budget, op.range)); + // A read that ended at its range's end reads on once the range widens. + if (reader.end === null && !this.instances[reader.instance].ended) { + reader.end = undefined; + reader.carried = new Set(); + } this.poll(op.reader); break; } case "bounds": { const reader = this.readers[op.reader]; if (!reader) return; - reader.cursor = op.range; + reader.local = op.range; + reader.cursor = intersect(reader.ranges, op.range); (reader.ordered ?? reader.sub).withRange(groups(op.range)); this.poll(op.reader); break; @@ -566,6 +578,7 @@ class Model { flag: false, gone: false, ranges: spec.range, + local: undefined, cursor: spec.range, handed: [], reading: new Map(), @@ -591,13 +604,20 @@ class Model { } /** Read everything `index` has ready, checking each step against the model. */ - /** The track ended cleanly: a group landing below the end afterwards is not owed, since a reader stops at it. */ + /** + * The read ended cleanly, with the track or, for a bounded range, once that range is read and a + * local cap holds nothing back: a group landing below the end afterwards is not owed, since a + * reader stops at it. + */ #endClean(index: number, reader: Reader): void { const instance = this.instances[reader.instance]; - if (!instance.ended) throw fail(index, "the track ended cleanly though no route declared an end"); + if (!instance.ended && reader.ranges === undefined) { + throw fail(index, "the track ended cleanly though no route declared an end"); + } + const covered = (group: number) => (instance.ended ? true : within(reader.ranges, group)); reader.carried = new Set(); for (let group = 0; group < GROUPS; group++) { - if (instance.live().every((copy) => complete(copy, group))) reader.carried.add(group); + if (covered(group) && instance.live().every((copy) => complete(copy, group))) reader.carried.add(group); } } @@ -763,14 +783,23 @@ class Model { const ended = reader.end === null || reader.parked; if (reader.doomed) { if (reader.end instanceof Error || ((ended || capped(reader)) && instance.ended)) return; + if (reader.end === null && reader.ranges !== undefined) return; throw fail(index, `outlived its instance's routes: ${reader.end}`); } // What every live route carries in full. const carried = (group: number) => instance.live().every((copy) => complete(copy, group)); + // A read that ended its range owed every group in it, whatever its cap, but an ordered one + // passes what its floor keeps out once it reads past it. + const readRange = reader.end === null && !instance.ended; + const owes = (group: number) => + readRange + ? within(reader.ranges, group) && + (!reader.spec.ordered || reader.local === undefined || group >= reader.local[0]) + : covers(reader, group); let owed: number[] = []; for (let group = 0; group < GROUPS; group++) { if (reader.floor !== undefined && group <= reader.floor) continue; - if (covers(reader, group) && carried(group)) owed.push(group); + if (owes(group) && carried(group)) owed.push(group); } if (reader.spec.ordered && before !== undefined) owed = owed.filter((group) => group > before); if (ended) owed = owed.filter((group) => reader.carried.has(group)); @@ -795,7 +824,7 @@ class Model { const ends = (end === undefined && (!live || capped(reader))) || (end === null && live) || - (end === null && !live && instance.ended); + (end === null && !live && (instance.ended || reader.ranges !== undefined)); if (!ends) problems += `track end ${end}; `; if (problems === "") return; throw fail( @@ -901,6 +930,18 @@ test("regressions", async () => { ], finish: true, }, + // Groups in range under the resume point that the local floor keeps out stay owed, so the + // read does not end once the copies read everything above them. Mirrors the Rust case. + { + ops: [ + { kind: "join", reader: 1, spec: spec(false, "hour", undefined, false) }, + { kind: "write", instance: 0, slot: 0, group: 6, frames: 1 }, + { kind: "up", instance: 0, slot: 1 }, + { kind: "ranges", reader: 1, range: [0, 8] }, + { kind: "bounds", reader: 1, range: [3, 8] }, + ], + finish: false, + }, // Local bounds below the subscription's range leave no drops to report. { ops: [ diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 7a418c23a3f..e1ee93c111a 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -273,7 +273,7 @@ test("a routed source start follows the serving copy and its subscriber floor", hooks.startTrack(tb.producer, { start: 5, requested: 3 }); const front = new Front(); front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe({ groups: { start: { included: 3 } } }); + const sub = front.consumer.track("seq").subscribe({ ranges: [{ start: { absolute: 3 } }] }); try { expect(hooks.sourceStart(sub)).toBe(3); const subscribed = tb.producer.subscription.changed(); @@ -288,6 +288,74 @@ test("a routed source start follows the serving copy and its subscriber floor", } }); +test("a routed reader serves its relative ranges as the serving copy resolves them", async () => { + const a = new BroadcastProducer(); + const ta = copy(a); + for (let sequence = 0; sequence < 5; sequence++) ta.write(sequence); + const front = new Front(); + front.serve(a.consume()); + const sub = front.consumer + .track("seq") + .subscribe({ delay: { max: null }, ranges: [{ start: { relative: 2 }, end: { relative: 1 } }] }); + try { + await settle(); + // The copy holds the groups the anchors count back from, so the ranges end there. + expect(hooks.servingEnd(sub)).toBe(4); + expect(hooks.servingRuns(sub, 0, 10)).toEqual([{ start: 2, end: 4 }]); + } finally { + sub.close(); + front.consumer.close(); + a.close(); + } +}); + +// Mirrors the Rust `Reader::exhausted`: the serving copy judges the window from its resume point, +// and a group under it the local floor keeps out stays owed. +test("a bounded read through routes ends once its window is read, across a route change", async () => { + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + for (let sequence = 0; sequence < 3; sequence++) ta.write(sequence); + const front = new Front(); + front.serve(a.consume()); + const sub = front.consumer + .track("seq") + .subscribe({ delay: { max: null }, ranges: [{ start: { absolute: 1 }, end: { absolute: 5 } }] }); + const floored = front.consumer + .track("seq") + .subscribe({ delay: { max: null }, ranges: [{ start: { absolute: 1 }, end: { absolute: 5 } }] }) + .withRange({ start: { included: 3 } }); + try { + expect((await sub.recvGroup())?.sequence).toBe(1); + expect((await sub.recvGroup())?.sequence).toBe(2); + const pending = sub.recvGroup(); + expect(await Promise.race([pending, settle().then(() => "pending")])).toBe("pending"); + for (let sequence = 0; sequence < 5; sequence++) tb.write(sequence); + front.serve(b.consume()); + expect((await pending)?.sequence).toBe(3); + expect((await sub.recvGroup())?.sequence).toBe(4); + // The track goes on, but the window is read. + expect(await sub.recvGroup()).toBeUndefined(); + + expect((await floored.recvGroup())?.sequence).toBe(3); + expect((await floored.recvGroup())?.sequence).toBe(4); + // Groups 1 and 2 are in range under the floor, which may still drop. + const held = floored.recvGroup(); + expect(await Promise.race([held, settle().then(() => "pending")])).toBe("pending"); + floored.withRange({}); + expect((await held)?.sequence).toBe(1); + expect((await floored.recvGroup())?.sequence).toBe(2); + expect(await floored.recvGroup()).toBeUndefined(); + } finally { + sub.close(); + floored.close(); + front.consumer.close(); + a.close(); + b.close(); + } +}); + test("a fork continues its open group across a route failure after its parent closes", async () => { const a = new BroadcastProducer(); const b = new BroadcastProducer(); @@ -350,7 +418,7 @@ test("a resumed fork has its own preferences and replay cursor", async () => { front.serve(a.consume()); const parent = front.consumer.track("seq").subscribe({ priority: 2, - groups: { start: { included: 1 } }, + ranges: [{ start: { absolute: 1 } }], }); ta.write(0); ta.write(1); @@ -2101,7 +2169,7 @@ test("the largest honours the reader's bounds on the serving copy", async () => .track("seq") .subscribe() .withRange({ end: { excluded: 3 } }); - const floored = front.consumer.track("seq").subscribe({ groups: { start: { included: 8 } } }); + const floored = front.consumer.track("seq").subscribe({ ranges: [{ start: { absolute: 8 } }] }); for (let sequence = 0; sequence < 5; sequence++) ta.write(sequence); const next = capped.largest(); @@ -2784,7 +2852,7 @@ test.each([ } return sequences; }; - const asked = () => tb.producer.subscription.peek()?.groups?.start?.included; + const asked = () => tb.producer.subscription.peek()?.ranges?.[0]?.start?.absolute; try { sub.withRange({ start: { included: 4 } }); await clock.advance(Nano.zero); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 87dd8e8f3c4..9c9af64eea6 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -55,7 +55,6 @@ import { type DropRun, finalSequence, type GroupRecover, - groupBounds, hooks, insertDrop, insertRun, @@ -73,6 +72,7 @@ import { import { grace } from "./ledger.ts"; import type { Lister, Tracks, TracksEvent } from "./listing.ts"; import * as track from "./track.ts"; +import { clamped, type Window } from "./window.ts"; import { type Broadcast as BroadcastWire, registerWire, wireOf } from "./wire.ts"; // Stream codes that report the route failing to deliver rather than judging what it carried: @@ -270,14 +270,18 @@ interface Copy { lease: number; } -/** Subscribe to a route's copy, or to a failed stand-in when that route can no longer serve. */ +/** + * Subscribe to a route's copy, or to a failed stand-in when that route can no longer serve. A + * copy parks past the end of a bounded window: the logical reader judges that end across its + * copies; see {@link Reader.exhausted}. + */ function subscribe(wire: BroadcastWire, name: string, subscription: track.Subscription): track.Subscriber { try { - return wire.subscribe(name, subscription); + return hooks.parked(wire.subscribe(name, subscription)); } catch (err) { const failed = new track.Producer(name); failed.abort(err instanceof Error ? err : new Error(String(err))); - return failed.subscribe(subscription); + return hooks.parked(failed.subscribe(subscription)); } } @@ -486,17 +490,9 @@ class Reader implements ResumeSource { /** What `copy` is asked for: the reader's own preferences, from its floor and through its replacement. */ #wire(copy: Pick): track.Subscription { const preferences = this.#shell.subscription.peek() ?? track.Subscription.default(); - const bounds = groupBounds(preferences.groups); - const start = Math.max(bounds.start, copy.floor ?? 0); - const replaced = copyEnd(copy); - const end = replaced === undefined ? bounds.end : Math.min(bounds.end ?? replaced, replaced); - return { - ...preferences, - groups: { - start: start > 0 ? { included: start } : undefined, - end: end === undefined ? undefined : { excluded: end }, - }, - }; + const floor = copy.floor ? copy.floor : undefined; + // The copies resolve relative anchors against the groups they hold. + return { ...preferences, ranges: clamped(preferences.ranges ?? [{}], floor, copyEnd(copy), []) }; } #update(copy: Copy): void { @@ -690,6 +686,35 @@ class Reader implements ResumeSource { return this.#end((copy) => copy.done); } + /** + * Whether the reader's bounded `window` has nothing left to deliver, once no replaced copy may + * still deliver a group under its bound. The serving copy reads every group of the window from + * its resume point, so it judges those, against what it and the reader (`delivered`) handed + * out. What lies under its resume point was the copies' it replaced, each let go once it had + * nothing more, though one there that the local floor or cap keeps out stays owed, as on one + * cursor. A backfill still to ask is not judged yet, and a route between copies has nothing to + * judge. Mirrors the Rust `Reader::exhausted`. + */ + exhausted(delivered: Delivered, window: Window): boolean { + this.sync(); + if (this.#closed || this.#ended || this.#resumed) return false; + const serving = this.#copies.at(-1); + if (!serving || serving.replaced || serving.done !== undefined || serving.generation !== this.#generation) { + return false; + } + if (this.#letGo(delivered)) return false; + const end = window.end(); + if (serving.floor !== undefined && end !== undefined) { + const cursor = this.#shell.cursor.peek(); + for (const run of window.runs(window.floor() ?? 0, Math.min(serving.floor, end))) { + for (const gap of delivered.gaps(run.start, run.end)) { + if (gap.start < cursor.start || (cursor.end !== undefined && gap.end > cursor.end)) return false; + } + } + } + return hooks.exhaustedFor(serving.sub, delivered); + } + tryDrop(delivered: Delivered): (Recv & { kind: "drop" }) | undefined { this.#delivered = delivered; this.sync(); @@ -982,7 +1007,7 @@ class Reader implements ResumeSource { * Whether `within` passed since `copy` was superseded; see {@link Copy.since}. Arms a wakeup * for then, so a read waiting on the copy looks again. */ - #expired(copy: Copy, within: Milli): boolean { + #expired(copy: Copy, within: number): boolean { if (copy.since === undefined) return false; // Plain arithmetic: an unenforced budget is infinite, which no duration brand holds. const at = copy.since + within * 1_000_000; @@ -1141,6 +1166,26 @@ class Reader implements ResumeSource { return this.#learnEnd(); } + // Each copy asks for the reader's ranges in turn, clamped to its floor and replacement. + starts(from: number): (number | undefined)[] | undefined { + this.sync(); + const serving = this.#serving(); + return serving && hooks.servingStarts(serving.sub, from); + } + + // The reader's ranges resolve on the copies, which hold the groups their anchors name. + runs(start: number, end: number): Run[] | undefined { + this.sync(); + const serving = this.#serving(); + return serving && hooks.servingRuns(serving.sub, start, end); + } + + servingEnd(): number | undefined { + this.sync(); + const serving = this.#serving(); + return serving && hooks.servingEnd(serving.sub); + } + /** The error the logical track's end carries, if it does not end cleanly. */ endError(): Error | undefined { this.sync(); diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 5cc81da4918..19a22c67693 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -492,7 +492,7 @@ test("joins without refresh ask for no group", () => { test("a fixed start is no join", () => { const producer = new TrackProducer("test").accept({ refresh: 1 }); refreshGroup(producer, 4); - const subscriber = producer.subscribe({ groups: { start: { included: 0 } } }); + const subscriber = producer.subscribe({ ranges: [{ start: { absolute: 0 } }] }); expect(producer.demand().wanted.peek()).toBe(0); subscriber.close(); }); @@ -701,14 +701,14 @@ test("appendDatagram rejects a payload over the QUIC datagram frame ceiling", () test("subscriber options and updates are forwarded to the producer's aggregate", async () => { const producer = new TrackProducer("test"); - const track = producer.subscribe({ delay: { max: Milli.zero }, groups: { start: { included: 0 } } }); + const track = producer.subscribe({ delay: { max: Milli.zero }, ranges: [{ start: { absolute: 0 } }] }); // The initial options are available before the request is accepted or put on the wire. expect(producer.subscription.peek()).toEqual({ priority: 0, delay: { min: Milli(0), max: Milli(0) }, order: "descending", - groups: { start: { included: 0 } }, + ranges: [{ start: { absolute: 0 } }], }); // The wire layer watches the producer's signal to emit SUBSCRIBE_UPDATE. @@ -717,13 +717,13 @@ test("subscriber options and updates are forwarded to the producer's aggregate", priority: 7, delay: { max: Milli(250) }, order: "ascending", - groups: { start: { included: 2 }, end: { excluded: 9 } }, + ranges: [{ start: { absolute: 2 }, end: { absolute: 9 } }], }); expect(await next).toEqual({ priority: 7, delay: { min: Milli(0), max: Milli(250) }, order: "ascending", - groups: { start: { included: 2 }, end: { excluded: 9 } }, + ranges: [{ start: { absolute: 2 }, end: { absolute: 9 } }], }); }); @@ -818,15 +818,15 @@ test("multiple subscriber options aggregate like Rust", async () => { priority: 2, delay: { min: Milli(400), max: Milli(100) }, order: "ascending", - groups: { start: { included: 10 }, end: { excluded: 20 } }, + ranges: [{ start: { absolute: 10 }, end: { absolute: 20 } }], }); const live = producer.subscribe({ priority: 7, delay: { min: Milli(50), max: Milli(250) }, - groups: { start: { included: 5 } }, + ranges: [{ start: { absolute: 5 } }], }); // A minimum that went unstated, as a legacy peer's does, never pulls the smallest to zero. - const legacy = producer.subscribe({ delay: { min: null, max: Milli(0) }, groups: { start: { included: 5 } } }); + const legacy = producer.subscribe({ delay: { min: null, max: Milli(0) }, ranges: [{ start: { absolute: 5 } }] }); // One descending subscriber makes the aggregate descending, so a live reader is never held // behind another's backlog. @@ -834,7 +834,7 @@ test("multiple subscriber options aggregate like Rust", async () => { priority: 7, delay: { min: Milli(50), max: Milli(250) }, order: "descending", - groups: { start: { included: 5 } }, + ranges: [{ start: { absolute: 5 } }], }); legacy.close(); @@ -844,7 +844,7 @@ test("multiple subscriber options aggregate like Rust", async () => { priority: 2, delay: { min: Milli(400), max: Milli(100) }, order: "ascending", - groups: { start: { included: 10 }, end: { excluded: 20 } }, + ranges: [{ start: { absolute: 10 }, end: { absolute: 20 } }], }); const none = producer.subscription.changed(); @@ -852,6 +852,282 @@ test("multiple subscriber options aggregate like Rust", async () => { expect(await none).toBeUndefined(); }); +test("each reader reads only its own ranges, whatever the union asks for", () => { + const producer = new TrackProducer("test"); + const sparse = producer.subscribe({ + delay: { max: null }, + ranges: [ + { start: { absolute: 1 }, end: { absolute: 3 } }, + { start: { absolute: 5 }, end: { absolute: 6 } }, + ], + }); + const tail = producer.subscribe({ delay: { max: null }, ranges: [{ start: { absolute: 4 } }] }); + expect(producer.subscription.peek()?.ranges).toEqual([ + { start: { absolute: 1 }, end: { absolute: 3 } }, + { start: { absolute: 4 } }, + ]); + for (let sequence = 0; sequence < 7; sequence++) producer.createGroup(sequence).finish(); + + const read = (subscriber: typeof sparse) => { + const got: number[] = []; + for (let group = subscriber.tryRecvGroup(); group; group = subscriber.tryRecvGroup()) got.push(group.sequence); + return got; + }; + expect(read(sparse)).toEqual([1, 2, 5]); + expect(read(tail)).toEqual([4, 5, 6]); + + // Replacing the ranges offers the held groups the new ones take in, never one read already. + sparse.update({ delay: { max: null }, ranges: [{ start: { absolute: 0 }, end: { absolute: 5 } }] }); + expect(read(sparse)).toEqual([0, 3, 4]); + producer.finish(); + expect(hooks.tryRecvGroup(sparse, false)).toEqual({ kind: "done" }); +}); + +// Mirrors the Rust `a_bounded_read_ends_once_its_window_is_read` and its siblings: a read whose +// ranges all end finishes once nothing in them is left, though the track goes on. +test("a bounded read ends once its window is read, unless a cap or a wire publisher holds it", () => { + const producer = new TrackProducer("test"); + const ranges = [{ start: { absolute: 1 }, end: { absolute: 5 } }]; + const read = (subscriber: ReturnType) => { + const got: number[] = []; + for (let group = subscriber.tryRecvGroup(); group; group = subscriber.tryRecvGroup()) got.push(group.sequence); + return got; + }; + const reader = producer.subscribe({ delay: { max: null }, ranges }); + const sequenced = producer.subscribe({ delay: { max: null }, ranges }); + const ordered = sequenced.ordered(); + const capped = producer.subscribe({ delay: { max: null }, ranges }).withRange({ end: { excluded: 3 } }); + const parked = hooks.parked(producer.subscribe({ delay: { max: null }, ranges })); + const open = producer.subscribe({ delay: { max: null }, ranges: [{ start: { absolute: 1 } }] }); + for (const sequence of [0, 1, 2]) producer.createGroup(sequence).finish(); + expect(read(reader)).toEqual([1, 2]); + // Groups 3 and 4 may still arrive. + expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "idle" }); + + hooks.dropTrackGroups(producer, 3, 4, DROPPED); + producer.createGroup(4).finish(); + expect(read(reader)).toEqual([4]); + expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "done" }); + expect(drainOrdered(ordered)).toEqual([1, 2, 4]); + expect(hooks.tryNextGroup(sequenced)).toEqual({ kind: "done" }); + // A wire publisher's peer would read the end of the stream as the track's. + expect(read(parked)).toEqual([1, 2, 4]); + expect(hooks.tryRecvGroup(parked, false)).toEqual({ kind: "idle" }); + expect(read(open)).toEqual([1, 2, 4]); + expect(hooks.tryRecvGroup(open, false)).toEqual({ kind: "idle" }); + // The cap keeps group 4 out, and may rise. + expect(read(capped)).toEqual([1, 2]); + expect(hooks.tryRecvGroup(capped, false).kind).not.toBe("done"); + capped.withRange({}); + expect(read(capped)).toEqual([4]); + expect(hooks.tryRecvGroup(capped, false)).toEqual({ kind: "done" }); + + // Replacing the ranges reads on. + reader.update({ delay: { max: null }, ranges: [{ start: { absolute: 0 }, end: { absolute: 6 } }] }); + expect(read(reader)).toEqual([0]); + expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "idle" }); + producer.createGroup(5).finish(); + expect(read(reader)).toEqual([5]); + expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "done" }); +}); + +// A subscribed copy owes a hole until its feed gives up on it, as Rust's bounded wait does. +test("a bounded read on a subscribed copy ends once its feed stops owing a hole", async () => { + const clock = manualTime(0); + try { + const producer = receivedTrack(); + const reader = producer.subscribe({ delay: { max: Milli(1000) }, ranges: [{ end: { absolute: 3 } }] }); + putGroup(producer, 0); + putGroup(producer, 2); + expect(reader.tryRecvGroup()?.sequence).toBe(0); + expect(reader.tryRecvGroup()?.sequence).toBe(2); + const ended = reader.recvGroup(); + await clock.advance(999); + expect(await Promise.race([ended, Promise.resolve("waiting")])).toBe("waiting"); + await clock.advance(1000); + expect(await ended).toBeUndefined(); + } finally { + clock.installed.close(); + } +}); + +test("an ordered reader keeps a group between its ranges until the cursor passes it", () => { + const producer = new TrackProducer("test"); + const track = producer + .subscribe({ + delay: { max: null }, + ranges: [ + { start: { absolute: 0 }, end: { absolute: 1 } }, + { start: { absolute: 3 }, end: { absolute: 4 } }, + ], + }) + .ordered(); + for (const sequence of [0, 1]) producer.createGroup(sequence).finish(); + expect(track.tryNextGroup()?.sequence).toBe(0); + expect(track.tryNextGroup()).toBeUndefined(); + + track.update({ delay: { max: null }, ranges: [{ start: { absolute: 0 }, end: { absolute: 4 } }] }); + expect(track.tryNextGroup()?.sequence).toBe(1); + track.close(); + producer.finish(); + producer.close(); +}); + +test("a relative range counts back from the latest group, or waits for the first", () => { + const producer = new TrackProducer("test"); + for (let sequence = 0; sequence < 5; sequence++) producer.createGroup(sequence).finish(); + const back = producer.subscribe({ + delay: { max: null }, + ranges: [{ start: { relative: 2 }, end: { relative: 1 } }], + }); + expect([back.tryRecvGroup()?.sequence, back.tryRecvGroup()?.sequence, back.tryRecvGroup()]).toEqual([ + 2, + 3, + undefined, + ]); + + const empty = new TrackProducer("test"); + const live = empty.subscribe({ delay: { max: null }, ranges: [{ start: { relative: 0 } }] }); + empty.createGroup(7).finish(); + empty.createGroup(6).finish(); + empty.createGroup(8).finish(); + // Resolved at the first group to arrive: group 6 sits below where it started. + expect([live.tryRecvGroup()?.sequence, live.tryRecvGroup()?.sequence, live.tryRecvGroup()]).toEqual([ + 7, + 8, + undefined, + ]); +}); + +test("a copy over a wire without ranges refuses a relative anchor it cannot carry", () => { + const producer = new TrackProducer("test"); + const back = producer.subscribe({ ranges: [{ start: { relative: 3 } }] }); + const live = producer.subscribe({ ranges: [{ start: { relative: 0 } }] }); + hooks.dropAnchors(producer); + producer.createGroup(0).finish(); + expect(() => back.tryRecvGroup()).toThrow("moq-lite-07"); + expect(live.tryRecvGroup()?.sequence).toBe(0); +}); + +test("a time range reads the groups containing its times, settling its end as groups start", () => { + const producer = new TrackProducer("test"); + producer.accept({ timescale: Timescale.MILLI }); + const start = (sequence: number, millis: number) => { + const group = producer.createGroup(sequence); + group.writeFrame(textFrame(`${sequence}`, Timestamp.fromMillis(millis))); + return group; + }; + for (let sequence = 0; sequence < 3; sequence++) start(sequence, sequence * 1000).finish(); + const reader = producer.subscribe({ + delay: { max: null }, + ranges: [{ start: { time: Timestamp.fromMillis(1500) }, end: { time: Timestamp.fromMillis(3500) } }], + }); + const read = () => { + const got: number[] = []; + for (let group = reader.tryRecvGroup(); group; group = reader.tryRecvGroup()) got.push(group.sequence); + return got; + }; + // Group 1 contains 1500, and group 3, the next to arrive, may still start inside the end. + expect(read()).toEqual([1, 2]); + const three = producer.createGroup(3); + expect(read()).toEqual([3]); + // Group 4 waits until group 3 shows where it starts. + const four = producer.createGroup(4); + expect(read()).toEqual([]); + three.writeFrame(textFrame("3", Timestamp.fromMillis(3000))); + expect(read()).toEqual([4]); + // Group 4 starts past the end, which settles before it, though it was already read on speculation. + producer.createGroup(5).writeFrame(textFrame("5", Timestamp.fromMillis(5000))); + expect(read()).toEqual([]); + expect(hooks.servingEnd(reader)).toBeUndefined(); + four.writeFrame(textFrame("4", Timestamp.fromMillis(4000))); + expect(read()).toEqual([]); + expect(hooks.servingEnd(reader)).toBe(4); +}); + +test("a datagram is judged against a time range by its own timestamp", () => { + const producer = new TrackProducer("test"); + producer.accept({ timescale: Timescale.MILLI }); + const reader = producer.subscribe({ + ranges: [{ start: { absolute: 0 }, end: { time: Timestamp.fromMillis(3500) } }], + }); + producer.insertDatagram(0, Timestamp.fromMillis(3600), enc.encode("late")); + producer.insertDatagram(1, Timestamp.fromMillis(3400), enc.encode("in")); + const recv = hooks.tryRecvDatagram(reader); + expect(recv.kind === "datagram" && dec.decode(recv.datagram.payload)).toBe("in"); + expect(hooks.tryRecvDatagram(reader)).toEqual({ kind: "idle" }); +}); + +// A reached end leaves a held group that may still start, so a time start still waits on it. +test("a time start waits on groups below a reached end that have yet to start", () => { + for (const order of ["ascending", "descending"] as const) { + const producer = new TrackProducer("test"); + producer.accept({ timescale: Timescale.MILLI }); + const groups = [0, 1, 2].map((sequence) => producer.createGroup(sequence)); + producer.finishAt({ group: 3, frame: 0 }); + const subscription = { delay: { max: null }, order, ranges: [{ start: { time: Timestamp.fromMillis(1500) } }] }; + const reader = producer.subscribe(subscription); + const sequenced = producer.subscribe(subscription); + const ordered = sequenced.ordered(); + // Idle rather than done: the groups may still start inside the window. + expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "idle" }); + expect(hooks.tryNextGroup(sequenced)).toEqual({ kind: "idle" }); + for (const [sequence, group] of groups.entries()) { + group.writeFrame(textFrame(`${sequence}`, Timestamp.fromMillis(sequence * 1000))); + } + const got: number[] = []; + for (let group = reader.tryRecvGroup(); group; group = reader.tryRecvGroup()) got.push(group.sequence); + expect(got.sort()).toEqual([1, 2]); + const inOrder: number[] = []; + for (let group = ordered.tryNextGroup(); group; group = ordered.tryNextGroup()) inOrder.push(group.sequence); + expect(inOrder).toEqual([1, 2]); + } +}); + +// A drop outside a time end's cap is reported once the end settles past it. +test("a drop a time end settles over is reported", () => { + const producer = new TrackProducer("test"); + producer.accept({ timescale: Timescale.MILLI }); + const zero = producer.createGroup(0); + const reader = producer.subscribe({ + delay: { max: null }, + order: "ascending", + ranges: [{ start: { absolute: 0 }, end: { time: Timestamp.fromMillis(500) } }, { start: { absolute: 10 } }], + }); + expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 0 } }); + // Past the cap while group 0 has yet to start. + hooks.dropTrackGroups(producer, 1, 2, DROPPED); + expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "idle" }); + zero.writeFrame(textFrame("0", Timestamp.fromMillis(0))); + producer.createGroup(2).writeFrame(textFrame("2", Timestamp.fromMillis(2000))); + expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", start: 1, end: 2, error: DROPPED }); +}); + +test("an untimed track or a moq-transport copy refuses a time anchor", () => { + const ranges = [{ start: { time: Timestamp.fromMillis(0) } }]; + const untimed = new TrackProducer("test"); + untimed.accept({}); + const reader = untimed.subscribe({ ranges }); + untimed.createGroup(0).finish(); + expect(() => reader.tryRecvGroup()).toThrow(TimestampMismatch); + + const copy = new TrackProducer("test"); + copy.accept({ timescale: Timescale.MILLI, ietf: IetfVersion.DRAFT_18 }); + expect(() => copy.subscribe({ ranges }).tryRecvGroup()).toThrow("moq-transport"); +}); + +// Waiting reads see the refusal once the track is accepted, with no group or datagram arriving. +test("accepting an untimed track refuses a waiting read's time anchor", async () => { + const producer = new TrackProducer("test"); + const reader = producer.subscribe({ ranges: [{ start: { time: Timestamp.fromMillis(0) } }] }); + const group = reader.recvGroup(); + const datagram = reader.recvDatagram(); + producer.accept({}); + const [grouped, received] = await Promise.allSettled([group, datagram]); + expect(grouped).toMatchObject({ status: "rejected", reason: expect.any(TimestampMismatch) }); + expect(received).toMatchObject({ status: "rejected", reason: expect.any(TimestampMismatch) }); +}); + test("only unstated minimums aggregate to an unstated one", () => { const producer = new TrackProducer("test"); producer.subscribe({ delay: { min: null } }); @@ -863,9 +1139,9 @@ test("only unstated minimums aggregate to an unstated one", () => { // joins after it: on a quiet track that group may be the only one for a long time. test("an unfloored subscriber clears a resume floor above the live edge", () => { const producer = new TrackProducer("test"); - producer.subscribe({ groups: { start: { included: 4 } } }); + producer.subscribe({ ranges: [{ start: { absolute: 4 } }] }); producer.subscribe({}); - expect(producer.subscription.peek()?.groups?.start).toBeUndefined(); + expect(producer.subscription.peek()?.ranges).toEqual([{}]); }); test("the producer aggregate is not clamped to the track's maxAge", () => { @@ -884,6 +1160,14 @@ test("the producer aggregate is not clamped to the track's maxAge", () => { expect(hooks.tailGrace(producer)).toBe(Milli(2_000)); }); +test("a null delay.max dominates when subscriptions combine", () => { + const producer = new TrackProducer("test"); + producer.subscribe({ delay: { max: null } }); + producer.subscribe({ delay: { max: Milli(250) } }); + // The most tolerant subscriber wins the aggregate, so no limit dominates. + expect(producer.subscription.peek()?.delay?.max).toBe(Milli(Number.MAX_SAFE_INTEGER)); +}); + test("nextGroup skips late arrivals", async () => { const producer = new TrackProducer("test"); const track = producer.subscribe().ordered(); @@ -1218,13 +1502,13 @@ test("a named start is a floor, not a request", async () => { // The budget is the only thing that asks for data; a named start only bounds how far // back it may reach. - const floored = producer.subscribe({ groups: { start: { included: 3 } }, delay: { max: Milli(60_000) } }); + const floored = producer.subscribe({ ranges: [{ start: { absolute: 3 } }], delay: { max: Milli(60_000) } }); expect((await floored.recvGroup())?.sequence).toBe(3); expect((await floored.recvGroup())?.sequence).toBe(4); // Naming group 1 at real time still delivers the live edge alone: the zero budget // calls everything older stale. - const named = producer.subscribe({ delay: { max: Milli.zero }, groups: { start: { included: 1 } } }); + const named = producer.subscribe({ delay: { max: Milli.zero }, ranges: [{ start: { absolute: 1 } }] }); expect((await named.recvGroup())?.sequence).toBe(4); }); @@ -1997,7 +2281,7 @@ test.each(["arrival", "ordered"] as const)("subscription bounds stay reader-loca const everything = producer.subscribe({ delay: { max: Milli(5000) } }); const subscriber = producer.subscribe({ delay: { max: Milli(5000) }, - groups: { start: { included: 1 }, end: { excluded: 3 } }, + ranges: [{ start: { absolute: 1 }, end: { absolute: 3 } }], }); const ordered = order === "ordered" ? subscriber.ordered() : undefined; const read = () => (ordered ? ordered.nextGroup() : subscriber.recvGroup()); @@ -2006,19 +2290,24 @@ test.each(["arrival", "ordered"] as const)("subscription bounds stay reader-loca expect((await everything.recvGroup())?.sequence).toBe(0); expect((await read())?.sequence).toBe(1); expect((await read())?.sequence).toBe(2); + // A bounded read ends once its window is read, as in Rust, though the track goes on. + expect(await read()).toBeUndefined(); + // Replacing the ranges reads on from the same reader. + subscriber.update({ delay: { max: Milli(5000) }, ranges: [{ start: { absolute: 3 }, end: { absolute: 4 } }] }); + expect((await read())?.sequence).toBe(3); + expect(await read()).toBeUndefined(); + subscriber.update({ delay: { max: Milli(5000) }, ranges: [{ end: { absolute: 5 } }] }); + // Arrival readers can now offer the unread group below the old floor; ordered ones cannot rewind. + if (!ordered) expect((await read())?.sequence).toBe(0); + expect((await read())?.sequence).toBe(4); + expect(await read()).toBeUndefined(); + // An open range parks at the live edge instead. + subscriber.update({ delay: { max: Milli(5000) }, ranges: [{ start: { absolute: 5 } }] }); const pending = read(); expect(await Promise.race([pending, Promise.resolve("pending")])).toBe("pending"); - subscriber.update({ delay: { max: Milli(5000) }, groups: { start: { included: 3 }, end: { excluded: 4 } } }); - expect((await pending)?.sequence).toBe(3); - const capped = read(); - expect(await Promise.race([capped, Promise.resolve("pending")])).toBe("pending"); - subscriber.update({ delay: { max: Milli(5000) }, groups: { end: { included: 4 } } }); - // Arrival readers can now offer the unread group below the old floor; ordered ones cannot rewind. - if (!ordered) expect((await capped)?.sequence).toBe(0); - expect((await (ordered ? capped : read()))?.sequence).toBe(4); producer.finish(); producer.close(); - expect(await read()).toBeUndefined(); + expect(await pending).toBeUndefined(); } finally { subscriber.close(); everything.close(); @@ -2033,12 +2322,10 @@ test("subscription updates narrow the cursor before its first read", async () => const track = producer.subscribe({ delay: { max: Milli(5000) } }); try { for (let sequence = 0; sequence < 5; sequence++) producer.createGroup(sequence); - track.update({ delay: { max: Milli(5000) }, groups: { start: { excluded: 2 }, end: { included: 3 } } }); + track.update({ delay: { max: Milli(5000) }, ranges: [{ start: { absolute: 3 }, end: { absolute: 4 } }] }); expect((await track.recvGroup())?.sequence).toBe(3); - const pending = track.recvGroup(); - expect(await Promise.race([pending, Promise.resolve("pending")])).toBe("pending"); - track.close(); - await pending; + // Its window read, the bounded read ends. + expect(await track.recvGroup()).toBeUndefined(); } finally { track.close(); everything.close(); @@ -2050,7 +2337,7 @@ test("subscription updates narrow the cursor before its first read", async () => test("a subscription cap chooses its own live edge", async () => { const producer = new TrackProducer("test").accept({ timescale: Timescale.MILLI, maxAge: Milli(5000) }); const everything = producer.subscribe({ delay: { max: Milli(5000) } }); - const track = producer.subscribe({ delay: { max: Milli.zero }, groups: { end: { excluded: 2 } } }); + const track = producer.subscribe({ delay: { max: Milli.zero }, ranges: [{ end: { absolute: 2 } }] }); try { for (let sequence = 0; sequence < 5; sequence++) { producer.writeFrame({ @@ -2059,7 +2346,7 @@ test("a subscription cap chooses its own live edge", async () => { }); } expect((await track.recvGroup())?.sequence).toBe(1); - track.update({ delay: { max: Milli.zero }, groups: { start: { included: 3 }, end: { excluded: 4 } } }); + track.update({ delay: { max: Milli.zero }, ranges: [{ start: { absolute: 3 }, end: { absolute: 4 } }] }); expect((await track.recvGroup())?.sequence).toBe(3); } finally { track.close(); @@ -2072,12 +2359,12 @@ test("a subscription cap chooses its own live edge", async () => { test("subscription updates bound each reader's datagrams", async () => { const producer = new TrackProducer("test"); const everything = producer.subscribe({ delay: { max: Milli(5000) } }); - const track = producer.subscribe({ groups: { start: { included: 1 }, end: { excluded: 2 } } }); + const track = producer.subscribe({ ranges: [{ start: { absolute: 1 }, end: { absolute: 2 } }] }); try { for (let sequence = 0; sequence < 4; sequence++) producer.insertDatagram(sequence, undefined, new Uint8Array([sequence])); expect((await track.recvDatagram())?.sequence).toBe(1); - track.update({ groups: { start: { included: 3 }, end: { excluded: 4 } } }); + track.update({ ranges: [{ start: { absolute: 3 }, end: { absolute: 4 } }] }); for (let sequence = 2; sequence < 5; sequence++) producer.insertDatagram(sequence, undefined, new Uint8Array([sequence])); expect((await track.recvDatagram())?.sequence).toBe(3); @@ -2096,8 +2383,7 @@ test("subscription updates bound each reader's datagrams", async () => { test("withRange with an excluded end of 0 is the empty range and does not ride another subscriber's demand", async () => { const producer = new TrackProducer("test"); const everything = producer.subscribe({ delay: { max: Milli(5000) } }); - const empty = producer.subscribe({ delay: { max: Milli(5000) }, groups: { end: { excluded: 0 } } }); - empty.withRange({ end: { excluded: 0 } }); + const empty = producer.subscribe({ delay: { max: Milli(5000) } }).withRange({ end: { excluded: 0 } }); for (let sequence = 0; sequence < 3; sequence++) producer.createGroup(sequence); @@ -2771,7 +3057,7 @@ test("group ranges preserve progress and spell out inclusion", async () => { // Mirrors the Rust `a_lowered_range_reoffers_the_unread_cached_groups`. test("a lowered range re-offers the unread cached groups", () => { const producer = new TrackProducer("test"); - const track = producer.subscribe({ delay: { max: Milli(5000) }, groups: { start: { included: 10 } } }); + const track = producer.subscribe({ delay: { max: Milli(5000) } }).withRange({ start: { included: 10 } }); for (const sequence of [5, 6, 10]) producer.createGroup(sequence); expect(track.tryRecvGroup()?.sequence).toBe(10); expect(track.tryRecvGroup()).toBeUndefined(); @@ -2789,7 +3075,7 @@ test("a lowered range re-offers the unread cached groups", () => { test("a lowered range re-offers a cached cut with its error", async () => { const producer = new TrackProducer("test"); - const track = producer.subscribe({ delay: { max: Milli(5000) }, groups: { start: { included: 1 } } }); + const track = producer.subscribe({ delay: { max: Milli(5000) } }).withRange({ start: { included: 1 } }); const cut = producer.createGroup(0); const error = new StreamError(StreamCode(70), { message: "cut" }); cut.abort(error); @@ -2807,7 +3093,7 @@ test("a lowered range re-offers a cached cut with its error", async () => { test("a lowered range still re-offers after the producer closes", () => { const producer = new TrackProducer("test"); - const track = producer.subscribe({ delay: { max: Milli(5000) }, groups: { start: { included: 1 } } }); + const track = producer.subscribe({ delay: { max: Milli(5000) } }).withRange({ start: { included: 1 } }); for (const sequence of [0, 1]) producer.createGroup(sequence).finish(); expect(track.tryRecvGroup()?.sequence).toBe(1); producer.finish(); @@ -2822,7 +3108,7 @@ test("a lowered range still re-offers after the producer closes", () => { for (const abort of [undefined, new Error("gone")]) { test(`a lowered range offers nothing to a reader that ${abort ? "aborted" : "closed"}`, () => { const producer = new TrackProducer("test"); - const track = producer.subscribe({ delay: { max: Milli(5000) }, groups: { start: { included: 1 } } }); + const track = producer.subscribe({ delay: { max: Milli(5000) } }).withRange({ start: { included: 1 } }); for (const sequence of [0, 1]) producer.createGroup(sequence); expect(track.tryRecvGroup()?.sequence).toBe(1); track.close(abort); @@ -3379,7 +3665,7 @@ test("an untimed subscriber starts at the oldest fresh group", () => { test("an untimed subscriber without a start begins at the latest group under its end", () => { const producer = untimedTrack(5); // As the lite publisher serves a SUBSCRIBE with an end and no start. - const subscriber = producer.subscribe({ delay: { max: Milli.zero }, groups: { end: { excluded: 3 } } }); + const subscriber = producer.subscribe({ delay: { max: Milli.zero }, ranges: [{ end: { absolute: 3 } }] }); subscriber.withRange({ end: { excluded: 3 } }); expect(drain(subscriber)).toEqual([2]); }); @@ -3417,9 +3703,9 @@ test("a single-frame write that throws aborts its group", () => { test("an explicit start floors an untimed track", () => { const producer = untimedTrack(5); - const resumed = producer.subscribe({ delay: { max: Milli(60_000) }, groups: { start: { included: 2 } } }); + const resumed = producer.subscribe({ delay: { max: Milli(60_000) }, ranges: [{ start: { absolute: 2 } }] }); expect(drain(resumed)).toEqual([2, 3, 4]); - const live = producer.subscribe({ delay: { max: Milli.zero }, groups: { start: { included: 2 } } }); + const live = producer.subscribe({ delay: { max: Milli.zero }, ranges: [{ start: { absolute: 2 } }] }); expect(drain(live)).toEqual([4]); }); @@ -3492,7 +3778,7 @@ test("an upstream drop ends a group still arriving with its error", async () => test("a drop below the window is reported once the window lowers to it", () => { const producer = new TrackProducer("video").accept({}); - const reader = producer.subscribe({ delay: { max: Milli(60_000) }, groups: { start: { included: 5 } } }); + const reader = producer.subscribe({ delay: { max: Milli(60_000) } }).withRange({ start: { included: 5 } }); hooks.dropTrackGroups(producer, 2, 3, DROPPED); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "idle" }); @@ -3876,7 +4162,7 @@ test("an untimed track takes groups in any order", () => { test("replay and backlogs deliver sparse out-of-order arrivals ascending without sorting", () => { const producer = new TrackProducer("test").accept({}); - const all = { groups: { start: { included: 0 } }, delay: { max: null } }; + const all = { ranges: [{ start: { absolute: 0 } }], delay: { max: null } }; const write = (sequence: number) => { const group = producer.createGroup(sequence); group.writeFrame({ payload: enc.encode(`${sequence}`) }); @@ -3894,8 +4180,12 @@ test("replay and backlogs deliver sparse out-of-order arrivals ascending without const sort = spyOn(Array.prototype, "sort"); try { const live = producer.subscribe(all); + // Subscribing merges the subscribers' ranges, which sorts them; only arrivals are judged. + sort.mockClear(); for (const sequence of [40, 10, 30, 0, 20]) write(sequence); + expect(sort).not.toHaveBeenCalled(); const joined = producer.subscribe(all); + sort.mockClear(); // Arrivals below, between and above the joiner's replayed cache. for (const sequence of [25, 50, 5, 35]) write(sequence); expect(sort).not.toHaveBeenCalled(); @@ -3999,11 +4289,11 @@ test("without a declared start, the first arrival stands in for it", async () => test("a source start follows the current requested floor after an update", async () => { const producer = new TrackProducer("test"); - const reader = producer.subscribe({ groups: { start: { included: 5 } } }); + const reader = producer.subscribe({ ranges: [{ start: { absolute: 5 } }] }); try { hooks.startTrack(producer, { start: 6, requested: 5 }); expect(hooks.sourceStart(reader)).toBe(6); - const lower = producer.subscribe({ groups: { start: { included: 2 } } }); + const lower = producer.subscribe({ ranges: [{ start: { absolute: 2 } }] }); try { expect(hooks.sourceStart(lower)).toBe(2); expect(hooks.sourceStart(reader)).toBe(5); @@ -4014,7 +4304,7 @@ test("a source start follows the current requested floor after an update", async } expect(hooks.sourceStart(reader)).toBe(6); const raised = producer.subscription.changed(); - reader.update({ groups: { start: { included: 9 } } }); + reader.update({ ranges: [{ start: { absolute: 9 } }] }); await raised; expect(hooks.sourceStart(reader)).toBe(9); } finally { @@ -4028,7 +4318,7 @@ test("a source start follows the current requested floor after an update", async // keeps that floor: a lowered floor owes the groups it newly asks for. test("a pending tail owes from SUBSCRIBE_START while the demand keeps its floor", async () => { const producer = new TrackProducer("test"); - const reader = producer.subscribe({ delay: { max: Milli(60_000) }, groups: { start: { included: 1 } } }); + const reader = producer.subscribe({ delay: { max: Milli(60_000) }, ranges: [{ start: { absolute: 1 } }] }); hooks.startTrack(producer, { start: 3, requested: 1 }); hooks.endTrack(producer, { group: 4, frame: 0 }); const group = producer.createGroup(3); @@ -4036,7 +4326,7 @@ test("a pending tail owes from SUBSCRIBE_START while the demand keeps its floor" expect((await reader.recvGroup())?.sequence).toBe(3); expect(await reader.recvGroup()).toBeUndefined(); - const lower = producer.subscribe({ delay: { max: Milli(60_000) }, groups: { start: { included: 0 } } }); + const lower = producer.subscribe({ delay: { max: Milli(60_000) }, ranges: [{ start: { absolute: 0 } }] }); expect((await lower.recvGroup())?.sequence).toBe(3); const next = lower.recvGroup(); expect(await settled(next)).toBe(false); @@ -4265,7 +4555,7 @@ test("a widened maxDelay re-offers nothing below a raised floor", () => { } expect(drainGroups(subscriber)).toEqual([3, 4]); - subscriber.update({ delay: { max: Milli(10_000) }, groups: { start: { included: 2 } } }); + subscriber.update({ delay: { max: Milli(10_000) }, ranges: [{ start: { absolute: 2 } }] }); expect(drainGroups(subscriber)).toEqual([2]); producer.close(); }); @@ -4385,7 +4675,7 @@ test("an errored end keeps its history", async () => { producer.close(); expect(producer.closed.peek()).toBe(failed); - const replay = { groups: { start: { included: 0 } }, delay: { max: Milli(30_000) } }; + const replay = { ranges: [{ start: { absolute: 0 } }], delay: { max: Milli(30_000) } }; const arrival = producer.subscribe(replay); expect((await arrival.recvGroup())?.sequence).toBe(0); expect((await arrival.recvGroup())?.sequence).toBe(1); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 3d5d56ae021..f429235ce5a 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -41,8 +41,20 @@ import { type StagedTrack, type TrackRequestOptions, } from "./internal.ts"; -import { grace, Ledger } from "./ledger.ts"; +import { CACHE_EXPIRY, grace, Ledger } from "./ledger.ts"; import { Timescale, type Timestamp } from "./time.ts"; +import { + type Holdings, + hasTime, + isWire, + joinsLive, + later, + rangesOf, + UNHELD, + union, + Window, + wireSpan, +} from "./window.ts"; import { type Broadcast as BroadcastWire, registerTrackConsumer } from "./wire.ts"; export type { Datagram } from "./datagram.ts"; @@ -59,9 +71,8 @@ function taken(read: () => Recv, ended: () => undefined): GroupConsumer | undefi } } -// Wall-clock bound for idle cached content, matching Rust cache::DEFAULT_EXPIRY. // Media delay.max only bounds how far behind the live edge a reader may fall. -const CACHE_WINDOW = Nano.fromMilli(Milli(30_000)); +const CACHE_WINDOW = Nano.fromMilli(CACHE_EXPIRY); // The cache scans at most this many times per idle cache window. const PRUNE_SLICES = 8; @@ -216,7 +227,7 @@ function refreshRatio(value: number): number { /** An explicitly included or excluded group sequence. */ export type Bound = { included: number; excluded?: never } | { excluded: number; included?: never }; -/** Group limits; an omitted endpoint is unbounded. */ +/** Group limits for {@link Subscriber.withRange}; an omitted endpoint is unbounded. */ export interface Groups { /** The first group boundary. */ start?: Bound; @@ -224,6 +235,48 @@ export interface Groups { end?: Bound; } +/** + * Where one end of a {@link Range} falls: a fixed group (`absolute`), a count of groups back from + * the latest one (`relative`, clamped to group 0, so `{ relative: 0 }` is the latest group), or a + * presentation time (`time`, the group containing it: the newest one starting at or before it). + * Mirrors the Rust `Anchor`. + * + * Every anchor names a whole group: a subscription always starts at a group's first frame. A + * relative or time anchor resolves at the hop that serves it, when it starts serving and again + * whenever the subscriber replaces its ranges. Before any group exists a relative anchor resolves + * at the first one, so `{ start: { relative: 0 } }` starts there. + * + * A time anchor resolves from where the groups the hop holds start. A start before every group + * held reaches back to the oldest one, and one resolved before any group has started waits for + * one that has, reading nothing of its range meanwhile. An end settles once a group starting + * after it is known and every sequence below that one is accounted for; until then reads stop + * at the first held group that may still start, so at most that one group past the end is read + * before its first frame shows where it starts. A datagram is judged by its own timestamp. A time anchor on an untimed track is refused + * with {@link TimestampMismatch}, and on a moq-transport copy, whose timestamps may rewind. + * + * Older moq-lite versions carry only absolute anchors, the live start, and the oldest start, + * refusing any other. + */ +export type Anchor = + | { absolute: number; relative?: never; time?: never } + | { relative: number; absolute?: never; time?: never } + | { time: Timestamp; absolute?: never; relative?: never }; + +/** + * A span of a track to deliver: from `start` up to `end`. Mirrors the Rust `Range`. + * + * An omitted start is the oldest group still around (Rust's `Range::oldest`), and an omitted end + * keeps following the live edge. An absolute end is exclusive, so `{absolute: 2}..{absolute: 6}` + * delivers groups 2 through 5; relative and time ends include their group, so + * `{absolute: 0}..{relative: 0}` delivers through the latest group. + */ +export interface Range { + /** Where delivery starts, or the oldest group when omitted. */ + start?: Anchor; + /** Where delivery stops, or onward when omitted. */ + end?: Anchor; +} + /** * How far behind the live edge a subscriber plays, in milliseconds: where it starts, and how * late it may fall. Mirrors the Rust `track::Delay`. Both default to zero, which starts at the @@ -264,14 +317,20 @@ export interface Subscription { /** How far behind the live edge this subscriber plays; see {@link Delay}. Omitted is all zero. */ delay?: Delay; /** - * The groups this reader may receive, with an optional floor and cap. + * The spans of the track to deliver; see {@link Range}. + * + * Defaults to `[{}]`: everything {@link Delay.max} still considers fresh. + * `[{ start: { relative: 0 } }]` starts at the latest group instead, and an empty list asks + * for nothing. {@link Delay.max} caps every range: a range names what may be delivered, and + * the budget skips what inside it has gone stale. A read whose ranges all end finishes once + * nothing in them is left to deliver, as in Rust, while one left open follows the live edge. * - * Only {@link Delay.max} asks for data; these bounds limit which groups it may reach. - * An omitted start means 0, and an omitted end leaves the range open. A floor above - * the live edge waits there (a resumed subscription naming where it left off). - * The latency budget uses the latest eligible group within these bounds. + * A request to the publisher, aggregated across every live subscriber as the union of their + * ranges, and this subscriber's read filter: it reads only the groups its own ranges cover, + * whatever another subscriber widens the union to. {@link Subscriber.withRange} narrows the + * reads further. */ - groups?: Groups; + ranges?: Range[]; /** * Which pending group the publisher sends first. Defaults to `"descending"`. * @@ -319,7 +378,7 @@ function subscriptionDefaults(subscription: Subscription = Subscription.default( if ("maxAge" in subscription || "maxDelay" in subscription) { throw new TypeError("Subscriber maxAge and maxDelay are unsupported; use delay.max"); } - const bounds = groupBounds(subscription.groups ?? {}); + if ("groups" in subscription) throw new TypeError("Subscription groups is unsupported; use ranges"); const delay = subscription.delay ?? {}; return { priority: priorityValue(subscription.priority ?? DEFAULT_PRIORITY), @@ -328,10 +387,7 @@ function subscriptionDefaults(subscription: Subscription = Subscription.default( max: delay.max === null ? NO_MAX_DELAY : wireMillis("delay.max", delay.max ?? Milli.zero), }, order: orderOf(subscription.order), - groups: { - start: subscription.groups?.start === undefined ? undefined : { included: bounds.start }, - end: bounds.end === undefined ? undefined : { excluded: bounds.end }, - }, + ranges: subscription.ranges === undefined ? [{}] : rangesOf(subscription.ranges), }; } @@ -370,18 +426,10 @@ function combineSubscriptions(states: Iterable): Subscription | unde combined.order = combined.order === "ascending" && subscription.order === "ascending" ? "ascending" : "descending"; - // A floor only restricts, so a subscriber without one clears the aggregate: - // its budget may reach below any floor the others set. - const a = groupBounds(combined.groups ?? {}); - const b = groupBounds(subscription.groups ?? {}); - combined.groups = { - start: - combined.groups?.start === undefined || subscription.groups?.start === undefined - ? undefined - : { included: Math.min(a.start, b.start) }, - end: a.end === undefined || b.end === undefined ? undefined : { excluded: Math.max(a.end, b.end) }, - }; + // A range above the live edge never hides the latest group from a merged live subscriber. + combined.ranges = [...(combined.ranges ?? []), ...(subscription.ranges ?? [])]; } + if (combined) combined.ranges = union(combined.ranges ?? []); return combined; } @@ -499,11 +547,11 @@ export class Consumer { /** * Open a live subscription to the track. * - * The cursor starts at the group the subscription named (its floor), or 0. - * {@link Delay.max} is what asks for data: delivery skips everything above - * the floor that the budget convicts. Without options this is {@link Subscription.default}, - * whose budget of zero delivers only the latest group; a larger one reaches back over what - * it can still use, and `delay: { max: null }` reaches back as far as the publisher retains. + * It reads only the groups its {@link Subscription.ranges} cover. + * {@link Delay.max} is what asks for data: delivery skips everything in range that the budget + * convicts. Without options this is {@link Subscription.default}, whose budget of zero + * delivers only the latest group; a larger one reaches back over what it can still use, and + * `delay: { max: null }` reaches back as far as the publisher retains. */ subscribe(options?: Subscription): Subscriber { return this.#broadcast.subscribe(this.name, options); @@ -580,6 +628,8 @@ class TrackState { // First timestamps mutate group state rather than track state, so held groups // watch this revision as well as arrivals when enforcing latency after handoff. timelineChanged = new Signal(0); + // A copy received over a wire without ranges: see `hooks.dropAnchors`. + dropsAnchors = new Signal(false); // Frames wake the sequence cursor only while an owed hole holds it. waitingOrdered = false; // Bumped when a wire subscription declares or resolves where its live feed starts, so a @@ -612,6 +662,9 @@ class TrackState { info = new Signal(undefined); /** Newest group head, datagram, frame, or location reported by the publisher. */ largest = new Signal(undefined); + // The group {@link largest} first named, where a relative anchor resolved before any group + // existed starts. + first?: number; // Set once {@link largest} can no longer advance while the track stays open. largestEnded = new Signal(false); @@ -689,6 +742,11 @@ let reofferSequences: (producer: Producer, sink: TrackState, sequences: readonly let mayLand: (producer: Producer, land: Land) => boolean; let owedGroup: (producer: Producer, start: number, end: number) => { sequence: number; folds?: number } | undefined; let sourceStart: (producer: Producer) => number | "pending" | undefined; +// The producer's checked group starts, sorted by sequence; assigned in its static block. +let startsOf: (producer: Producer) => readonly Start[]; +// The exclusive end of the run of sequences from `sequence` the producer's feed can no longer +// deliver, undefined while it may; assigned in its static block. +let goneAt: (producer: Producer, sequence: number) => number | undefined; // Actual object metadata outlives cached payloads, unlike empty group heads in model largest. let objectLargest: (producer: Producer) => Position | undefined; @@ -707,11 +765,6 @@ let ordered_: { recvDatagram(subscriber: Subscriber): Promise; }; -// Whether a subscription joins at the live edge rather than resuming from a fixed group. -function joinsLive(subscription: Subscription | undefined): boolean { - return subscription?.groups?.start === undefined && subscription?.groups?.end === undefined; -} - // Constructs a Demand from within this module; assigned in the class's static block. let makeDemand: (name: string, wanted: Getter, state: TrackState) => Demand; @@ -862,6 +915,14 @@ export class Producer { hooks.receiveGroup = (producer, { sequence, start }) => producer.#createGroup(sequence, start); hooks.writeDatagram = (producer, datagram, inflated) => producer.#insertDatagram(datagram, inflated); sourceStart = (producer) => producer.#served && producer.#feedStart(); + startsOf = (producer) => producer.#bounds; + goneAt = (producer, sequence) => { + // Without a declared start, the first arrival stands in for it, as the tail's does. + const floor = producer.#served?.start ?? producer.#arrivals.first() ?? 0; + if (sequence < floor) return floor; + const dropped = producer.#state.drops.peek().find((run) => run.start <= sequence && sequence < run.end); + return dropped?.end ?? producer.#arrivals.runEnd(sequence); + }; hooks.producerLargest = (producer) => producer.#state.largest; hooks.joinProducer = (producer) => producer.#join(); hooks.endTrack = (producer, end, error) => producer.#endPending(end, error); @@ -870,6 +931,9 @@ export class Producer { hooks.trackReleased = (producer) => producer.#state.closed; hooks.trackFinished = (producer) => producer.#state.end.peek() !== undefined; hooks.dropTrackGroups = (producer, start, end, error) => producer.#drop(start, end, error, true); + hooks.dropAnchors = (producer) => { + for (const state of [producer.#state, ...producer.#sinks]) state.dropsAnchors.set(true); + }; hooks.startTrack = (producer, served) => { producer.#settle(); producer.#served = served; @@ -1062,11 +1126,11 @@ export class Producer { /** * An independent {@link Subscriber} reading this track's groups. * - * Its cursor starts at the group the subscription named (its floor), or 0. - * {@link Delay.max} is what asks for data: delivery skips everything above - * the floor that the budget convicts. Without options this is {@link Subscription.default}, - * whose budget of zero delivers only the latest group; a larger one reaches back over what - * it can still use, and `delay: { max: null }` reaches back as far as the publisher retains. + * It reads only the groups its {@link Subscription.ranges} cover. + * {@link Delay.max} is what asks for data: delivery skips everything in range that the budget + * convicts. Without options this is {@link Subscription.default}, whose budget of zero + * delivers only the latest group; a larger one reaches back over what it can still use, and + * `delay: { max: null }` reaches back as far as the publisher retains. */ subscribe(options: Subscription = Subscription.default()): Subscriber { const sink = new TrackState(options); @@ -1107,11 +1171,13 @@ export class Producer { const info = this.#state.info.peek(); if (info) sink.info.set(info); sink.largest.set(this.#state.largest.peek()); + sink.first = this.#state.largest.peek()?.group; sink.largestEnded.set(this.#state.largestEnded.peek()); + sink.dropsAnchors.set(this.#state.dropsAnchors.peek()); const closed = this.#state.closed.peek(); if (closed === undefined) { - if (joinsLive(sink.update.peek())) this.#join(); + if (joinsLive(sink.update.peek()?.ranges)) this.#join(); this.#sinks.add(sink); this.#reportWanted(); @@ -1310,8 +1376,10 @@ export class Producer { const current = this.#state.largest.peek(); if (current && !after(largest, current)) return; this.#largestOpen = true; - this.#state.largest.set(largest); - for (const sink of this.#sinks) sink.largest.set(largest); + for (const state of [this.#state, ...this.#sinks]) { + state.first ??= largest.group; + state.largest.set(largest); + } } #advanceObject(largest: Position | undefined): void { @@ -1716,8 +1784,8 @@ export class Producer { // back restores the START. Without either, the first arrival stands in for it. // Until the START arrives, the feed start is pending; the requested floor still bounds it. #feedStart(): number | "pending" | undefined { - const groups = this.#state.update.peek()?.groups; - const requested = groups?.start === undefined ? undefined : groupBounds(groups).start; + const ranges = this.#state.update.peek()?.ranges; + const requested = ranges && wireSpan(ranges).start; const served = this.#served; if (!served || served.requested !== requested) return requested; if (served.start === null) return requested; @@ -1916,6 +1984,18 @@ export class Subscriber { #largestSeen?: Position; #largestDone = false; #nextSequence = 0; + // A wire publisher's reader parks past the end of a bounded window rather than end there: + // its peer would read the end of the stream as the track's. Set through `hooks.parked`. + #parks = false; + // The local bounds {@link withRange} set. + #local: { start: number; end?: number } = { start: 0 }; + // This reader's own ranges resolved to the groups it reads, whatever another subscriber + // widens the aggregate to. Assigned in the constructor. + #window!: Window; + // Bumped whenever the window moves, so the drop scan knows to look again. + #revision = 0; + // The hull of what this reader reads now: its local bounds narrowed to the window's floor + // and end. A group inside it is read only if the window contains it too. #cursor = new Signal<{ start: number; end?: number }>({ start: 0 }); // A FETCH's own budget in place of the subscription's; unlike SUBSCRIBE's zero // default, an omitted one is unbounded, leaving publisher retention alone. @@ -1929,7 +2009,7 @@ export class Subscriber { // the track's drops last folded in, and every sequence already reported. Drops are // persistent track state, so they are tracked here rather than removed for everyone. #pending: DropRun[] = []; - #dropsSeen?: { drops: readonly DropRun[]; start: number; end?: number }; + #dropsSeen?: { drops: readonly DropRun[]; start: number; end?: number; revision: number }; #reported: Run[] = []; // The group the frame-level helpers are currently draining, acquired through the // sequence cursor so frame reads and {@link Ordered.nextGroup} share one floor. @@ -1957,7 +2037,7 @@ export class Subscriber { // stamped group's latest frame: walk back from the cap to the first one. for (let i = (end === undefined ? timeline.length : timelineIndex(timeline, end)) - 1; i >= 0; i--) { const group = timeline[i]; - if (group.closed.peek() instanceof Error) continue; + if (group.closed.peek() instanceof Error || !this.#window.contains(group.sequence)) continue; const timestamp = hooks.groupTimestamp(group); if (timestamp === undefined) continue; presentation = { sequence: group.sequence, timestamp: hooks.groupLatest(group) ?? timestamp }; @@ -2005,7 +2085,7 @@ export class Subscriber { for (let i = timelineIndex(timeline, sequence + 1); i < timeline.length; i++) { const successor = timeline[i]; if (end !== undefined && successor.sequence >= end) break; - if (successor.closed.peek() instanceof Error) continue; + if (successor.closed.peek() instanceof Error || !this.#window.contains(successor.sequence)) continue; if (counts(successor)) return successor; } return undefined; @@ -2238,7 +2318,159 @@ export class Subscriber { this.name = name; this.#state = state; // Each reader filters locally even when another reader widens upstream demand. - this.#cursor.set(groupBounds(state.update.peek()?.groups)); + this.#resolve(); + } + + // The latest group to count relative anchors back from. A reader following route copies + // leaves them to the copies, which hold the groups. + #latestGroup(): number | undefined { + return this.#resume ? undefined : this.#state.largest.peek()?.group; + } + + // Resolve this reader's ranges again, as when they are replaced. + #resolve(): void { + this.#window = new Window(this.#state.update.peek()?.ranges ?? [{}], this.#latestGroup(), this.#holdings()); + this.#moved(); + } + + // The window moved: narrow the bounds to it. + #moved(): void { + this.#revision++; + this.#bound(); + } + + // Where the groups this reader's track holds start, for resolving its time anchors: the + // producer's checked starts, and this reader's mirror of its cache. A reader following route + // copies leaves them to the copies, which hold the groups. + #holdings(): Holdings { + const state = this.#state; + const producer = state.producer; + if (this.#resume || !producer) return UNHELD; + return { + containing: (pts) => { + const starts = startsOf(producer); + if (starts.length === 0) return "unknown"; + if (later(starts[0].start, pts)) return "before"; + // Checked starts never rewind, so they ascend with the sequence. + let lo = 1; + let hi = starts.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if (later(starts[mid].start, pts)) hi = mid; + else lo = mid + 1; + } + return { group: starts[lo - 1].sequence }; + }, + nextStart: (from) => { + const starts = startsOf(producer); + return starts[timelineIndex(starts, from)]; + }, + nextHeld: (from) => { + const group = state.timeline[timelineIndex(state.timeline, from)]; + if (!group) return undefined; + const started = hooks.groupTimestamp(group); + if (started) return { sequence: group.sequence, held: { started } }; + return { sequence: group.sequence, held: hooks.groupStarted(group) ? "never" : "pending" }; + }, + gone: (sequence) => goneAt(producer, sequence), + // Only once the producer is gone: a group held below a reached end may still start. + complete: () => state.closed.peek() !== undefined, + }; + } + + // Resolve what the groups now held settle: a relative anchor waiting on a first group once + // one exists, and a time end as groups start. Returns the refusal of ranges this copy cannot + // resolve, which every read reports. + #settle(): Error | undefined { + const ranges = this.#window.ranges; + if (this.#state.dropsAnchors.peek() && !ranges.every(isWire)) { + return new Error("a relative or time anchor other than the live start needs moq-lite-07"); + } + const info = this.#state.info.peek(); + if (info && info.timescale == null && hasTime(ranges)) return new TimestampMismatch(); + // Its timestamps may rewind, so its group starts are never checked or indexed. + if (info?.ietf !== undefined && hasTime(ranges)) { + return new Error("a time anchor needs timestamps that never rewind, which moq-transport does not promise"); + } + if (this.#resume || this.#window.settled) return undefined; + // The live feed had no group when the ranges resolved, so the newest one then was the first. + const latest = this.#window.pending ? this.#state.first : this.#latestGroup(); + if (this.#window.settle(latest, this.#holdings())) this.#moved(); + return undefined; + } + + // Narrow the local bounds to the window into the hull every read starts from. + #bound(): void { + const end = this.#window.end(); + const local = this.#local; + this.#cursor.set({ + start: Math.max(local.start, this.#window.floor() ?? 0), + end: end === undefined ? local.end : Math.min(local.end ?? end, end), + }); + } + + // Whether a held group waits on the local cap rising: one past it may still be read. A + // group outside the window alone never holds a finished track open. + #parked(groups: readonly GroupConsumer[]): boolean { + const cap = this.#local.end; + return cap !== undefined && (groups.at(-1)?.sequence ?? -1) >= cap; + } + + // Whether a held group from `from` on may still start, and so settle a time anchor the window + // waits on, which a reached end does not decide: until it starts or the track closes, as + // {@link Holdings.complete} says, the window may still take it in. + #starting(groups: readonly GroupConsumer[], from = 0): boolean { + if (this.#window.settled || this.#state.closed.peek() !== undefined) return false; + return groups.some((group) => group.sequence >= from && !hooks.groupStarted(group)); + } + + // Take the group at `index` off the ordered buffer, closing the out-of-range ones below it, + // which sequence order passes for good. + #pass(groups: GroupConsumer[], index: number): void { + const [group] = groups.splice(index, 1); + for (const passed of groups.splice(0, index)) passed.close(); + this.#nextSequence = group.sequence + 1; + } + + // Whether this read ends because its bounded window has nothing left to deliver; see + // {@link #drained}. A wire publisher's reader never ends there. + #exhausted(): boolean { + if (this.#parks) return false; + if (this.#resume) return this.#resume.exhausted(this.#delivered, this.#window); + return this.#drained([this.#delivered]); + } + + // Whether a bounded window has nothing left to deliver: every group in its ranges was handed + // out (in any of `taken`), dropped, or never will be, as one skipped as stale, one below where + // the feed started, or a hole the feed no longer owes. An open window always has more, and so + // does one the cap keeps a group out of, since the reader may raise it again. Mirrors the + // Rust `Cursor::is_exhausted`. + #drained(taken: readonly Delivered[]): boolean { + const end = this.#window.end(); + if (end === undefined) return false; + const held = this.#state.groups.peek(); + const received = this.#state.received.peek(); + const drops = this.#state.drops.peek(); + const producer = this.#state.producer; + for (const run of this.#window.runs(this.#window.floor() ?? 0, end)) { + let gaps = [run]; + for (const delivered of taken) gaps = gaps.flatMap((gap) => delivered.gaps(gap.start, gap.end)); + for (const gap of gaps) { + for (const open of runGaps(drops, gap.start, gap.end)) { + // Not received yet, so it may still arrive. + if (open.end > received) return false; + // Held, so a read still hands it out, skips it, or waits for the cap to rise. + const index = timelineIndex(held, open.start); + if (held[index] && held[index].sequence < open.end) return false; + const owed = producer && owedGroup(producer, open.start, open.end); + if (owed) { + if (owed.folds !== undefined) this.#wakeAt(owed.folds); + return false; + } + } + } + } + return true; } static { @@ -2259,7 +2491,37 @@ export class Subscriber { return Math.max(start, subscriber.#cursor.peek().start); }; hooks.tryRecvGroup = (subscriber, dispositions) => subscriber.#tryRecvGroup(dispositions); + hooks.parked = (subscriber) => { + subscriber.#parks = true; + return subscriber; + }; + // A route copy judged for the logical reader that handed out `delivered`: what the copy + // handed out itself counts too. A copy that is itself a front judges across its own copies. + hooks.exhaustedFor = (subscriber, delivered) => + subscriber.#resume + ? subscriber.#resume.exhausted(subscriber.#delivered.union(delivered), subscriber.#window) + : subscriber.#drained([subscriber.#delivered, delivered]); hooks.tryDrop = (subscriber) => subscriber.#tryDrop(); + // A reader following route copies leaves its ranges to the serving copy, which resolves + // their anchors against the groups it holds, then narrows them to its own local bounds. + hooks.servingRuns = (subscriber, start, end) => { + const cursor = subscriber.#cursor.peek(); + const from = Math.max(start, cursor.start); + const to = Math.min(end, cursor.end ?? end); + return subscriber.#resume?.runs(from, to) ?? subscriber.#window.runs(from, to); + }; + hooks.servingEnd = (subscriber) => { + const cursor = subscriber.#cursor.peek(); + if (!subscriber.#resume) return subscriber.#window.end() === undefined ? undefined : cursor.end; + const end = subscriber.#resume.servingEnd(); + return end === undefined ? undefined : Math.min(end, cursor.end ?? end); + }; + hooks.servingStarts = (subscriber, from) => { + const resumed = subscriber.#resume?.starts(from); + if (resumed) return resumed; + subscriber.#settle(); + return subscriber.#window.starts(from); + }; hooks.tryNextGroup = (subscriber, floor) => { subscriber.#nextSequence = Math.max(subscriber.#nextSequence, floor ?? 0); return subscriber.#tryNextGroup(); @@ -2308,7 +2570,8 @@ export class Subscriber { .some( (datagram) => datagram.sequence >= cursor.start && - (cursor.end === undefined || datagram.sequence < cursor.end), + (cursor.end === undefined || datagram.sequence < cursor.end) && + subscriber.#window.admitsDatagram(datagram.sequence, datagram.timestamp), ); return ( (state.closed.peek() !== undefined || subscriber.#complete()) && @@ -2539,22 +2802,31 @@ export class Subscriber { * Rust `with_range`. * * Both bounds are assigned, in either direction: an omitted start is group 0 and an - * omitted end removes the cap. Lowering the floor or raising the cap offers the groups it - * newly covers that the track still caches and this reader never handed out; a group is - * never handed out twice. A local filter, not a request: it leaves the wire subscription - * alone. One exception: on a track served across route changes, lowering the floor asks the - * serving route again for the groups it newly covers within the subscription's groups, since - * the route that delivered them may be gone. {@link largest} reports within these bounds. + * omitted end removes the cap. They narrow the subscription's ranges, never widen them. + * Lowering the floor or raising the cap offers the groups it newly covers that the track + * still caches and this reader never handed out; a group is never handed out twice. A local + * filter, not a request: it leaves the wire subscription alone, so to change what is asked + * for, replace {@link Subscription.ranges} with {@link update}. One exception: on a track + * served across route changes, lowering the floor asks the serving route again for the groups + * it newly covers within the subscription's ranges, since the route that delivered them may + * be gone. {@link largest} reports within these bounds. */ withRange(groups: Groups): this { - const { start, end } = groupBounds(groups); const before = this.#cursor.peek(); - this.#cursor.set({ start, end }); + this.#local = groupBounds(groups); + this.#bound(); + const { start } = this.#cursor.peek(); const producer = this.#state.producer; if (!this.#resume && !this.#closed && producer && start < before.start) { // The ordered cursor let go of what it passed below the old floor; the arrival // cursor keeps it buffered, which this skips. - reofferCached(producer, this.#state, start, before.start, (sequence) => !this.#delivered.has(sequence)); + reofferCached( + producer, + this.#state, + start, + before.start, + (sequence) => !this.#delivered.has(sequence) && this.#window.contains(sequence), + ); } return this; } @@ -2600,7 +2872,11 @@ export class Subscriber { * Honors the range set by {@link withRange}: a group * beyond the cap stays buffered (not dropped) and is offered once the cap rises, even * after a clean close, without blocking in-range groups that arrive behind it. - * Once the track closes cleanly, a reader with nothing left in range finishes. + * Once the track closes cleanly, a reader with nothing left in range finishes. So does a + * read whose {@link Subscription.ranges} all end, once every group in them was handed out, + * dropped, skipped as stale, or will never arrive, while the track goes on, as in Rust; + * replacing the ranges with {@link update} reads on. A cap keeping a group out holds it + * open, since the cap may rise. * A group whose presentation time is further behind the live edge than this * subscriber's `delay.max` is skipped. The default subscription's zero takes the live edge. * The budget remains attached after return, so a pending frame read rejects if a stalled @@ -2654,6 +2930,8 @@ export class Subscriber { // control state, popping the group, and positioning its frames are one JavaScript turn. #tryRecvGroup(dispositions = false): Recv { if (this.#resume) return this.#tryResumed(false, dispositions); + const refused = this.#settle(); + if (refused) return { kind: "error", error: refused }; const groups = this.#state.groups.peek(); const { start, end } = this.#cursor.peek(); const old = () => new StreamError(StreamCode.Old, { message: "group is past the budget" }); @@ -2667,13 +2945,18 @@ export class Subscriber { // Keep unread groups below the floor until cache eviction: replacing the // subscription can lower it, just as raising the cap re-offers held groups. Found // after the re-offer, which may insert below the floor. - const index = timelineIndex(groups, start); + let index = timelineIndex(groups, start); for (;;) { // The buffer is sequence-sorted, so an in-range group that arrives behind a // beyond-cap one sorts in front of it and is never blocked by it. const group = groups[index]; if (!group || (end !== undefined && group.sequence >= end)) break; + // Kept, like a group below the floor: replacing the ranges can take it in. + if (!this.#window.contains(group.sequence)) { + index++; + continue; + } groups.splice(index, 1); const failed = group.closed.peek(); if (dispositions && failed instanceof Error) { @@ -2705,19 +2988,25 @@ export class Subscriber { const closed = this.#state.closed.peek(); if (closed instanceof Error) return { kind: "error", error: closed }; + // A clean close completes this range, retaining groups a later cap raise can read. if (closed !== undefined) return { kind: "done" }; - if (!this.#complete()) return { kind: "idle" }; - // An open track can still widen its range. Keep the groups beyond its cap. - return groups[index] ? { kind: "boundary" } : { kind: "done" }; + // A bounded window ends once nothing in it is left to deliver, as in Rust. + if (this.#exhausted()) return { kind: "done" }; + if (!this.#complete() || this.#starting(groups)) return { kind: "idle" }; + // An open track can still raise the cap. Keep the groups beyond it. + return this.#parked(groups) ? { kind: "boundary" } : { kind: "done" }; } // The first part of `[from, to)` this cursor has not reported yet, now reported, with why. #report(from: number, to: number, error: Error): (Recv & { kind: "drop" }) | undefined { const { start, end } = this.#cursor.peek(); - const [gap] = runGaps(this.#reported, Math.max(from, start), Math.min(to, end ?? to)); - if (!gap) return undefined; - insertRun(this.#reported, gap.start, gap.end); - return { kind: "drop", ...gap, error }; + for (const run of this.#window.runs(Math.max(from, start), Math.min(to, end ?? to))) { + const [gap] = runGaps(this.#reported, run.start, run.end); + if (!gap) continue; + insertRun(this.#reported, gap.start, gap.end); + return { kind: "drop", ...gap, error }; + } + return undefined; } // The next run this cursor gave up and has not reported, without taking a group. From the @@ -2727,13 +3016,15 @@ export class Subscriber { if (this.#resume) { return this.#resume.tryDrop(this.#delivered); } + this.#settle(); const { start, end } = this.#cursor.peek(); // Rescanned only after it changed, so an idle poll costs nothing per drop held. // Out-of-window runs were skipped, so a moved window rescans too. const drops = this.#state.drops.peek(); + const revision = this.#revision; const seen = this.#dropsSeen; - if (!seen || seen.drops !== drops || seen.start !== start || seen.end !== end) { - this.#dropsSeen = { drops, start, end }; + if (!seen || seen.drops !== drops || seen.start !== start || seen.end !== end || seen.revision !== revision) { + this.#dropsSeen = { drops, start, end, revision }; for (const run of drops) this.#pending = insertDrop(this.#pending, run.start, run.end, run.error); } for (let run = this.#pending.shift(); run; run = this.#pending.shift()) { @@ -2753,6 +3044,7 @@ export class Subscriber { if (closed instanceof Error) return { kind: "error", error: closed }; const recv = resume.tryGroup(ordered, this.#delivered, dispositions); if (recv.kind === "group") this.#handOut(recv.group.sequence, ordered); + else if (recv.kind === "idle" && this.#exhausted()) return { kind: "done" }; return recv; } @@ -2778,6 +3070,11 @@ export class Subscriber { this.#state.groups.changed(fn), this.#state.drops.changed(fn), this.#state.startChanged.changed(fn), + // A group starting may settle a time end, or move the cap it holds reads back with. + ...(this.#window.settled ? [] : [this.#state.timelineChanged.changed(fn)]), + // Accepting the track may refuse a time anchor, with no group arriving to say so. + this.#state.info.changed(fn), + this.#state.dropsAnchors.changed(fn), this.#cursor.changed(fn), // A widened budget re-offers the groups it skipped as stale. this.#state.update.changed(fn), @@ -2790,10 +3087,11 @@ export class Subscriber { dispose.push( this.#state.timelineChanged.changed(fn), this.#wall.revision.changed(fn), - this.#state.info.changed(fn), // A datagram shares the sequence namespace, so it can fill the owed hole. this.#state.datagrams.changed(fn), ); + // A bounded window ends once the feed stops owing its last hole, which only time decides. + else if (this.#window.end() !== undefined) dispose.push(this.#wall.revision.changed(fn)); return () => { for (const close of dispose) close(); }; @@ -2849,6 +3147,9 @@ export class Subscriber { else await Signal.race( this.#state.datagrams, + // Either may refuse the ranges, with no datagram arriving to say so. + this.#state.info, + this.#state.dropsAnchors, this.#state.closed, this.#state.end, this.#state.received, @@ -2863,9 +3164,13 @@ export class Subscriber { if (this.#resume && closed instanceof Error) return { kind: "error", error: closed }; // The group range bounds datagrams too, judged when each is read. One outside it is // gone for good: nothing holds it for a later raise, unlike a group past the cap. + const refused = this.#settle(); + if (refused) return { kind: "error", error: refused }; const { start, end } = this.#cursor.peek(); const inRange = (datagram: Datagram) => - datagram.sequence >= start && (end === undefined || datagram.sequence < end); + datagram.sequence >= start && + (end === undefined || datagram.sequence < end) && + this.#window.admitsDatagram(datagram.sequence, datagram.timestamp); if (this.#resume) { for (;;) { const recv = this.#resume.tryDatagram(this.#delivered); @@ -2902,7 +3207,12 @@ export class Subscriber { #waitOwed(start: number, end: number, drift: Drift): boolean { const producer = this.#state.producer; if (start >= end || !producer) return false; - const owed = owedGroup(producer, start, end); + // Only a hole in this reader's ranges is owed to it. + let owed: { sequence: number; folds?: number } | undefined; + for (const run of this.#window.runs(start, end)) { + owed = owedGroup(producer, run.start, run.end); + if (owed) break; + } if (!owed || this.#staleAt(owed.sequence, drift, true)) return false; if (owed.folds !== undefined) this.#wakeAt(owed.folds); this.#state.waitingOrdered = true; @@ -2912,11 +3222,12 @@ export class Subscriber { // The lowest buffered candidate in range that the budget spares. An ordered take may // still wait on an owed hole below it, so a resumed reader must keep its place. #peekGroup(ordered: boolean): number | undefined { + this.#settle(); const cursor = this.#cursor.peek(); const start = ordered ? Math.max(cursor.start, this.#nextSequence) : cursor.start; const drift = this.#drift(); for (const group of this.#state.groups.peek()) { - if (group.sequence < start) continue; + if (group.sequence < start || !this.#window.contains(group.sequence)) continue; if (cursor.end !== undefined && group.sequence >= cursor.end) return undefined; if (!this.#delivered.has(group.sequence) && !this.#staleAt(group.sequence, drift)) { return group.sequence; @@ -2938,6 +3249,8 @@ export class Subscriber { } #tryNextBuffered(): Recv { + const refused = this.#settle(); + if (refused) return { kind: "error", error: refused }; this.#state.waitingOrdered = false; const groups = this.#state.groups.peek(); const cursor = this.#cursor.peek(); @@ -2946,23 +3259,28 @@ export class Subscriber { // One anchor for the whole pass, so walking a backlog off stays linear. const drift = this.#drift(); - for (;;) { - const group = groups[0]; + for (let index = 0; ; ) { + const group = groups[index]; if (!group || (cursor.end !== undefined && group.sequence >= cursor.end)) break; + // Out of range for now, but kept until the cursor passes it: an anchor still + // resolving, or an update widening the ranges, may take it in. + if (!this.#window.contains(group.sequence)) { + index++; + continue; + } // Every frame this group could still hold is past the budget, so keep // scanning: one pass walks a whole backlog off rather than replaying it. if (this.#staleAt(group.sequence, drift)) { - groups.shift(); - this.#nextSequence = group.sequence + 1; + this.#pass(groups, index); // Holes below a skipped group are undeliverable, even ones the wall clock // still spares because their successor arrived late. start = this.#nextSequence; + index = 0; group.close(); continue; } if (this.#waitOwed(start, group.sequence, drift)) return { kind: "idle" }; - groups.shift(); - this.#nextSequence = group.sequence + 1; + this.#pass(groups, index); this.#handOut(group.sequence, true); return { kind: "group", group: this.#guard(group) }; } @@ -2972,7 +3290,12 @@ export class Subscriber { // A clean close completes this range, retaining groups a later cap raise can read. if (closed !== undefined) return { kind: "done" }; const end = this.#state.end.peek(); - if (!groups[0] && (this.#complete() || (end !== undefined && start >= finalSequence(end)))) { + if (this.#exhausted()) return { kind: "done" }; + if ( + !this.#parked(groups) && + !this.#starting(groups, start) && + (this.#complete() || (end !== undefined && start >= finalSequence(end))) + ) { return { kind: "done" }; } return { kind: "idle" }; @@ -3073,14 +3396,16 @@ export class Subscriber { /** * Update this subscription's options (e.g. priority), triggering a SUBSCRIBE_UPDATE to the - * publisher. Replaces this reader's group bounds without redelivering groups already - * read; an arrival reader can receive unread cached groups below its previous floor. + * publisher. Replacing the ranges resolves them again, without redelivering groups already + * read; an arrival reader can receive the unread groups it still holds that the new ranges + * take in. * Mirrors the Rust `Subscriber::update`. */ update(options: Subscription) { const subscription = subscriptionDefaults(options); - this.#cursor.set(groupBounds(subscription.groups)); + const replaced = JSON.stringify(subscription.ranges) !== JSON.stringify(this.#window.ranges); this.#state.update.set(subscription); + if (replaced) this.#resolve(); } } @@ -3174,7 +3499,8 @@ export class Ordered { * Late arrivals (sequence at or below the last returned) are silently skipped, as is a * group whose every frame is further behind the live edge than `delay.max` (the default * subscription's zero keeps only what nothing newer has superseded). Honors the bounds set by - * {@link withRange}. + * {@link withRange}. Like {@link Subscriber.recvGroup}, a read whose ranges all end + * finishes once nothing in them is left to deliver. */ nextGroup(): Promise { return ordered_.nextGroup(this.#subscriber); diff --git a/js/net/src/window.test.ts b/js/net/src/window.test.ts new file mode 100644 index 00000000000..6bf429e9a16 --- /dev/null +++ b/js/net/src/window.test.ts @@ -0,0 +1,357 @@ +import { expect, test } from "bun:test"; +import { Timestamp } from "./time.ts"; +import type { Anchor, Range } from "./track.ts"; +import { + clamped, + type Held, + type Holdings, + isWire, + joinsLive, + rangesOf, + union, + Window, + wireRanges, + wireSpan, +} from "./window.ts"; + +const abs = (start: number, end?: number): Range => + end === undefined ? { start: { absolute: start } } : { start: { absolute: start }, end: { absolute: end } }; + +test("union merges absolute spans and drops what an open range covers", () => { + expect(union([abs(4, 6), abs(0, 2), abs(2, 3), abs(5, 9)])).toEqual([abs(0, 3), abs(4, 9)]); + // The open absolute span drops a relative-ended range starting inside it. + expect(union([abs(3), { start: { absolute: 5 }, end: { relative: 0 } }])).toEqual([abs(3)]); + // The farthest open relative range drops relative ranges starting no farther back. + expect( + union([ + { start: { relative: 2 } }, + { start: { relative: 5 } }, + { start: { relative: 1 }, end: { relative: 0 } }, + ]), + ).toEqual([{ start: { relative: 5 } }]); + // A relative range above the live edge never hides an absolute one, nor the reverse. + expect(union([abs(10), { start: { relative: 0 } }])).toEqual([abs(10), { start: { relative: 0 } }]); + // The oldest start covers everything, and empty ranges vanish. + expect(union([abs(2, 4), {}])).toEqual([{}]); + expect(union([abs(4, 4), { end: { absolute: 0 } }])).toEqual([]); +}); + +test("a window reads only its ranges, counting relative anchors back from the latest group", () => { + const window = new Window( + [abs(2, 4), { start: { relative: 1 } }, { start: { absolute: 0 }, end: { relative: 8 } }], + 10, + ); + expect([0, 1, 2, 3, 4, 8, 9, 10, 11].map((sequence) => window.contains(sequence))).toEqual([ + true, + true, + true, + true, + false, + false, + true, + true, + true, + ]); + expect(window.floor()).toBe(0); + expect(window.start()).toBe(0); + expect(window.end()).toBeUndefined(); + expect(window.next(4)).toBe(9); + expect(window.runs(1, 12)).toEqual([ + { start: 1, end: 4 }, + { start: 9, end: 12 }, + ]); + expect(window.resolved()).toEqual([{ start: 2, end: 4 }, { start: 9 }, { start: 0, end: 3 }]); +}); + +test("a relative anchor clamps to group 0 and waits for a first group", () => { + expect(new Window([{ start: { relative: 50 }, end: { relative: 40 } }], 10).runs()).toEqual([{ start: 0, end: 1 }]); + + const window = new Window([{ start: { relative: 0 } }], undefined); + expect(window.pending).toBe(true); + expect(window.contains(3)).toBe(true); + expect(window.floor()).toBeUndefined(); + expect(window.settle(7)).toBe(true); + expect(window.pending).toBe(false); + expect(window.contains(6)).toBe(false); + expect(window.floor()).toBe(7); + // Resolved once: a later group never moves it. + expect(window.settle(9)).toBe(false); +}); + +test("a bounded window ends past its last group, and nothing in range ends at 0", () => { + expect(new Window([abs(2, 4), abs(6, 9)], undefined).end()).toBe(9); + expect(new Window([], undefined).end()).toBe(0); + // The oldest start names no floor, so a wire without ranges asks with none. + expect(new Window([{ end: { absolute: 5 } }], undefined).start()).toBeUndefined(); +}); + +test("a wire without ranges carries one span covering them all", () => { + expect(wireSpan([abs(4, 6), abs(2, 3)])).toEqual({ start: 2, end: 6 }); + expect(wireSpan([{ start: { relative: 0 } }])).toEqual({ start: undefined, end: undefined }); + // No single start covers both an absolute and a relative one. + expect(wireSpan([abs(9), { start: { relative: 0 } }])).toEqual({ start: 0, end: undefined }); + // A relative end goes out open, since the copy resolves it against what it receives. + expect(wireRanges([{ start: { relative: 3 }, end: { relative: 1 } }, abs(5, 5)])).toEqual([ + { start: { relative: 3 } }, + ]); +}); + +test("clamping raises starts to the floor and caps ends without widening", () => { + const ranges: Range[] = [abs(1, 9), {}, { start: { relative: 2 }, end: { relative: 0 } }]; + expect(clamped(ranges, 4, 6, [undefined, undefined, { start: 8, end: 11 }])).toEqual([ + abs(4, 6), + abs(4, 6), + abs(8, 6), + ]); + // A relative start not resolved keeps its anchor for the caller to floor. + expect(clamped([{ start: { relative: 2 } }], 4, undefined, [])).toEqual([{ start: { relative: 2 } }]); +}); + +test("ranges are checked rather than served widened", () => { + expect(() => rangesOf([{ start: { absolute: -1 } }])).toThrow(TypeError); + expect(() => rangesOf([{ start: { absolute: 1, relative: 2 } as never }])).toThrow(TypeError); + expect(() => rangesOf([{ end: { time: 5 } as never }])).toThrow(TypeError); + expect(joinsLive([{}])).toBe(true); + expect(joinsLive([abs(3)])).toBe(false); +}); + +const time = (millis: number): Anchor => ({ time: Timestamp.fromMillis(millis) }); +const started = (millis: number): Held => ({ started: Timestamp.fromMillis(millis) }); + +// A hop's groups by sequence, the runs that can no longer arrive, and how many queries a walk made. +class Groups implements Holdings { + held = new Map(); + gones: { start: number; end: number }[] = []; + done = false; + asked = 0; + + constructor(held: [number, Held][] = []) { + for (const [sequence, state] of held) this.held.set(sequence, state); + } + + #sorted(from: number): [number, Held][] { + return [...this.held].filter(([sequence]) => sequence >= from).sort((a, b) => a[0] - b[0]); + } + + containing(pts: Timestamp) { + const starts = this.#sorted(0).filter(([, held]) => typeof held === "object"); + if (starts.length === 0) return "unknown" as const; + const at = starts.filter(([, held]) => !pts.isBefore((held as { started: Timestamp }).started)).at(-1); + return at ? { group: at[0] } : ("before" as const); + } + + nextStart(from: number) { + this.asked++; + const next = this.#sorted(from).find(([, held]) => typeof held === "object"); + return next && { sequence: next[0], start: (next[1] as { started: Timestamp }).started }; + } + + nextHeld(from: number) { + this.asked++; + const next = this.#sorted(from)[0]; + return next && { sequence: next[0], held: next[1] }; + } + + gone(sequence: number) { + return this.gones.find((run) => run.start <= sequence && sequence < run.end)?.end; + } + + complete() { + return this.done; + } +} + +const covered = (window: Window, upto: number) => + Array.from({ length: upto }, (_, sequence) => sequence).filter((sequence) => window.contains(sequence)); + +test("a time start is the group containing it, and an end stops past it once a later group starts after it", () => { + const groups = new Groups([ + [2, started(2000)], + [3, started(3000)], + [4, started(4000)], + [5, started(5000)], + [6, "pending"], + ]); + let window = new Window([{ start: time(3500), end: time(4500) }], 6, groups); + expect(covered(window, 8)).toEqual([3, 4]); + expect(window.settled).toBe(true); + expect(window.resolved()).toEqual([{ start: 3, end: 5 }]); + + // Group 6 may yet start at 5000 too: read on speculation, the end left open. + window = new Window([{ start: time(3500), end: time(5000) }], 6, groups); + expect(window.settled).toBe(false); + expect(covered(window, 9)).toEqual([3, 4, 5, 6]); + expect(window.resolved()).toEqual([{ start: 3, end: undefined }]); + expect(window.end()).toBeUndefined(); + + // Before every start held: reach back to the oldest. + expect(covered(new Window([{ start: time(1000), end: time(2500) }], 6, groups), 8)).toEqual([0, 1, 2]); + + // Nothing started yet: a time start waits on a first group, as a relative one does, then on + // one starting, reading none meanwhile, since a group arrives before the frame that shows + // where it starts. + window = new Window([{ start: time(1000) }], undefined, new Groups()); + expect([window.pending, window.floor(), window.settled]).toEqual([true, undefined, false]); + expect(window.settle(4, new Groups([[4, "pending"]]))).toBe(true); + expect([window.pending, window.contains(4), window.settled]).toEqual([false, false, false]); + expect(window.settle(4, new Groups([[4, started(500)]]))).toBe(true); + expect([window.floor(), window.settled]).toEqual([4, true]); + // Nothing can start any more: it reaches back to the oldest. + const complete = new Groups([[4, "never"]]); + complete.done = true; + expect(new Window([{ start: time(1000) }], 4, complete).floor()).toBe(0); +}); + +test("a waiting time start keeps the relative end it was asked with", () => { + for (const latest of [undefined, 9]) { + const groups = new Groups(); + const window = new Window([{ start: time(2500), end: { relative: 0 } }], latest, groups); + expect(window.pending).toBe(latest === undefined); + for (let sequence = 2; sequence <= 9; sequence++) groups.held.set(sequence, "pending"); + window.settle(9, groups); + expect(window.pending).toBe(false); + groups.held.set(10, "pending"); + expect(window.settle(10, groups)).toBe(false); + + groups.held.set(2, started(2000)); + expect(window.settle(10, groups)).toBe(true); + expect(window.resolved()).toEqual([{ start: 2, end: 10 }]); + } +}); + +test("a time end settles once nothing below its end can arrive", () => { + const groups = new Groups([ + [3, started(3000)], + [6, started(6000)], + ]); + const window = new Window([{ start: time(3000), end: time(4000) }], 6, groups); + // 4 and 5 may still arrive inside the end: only 4 is read, on speculation, and never 6. + expect(window.settled).toBe(false); + expect(covered(window, 8)).toEqual([3, 4]); + + // 4 arrives yet to start: still the one read on speculation, so 5 waits. + groups.held.set(4, "pending"); + expect(window.settle(6, groups)).toBe(false); + groups.held.set(4, started(3500)); + expect(window.settle(6, groups)).toBe(true); + expect(window.settled).toBe(false); + groups.gones.push({ start: 5, end: 6 }); + expect(window.settle(6, groups)).toBe(true); + expect(window.settled).toBe(true); + expect(covered(window, 8)).toEqual([3, 4, 5]); + + // A group that never starts proves nothing, so it never holds the end open. + const never = new Groups([ + [3, started(3000)], + [4, "never"], + [5, started(5000)], + ]); + const settled = new Window([{ start: time(3000), end: time(4000) }], 5, never); + expect(settled.settled).toBe(true); + expect(covered(settled, 8)).toEqual([3, 4]); +}); + +test("an unsettled time end reads one group on speculation", () => { + const groups = new Groups([ + [3, "pending"], + [4, "pending"], + [5, "pending"], + ]); + const window = new Window([{ start: { absolute: 3 }, end: time(3500) }], 5, groups); + expect(covered(window, 8)).toEqual([3]); + + groups.held.set(3, started(3000)); + expect(window.settle(5, groups)).toBe(true); + expect(covered(window, 8)).toEqual([3, 4]); + + // 4 never starts, so 5 is the one read on speculation. + groups.held.set(4, "never"); + expect(window.settle(5, groups)).toBe(true); + expect(covered(window, 8)).toEqual([3, 4, 5]); + + // 5 stalls and 6 starts inside the end: 5 is in range then, and 7 is speculative. + groups.held.set(6, started(3400)); + groups.held.set(7, "pending"); + expect(window.settle(7, groups)).toBe(true); + expect(covered(window, 9)).toEqual([3, 4, 5, 6, 7]); + + // 7 starts past the end, which settles with 5 still stalled below it. + groups.held.set(7, started(3600)); + expect(window.settle(7, groups)).toBe(true); + expect(window.settled).toBe(true); + expect(covered(window, 9)).toEqual([3, 4, 5, 6]); + + // Nothing more can arrive or start: uncapped. + const complete = new Groups([ + [3, started(3000)], + [4, "pending"], + [5, "pending"], + ]); + complete.done = true; + expect(covered(new Window([{ start: time(3000), end: time(3500) }], 5, complete), 7)).toEqual([3, 4, 5, 6]); +}); + +test("a time end walks each group once behind a missing sequence", () => { + const groups = new Groups([[0, started(0)]]); + const window = new Window([{ start: time(0), end: time(1_000_000) }], 0, groups); + const count = 2_000; + for (let sequence = 2; sequence < count; sequence++) { + groups.held.set(sequence, started(sequence * 10)); + window.settle(sequence, groups); + } + // Group 1 never arrived. + expect(window.settled).toBe(false); + expect(window.contains(count - 1)).toBe(true); + expect(groups.asked).toBeLessThan(8 * count); +}); + +test("a datagram is judged by its own timestamp, one range at a time", () => { + const window = new Window([{ start: { absolute: 0 }, end: time(3500) }], undefined, new Groups()); + expect(window.admitsDatagram(9, Timestamp.fromMillis(3500))).toBe(true); + expect(window.admitsDatagram(9, Timestamp.fromMillis(3600))).toBe(false); + + // A start still waiting judges a datagram by its time alone. + expect(window.admitsDatagram(0, Timestamp.fromMillis(0))).toBe(true); + const waiting = new Window([{ start: time(3000), end: time(3500) }], undefined, new Groups()); + expect(waiting.admitsDatagram(0, Timestamp.fromMillis(2900))).toBe(false); + expect(waiting.admitsDatagram(1, Timestamp.fromMillis(3000))).toBe(true); + expect(waiting.admitsDatagram(3, Timestamp.fromMillis(3600))).toBe(false); + + // Group 1 arrived only as a datagram: the end settles before 2 by sequence, but 1 presents past it. + const groups = new Groups([ + [0, started(0)], + [2, started(10_000)], + ]); + groups.gones.push({ start: 1, end: 2 }); + const settled = new Window([{ start: time(0), end: time(5000) }], undefined, groups); + expect(settled.resolved()).toEqual([{ start: 0, end: 2 }]); + expect(settled.admitsDatagram(1, Timestamp.fromMillis(4000))).toBe(true); + expect(settled.admitsDatagram(1, Timestamp.fromMillis(7000))).toBe(false); + + // The sequence and the time must pass the same range. + const both = new Window([{ start: { absolute: 6 } }, { start: time(5500) }], 5, new Groups([[5, started(5000)]])); + expect(both.admitsDatagram(5, Timestamp.fromMillis(5000))).toBe(false); + expect(both.admitsDatagram(5, Timestamp.fromMillis(5500))).toBe(true); + expect(both.admitsDatagram(6, Timestamp.fromMillis(5000))).toBe(true); +}); + +test("a time anchor needs moq-lite-07, is resolved by the copy, and clamps to its group", () => { + expect(isWire({ start: time(1) })).toBe(false); + expect(isWire({ start: { absolute: 1 }, end: time(1) })).toBe(false); + // The copy resolves a time end itself, so it asks through the live edge. + expect(wireRanges([{ start: time(1), end: time(5) }])).toEqual([{ start: time(1) }]); + expect(joinsLive([{ start: time(1) }])).toBe(false); + // A time can't be placed against the others without the hop's groups. + expect(union([{ start: time(5) }, {}])).toEqual([{}]); + expect(union([{ start: time(5) }, { start: { relative: 2 } }])).toEqual([ + { start: { relative: 2 } }, + { start: time(5) }, + ]); + // Resolved anchors become their groups; an unresolved time end stays so the cap applies apart. + expect(clamped([{ start: time(1), end: time(5) }], 2, 9, [{ start: 3, end: 7 }])).toEqual([ + { start: { absolute: 3 }, end: { absolute: 7 } }, + ]); + expect(clamped([{ start: time(1), end: time(5) }], 2, 9, [])).toEqual([{ start: time(1), end: time(5) }]); + expect(rangesOf([{ start: time(1) }])).toEqual([{ start: time(1) }]); + expect(() => rangesOf([{ start: { time: 1 } as unknown as Anchor }])).toThrow(TypeError); +}); diff --git a/js/net/src/window.ts b/js/net/src/window.ts new file mode 100644 index 00000000000..698246780db --- /dev/null +++ b/js/net/src/window.ts @@ -0,0 +1,797 @@ +/** + * A subscriber's own ranges, resolved to the group sequences it reads, and the helpers that + * translate ranges between the model and the wire. Mirrors the Rust `subscription` module. + * + * The aggregate a publisher sees is the union of every subscriber's ranges, so it says what + * arrives, not what one subscriber asked for. A {@link Window} is the per-subscriber side: + * resolved at the hop serving the subscriber, it filters that subscriber's reads. + * + * @module + */ + +import { Timestamp } from "./time.ts"; +import type { Anchor, Range } from "./track.ts"; + +/** A run of group sequences `[start, end)`; an open run ends at infinity. */ +export type Run = { start: number; end: number }; + +/** One range resolved to groups: from `start` up to the exclusive `end`, undefined while open. */ +export type Resolved = { start: number; end?: number }; + +/** Everything from the latest group onward. */ +export const LIVE: Range = Object.freeze({ start: Object.freeze({ relative: 0 }) }); + +/** Check one anchor's shape, naming `field` in the error. */ +function anchorOf(field: string, anchor: Anchor): Anchor { + const keys = typeof anchor === "object" && anchor !== null ? Object.keys(anchor) : []; + const value = keys.length === 1 ? (anchor as Record)[keys[0]] : undefined; + if (keys[0] === "time" && value instanceof Timestamp) return { time: value }; + if ((keys[0] !== "absolute" && keys[0] !== "relative") || !Number.isSafeInteger(value) || (value as number) < 0) { + throw new TypeError( + `${field} must be {absolute} or {relative}, a non-negative safe integer, or {time}, a Timestamp`, + ); + } + return keys[0] === "absolute" ? { absolute: value as number } : { relative: value as number }; +} + +/** Check and copy a caller's ranges, refusing a malformed one rather than serving it widened. */ +export function rangesOf(ranges: readonly Range[]): Range[] { + if (!Array.isArray(ranges)) throw new TypeError("ranges must be an array"); + return ranges.map((range) => { + if (typeof range !== "object" || range === null) throw new TypeError("a range must be an object"); + const out: Range = {}; + if (range.start !== undefined) out.start = anchorOf("range start", range.start); + if (range.end !== undefined) out.end = anchorOf("range end", range.end); + return out; + }); +} + +/** The exclusive end of an absolute anchor, or undefined for any other end. */ +function absolute(anchor: Anchor | undefined): number | undefined { + return anchor?.absolute; +} + +/** How many groups back a relative anchor counts, or undefined for any other anchor. */ +function relative(anchor: Anchor | undefined): number | undefined { + return anchor?.relative; +} + +/** The presentation time a time anchor names, or undefined for any other anchor. */ +function time(anchor: Anchor | undefined): Timestamp | undefined { + return anchor?.time; +} + +/** Whether `time` presents strictly after `other`, as instants, whatever their scales. */ +export function later(time: Timestamp, other: Timestamp): boolean { + return other.isBefore(time); +} + +/** Whether any range starts or ends at a presentation time, which needs a timed track. */ +export function hasTime(ranges: readonly Range[]): boolean { + return ranges.some((range) => time(range.start) !== undefined || time(range.end) !== undefined); +} + +/** Whether `range` certainly delivers nothing, wherever the latest group lands. */ +export function isEmpty(range: Range): boolean { + const end = absolute(range.end); + if (end === undefined) return false; + // Nothing sorts below group 0, whatever the start. + if (end === 0) return true; + // Nothing sorts below group 0, so the oldest start floors there. + const start = range.start === undefined ? 0 : absolute(range.start); + return start !== undefined && end <= start; +} + +/** An anchor as a sort key: absolute, then relative, then time, with `none` for an omitted one. */ +function anchorKey(anchor: Anchor | undefined, none: [number, number]): [number, number] { + if (anchor === undefined) return none; + if (anchor.absolute !== undefined) return [0, anchor.absolute]; + if (anchor.relative !== undefined) return [1, anchor.relative]; + return [2, anchor.time.value / anchor.time.scale]; +} + +/** + * A canonical order: by start, absolute ones first, then relative ones, the oldest (which counts + * back the farthest), and times; then by end, open last. + */ +function compare(a: Range, b: Range): number { + const [x, y] = [ + anchorKey(a.start, [1, Number.POSITIVE_INFINITY]), + anchorKey(b.start, [1, Number.POSITIVE_INFINITY]), + ]; + const [p, q] = [anchorKey(a.end, [3, 0]), anchorKey(b.end, [3, 0])]; + return x[0] - y[0] || x[1] - y[1] || p[0] - q[0] || p[1] - q[1]; +} + +/** + * The union of `ranges` in a canonical order, without empty or redundant ranges. + * + * Absolute spans merge where they overlap or touch. The rest can't be placed without the + * latest group, so only an open range drops what it certainly covers: the open absolute span + * drops ranges starting inside it, and the farthest open relative range drops relative ranges + * starting no farther back. The oldest start covers everything, and a time can't be placed + * against the others without the hop's groups. Mirrors the Rust `union`. + */ +export function union(ranges: readonly Range[]): Range[] { + const kept = ranges.filter((range) => !isEmpty(range)); + if (kept.some((range) => range.start === undefined && range.end === undefined)) return [{}]; + + const spans: Run[] = []; + for (const range of kept) { + const start = absolute(range.start); + if (start === undefined) continue; + if (range.end === undefined) spans.push({ start, end: Number.POSITIVE_INFINITY }); + else { + const end = absolute(range.end); + if (end !== undefined) spans.push({ start, end }); + } + } + spans.sort((a, b) => a.start - b.start || a.end - b.end); + const runs: Run[] = []; + for (const span of spans) { + const last = runs.at(-1); + if (last && span.start <= last.end) last.end = Math.max(last.end, span.end); + else runs.push({ ...span }); + } + const tail = runs.at(-1)?.end === Number.POSITIVE_INFINITY ? runs.at(-1)?.start : undefined; + let farthest: number | undefined; + for (const range of kept) { + const back = relative(range.start); + if (back !== undefined && range.end === undefined) farthest = Math.max(farthest ?? back, back); + } + + const out: Range[] = runs.map((run) => + run.end === Number.POSITIVE_INFINITY + ? { start: { absolute: run.start } } + : { start: { absolute: run.start }, end: { absolute: run.end } }, + ); + for (const range of kept) { + const start = absolute(range.start); + const back = relative(range.start); + if (start !== undefined) { + // Merged above unless it ends relative to the latest group. + if (range.end === undefined || absolute(range.end) !== undefined) continue; + if (tail === undefined || start < tail) out.push(range); + } else if (back !== undefined) { + if (range.end === undefined) continue; + if (farthest === undefined || back > farthest) out.push(range); + } else out.push(range); + } + if (farthest !== undefined) out.push({ start: { relative: farthest } }); + + out.sort(compare); + return out.filter((range, index) => index === 0 || JSON.stringify(range) !== JSON.stringify(out[index - 1])); +} + +/** + * The lowest absolute group any range asks for, or undefined when one counts back from the + * latest group or reaches back to the oldest one: the floor a wire without ranges carries. + */ +export function rangesStart(ranges: readonly Range[]): number | undefined { + let floor: number | undefined; + for (const range of ranges) { + const start = absolute(range.start); + if (start === undefined) return undefined; + floor = Math.min(floor ?? start, start); + } + return floor; +} + +/** + * The exclusive end past every range, or undefined when one is open or ends relative to the + * latest group: the cap a wire without ranges carries. No range at all ends at group 0. + */ +export function rangesEnd(ranges: readonly Range[]): number | undefined { + let end = 0; + for (const range of ranges) { + const next = absolute(range.end); + if (next === undefined) return undefined; + end = Math.max(end, next); + } + return end; +} + +/** + * Whether a wire without ranges carries `range` upstream: absolute anchors, and the live and + * oldest starts it spells with no floor. Mirrors the Rust `Range::is_wire`. + */ +export function isWire(range: Range): boolean { + const back = relative(range.start); + return ( + (back === undefined || back === 0) && + time(range.start) === undefined && + relative(range.end) === undefined && + time(range.end) === undefined + ); +} + +/** + * The ranges a SUBSCRIBE carries for `ranges`, without the empty ones. Mirrors the Rust + * `wire_ranges`. + * + * The copy resolves a relative or time end itself, against the groups it receives past it, so + * it asks through the live edge: SUBSCRIBE_START names a resolved start, but nothing names a + * resolved end, so one resolved upstream would leave the copy short. + */ +export function wireRanges(ranges: readonly Range[]): Range[] { + return ranges + .filter((range) => !isEmpty(range)) + .map((range) => + range.end === undefined || absolute(range.end) !== undefined + ? range + : range.start + ? { start: range.start } + : {}, + ); +} + +/** Whether every range has an end, so a lite-07 publisher ends the subscription once it covers them. */ +export function bounded(ranges: readonly Range[]): boolean { + return ranges.every((range) => range.end !== undefined); +} + +/** + * Whether a subscriber joins at the live edge: a range counting back from the latest group (or + * reaching back to the oldest) and following it. A fixed start resumes a known group instead, + * such as a route switch picking up where the reader stopped. + */ +export function joinsLive(ranges: readonly Range[] | undefined): boolean { + return (ranges ?? [{}]).some( + (range) => absolute(range.start) === undefined && time(range.start) === undefined && range.end === undefined, + ); +} + +/** + * The one span a wire without ranges carries for `ranges`, which the subscribers' own reads + * narrow again: from the lowest absolute start up to {@link rangesEnd}. + * + * A subscription counting only back from the latest group asks with no start, which a pre-06 + * publisher reads as the latest group and a lite-06 one as no floor. One mixing the two asks + * from group 0, since no single start covers both: the latest group may sit below the absolute + * start. Mirrors the Rust `wire_span`. + */ +export function wireSpan(ranges: readonly Range[]): { start?: number; end?: number } { + let start = rangesStart(ranges); + if (start === undefined && ranges.some((range) => absolute(range.start) !== undefined)) start = 0; + return { start, end: rangesEnd(ranges) }; +} + +/** + * `ranges` with every range raised to start no earlier than `floor` and stop no later than the + * exclusive `end`, without ever widening one. Mirrors the Rust `Subscription::clamped`. + * + * `resolved` names where each range resolved, by index, so a relative or time anchor becomes its + * own group before it is clamped. A start not resolved keeps its anchor, since the floor may lie + * before it, so the caller drops what it resolves to below `floor`. The oldest start never lies + * above the floor, so it becomes the floor. An end not resolved becomes the cap, except a time + * end, which stays so the caller enforces `end` apart from it. + */ +export function clamped( + ranges: readonly Range[], + floor: number | undefined, + end: number | undefined, + resolved: readonly (Resolved | undefined)[], +): Range[] { + if (floor === undefined && end === undefined) return [...ranges]; + return ranges.map((range, index) => { + const at = resolved[index]; + let start = range.start; + if (at && absolute(start) === undefined && start !== undefined) start = { absolute: at.start }; + let requested = range.end; + if (at?.end !== undefined && requested !== undefined && absolute(requested) === undefined) + requested = { absolute: at.end }; + const named = absolute(start); + if (floor !== undefined && named !== undefined) start = { absolute: Math.max(named, floor) }; + else if (floor !== undefined && start === undefined) start = { absolute: floor }; + if (end !== undefined && time(requested) === undefined) { + const capped = absolute(requested); + requested = { absolute: capped === undefined ? end : Math.min(capped, end) }; + } + const out: Range = {}; + if (start !== undefined) out.start = start; + if (requested !== undefined) out.end = requested; + return out; + }); +} + +/** Where a presentation time falls among the groups a hop holds. Mirrors the Rust `Containing`. */ +export type Containing = + /** The newest group starting at or before it. */ + | { group: number } + /** Every group with a known start begins after it. */ + | "before" + /** No group has a known start. */ + | "unknown"; + +/** What a held group says about where it starts. Mirrors the Rust `Held`. */ +export type Held = + /** Its frame 0 presented at this time. */ + | { started: Timestamp } + /** It holds frame 0 and has not presented it yet. */ + | "pending" + /** It never will: it ended empty or failed. */ + | "never"; + +/** + * When a serving hop's groups start, for resolving time anchors. Mirrors the Rust `Holdings`. + * + * Each query answers from ordered indexes, so a time end that walks them forward a step at a + * time pays for each group or run it passes once, however long it stays unsettled. + */ +export interface Holdings { + /** Where `pts` falls among the groups with a known start. */ + containing(pts: Timestamp): Containing; + /** The lowest group at or above `from` known to start, with where it starts. */ + nextStart(from: number): { sequence: number; start: Timestamp } | undefined; + /** The lowest held group at or above `from`, with what it says about its start. */ + nextHeld(from: number): { sequence: number; held: Held } | undefined; + /** + * The exclusive end, past `sequence`, of the run of sequences from it that can no longer + * arrive (dropped, given up, or below where the feed starts), undefined while it may. + */ + gone(sequence: number): number | undefined; + /** Whether nothing more can arrive or start here, so a time end has nothing to wait on. */ + complete(): boolean; +} + +/** + * A hop that holds no groups of its own, such as a reader following route copies that resolve + * its ranges for it: a time end there filters nothing. + */ +export const UNHELD: Holdings = Object.freeze({ + containing: () => "unknown" as const, + nextStart: () => undefined, + nextHeld: () => undefined, + gone: () => undefined, + complete: () => true, +}); + +/** + * A time end not settled yet, and how far the walk to settle it got. Mirrors the Rust `TimeEnd`. + * + * Timestamps never rewind, so the end falls before the lowest group starting past `pts`, once + * every sequence between it and the newest one starting at or before `pts` is accounted for. + * Every position below only moves forward, so each group or run of sequences is passed once, + * whatever a peer leaves missing. + */ +class TimeEnd { + readonly pts: Timestamp; + /** The newest group known to start at or before `pts`: it and everything below it in the span is in range. */ + known?: number; + /** The lowest group known to start after `pts`: where the end falls once settled. */ + after?: number; + /** Where the walk over held groups resumes: those below it started, or never will. */ + held: number; + /** Where settling resumes: every sequence from above `known` up to it is accounted for. */ + accounted: number; + /** The exclusive cap on what is read meanwhile, undefined for none. */ + limit?: number; + /** The group the cap admits on speculation: the first above `known` that may still start, which may start past `pts`. */ + speculative?: number; + + constructor(pts: Timestamp, start: number) { + this.pts = pts; + this.held = start; + this.accounted = start; + this.limit = start; + } + + /** The lowest sequence not known to be in range. */ + unknown(start: number): number { + return this.known === undefined ? start : Math.max(this.known + 1, start); + } + + /** Walk the held groups on from where the last call stopped. Returns where the end falls once it settles. */ + advance(start: number, holdings: Holdings): number | undefined { + // The starts index passes a group that may still start, so a stalled one never keeps a + // later group known to start inside the end from being read. + for (;;) { + const next = holdings.nextStart(this.unknown(start)); + if (!next || (this.after !== undefined && next.sequence >= this.after)) break; + if (later(next.start, this.pts)) this.after = next.sequence; + else this.known = next.sequence; + } + + // The first sequence above `known` that may still start is the one read on speculation: + // a held group yet to start, or a missing one that may arrive. Only a group starting + // inside the end is jumped to, since it puts all below it in range. + this.speculative = undefined; + for (;;) { + const at = Math.max(this.held, this.unknown(start)); + if (this.after !== undefined && at >= this.after) break; + let next = holdings.nextHeld(at); + if (next && this.after !== undefined && next.sequence >= this.after) next = undefined; + const held = next?.held; + if (next && next.sequence === at) { + if (held === "pending") { + this.held = at; + this.speculative = at; + break; + } + if (typeof held === "object") { + if (later(held.started, this.pts)) this.after = at; + else this.known = at; + } + } else if (next && typeof held === "object" && !later(held.started, this.pts)) { + this.known = next.sequence; + } else if (!next && this.after === undefined) { + // Past every held group: the next to arrive is the speculative one, judged once a + // settle sees it, before it is read. + this.held = at; + break; + } else { + const end = holdings.gone(at); + if (end !== undefined && end > at) { + this.held = end; + continue; + } + this.held = at; + this.speculative = holdings.complete() ? undefined : at; + break; + } + this.held = at + 1; + } + if (holdings.complete()) this.limit = undefined; + else if (this.speculative !== undefined) this.limit = this.speculative + 1; + // Every held group is accounted for: the next to arrive is the speculative one. + else this.limit = this.after; + + // Settle once every sequence up to the first group starting past `pts` can no longer + // start inside it. A sequence that arrived only as a datagram never indexes where its + // group starts, so it counts as gone here. + const after = this.after; + if (after === undefined) return undefined; + // Everything at or below `known` is in range, and `after` lies above it. + this.accounted = Math.max(this.accounted, this.unknown(start)); + while (this.accounted < after) { + const next = holdings.nextHeld(this.accounted); + if (next && next.sequence === this.accounted) { + if (next.held === "pending") return undefined; + // It started past `pts` since the walk above stopped on it: everything below is + // accounted for, so the end falls here. + if (typeof next.held === "object" && later(next.held.started, this.pts)) { + this.after = next.sequence; + return this.after; + } + this.accounted++; + } else { + const below = next ? Math.min(next.sequence, after) : after; + if (holdings.complete()) this.accounted = below; + else { + const gone = holdings.gone(this.accounted); + if (gone === undefined || gone <= this.accounted) return undefined; + this.accounted = Math.min(gone, below); + } + } + } + return this.after; + } +} + +/** Where a resolved range stops. */ +type End = + /** Before this group. */ + | { before: number } + /** Past every group. */ + | "open" + /** Before the first group starting after a time, once that is known. */ + | { time: TimeEnd }; + +/** One non-empty range, resolved. */ +type Span = { + /** Which of the ranges it came from. */ + index: number; + start: number; + end: End; + /** Whether the start names a group, rather than reaching back to the oldest one. */ + floored: boolean; +}; + +/** The exclusive end of the groups a span reads now, infinity for none: an unsettled time end reads up to its cap. */ +function limitOf(span: Span): number { + if (span.end === "open") return Number.POSITIVE_INFINITY; + if ("before" in span.end) return span.end.before; + return span.end.time.limit ?? Number.POSITIVE_INFINITY; +} + +/** + * A subscriber's ranges resolved to the group sequences it reads. Mirrors the Rust `Window`. + * + * A relative or time anchor resolves when the window is built, and again only when the ranges + * are replaced. Before any group exists a relative anchor waits, reading everything, until + * {@link settle} names the first one. A time start before any group has started waits too, but + * reading nothing of its range, since a group arrives before the frame that shows where it + * starts. A time end settles as the groups it waits on start. + */ +export class Window { + /** What these were resolved from, so only replacing the ranges resolves again. */ + readonly ranges: readonly Range[]; + #spans: Span[] = []; + /** The groups the spans cover, an unsettled time end through its cap: merged, sorted, and disjoint. */ + #runs: Run[] = []; + /** A relative anchor waits on a first group, so nothing is filtered yet. */ + #pending = false; + /** + * The ranges, by index, whose time start waits on a group starting, so they read nothing yet, + * with the latest group their relative end counts back from. + */ + #waiting: { index: number; latest: number | undefined }[] = []; + /** Nothing is left to resolve as groups arrive and start. */ + #settled = false; + + /** Resolve `ranges`, counting relative anchors back from `latest` and time anchors from `holdings`. */ + constructor(ranges: readonly Range[], latest: number | undefined, holdings: Holdings = UNHELD) { + this.ranges = ranges; + this.#place(latest, holdings); + } + + /** Whether a relative anchor still waits on a first group. */ + get pending(): boolean { + return this.#pending; + } + + /** Whether nothing is left to resolve as groups arrive and start. */ + get settled(): boolean { + return this.#settled; + } + + /** + * Resolve what the groups now held settle: relative anchors once `latest` exists, and time + * starts and ends as groups start. Returns whether the groups in range moved. + */ + settle(latest: number | undefined, holdings: Holdings = UNHELD): boolean { + if (this.#pending) { + if (latest === undefined) return false; + this.#place(latest, holdings); + return !this.#pending; + } + let moved = false; + const waiting = this.#waiting; + this.#waiting = []; + // A relative end counts back from the latest group when asked, or the first observed + // group on an empty track, even while the time start waits. + for (const { index, latest } of waiting) { + if (this.#span(index, latest, holdings)) moved = true; + } + for (const span of this.#spans) { + if (span.end === "open" || !("time" in span.end)) continue; + const time = span.end.time; + const limit = time.limit; + const end = time.advance(span.start, holdings); + if (end !== undefined) { + span.end = { before: end }; + moved = true; + } else if (time.limit !== limit) moved = true; + } + if (moved) this.#rebuild(); + return moved; + } + + /** Whether group `sequence` is in range. Everything is while a relative anchor waits. */ + contains(sequence: number): boolean { + if (this.#pending) return true; + const run = this.#runAt(sequence); + return run !== undefined && sequence >= run.start; + } + + /** + * Whether a datagram is in range, judged on its own and never through a time end's settling: + * its `timestamp` against each range's times, and its `sequence` against the range's absolute + * groups and the relative ones resolved so far. + */ + admitsDatagram(sequence: number, timestamp: Timestamp | undefined): boolean { + if (timestamp === undefined) { + return this.#pending || this.#spans.some((span) => sequence >= span.start && sequence < limitOf(span)); + } + return this.ranges.some((range, index) => { + if (isEmpty(range)) return false; + // A relative anchor still waiting on a latest group bounds nothing yet. + const span = this.#spans.find((span) => span.index === index); + const { start, end } = range; + const started = + start?.time !== undefined + ? !later(start.time, timestamp) + : start?.absolute !== undefined + ? sequence >= start.absolute + : start === undefined || !span || sequence >= span.start; + const ended = + end === undefined + ? true + : end.time !== undefined + ? !later(timestamp, end.time) + : end.absolute !== undefined + ? sequence < end.absolute + : !span || span.end === "open" || !("before" in span.end) || sequence < span.end.before; + return started && ended; + }); + } + + /** The lowest group in range at or above `from`, undefined when none is. `from` itself while a relative anchor waits. */ + next(from: number): number | undefined { + if (this.#pending) return from; + const run = this.#runAt(from); + return run === undefined ? undefined : Math.max(run.start, from); + } + + /** The lowest group in range, undefined while an anchor waits. Nothing in range floors at group 0. */ + floor(): number | undefined { + if (this.#pending || this.#waiting.length > 0) return undefined; + return this.#runs[0]?.start ?? 0; + } + + /** + * The lowest group the ranges name, undefined while one reaches back to the oldest group or + * a relative anchor waits on a first group. + */ + start(): number | undefined { + if (this.#pending || this.#waiting.length > 0 || this.#spans.some((span) => !span.floored)) return undefined; + let start: number | undefined; + for (const span of this.#spans) start = Math.min(start ?? span.start, span.start); + return start; + } + + /** + * The exclusive end past every group in range, undefined when one stays open. An unsettled + * time end counts as open: its cap only holds reads back meanwhile. Nothing in range ends at 0. + */ + end(): number | undefined { + if (this.#pending || this.#waiting.length > 0) return undefined; + if (this.#spans.some((span) => span.end !== "open" && "time" in span.end)) return undefined; + const end = this.#runs.at(-1)?.end ?? 0; + return end === Number.POSITIVE_INFINITY ? undefined : end; + } + + /** The runs in range from `from` up to the exclusive `end`, lowest first. */ + runs(from = 0, end = Number.POSITIVE_INFINITY): Run[] { + if (from >= end) return []; + if (this.#pending) return [{ start: from, end }]; + const out: Run[] = []; + for (const run of this.#runs) { + if (run.end <= from) continue; + if (run.start >= end) break; + out.push({ start: Math.max(run.start, from), end: Math.min(run.end, end) }); + } + return out; + } + + /** + * Each range resolved, in the order given: undefined for one that is empty or whose start + * still waits. An unsettled time end is reported open, never at its cap. + */ + resolved(): (Resolved | undefined)[] { + const resolved: (Resolved | undefined)[] = this.ranges.map(() => undefined); + for (const span of this.#spans) { + resolved[span.index] = { + start: span.start, + end: span.end !== "open" && "before" in span.end ? span.end.before : undefined, + }; + } + return resolved; + } + + /** + * Where each range starts once delivery starts at `from`, in the order given: undefined for + * one with no group there, or whose start still waits. Mirrors the Rust `Window::starts`. + */ + starts(from: number): (number | undefined)[] { + return this.resolved().map((range) => { + if (!range) return undefined; + const start = Math.max(range.start, from); + return range.end === undefined || range.end > start ? start : undefined; + }); + } + + // The first run ending past `sequence`. + #runAt(sequence: number): Run | undefined { + let lo = 0; + let hi = this.#runs.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if (this.#runs[mid].end <= sequence) lo = mid + 1; + else hi = mid; + } + return this.#runs[lo]; + } + + #place(latest: number | undefined, holdings: Holdings): void { + this.#spans = []; + this.#pending = false; + this.#waiting = []; + for (let index = 0; index < this.ranges.length; index++) this.#span(index, latest, holdings); + this.#rebuild(); + } + + // Resolve range `index` into a span, or note what it waits on. Returns whether it resolved. + #span(index: number, latest: number | undefined, holdings: Holdings): boolean { + const range = this.ranges[index]; + if (isEmpty(range)) return false; + const until = endOf(range.end, latest); + if (until === undefined) { + this.#pending = true; + return false; + } + const start = startOf(range.start, latest, holdings); + if (start === "waiting") { + this.#waiting.push({ index, latest }); + return false; + } + if (start === undefined) { + this.#pending = true; + return false; + } + let end: End = until === "open" || "before" in until ? until : { time: new TimeEnd(until.pts, start.group) }; + if (end !== "open" && "time" in end) { + const settled = end.time.advance(start.group, holdings); + if (settled !== undefined) end = { before: settled }; + } + this.#spans.push({ index, start: start.group, end, floored: start.floored }); + return true; + } + + #rebuild(): void { + this.#settled = + !this.#pending && + this.#waiting.length === 0 && + this.#spans.every((span) => span.end === "open" || !("time" in span.end)); + const spans = this.#spans + .map((span) => ({ start: span.start, end: limitOf(span) })) + .filter((span) => span.end > span.start) + .sort((a, b) => a.start - b.start); + this.#runs = []; + for (const span of spans) { + const last = this.#runs.at(-1); + if (last && span.start <= last.end) last.end = Math.max(last.end, span.end); + else this.#runs.push({ start: span.start, end: span.end }); + } + } +} + +/** + * The first group a range starts at, and whether it names that group rather than reaching back + * to the oldest one. Undefined while it waits on a latest group, and `"waiting"` for a time + * start while no group has started. + */ +function startOf( + anchor: Anchor | undefined, + latest: number | undefined, + holdings: Holdings, +): { group: number; floored: boolean } | "waiting" | undefined { + const group = absolute(anchor); + if (group !== undefined) return { group, floored: true }; + const pts = time(anchor); + if (pts !== undefined) { + const containing = holdings.containing(pts); + if (containing !== "before" && containing !== "unknown") return { group: containing.group, floored: true }; + // The group holding it is older than any held, so reach back to the oldest. + if (containing === "before") return { group: 0, floored: false }; + // Before any group, as for a relative anchor, which a hop that holds none (such as a + // reader following route copies) leaves to the copies to resolve. + if (latest === undefined) return undefined; + // Nothing can start any more, so nothing held reaches past it. + if (holdings.complete()) return { group: 0, floored: false }; + // Nothing has started yet, so wait for a group that has. A copy may hear the live edge + // before the groups its publisher resolved the time to, which the edge would skip. + return "waiting"; + } + const back = relative(anchor); + if (back === undefined) return { group: 0, floored: false }; + if (latest === undefined) return undefined; + return { group: Math.max(0, latest - back), floored: true }; +} + +/** + * Where a range stops, exclusive, or undefined while it waits on a latest group. A time end's + * walk starts once its start is known. + */ +function endOf( + anchor: Anchor | undefined, + latest: number | undefined, +): { before: number } | "open" | { pts: Timestamp } | undefined { + if (anchor === undefined) return "open"; + const end = absolute(anchor); + if (end !== undefined) return { before: end }; + const pts = time(anchor); + if (pts !== undefined) return { pts }; + if (latest === undefined) return undefined; + return { before: Math.max(0, latest - (relative(anchor) ?? 0)) + 1 }; +} diff --git a/quest/m1/subscribe-ranges/README.md b/quest/m1/subscribe-ranges/README.md index d3051609410..8bdcd92f4e3 100644 --- a/quest/m1/subscribe-ranges/README.md +++ b/quest/m1/subscribe-ranges/README.md @@ -103,7 +103,7 @@ every sequence delivered or dropped. ## Required - [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) - the lite-07 wire, Rust publisher and subscriber, and the draft -- [JS ranges](/quest/m1/subscribe-ranges/js.md) - `@moq/net` model and lite-07 parity, and the on-demand request surface +- [JS ranges](/quest/m1/subscribe-ranges/js.md) - `@moq/net`'s on-demand request surface and `fetchGroup` as a range - [Remove FETCH from lite-07](/quest/m1/subscribe-ranges/lite07-fetch.md) - FETCH removed from lite-07 in Rust, JS, and the draft - [Delete the old start bookkeeping](/quest/m1/subscribe-ranges/start-bookkeeping.md) - what each lite cursor is owed and dropped lives in the model, and `TrackRun` only translates it per version diff --git a/quest/m1/subscribe-ranges/js.md b/quest/m1/subscribe-ranges/js.md index 3461e56f57e..0fe7fa8390f 100644 --- a/quest/m1/subscribe-ranges/js.md +++ b/quest/m1/subscribe-ranges/js.md @@ -1,67 +1,94 @@ -# [L] JS ranges +# [M] JS ranges ## Goal -`@moq/net` matches Rust: `Subscription` carries ranges beside its existing -`order`, the lite-07 SUBSCRIBE carries them on the wire, and `fetchGroup` -becomes a one-range subscription or is removed. A JavaScript publisher serves -requested ranges that are no longer in its live cache. +`@moq/net` matches Rust: `fetchGroup` becomes a one-range subscription or is +removed, and a JavaScript publisher serves requested ranges that are no longer +in its live cache. ## Plan -Decided 2026-10-27 (maintainer): removing FETCH from lite-07 moved to -[Remove FETCH from lite-07](/quest/m1/subscribe-ranges/lite07-fetch.md), and -this quest no longer waits on -[Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md). The rest starts now: -the ranges model, anchors, `delay: { max: null }`, and the on-demand request -surface. Rust's lite-07 SUBSCRIBE already carries ranges -([moq-dev/moq#6715](https://github.com/moq-dev/moq/pull/6715)). +Landed in [moq-dev/moq#6949](https://github.com/moq-dev/moq/pull/6949): +`Subscription.ranges` of `absolute`, `relative`, and `time` anchors in place of +`groups`, a per-reader window (`js/net/src/window.ts`) with the Rust time-end +walk, the union aggregate, lite-07 SUBSCRIBE carrying the list with times in +the track's timescale (older wires the covering span, refusing other relative +and time anchors per reader), the lite-07 publisher resolving relative and +time anchors and naming each range's start in SUBSCRIBE_START, FINning a +covered bounded subscription without SUBSCRIBE_END, and the subscriber +keeping that copy and opening a fresh SUBSCRIBE on the same track once the +demand moves past what it covered. `delay: { max: null }` landed with +[moq-dev/moq#7597](https://github.com/moq-dev/moq/pull/7597). -Mirror lite-07 bounded completion as the Rust copy implements it when this -lands. The target is [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md)' -"Ends are signaled" design: a covered bounded subscription sends -SUBSCRIBE_FINISH, keeps the Subscribe Stream open, and never FINs it. This -quest does not wait on that one, so if it lands first, the wire still FINs a -covered bounded subscription without SUBSCRIBE_END. The JS subscriber already -accepts that FIN, closing the request's track once its tail settles, and -failing it when an update moved the range and the groups that arrived leave it -uncovered; until SUBSCRIBE_FINISH ships it reopens that demand on the same -track instead, as the Rust copy does. +Reader completion, decided 2026-11-02 (maintainer, on +[moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545) and +[moq-dev/moq#7625](https://github.com/moq-dev/moq/pull/7625)), replacing the +2026-10-31 decision that a range-bounded reader parks: -Mirror the Rust names, including `Time(pts)` ends and their refusal on an untimed track. -The publisher resolves no relative or time start, serving it from the oldest -group its budget allows, and SUBSCRIBE_OK `Range Start` says so; resolve them. -Also mirror Rust's optional `Delay::max` -([moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171)): a `null` -`delay.max` asks for the publisher's max age alone, dominates when subscriptions -combine, and goes out as the largest delay the wire allows, where today an -omitted one means the live edge only. +1. Park in both: keep the copy on a covered FIN, and leave a read that should + end to the request surface and `fetchGroup`. +2. ✅ Complete in both: a bounded read ends once every group in its ranges is + handed out, dropped, skipped as stale, or given up, and only wire + publishers park. +3. Leave it: JS copies keep releasing on a covered FIN. -Fills mirror Rust's rule from [#7168](https://github.com/moq-dev/moq/pull/7168) -(decided 2026-10-29, maintainer): give up a run below a start already -resolved under a budget at least as loose as the reader's, and on a pre-06 -wire fill below the start only when the reader names a floor. `js/net` -requests no fills today, so nothing over-fetches yet; this keeps it that way -once it does. +JavaScript matches Rust: a plain reader judges its own window, as the Rust +`Cursor::is_exhausted` does, and a reader following route copies judges it on +the serving copy, as the Rust `Reader::exhausted` does. Wire publishers and +route copies read through the package-internal `hooks.parked`, the Rust +`track::Subscriber::parked`. A lite-07 copy still outlives a covered FIN, so +the cache answers the same range and a widening update still delivers. -Decided in the 2026-10-05 audit: the producer half of -JavaScript FETCH folds in here, so `@moq/net`'s -on-demand request surface takes range requests from the start instead of the -per-group `requested_group` shape Rust replaced with `requested_fill`. IETF -FETCH dispatch onto it is -[on-demand JavaScript FETCH](/quest/m1/js-fetch-on-demand.md). Resized from -[M] to [L] for it. +Decided 2026-11-03 (maintainer): main moved the cursor paths under +[moq-dev/moq#6949](https://github.com/moq-dev/moq/pull/6949) +([moq-dev/moq#7486](https://github.com/moq-dev/moq/pull/7486), +[moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), +[moq-dev/moq#7597](https://github.com/moq-dev/moq/pull/7597), +[moq-dev/moq#7609](https://github.com/moq-dev/moq/pull/7609)), so the PR +re-ports its ranges onto them: -- Expose an owned request handle rather than a storage callback; dropping a - request refuses it rather than leaving the subscriber waiting. Preserve - group sequence, frame boundaries, payload bytes, and track properties. - Storage, retention, and media catalog interpretation stay outside `js/net`. -- One logical dynamic track per broadcast and name: a cache-miss request - never creates a duplicate live producer. -- Verify with an in-memory responder, without OPFS or an archive writer: a - browser publisher serves a native subscriber after a group is evicted or - was never cached, covering exact replay, empty and missing groups, - concurrent requests, and cancellation while awaiting a reply. +1. ✅ Re-port in place, on the same PR. +2. A fresh branch, closing and deleting this one. +3. Park it. + +Mirror lite-07 bounded completion as the Rust copy implements it. The target +is [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md)' "Ends are signaled" +design: a covered bounded subscription sends SUBSCRIBE_FINISH, keeps the +Subscribe Stream open, and never FINs it. Until SUBSCRIBE_FINISH ships, the +wire FINs a covered bounded subscription without SUBSCRIBE_END, and the JS +subscriber reopens demand it no longer covers on the same track, as the Rust +copy does. + +What remains: + +- **On-demand request surface.** Decided in the 2026-10-05 audit: the producer + half of JavaScript FETCH folds in here, so + `@moq/net`'s on-demand request surface takes range requests from the start + instead of the per-group `requested_group` shape Rust replaced with + `requested_fill`. IETF FETCH dispatch onto it is + [on-demand JavaScript FETCH](/quest/m1/js-fetch-on-demand.md). + - Expose an owned request handle rather than a storage callback; dropping a + request refuses it rather than leaving the subscriber waiting. Preserve + group sequence, frame boundaries, payload bytes, and track properties. + Storage, retention, and media catalog interpretation stay outside `js/net`. + - One logical dynamic track per broadcast and name: a cache-miss request + never creates a duplicate live producer. + - Verify with an in-memory responder, without OPFS or an archive writer: a + browser publisher serves a native subscriber after a group is evicted or + was never cached, covering exact replay, empty and missing groups, + concurrent requests, and cancellation while awaiting a reply. +- **`fetchGroup`.** Becomes a one-range subscription on the request surface + above, or is removed. +- **Fills.** Mirror Rust's rule from [#7168](https://github.com/moq-dev/moq/pull/7168) + (decided 2026-10-29, maintainer): give up a run below a start already + resolved under a budget at least as loose as the reader's, and on a pre-06 + wire fill below the start only when the reader names a floor. `js/net` + requests no fills today, so nothing over-fetches yet; this keeps it that way + once it does. With fills, a bounded read also mirrors Rust's head fill + ([moq-dev/moq#7625](https://github.com/moq-dev/moq/pull/7625)): it hands out + a cached group missing its first frames only once a fill answers from frame + 0, and gives one whose fill is refused up as a miss. Until then a JS reader + counts a hole the feed no longer owes as given up, with no fill asked. Ranges stay whole groups ([moq-dev/moq#6465](https://github.com/moq-dev/moq/pull/6465)), so the group model keeps refusing a nonzero lite `frameStart` as a protocol diff --git a/rs/moq-net/src/model/subscription.rs b/rs/moq-net/src/model/subscription.rs index acd89f3a81d..cd07a2368d4 100644 --- a/rs/moq-net/src/model/subscription.rs +++ b/rs/moq-net/src/model/subscription.rs @@ -196,8 +196,8 @@ pub enum Anchor { /// or before it, a hop whose [`Requests`](crate::track::Requests) handler serves its /// history looks the start up there (see [`Fill::time`](crate::track::Fill::time)), /// reading nothing until it answers; otherwise, or when the handler can't tell, a start - /// reaches back to the oldest group. While no group has started at all it waits like - /// `Relative(0)`. An end settles once a group starting + /// reaches back to the oldest group. While no group has started at all it waits for one + /// that has, reading nothing of its range meanwhile. An end settles once a group starting /// after it is known and every sequence below that one is accounted for (started, /// dropped, or never to start). Until then reads stop at the first held group that /// may still start, so at the live edge at most that one group past the end is read diff --git a/rs/moq-net/src/model/subscription/window.rs b/rs/moq-net/src/model/subscription/window.rs index 6b7d76727ac..1cbc2fa50b9 100644 --- a/rs/moq-net/src/model/subscription/window.rs +++ b/rs/moq-net/src/model/subscription/window.rs @@ -172,6 +172,10 @@ pub(in crate::model) struct Window { /// The spans still looking up their time start (see [`Self::found`]), which read nothing /// meanwhile: kept apart so a probe finds them without visiting every span. looking: Vec, + /// The spans whose time start waits on a group starting, which read nothing meanwhile. + /// Their end resolved when asked, so a relative one counts back from that latest group + /// however many arrive while they wait. + waiting: Vec, /// The unsettled time ends, grouped by walk. walks: Vec, /// The walks waiting out a gap below the group their ends fall before, which move only @@ -232,6 +236,8 @@ struct Span { /// A time start the groups held can't place, looked up from the track's history: the /// span reads nothing until the lookup answers. looking: Option, + /// A time start no group has started for yet: the span reads nothing until one has. + waiting: bool, } /// A time start being looked up: its time, and the exclusive end of the run below every @@ -573,6 +579,7 @@ impl Window { ranges: ranges.to_vec(), spans: Vec::new(), looking: Vec::new(), + waiting: Vec::new(), walks: Vec::new(), parked: BTreeMap::new(), seen: Seen::default(), @@ -603,7 +610,7 @@ impl Window { } /// Resolve what the groups now held settle: relative anchors once `latest` exists, and - /// time ends as groups start. Returns whether the groups in range moved. + /// time starts and ends as groups start. Returns whether the groups in range moved. pub fn settle(&mut self, latest: Option, holdings: &impl Holdings) -> bool { if self.pending { if latest.is_none() { @@ -613,16 +620,86 @@ impl Window { let _ = self.place(latest, holdings); return !self.pending; } - if self.walks.is_empty() && self.parked.is_empty() { - return false; + let mut moved = !self.waiting.is_empty() && self.start_waiting(holdings); + if !self.walks.is_empty() || !self.parked.is_empty() { + moved |= self.walk(holdings); } - let moved = self.walk(holdings); if moved || self.stale { self.rebuild(); } moved } + /// Start the spans whose time start waits on a group starting, once one has, or once + /// nothing can start any more. Returns whether any did. + fn start_waiting(&mut self, holdings: &impl Holdings) -> bool { + let mut started = false; + for position in std::mem::take(&mut self.waiting) { + let Anchor::Time(pts) = self.ranges[self.spans[position].index].start else { + unreachable!("only a time start waits on a group starting"); + }; + let (start, floored) = match holdings.containing(pts) { + Containing::Group(group) => (group, true), + // The group holding it is older than any held, or nothing held can start. + Containing::Before => (0, false), + Containing::Unknown if holdings.complete() => (0, false), + Containing::Unknown => { + self.waiting.push(position); + continue; + } + }; + started = true; + self.start_span(position, pts, start, floored, holdings); + } + if started { + self.stale = true; + self.name(); + } + started + } + + /// Start the span at `position`, whose time start `pts` waited, at `start`: looked up + /// where the groups held can't place it, as when first resolved, and otherwise reading + /// its groups, its time end walking from there. + fn start_span(&mut self, position: usize, pts: Timestamp, start: u64, floored: bool, holdings: &impl Holdings) { + let span = &mut self.spans[position]; + (span.start, span.floored, span.waiting) = (start, floored, false); + if let Some(below) = unresolved(start, floored, holdings) { + span.looking = Some(Looking { pts, below }); + self.looking.push(position); + return; + } + let reach = match floored { + true => start, + false => holdings.next_start(0).map_or(start, |(oldest, _)| oldest), + }; + let limit = match span.end { + End::Before(end) => Some(end), + End::Open => None, + End::Time(end) => { + self.walks.push(Walk { + ends: BTreeSet::from([Ending { + pts: end, + span: position, + }]), + trail: Trail::new(start, reach), + low: start, + low_reach: reach, + was: Some(start), + }); + return; + } + }; + if let Some(last) = limit.map_or(Some(u64::MAX), |end| end.checked_sub(1)) + && last >= start + { + self.resolved.insert(start..=last); + if reach <= last { + self.resolved_missable.insert(reach..=last); + } + } + } + /// Walk each unsettled time end on, settling those that can be, and merge the walks /// that come to agree. Returns whether the groups in range moved. fn walk(&mut self, holdings: &impl Holdings) -> bool { @@ -745,9 +822,12 @@ impl Window { self.walks.iter().filter_map(|walk| walk.trail.speculative) } - /// The lowest sequence whose start could still move an unsettled time end not parked, - /// `None` without one or once the last sequence is in range. + /// The lowest sequence whose start could still move an unsettled time start, or a time + /// end not parked, `None` without one or once the last sequence is in range. pub fn watch_starts(&self) -> Option { + if !self.waiting.is_empty() { + return Some(0); + } self.walks.iter().filter_map(|walk| walk.trail.unknown()).min() } @@ -766,17 +846,17 @@ impl Window { self.admission.admits(sequence, timestamp) } - /// The lowest group in range, `None` while a relative anchor waits. Nothing in range - /// floors nowhere: group 0. + /// The lowest group in range, `None` while an anchor waits. Nothing in range floors + /// nowhere: group 0. pub fn floor(&self) -> Option { - match self.pending { + match self.pending || !self.waiting.is_empty() { true => None, false => Some(self.groups.lowest().unwrap_or(0)), } } /// The lowest group the ranges name, `None` while one reaches back to the oldest group - /// or a relative anchor waits on a first group. + /// or an anchor waits. pub fn start(&self) -> Option { match self.pending || !self.all_named { true => None, @@ -784,22 +864,22 @@ impl Window { } } - /// The lowest group a range names, `None` when none does or a relative anchor waits. + /// The lowest group a range names, `None` when none does or an anchor waits. /// Unlike [`Self::start`], a range reaching back to the oldest group names nothing. pub fn named(&self) -> Option { - match self.pending { + match self.pending || !self.waiting.is_empty() { true => None, false => self.named, } } - /// Each range resolved, in the order given: `None` for one empty as given, still - /// waiting on a first group, or still looking up its start. One that only resolves - /// empty keeps its span, ending at or below its start. An unsettled time end is reported - /// open, never at its cap. + /// Each range resolved, in the order given: `None` for one empty as given, whose start + /// still waits, or still looking up its start. One that only resolves empty keeps its + /// span, ending at or below its start. An unsettled time end is reported open, never at + /// its cap. pub fn resolved(&self) -> Vec> { let mut resolved = vec![None; self.ranges.len()]; - for span in self.spans.iter().filter(|span| span.looking.is_none()) { + for span in self.spans.iter().filter(|span| span.looking.is_none() && !span.waiting) { resolved[span.index] = Some(Resolved { start: span.start, end: match span.end { @@ -812,7 +892,7 @@ impl Window { } /// Where each range starts once delivery starts at `from`, in the order given: `None` - /// for one with no group there, or still waiting on a first group. + /// for one with no group there, or whose start still waits. pub fn starts(&self, from: u64) -> Vec> { self.resolved() .into_iter() @@ -824,8 +904,8 @@ impl Window { .collect() } - /// The exclusive end past every group in range, `None` when one stays open. An - /// unsettled time end counts as open: its cap only holds reads back meanwhile. + /// The exclusive end past every group in range, `None` when one stays open or an anchor + /// waits. An unsettled time end counts as open: its cap only holds reads back meanwhile. pub fn end(&self) -> Option { if !self.settled { return None; @@ -1000,6 +1080,7 @@ impl Window { } self.spans.clear(); self.looking.clear(); + self.waiting.clear(); self.walks.clear(); self.parked.clear(); let (mut fixed, mut fixed_missable) = (Vec::new(), Vec::new()); @@ -1007,11 +1088,30 @@ impl Window { // The time ends that start alike walk alike, until a group starts between them. let mut walks = std::collections::BTreeMap::<(u64, u64), BTreeSet>::new(); for (index, range) in self.ranges.iter().enumerate().filter(|(_, range)| !range.is_empty()) { - let (Some((start, floored)), Some(end)) = (start(range.start, latest, holdings), end(range.end, latest)) - else { + let Some(end) = end(range.end, latest) else { self.pending = true; continue; }; + let (start, floored) = match start(range.start, latest, holdings) { + Start::At(start, floored) => (start, floored), + Start::Pending => { + self.pending = true; + continue; + } + // Reads nothing until a group starts; see [`Self::start_waiting`]. + Start::Waiting => { + self.waiting.push(self.spans.len()); + self.spans.push(Span { + index, + start: 0, + end, + floored: false, + looking: None, + waiting: true, + }); + continue; + } + }; // A time start the groups held can't place reads nothing until its lookup answers. let looking = match range.start { Anchor::Time(pts) => unresolved(start, floored, holdings).map(|below| Looking { pts, below }), @@ -1025,6 +1125,7 @@ impl Window { end, floored, looking, + waiting: false, }); continue; } @@ -1057,6 +1158,7 @@ impl Window { end, floored, looking: None, + waiting: false, }); } self.admission = Admission::new(&self.ranges, &self.spans); @@ -1097,8 +1199,13 @@ impl Window { /// parking or waking changed them. fn rebuild(&mut self) { self.covered = None; - // A span still looking up its start reads nothing until it knows which group that is. - self.settled = !self.pending && self.walks.is_empty() && self.parked.is_empty() && self.looking.is_empty(); + // A span still looking up or waiting on its start reads nothing until it knows which + // group that is. + self.settled = !self.pending + && self.walks.is_empty() + && self.parked.is_empty() + && self.looking.is_empty() + && self.waiting.is_empty(); let (groups, missable): (Vec<_>, Vec<_>) = self.walks.iter().map(Walk::runs).unzip(); self.groups.set_layer(Runs::new(groups.into_iter().flatten())); self.missable.set_layer(Runs::new(missable.into_iter().flatten())); @@ -1252,20 +1359,36 @@ impl Stab { } } -/// The first group a range starts at, and whether it names that group rather than -/// reaching back to the oldest one. `None` while it waits on a latest group. -fn start(anchor: Anchor, latest: Option, holdings: &impl Holdings) -> Option<(u64, bool)> { +/// Where a range starts. +enum Start { + /// At this group, and whether it names that group rather than reaching back to the + /// oldest one. + At(u64, bool), + /// A time start no group has started for yet, so it reads nothing. + Waiting, + /// Waiting on a latest group, reading everything meanwhile. + Pending, +} + +fn start(anchor: Anchor, latest: Option, holdings: &impl Holdings) -> Start { match anchor { - Anchor::Absolute(start) => Some((start, true)), + Anchor::Absolute(start) => Start::At(start, true), // Nothing sits below group 0, so the farthest offset needs no latest group. - Anchor::Relative(u64::MAX) => Some((0, false)), - Anchor::Relative(back) => Some((latest?.saturating_sub(back), true)), + Anchor::Relative(u64::MAX) => Start::At(0, false), + Anchor::Relative(back) => latest.map_or(Start::Pending, |latest| Start::At(latest.saturating_sub(back), true)), Anchor::Time(pts) => match holdings.containing(pts) { - Containing::Group(group) => Some((group, true)), + Containing::Group(group) => Start::At(group, true), // The group holding it is older than any held, so reach back to the oldest. - Containing::Before => Some((0, false)), - // Nothing has started yet: like the live edge, the latest group. - Containing::Unknown => Some((latest?, true)), + Containing::Before => Start::At(0, false), + // Before any group, as for a relative anchor, which a hop that holds none (such as + // a reader following route copies) leaves to the copies to resolve. + Containing::Unknown if latest.is_none() => Start::Pending, + // Nothing can start any more, so nothing held reaches past it. + Containing::Unknown if holdings.complete() => Start::At(0, false), + // Nothing has started yet, so wait for a group that has, rather than guess the + // latest: groups arrive before the frames that show where they start, so a copy + // can see the live edge before the groups its publisher resolved the time to. + Containing::Unknown => Start::Waiting, }, } } @@ -1549,10 +1672,14 @@ mod tests { let window = Window::new(&[range(time(1000), time(2500))], Some(6), &groups).unwrap(); assert_eq!(covered(&window, 8), vec![0, 1, 2]); - // Nothing started yet: a time start waits like the live edge. + // Nothing started yet: a time start waits on a first group, then on one starting. let mut window = Window::new(&[range(time(1000), None)], None, &Groups::default()).unwrap(); assert!(window.is_pending()); - assert!(window.settle(Some(4), &Groups::new([(4, Held::Pending)]))); + let mut held = Groups::new([(4, Held::Pending)]); + assert!(window.settle(Some(4), &held)); + assert!(!window.contains(4), "4 has not started"); + held.held.insert(4, Held::Started(at(1000))); + assert!(window.settle(Some(4), &held)); assert_eq!(window.floor(), Some(4)); } @@ -1925,6 +2052,104 @@ mod tests { assert!(!window.admits_datagram(1, Some(at(7000)))); } + /// A time start no held group has started for yet reads nothing of its range, rather + /// than guess the latest group, and resolves once one starts. A track that can start + /// nothing more reaches back to the oldest group. + #[test] + fn a_time_start_waits_for_a_group_to_start() { + let pending = (2..10).map(|sequence| (sequence, Held::Pending)); + let mut groups = Groups::new(pending.clone()); + let ranges = [ + range(time(2500), None), + range(Anchor::Absolute(20), Anchor::Absolute(21)), + ]; + let mut window = Window::new(&ranges, Some(9), &groups).unwrap(); + assert!(!window.is_pending() && !window.is_settled()); + assert_eq!(covered(&window, 22), vec![20], "only the other range reads"); + assert_eq!((window.floor(), window.start(), window.end()), (None, None, None)); + assert_eq!( + window.resolved(), + vec![ + None, + Some(Resolved { + start: 20, + end: Some(21) + }) + ] + ); + assert_eq!(window.watch_starts(), Some(0), "any group starting may resolve it"); + assert!(!window.settle(Some(9), &groups), "nothing started"); + + groups.held.insert(3, Held::Started(at(3000))); + assert!(window.settle(Some(9), &groups)); + assert_eq!( + window.resolved()[0], + Some(Resolved { start: 0, end: None }), + "3 starts past it" + ); + groups.held.insert(2, Held::Started(at(2000))); + let window = Window::new(&ranges, Some(9), &groups).unwrap(); + assert_eq!(window.resolved()[0], Some(Resolved { start: 2, end: None })); + assert!(window.is_settled()); + + let groups = Groups { + complete: true, + ..Groups::new(pending) + }; + let window = Window::new(&ranges, Some(9), &groups).unwrap(); + assert_eq!(window.resolved()[0], Some(Resolved { start: 0, end: None })); + } + + /// A relative end counts back from the latest group it was asked against, however many + /// arrive while its time start waits. + #[test] + fn a_waiting_time_start_keeps_its_relative_end() { + for latest in [None, Some(9)] { + let mut groups = Groups::default(); + let ranges = [range(time(2500), Anchor::Relative(0))]; + let mut window = Window::new(&ranges, latest, &groups).unwrap(); + assert_eq!(window.is_pending(), latest.is_none()); + groups.held.extend((2..10).map(|sequence| (sequence, Held::Pending))); + window.settle(Some(9), &groups); + assert!(!window.is_pending()); + groups.held.insert(10, Held::Pending); + assert!(!window.settle(Some(10), &groups), "nothing started"); + + groups.held.insert(2, Held::Started(at(2000))); + assert!(window.settle(Some(10), &groups)); + assert_eq!( + window.resolved()[0], + Some(Resolved { + start: 2, + end: Some(10) + }) + ); + } + } + + /// A waiting time start that resolves after a later range still bounds datagrams by its + /// own relative end. + #[test] + fn a_resolved_waiting_start_judges_datagrams_by_its_own_end() { + let mut groups = Groups::new((2..10).map(|sequence| (sequence, Held::Pending))); + let ranges = [ + range(time(2500), Anchor::Relative(0)), + range(Anchor::Absolute(20), Anchor::Absolute(21)), + ]; + let mut window = Window::new(&ranges, Some(9), &groups).unwrap(); + groups.held.insert(2, Held::Started(at(2000))); + assert!(window.settle(Some(9), &groups)); + assert_eq!( + window.resolved()[0], + Some(Resolved { + start: 2, + end: Some(10) + }) + ); + assert!(window.admits_datagram(9, Some(at(3000)))); + assert!(!window.admits_datagram(15, Some(at(3000))), "past the relative end"); + } + #[test] fn a_waiting_window_judges_a_datagram_against_both_times() { let window = Window::new(&[range(time(3000), time(3500))], None, &Groups::default()).unwrap(); diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index c1f6b13ae63..46628a49d2f 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -22930,6 +22930,49 @@ mod test { assert_eq!(drain(&mut seek), vec![1, 2]); } + /// A time start resolves once a group starts, not at the latest group: a copy can show + /// the route's answer (here an empty group) before the groups its publisher resolved the + /// time to start, and a producer can open groups before their first frames. + #[test] + #[cfg(feature = "async")] + fn a_time_start_waits_for_a_group_to_start() { + let at = |millis| Timestamp::from_millis(millis).unwrap(); + let since = [Range { + start: Anchor::Time(at(2500)), + end: None, + }]; + for (copy, order) in [ + (false, Order::Ascending), + (true, Order::Ascending), + (true, Order::Descending), + ] { + let mut producer = track_producer("test", Info::timed()); + if copy { + producer.set_idle(); + producer.set_live(Some(Position::group(9)), Some(2)); + } + let mut groups: Vec<_> = (2..9) + .map(|sequence| producer.create_group(group::Info { sequence }).unwrap()) + .collect(); + producer + .create_group(group::Info { sequence: 9 }) + .unwrap() + .finish() + .unwrap(); + + let mut subscriber = producer.subscribe(replay().with_ranges(since).with_order(order)); + assert_eq!(drain(&mut subscriber), Vec::::new(), "copy: {copy}, {order:?}"); + for (group, sequence) in groups.iter_mut().zip(2..) { + group + .write_frame(at(sequence * 1000), bytes::Bytes::from_static(b"x")) + .unwrap(); + } + let mut read = drain(&mut subscriber); + read.sort_unstable(); + assert_eq!(read, (2..10).collect::>(), "copy: {copy}, {order:?}"); + } + } + /// A time end stays open across a group that has yet to start: it may start before the /// end, so a later group starting after it settles nothing. #[test] diff --git a/test/interop/clients/js-native/finish.ts b/test/interop/clients/js-native/finish.ts index b943dae595e..aeca88282ec 100644 --- a/test/interop/clients/js-native/finish.ts +++ b/test/interop/clients/js-native/finish.ts @@ -37,7 +37,7 @@ export async function readFinish( if (!format) throw new Error("unsupported finish media container"); const subscription = broadcast .track(Catalog.track(name, config)) - .subscribe({ delay: { max: maxDelay }, groups: { start: { included: 0 } } }); + .subscribe({ delay: { max: maxDelay }, ranges: [{ start: { absolute: 0 } }] }); const media = new Container.Consumer(subscription, { format, delay: { max: maxDelay } }); let count = 0; try { diff --git a/test/interop/clients/js-native/goaway.ts b/test/interop/clients/js-native/goaway.ts index 52af4e03c22..570f8729e6d 100644 --- a/test/interop/clients/js-native/goaway.ts +++ b/test/interop/clients/js-native/goaway.ts @@ -31,7 +31,7 @@ try { const initial = await active(requests[0]); // moq-transport carries no instance, so the session mints one for each announcement. assert(initial.instance && Moq.Instance.isMinted(initial.instance), "moq-transport carries no instance"); - let track = initial.track("video").subscribe({ groups: { start: { included: 0 } } }); + let track = initial.track("video").subscribe({ ranges: [{ start: { absolute: 0 } }] }); try { for (let sequence = 0; sequence < 3; sequence++) { const group = await track.recvGroup(); @@ -67,7 +67,7 @@ try { replacementProbe = wireOf(replacement) .consume(Moq.Path.from("handover")) .track("video") - .subscribe({ groups: { start: { included: 1 } } }); + .subscribe({ ranges: [{ start: { absolute: 1 } }] }); const probe = await replacementProbe.recvGroup(); assert(probe, "replacement has no group"); try { @@ -87,7 +87,7 @@ try { restarted.instance !== initial.instance, "the replacement announcement is another minted instance", ); - const next = restarted.track("video").subscribe({ groups: { start: { included: 2 } } }); + const next = restarted.track("video").subscribe({ ranges: [{ start: { absolute: 2 } }] }); track.close(); track = next; process.stdout.write("replacement ready\n"); diff --git a/test/interop/clients/js-native/tail.ts b/test/interop/clients/js-native/tail.ts index ad45e4f993b..11e1a90c23d 100644 --- a/test/interop/clients/js-native/tail.ts +++ b/test/interop/clients/js-native/tail.ts @@ -147,7 +147,7 @@ try { // head predates the subscription. const track = broadcast.track("tail").subscribe({ delay: { max: Milli(drop ? 60_000 : 5000) }, - ...(late ? {} : { groups: { start: { included: 0 } } }), + ...(late ? {} : { ranges: [{ start: { absolute: 0 } }] }), }); if (compress) assert.equal((await track.info()).compress, version === "moq-lite-07-wip"); const seen: number[] = [];