feat(plugin): one render job per URL — URL-keyed schedule, one result schedules every device (v0.66.0)

[harper-joseph]

What

A render job is one URL. RenderSchedule now holds one row per URL (was one per <url>|<device> cacheKey). A claim hands the renderer one job naming every device in deviceTypes.default; the renderer (browser >= 1.23.0, #155) renders them in turn and posts one result — { id, url, deviceTypes, variants: [...] } — and the plugin makes one scheduling decision for the URL. Pages are still stored per device (PrerenderedPage stays keyed by <url>|<device>).

Depends on #155 being deployed to the render fleet first (see rollout order).

Why

Two independent rows per URL drift apart through every per-device path — render-now and revalidate write one device key on purpose, reconcile repairs a missing row with fresh jitter, each retry lane delays only its own device, and completion-relative rescheduling re-anchors each device to its own finish time. The code calls a split pair "a normal production state", and it costs:

• Correctness of the per-URL strikes counter, which was fed per device: both devices failing burned the fast-retry lane in one cycle, and one device succeeding reset the counter under the other. One result per URL is one strike per failed cycle.
• Mechanisms that exist only because of the split: PageVerification.basisAt, the per-row verdicts in the reenqueue accelerator, the probe's per-URL page claim written by whichever device rendered last. They still work; they now cover a rare transitional state instead of the steady state.
• Measured divergence: desktop and mobile structured data agreed byte-for-byte in 39/40 customer samples; the exception was a pair rendered 41h apart.
• Queue cost: half the schedule rows (~2.03M → ~1.0M fleet-wide), one reschedule write per URL render instead of two, and half the claim passes per render — the claim-scan degradation grows per reschedule at the seek point, so both halve.

How

• Schedule key: the URL. CacheKey.urlOf / deviceOf / isCacheKey are the only readers of a schedule key; extractUrl on a URL-shaped key would return ''. Residency hashes the URL either way, so ownership never moves. The primary-key attribute keeps its name cacheKey: Harper refuses to rename the primary key of a populated table (verified in databases.ts), so the schema comment says to read it as "the schedule key".
• No migration sweep. Pre-0.66.0 per-device rows convert the first time they render: the job renders exactly the device its key names, and its result writes the URL row and deletes the device row — two siblings fold into one URL row within a cycle at no extra renders. Both key shapes coexist for one cycle and every reader tolerates both (claim, sweep, unpin hatch, reconcile, orphan sweep, purge, accelerator, explainer). A pinned test walks the whole conversion.
• Result processing (processDecodedJobResult, rewritten around a list of variants): a legacy flat result is normalized to a one-variant list, so old and new browsers share one path. Precedence across variants is the precedence one device's result used to impose on the whole URL: a bail-at-nav redirect on any device decides the URL; a genuine non-indexable verdict on any device suppresses it (nothing stored); otherwise every rendered variant's page is stored (one lastCached for the pair, so the verification basis is exact), and a failed variant — renderer error, auth/transient status, or a device the browser was asked for and did not post back (not-attempted) — puts the URL in the retry lanes with the good pages kept; all rendered reschedules once. One render outcome emit per result (contract unchanged); one time_ms sample per variant.
• Per-device rows that remain by design: renderNow for a device outside deviceTypes.default writes a <url>|<device> row — a one-off render of that device, stored and retired without touching the URL row. For a default device, render-now now pulls the URL row forward (every device together) rather than adding a one-device render beside it.
• Wire format: claim returns { id: <row key>, url, deviceTypes, deviceType: deviceTypes[0], … } (deviceType for renderers that predate the list); job_result accepts the variants envelope (bodies concatenated, walked by contentLength, refused with a 400 when the lengths do not add up) or the legacy flat shape.
• Admin/explain: POST /prerender_admin/schedule takes { url } or { cacheKey }; the schedule read tries the URL row then the per-device row and reports scheduleKey + perDevice; revalidate writes the URL row. The console needs no change (it reads the same fields and still sends cacheKey).
• Docs: schema comments, README ("A job is one URL" + upgrade notes), METRICS (not-attempted detail; time_ms per variant), config descriptions.

Rollout order

Render fleet first (browser 1.23.0, #155), then this. A 1.23.0 renderer given a legacy per-device job posts the legacy shape, so it runs correctly against 0.64.x/0.65.x. The reverse is degraded, not broken: a pre-1.23.0 renderer given a URL job renders deviceType (the first device) and posts flat; this plugin attributes that to the first default device and the other devices go unrendered until the fleet is upgraded (pinned by a test).

On the customer cluster the conversion completes within one PDP cycle (96h) with no extra renders; reconcile (shipped disabled) treats a URL scheduled under a leftover device row as scheduled, so enabling it mid-conversion cannot double-render.

Tests

• 1019/1019 plugin tests locally; npm run lint, npm run format:check (this repo has no PR CI).
• New test/renderQueueVariants.test.js (20 tests): claim job shape for URL and per-device rows; body framing and its 400; the precedence matrix (all rendered / one failed / both failed = one strike / auth outranks transient / verdict wins / redirect wins / temp redirect / client-side refile); partial results (not-attempted); the per-device → URL row conversion incl. the fast lane (row kept) and slow lane (converted); a non-default one-device row leaving the URL row untouched; a targetless URL row dropped; an older renderer's flat result attributed to the first device.
• Updated fixtures in reconcile, renderQueueRedirect, suppressionStatus, targetRevalidate — those files deliberately keep seeding the pre-0.66.0 shape and posting legacy results, so they now also pin the conversion.

Version

@harperfast/prerender 0.64.1 → 0.66.0 (0.65.0 is reserved by #154). Release tag prerender-v0.66.0. Follow-ups after release: the customer component bump (deploy the render-service bump to 1.23.0 first).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request transitions the prerender scheduling queue from a per-device model to a per-URL model (v0.66.0), consolidating multiple device variants into a single job per URL to keep device snapshots aligned. It includes on-the-fly conversion of legacy per-device schedule rows during their first render, updates to metrics, administration endpoints, and extensive test coverage. Feedback suggests validating and rejecting unusable configuration overrides at the entry point where configuration is parsed or applied, rather than adding defensive guards or fallbacks (such as config.deviceTypes.default[0]) in downstream utility functions.

[gemini-code-assist]

Avoid adding defensive guards or fallbacks for config.deviceTypes.default in downstream utility functions. Instead, validate and reject unusable or structural configuration overrides at the entry point where the configuration is parsed or applied, restoring them to safe defaults.

References

1. Validate and reject unusable or structural configuration overrides at the entry point (where configuration is parsed or applied) by restoring them to safe defaults, rather than adding defensive guards in downstream utility functions.

[harper-joseph]

Agreed, and done in f9b0fa7: deviceTypes.default is now nonEmpty in the schema, so an emptied list is refused where config is applied (default kept — same mechanism as cacheKey.delimiter), and the downstream fallback in RenderQueue.js is gone. That also closes a related hole: claim and describeJob's fold rule read the same list, and a fallback in only one of them let them disagree. readScheduleRowLocal here reads config.deviceTypes.default[0] for the fallback per-device key, which is now guaranteed to exist. Tests: config.test.js (rejection at apply) and renderQueueVariants.test.js.

[harper-joseph]

Pushed f9b0fa7 addressing the auto-review note and a fresh-context adversarial pass over the diff (the outside-model review CLIs on this machine are currently non-functional, so that pass is the mitigation). Three real defects were found and fixed, each with a pinned test:

1. Precedence hole — a variant with an outcome this code does not know, or a redirected whose destination re-keyed to the same cache key, fell through to the all-rendered branch and rescheduled the URL as a success that stored nothing. failed is now the complement of rendered, as the per-key code's final else was; a result naming no device is a 400.
2. Non-default one-device row on a URL that redirects — Target.delete only removes the URL row and the default device keys, so a |tablet one-off was marked gone without being deleted and would have been re-granted every claim pass. retireSource now marks the row gone only when the job folds, and the end-of-result cleanup retires the rest.
3. Refile with a failed sibling — held the source lease (for a row just deleted) and charged the failure to the destination's target. The refile case now skips the retry lanes and logs the sibling.

Also: no double delete on suppress→maxStrikes for a folding row; the key-shape test keys on a supported device tail so a configurable delimiter that can appear inside a URL cannot misread a URL row; README carve-outs for bulk revalidate during the conversion cycle and render-now latency; configSchema arithmetic at one row per URL.

1029/1029 plugin tests, lint and format clean (the earlier figure in this comment was posted before one new test's final assertion was corrected in 99b2682 — the destination adopted after a 301 is legitimately claimable).