[ankurmans]

What it checks

This adds a claim check for any issue: an editor draft, an agent's create_campaign, a briefing's draft or an import. It runs as part of the ready-to-send check. It is the code half of the Briefings engine's check step, without a briefing's records: the issue's own links are its sources. The design note is docs/CLAIM-CHECK.md.

It reads every paragraph, list item, heading and table cell, plus the subject and preview text, and reports:

1. Dead links. A link counts as dead when it answers 404, 410 or another 4xx, can't be reached, has a name that doesn't resolve or redirects too many times. A link to an address inside a private network is refused by the guard and never requested. A 401, 403, 429, a 5xx, a slow answer or a robots.txt refusal is listed as "couldn't be confirmed" (unsureLinks). That matches the existing link check (judgeStatus): those answers don't tell us whether a reader can open the link.
2. Figures not on their pages. Each figure in a paragraph is looked for on every page that paragraph links to. It is reported only when all of those pages were read. If a page can't be read (a file, robots.txt, plain http, a refusal, or the caps), the figure is marked "unconfirmed", not failed. A figure in a heading, the subject or the preview may be backed by any page the issue links to.
3. Figures with no source. A paragraph or list item that has a figure and no link.

Figure formats. These all count as the same figure: $1.7 billion, 1.7bn, $1.7B, 1,700 million and 1,700,000,000. So do 8.0%, 8% and 8 percent. A figure written with a scale also matches a more exact figure on the page that rounds to it, so 1.71bn matches 1,712,000,000.

Ignored:

• years, dates (month names and numeric), times, ordinals and decades
• a list's numbering and labels (issue 12, page 40)
• numbers joined to a word (Q3, A320, COVID-19, v1.2.3)
• footnote marks, merge tags and numbers with a leading zero
• bare whole numbers under 10
• anything inside a URL or an email address

The check is code only and makes no model call. The doc describes a possible model step as a later option and leaves it out. It would check that a sentence says what its page says, read figures written in words, and read tables in "$ millions".

How it is bounded

• Per check: 25 links judged, and 12 pages read whole, starting with the pages that figures need. 60 requests in all, robots.txt and redirects included. No request starts after 25 s. Anything not reached is marked "not checked", never failed.

• Per request: the Briefings fetcher (lib/briefings/fetch.js createFetcher), reused as is:

  • https only for page reads
  • every resolved address must be public (DNS over HTTPS in Workers), checked again on each redirect
  • robots.txt is read for the user agent FlockStack-Briefings/1.0 (+https://flockstack.io/docs/briefings-bot)
  • 1 MB and 8 s per request, 2 at a time per host, no cookies

  The briefings-bot docs page now says it also reads the pages an issue cites when the issue is checked.

• Per account: at most 30 checks started an hour.

• Cache: the result is kept in a new ClaimCheck row per campaign for a day, keyed by campaignContentHash (the existing campaign content hash). The same words never fetch again; any edit checks again. The hourly cron deletes rows nobody has looked at for a week, and the account purge deletes them too.

• No job needed: an issue with figures but no links is answered at once. An issue with neither passes at once.

Blocking decision

Findings are warnings by default. The check runs in its own QStash job (check-claims), started by the ready-to-send check like the link check. A send never starts it, never waits for it and never fetches anything for it.

A newsletter can turn on requireClaimCheck, with the new "Claim check" switch in newsletter settings or the update_newsletter / get_newsletter field. When it is on, getSendBlocker refuses with send_check_failed until a finished check for the current content hash has no failures. getSendBlocker is behind send_campaign, schedule_campaign, the dashboard's Send, approved agent sends and get_campaign's sendProblem. The ready-to-send item shows as a blocker while the check runs or fails. This costs one indexed ClaimCheck read, only for those newsletters. "Couldn't be confirmed" never blocks. There is no per-campaign option and no plan gating.

Known limit: a scheduled issue is checked when it is scheduled. If it is edited after that, it is not checked again when its send time comes.

Parity

• Registry, REST and MCP: check_campaign gains three things:

  • claims: state, required, passed, deadLinks, unsureLinks, unmatchedFigures, unconfirmedFigures, uncitedClaims, counts, figuresChecked, linksChecked, linksTotal, pagesRead and checkedAt
  • a claims item in its list
  • checking, which says whether links, claims or both are still running

  GET /api/v1/campaigns/{id}/check and the MCP tool are the same action; the tests cover the REST API.

• Dashboard: the review step's "Ready to send?" card and the approval card show the item through PanelReady. PanelReady now shows a spinner only on rows that are still running. use-ready-check now polls for up to 50 s instead of 30 s, to cover the job.

• Link check: the existing link check is unchanged. When the Links row already names the same dead links, the claim row says "(see Links)" instead of listing them again.

Tests

New tests/claim-check.test.mjs with 16 tests. Fetch and the DNS resolver are stubbed, so nothing reaches the network. They cover:

• Links: a dead link (404, and an unreachable host) and a 403, which is only unconfirmed
• Figures: a figure not on its cited page, the same figure in a different format, and an uncited figure
• Ignored numbers: URL, date, year, ordinal and list numbers, and words that start like a month name
• Cache: an unchanged issue starts no job and makes no request; an edit checks again
• Guards:
  • private addresses are refused and none of them is requested: a private IP, a name that resolves to 10.x, and a redirect to 169.254.169.254
  • a robots.txt refusal
  • the caps on links, pages and time
• Jobs: a stale job opens nothing, and an invalid job is skipped through the route
• Whole flow: a plain editor draft (not a briefing's) through check_campaign and the REST API
• Require mode: requireClaimCheck refuses send and schedule while the check runs or fails, and allows the send after a fix
• Dashboard: the settings form saves the switch and clears it, and PanelReady spins only the running row

One existing expectation changed, because changing that behavior is the point of this PR. tests/ready-check.test.mjs asserted isChecking === false after the link job ran. isChecking now also covers the claim check, so the test asserts that checking is ["claims"], which means the links are done.

Full suite

npm test: 3961 tests, 3957 pass, 0 fail, 4 skipped (the existing skips). It ran around 23:40 UTC, inside the known automations-v2 clock window, and that test passed anyway.

A few small fixes landed while the suite was running: exact month-name matching, links kept on headings, a case-insensitive https check and one new test. After them, tests/claim-check.test.mjs, tests/ready-check.test.mjs and tests/ready-check-action.test.mjs were run again: 51/51 pass. npx eslint on every changed file is clean.

Bundle impact

Both builds use FLOCKSTACK_CLOUD_BUILD=true and the same node_modules. Each runs npx opennextjs-cloudflare build, npm run build:api and npm run bundle:app, once on origin/main (23b1e77) and once on this branch:

Script	origin/main	this branch	change	budget
app handler.mjs	34,284,107	34,300,313	+16,206	34.75 MB
app upload (.worker/worker-entry.js)	36,408,198	36,424,234	+16,036	37.10 MB
API Worker	7,570,980	7,590,254	+19,274	8.15 MB

perf-budgets.json is unchanged, and every budget passes. The page-reading code is in the app only behind a lazy import() in the new job route: the Briefings fetcher, robots.txt, the page-text reader and lib/claim-check-run.js. The API Worker gets only what check_campaign and the send check need: the figure rules and the claim state and send check. It does not get the fetcher.

[ankurmans]

CI: the self-hosted runner had not picked up this PR's check job when this was written (it was still queued), so there is no CI result yet. Local results are in the description: full suite 3957 pass, 0 fail, 4 skipped; ESLint clean; size budgets pass.