Claim check for any campaign in check_campaign and the ready-to-send check

AGENTS.md

@@ -39,7 +39,7 @@ Next.js App Router in JavaScript, Tailwind CSS with shadcn/ui, Cloudflare D1 thr
 - **Storage**: models in `models/` (D1 in production, SQLite for tests and local scripts), images in the R2 bucket bound as `IMAGES`, or S3 behind CloudFront, or Vercel Blob (`lib/image-storage.js`), rate limits in Upstash Redis or memory (`lib/rate-limit.js`).
 - **Comments**: subscribers comment on the web issues with no account: a signed link (`lib/reader-links.js`, four kinds, each signed with a key of its own) starts a session for one newsletter (`lib/reader-session.js`), and every function checks the contact is still an active subscriber. A comment's text is parsed to parts and printed as text, never HTML (`lib/comment-body.js`). Readers act through `lib/comments.js`, the owner through `lib/comment-moderation.js` and the `Comments` actions; hiding and removing always carry a reason the author sees. Emails (`lib/comment-notify.js`) are one digest an hour at most to the owner and one reply email to an author, never one per comment. A reader's email never appears in anything public.
 - **Undo send**: the dashboard's Send (the `startSend` server action, `send_campaign` with its dashboard-only `holdForUndo`) puts a draft in the `held` status for the newsletter's `sendUndoSeconds` (`lib/send-hold-rules.js`, default 30) instead of queueing it (`lib/send-hold.js`), and a `release-held-send` job hands it to `queueCampaign` like any send. Release and `undo_campaign_send` are each one write on the same row (held and not yet claimed), so exactly one wins; the hourly cron releases a hold whose job never ran. API, MCP and approved agent sends are never held, and a scheduled send is unaffected. Limits are checked at hold time and again by `queueCampaign`. `BarSend` (in `LayoutDashboard`) is the countdown.
-- **Ready-to-send check**: `check_campaign` (`lib/ready-check-data.js` gathers, `lib/ready-check.js` words each item as pass, warning, blocker or info; the review step's `PanelReady` and the approval card show it). A blocker is what the send itself refuses (`checkCampaignSend` and `getPresendChecks` give the words); a warning never stops a send. The links (`lib/link-check.js`, the `check-links` job, results in `LinkCheck`, kept for the same set of addresses) are the one fetch of an address in an issue, held to `lib/link-check-rules.js`: public hosts, default ports, three redirects followed by hand and each checked.
+- **Ready-to-send check**: `check_campaign` (`lib/ready-check-data.js` gathers, `lib/ready-check.js` words each item as pass, warning, blocker or info; the review step's `PanelReady` and the approval card show it). A blocker is what the send itself refuses (`checkCampaignSend` and `getPresendChecks` give the words); a warning never stops a send. The links (`lib/link-check.js`, the `check-links` job, results in `LinkCheck`, kept for the same set of addresses) are one fetch of an address in an issue, held to `lib/link-check-rules.js`: public hosts, default ports, three redirects followed by hand and each checked. The claim check is the other (`docs/CLAIM-CHECK.md`: `lib/claim-check-rules.js` reads the issue's figures and words the report, `lib/claim-check.js` keeps it in `ClaimCheck` per `campaignContentHash`, the `check-claims` job runs `lib/claim-check-run.js` through the Briefings fetcher's guards): dead links, figures not on the pages their paragraph links to, figures with no source, code only. It never runs in a send; its findings are warnings unless the newsletter has `requireClaimCheck`, which `getSendBlocker` reads.
 - **Editing a sent issue's page**: `update_published_issue` (`lib/actions/published-issues.js`, class `write`) changes only the web version of a sent issue (title, subtitle, body, tags, search settings) in place, once its send is over; the emails that went out are built from the campaign at send time, so nothing already sent changes, but a later step or resend reads the new words. The page as it was goes into `CampaignVersion` (reason `edit`, the same 30) and `restore_campaign_version` works on a sent issue the same way. The RSS feed has no updated date: the entry keeps its `pubDate` and guid and carries the new content; the sitemap's lastmod follows `seo.updatedAt`.
 - **Content calendar**: issues and LinkedIn posts planned in one place (`docs/CONTENT-CALENDAR.md` has the model). `ContentItem` (channel `newsletter` or `linkedin`; copy, funnel stage, pillar, references, private notes, visuals as pointers to `Upload` records, a plan, review), `EditorialBatch` (with an optional recurring pattern that only `apply_editorial_pattern` uses) and `ContentRevision` (the last 50 states of an item's content), in `lib/content-items.js`, `lib/content-batches.js`, `lib/content-plan.js` and the rules in `lib/content-rules.js`; the actions are `lib/actions/content-items.js` and `content-batches.js`, the pages are under `app/(app)/campaigns/calendar`. Planning never sends, schedules or publishes: nothing in these files writes a campaign or queues a job, every planning action is class `write-draft` (reads are `read`, deletes `destructive`), and approving is `decide_content_review`, a `dashboardOnly` action, so no key can approve. Editorial state (`idea` to `delivered`) is apart from a campaign's delivery; a LinkedIn post is posted by hand and only recorded (`record_content_publication`, which needs an approved item and a LinkedIn address), and nothing may call it scheduled or published otherwise. A planned day is a `YYYY-MM-DD` plus an optional wall-clock time in an IANA zone, with its instant worked out by `zonedTimeToInstant` (`lib/time-zone.js`). An edit to the title, hook, body, call to action, destination, visuals or linked issue of an approved item takes the approval back (`changes_requested`, `editedAfterApproval`); a move does not. `externalId` makes creates and imports repeat safely (unique per account). `get_content_calendar` and `lib/calendar.js` are unchanged and leave drafts out; `get_content_plan` is the one that shows plans beside real deliveries.
 - **Growth**: each newsletter's RSS feed (`lib/archive-feed.js`), the account export (any account may run it, even blocked or without a plan: `export_account_data` sets `whileBlocked` and `requiresPlan: false`, and that stays so) and the referral program (`lib/referrals.js`; reward emails are `Message` kind `referral`, sent like welcome emails). Newsletters recommend each other for free (`lib/recommendations.js`, rules in `lib/recommendation-rules.js`): a reader's email reaches another newsletter only through their own tick on the "You might also like" step, as a normal signup with that newsletter's confirmation email, and the dashboard is at `/recommending` because `/recommendations` is the public directory. Recommendations stay inside a region.

app/(app)/newsletters/actions.js

@@ -58,6 +58,7 @@ function readSettings(formData) {
 		preferencesLink: formData.get("preferencesLink") === "on",
 		utmLinks: formData.get("utmLinks") === "on",
 		shareButtons: formData.get("shareButtons") === "on",
+		requireClaimCheck: formData.get("requireClaimCheck") === "on",
 		...(formData.has("customHeaders") && { customHeaders: readHeaderLines(String(formData.get("customHeaders") ?? "")) }),
 		...(formData.has("sendUndoSeconds") && { sendUndoSeconds: Number(readText(formData, "sendUndoSeconds")) }),
 		author: { bio: readText(formData, "authorBio"), imageUrl: readText(formData, "authorImageUrl") },

app/(site)/docs/briefings-bot/page.js

@@ -26,11 +26,16 @@ export default function BriefingsBotPage() {
 
 			<Section title="What it reads" stagger={2}>
 				<p>
-					Only pages that a FlockStack user&apos;s briefing found: a result of a search for the briefing&apos;s topic, a story in a feed the
+					Pages that a FlockStack user&apos;s briefing found: a result of a search for the briefing&apos;s topic, a story in a feed the
 					briefing reads, or a page linked from one of those. It does not crawl a site, follow its links beyond one step, or keep a copy of
 					a page after the briefing&apos;s run: it keeps the facts it needs for a short summary, at most 25 words of quotation, and the
 					link to the page.
 				</p>
+				<p className="mt-4">
+					It also opens the pages a FlockStack issue links to when its writer checks the issue before sending it: it looks for the
+					issue&apos;s figures on the pages it cites and checks that its links load, at most 25 links and 12 pages per check, and keeps
+					nothing of the pages it read.
+				</p>
 				<p className="mt-4">
 					It reads at most 1 MB of a page, waits at most 8 seconds, makes at most 2 requests to one host at a time and at most 300
 					requests in one run, and only to public addresses over https.

app/api/cron/hourly/route.js

@@ -4,6 +4,7 @@ import { runAgentHousekeeping } from "@/lib/agent-housekeeping";
 import { announceTelegram } from "@/lib/alerts";
 import { sendCommentDigests } from "@/lib/comment-notify";
 import { sendConfirmationReminders } from "@/lib/confirmation-reminders";
+import { purgeOldClaimChecks } from "@/lib/claim-check";
 import { withCronSecret } from "@/lib/cron";
 import { purgeCustomRequests } from "@/lib/custom-requests";
 import { followDedicatedIps } from "@/lib/dedicated-ip";
@@ -44,10 +45,10 @@ async function runBriefingsInCron() {
 // exports whose links ran out, the owners' digests of new comments and reports, the extra emails Polar hasn't been told
 // about yet, dedicated IPs that no longer match their add-on, the day of every IP warm-up and the stock of spare IPs, the reminders to signups that haven't confirmed, and the
 // Custom plan requests past their two years, the RSS feeds' kept copies nobody has read for a day, comps and gifts that ran out, and
-// pledge charging whose job went quiet, and the agent approvals nobody decided and the agent activity past its 90 days, the held sends whose release job never ran, and the link checks nobody looked at for a week.
+// pledge charging whose job went quiet, and the agent approvals nobody decided and the agent activity past its 90 days, the held sends whose release job never ran, and the link and claim checks nobody looked at for a week.
 // On a self-hosted server with BRIEFINGS_IN_CRON=true, also a tick of the briefings engine.
 export const GET = withCronSecret(async () => {
-	const [postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, editorSnapshots, briefings] = await Promise.all([
+	const [postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, claimChecks, editorSnapshots, briefings] = await Promise.all([
 		checkPostshibaStatus(),
 		announceTelegram(),
 		expireAccountExports(),
@@ -66,9 +67,10 @@ export const GET = withCronSecret(async () => {
 		runAgentHousekeeping(),
 		sweepHeldSends(),
 		purgeOldLinkChecks(),
+		purgeOldClaimChecks(),
 		expireCollaborationSnapshots(),
 		runBriefingsInCron(),
 	]);
-	console.info(`[hourly] ${JSON.stringify({ postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, editorSnapshots, briefings })}`);
+	console.info(`[hourly] ${JSON.stringify({ postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, claimChecks, editorSnapshots, briefings })}`);
 	return { postshiba, telegram };
 });

app/api/jobs/check-claims/route.js

@@ -0,0 +1,18 @@
+import { NextResponse } from "next/server";
+import { withQStashSignature } from "@/lib/qstash-route";
+
+export const dynamic = "force-dynamic";
+export const maxDuration = 60;
+
+// Handled outcomes answer 200 so QStash doesn't retry them; a thrown error (database down) is retried. The checker and the
+// fetcher load only when a job comes in.
+async function handleCheck(request) {
+	const body = await request.json().catch(() => null);
+	const job = { accountId: body?.accountId, campaignId: body?.campaignId, hash: body?.hash };
+	const { runClaimCheck } = await import("@/lib/claim-check-run");
+	const result = await runClaimCheck(job);
+	console.info(`[claim-check] ${JSON.stringify({ ...job, ...result })}`);
+	return NextResponse.json(result);
+}
+
+export const POST = withQStashSignature(handleCheck);

components/FormNewsletter.jsx

@@ -429,6 +429,12 @@ export default function FormNewsletter({
 						In the email header
 					</span>
 				</Field>
+				<Field label="Claim check" hint="Before a send, the ready-to-send check opens the issue's links and looks for each figure on the pages its paragraph links to. Its findings are warnings unless you require it to pass." as="div">
+					<span className="flex items-center gap-2 text-sm">
+						<Switch name="requireClaimCheck" checked={draft.isClaimCheckRequired} onCheckedChange={(isChecked) => set("isClaimCheckRequired", isChecked)} aria-label="Require the claim check to pass before a send" />
+						Required before a send
+					</span>
+				</Field>
 				<Field label="Custom headers" hint="Extra headers on every email, one Name: value a line, 10 at most. Names starting with X-, List-Id, List-Archive, List-Help, List-Owner, List-Subscribe, Organization and Keywords are allowed. The sender, signature and unsubscribe headers can't be changed." as="div">
 					<Textarea name="customHeaders" value={draft.customHeaders} rows={3} placeholder="X-Newsletter-Group: weekly" aria-label="Custom headers" className="font-mono text-xs" onChange={update("customHeaders")} />
 				</Field>

components/PanelReady.jsx

@@ -55,7 +55,7 @@ export default function PanelReady({ ready, extra = [], isApproval = false, onFi
 			{[...blockers, ...warnings, ...infos].length > 0 && (
 				<ul className="divide-y divide-border/60">
 					{[...blockers, ...warnings, ...infos].map((entry) => (
-						<RowItem key={entry.id} entry={entry} isApproval={isApproval} isChecking={result?.isChecking && entry.id === "links"} onFix={onFix} onTest={onTest} />
+						<RowItem key={entry.id} entry={entry} isApproval={isApproval} isChecking={isStillChecking(result, entry.id)} onFix={onFix} onTest={onTest} />
 					))}
 				</ul>
 			)}
@@ -73,6 +73,13 @@ export default function PanelReady({ ready, extra = [], isApproval = false, onFi
 	);
 }
 
+// The links and the claim check are opened by a job: their rows spin until the result is in. A result from before the claim
+// check names no list and only the links are being opened.
+function isStillChecking(result, id) {
+	if (!result?.isChecking) return false;
+	return Array.isArray(result.checking) ? result.checking.includes(id) : id === "links";
+}
+
 const ICONS = {
 	blocker: { Icon: CircleX, label: "To fix", className: "text-red-500" },
 	warning: { Icon: CircleAlert, label: "To check", className: "text-amber-500" },

docs/BRIEFINGS-DESIGN.md

@@ -13,7 +13,7 @@ A briefing belongs to a newsletter. From a topic brief (no fixed source list), i
 | Encrypting a stored credential | `lib/credentials.js` (AES-256-GCM, `CREDENTIALS_ENCRYPTION_KEY`, AAD context per value), hidden fields with `select: false` |
 | An idempotent story queue | content items with `externalId` (`lib/content-items.js`) |
 | A draft issue with a retry key | `create_campaign` with `externalId`, called through `runAction` with its own `via` (as `lib/agent-bootstrap.js` does) |
-| Checks shown before a send | `check_campaign` (`lib/ready-check.js`, `lib/ready-check-data.js`) |
+| Checks shown before a send | `check_campaign` (`lib/ready-check.js`, `lib/ready-check-data.js`), with the claim check of any issue (docs/CLAIM-CHECK.md) |
 | Agent send approval | `agentApproval` on the newsletter, `guardCall` in `lib/agent-guard.js` |
 | Other languages for readers | reader translation (`readerTranslations`, `lib/reader-translation.js`; en, es, fr, de, it, pt) |
 | Lean Worker builds without Next.js | `scripts/build-api-worker.mjs` with a different entry, as the readers Worker does |
@@ -116,7 +116,7 @@ Why its own Worker: the app handler has about 250 KB left in its size budget, an
 4. **Rank** (see "Ranking" below), and keep the `shortlist` best stories, then after triage the full stories (`minFull` to `maxFull`) and the brief items. None left means `no_news`: no draft, and the owner gets a one-line email.
 5. **Extract.** For the survivors only, fetch the primary page (and at most one primary source it links to), reduce it to readable text (at most 20,000 characters), and ask the extract model for a record: what happened, who, numbers with the sentence each comes from, and the URL. Quotes are capped at 25 words. A paywalled page is never worked around: the story keeps only facts found on public pages, or is dropped.
 6. **Write.** One call to the write model with the style guide, the brief and the records only (no raw pages): full stories first, then the brief items as one line each with their link. It returns Markdown in which every paragraph cites at least one record by number; code turns the citations into links to the originals.
-7. **Check.** Code checks first: every number in the draft appears in the text of a page it cites; every link resolves (the link checker's rules); every story links to its original; no quote is over 25 words; length within bounds. Then the check model, which must differ from the writer, judges each claim against the record it cites (with a TypeSafe key: one Noul per claim, as in TypeSafe's citation check). Each check is recorded as `pass`, `warning` or `blocker`.
+7. **Check.** Code checks first: every number in the draft appears in the text of a page it cites; every link resolves (the link checker's rules); every story links to its original; no quote is over 25 words; length within bounds. Then the check model, which must differ from the writer, judges each claim against the record it cites (with a TypeSafe key: one Noul per claim, as in TypeSafe's citation check). Each check is recorded as `pass`, `warning` or `blocker`. The same code checks of links and figures, without a briefing's records (the issue's own links are its sources), run on every issue as the ready-to-send check's claim check (docs/CLAIM-CHECK.md, `lib/claim-check-rules.js`), through this engine's fetcher.
 8. **Translate** (beta). When `languages` has more than one, the translate model writes each other language from the checked draft, within its share of the budget. The translations are stored against the draft for readers who choose that language, labelled beta; one that does not fit the budget is skipped and noted.
 9. **Deliver.** Through `BriefingsDelivery`: the issue as a draft campaign (`externalId` `briefing:<briefingId>:<slot>`, so a retried run returns the same draft), one newsletter content item linked to it carrying every story's references (the calendar allows one newsletter item per issue, and per-story items would fill the 5,000-item cap within a year of daily runs; story keys stay on the run), a test email of the draft to `testEmail`, and a separate note to the owner with the run's checks, cost, discovery notes and a keep or drop link per story. The note is never part of the draft. With `send_if_checks_pass` (Phase 3), a run with no blocker and no warning is queued for sending, unless the newsletter has "ask me before an agent sends" on, which turns it into an approval request. A thin run, a run that stopped at the budget, a run with any blocker, and a run with a discovery provider that failed never send: they are held as a draft, with `heldReason` and the reason in the owner's email.