diff --git a/AGENTS.md b/AGENTS.md index 0116f87f..6ce6a79d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,7 +39,7 @@ Next.js App Router in JavaScript, Tailwind CSS with shadcn/ui, Cloudflare D1 thr - **Storage**: models in `models/` (D1 in production, SQLite for tests and local scripts), images in the R2 bucket bound as `IMAGES`, or S3 behind CloudFront, or Vercel Blob (`lib/image-storage.js`), rate limits in Upstash Redis or memory (`lib/rate-limit.js`). - **Comments**: subscribers comment on the web issues with no account: a signed link (`lib/reader-links.js`, four kinds, each signed with a key of its own) starts a session for one newsletter (`lib/reader-session.js`), and every function checks the contact is still an active subscriber. A comment's text is parsed to parts and printed as text, never HTML (`lib/comment-body.js`). Readers act through `lib/comments.js`, the owner through `lib/comment-moderation.js` and the `Comments` actions; hiding and removing always carry a reason the author sees. Emails (`lib/comment-notify.js`) are one digest an hour at most to the owner and one reply email to an author, never one per comment. A reader's email never appears in anything public. - **Undo send**: the dashboard's Send (the `startSend` server action, `send_campaign` with its dashboard-only `holdForUndo`) puts a draft in the `held` status for the newsletter's `sendUndoSeconds` (`lib/send-hold-rules.js`, default 30) instead of queueing it (`lib/send-hold.js`), and a `release-held-send` job hands it to `queueCampaign` like any send. Release and `undo_campaign_send` are each one write on the same row (held and not yet claimed), so exactly one wins; the hourly cron releases a hold whose job never ran. API, MCP and approved agent sends are never held, and a scheduled send is unaffected. Limits are checked at hold time and again by `queueCampaign`. `BarSend` (in `LayoutDashboard`) is the countdown. -- **Ready-to-send check**: `check_campaign` (`lib/ready-check-data.js` gathers, `lib/ready-check.js` words each item as pass, warning, blocker or info; the review step's `PanelReady` and the approval card show it). A blocker is what the send itself refuses (`checkCampaignSend` and `getPresendChecks` give the words); a warning never stops a send. The links (`lib/link-check.js`, the `check-links` job, results in `LinkCheck`, kept for the same set of addresses) are the one fetch of an address in an issue, held to `lib/link-check-rules.js`: public hosts, default ports, three redirects followed by hand and each checked. +- **Ready-to-send check**: `check_campaign` (`lib/ready-check-data.js` gathers, `lib/ready-check.js` words each item as pass, warning, blocker or info; the review step's `PanelReady` and the approval card show it). A blocker is what the send itself refuses (`checkCampaignSend` and `getPresendChecks` give the words); a warning never stops a send. The links (`lib/link-check.js`, the `check-links` job, results in `LinkCheck`, kept for the same set of addresses) are one fetch of an address in an issue, held to `lib/link-check-rules.js`: public hosts, default ports, three redirects followed by hand and each checked. The claim check is the other (`docs/CLAIM-CHECK.md`: `lib/claim-check-rules.js` reads the issue's figures and words the report, `lib/claim-check.js` keeps it in `ClaimCheck` per `campaignContentHash`, the `check-claims` job runs `lib/claim-check-run.js` through the Briefings fetcher's guards): dead links, figures not on the pages their paragraph links to, figures with no source, code only. It never runs in a send; its findings are warnings unless the newsletter has `requireClaimCheck`, which `getSendBlocker` reads. - **Editing a sent issue's page**: `update_published_issue` (`lib/actions/published-issues.js`, class `write`) changes only the web version of a sent issue (title, subtitle, body, tags, search settings) in place, once its send is over; the emails that went out are built from the campaign at send time, so nothing already sent changes, but a later step or resend reads the new words. The page as it was goes into `CampaignVersion` (reason `edit`, the same 30) and `restore_campaign_version` works on a sent issue the same way. The RSS feed has no updated date: the entry keeps its `pubDate` and guid and carries the new content; the sitemap's lastmod follows `seo.updatedAt`. - **Content calendar**: issues and LinkedIn posts planned in one place (`docs/CONTENT-CALENDAR.md` has the model). `ContentItem` (channel `newsletter` or `linkedin`; copy, funnel stage, pillar, references, private notes, visuals as pointers to `Upload` records, a plan, review), `EditorialBatch` (with an optional recurring pattern that only `apply_editorial_pattern` uses) and `ContentRevision` (the last 50 states of an item's content), in `lib/content-items.js`, `lib/content-batches.js`, `lib/content-plan.js` and the rules in `lib/content-rules.js`; the actions are `lib/actions/content-items.js` and `content-batches.js`, the pages are under `app/(app)/campaigns/calendar`. Planning never sends, schedules or publishes: nothing in these files writes a campaign or queues a job, every planning action is class `write-draft` (reads are `read`, deletes `destructive`), and approving is `decide_content_review`, a `dashboardOnly` action, so no key can approve. Editorial state (`idea` to `delivered`) is apart from a campaign's delivery; a LinkedIn post is posted by hand and only recorded (`record_content_publication`, which needs an approved item and a LinkedIn address), and nothing may call it scheduled or published otherwise. A planned day is a `YYYY-MM-DD` plus an optional wall-clock time in an IANA zone, with its instant worked out by `zonedTimeToInstant` (`lib/time-zone.js`). An edit to the title, hook, body, call to action, destination, visuals or linked issue of an approved item takes the approval back (`changes_requested`, `editedAfterApproval`); a move does not. `externalId` makes creates and imports repeat safely (unique per account). `get_content_calendar` and `lib/calendar.js` are unchanged and leave drafts out; `get_content_plan` is the one that shows plans beside real deliveries. - **Growth**: each newsletter's RSS feed (`lib/archive-feed.js`), the account export (any account may run it, even blocked or without a plan: `export_account_data` sets `whileBlocked` and `requiresPlan: false`, and that stays so) and the referral program (`lib/referrals.js`; reward emails are `Message` kind `referral`, sent like welcome emails). Newsletters recommend each other for free (`lib/recommendations.js`, rules in `lib/recommendation-rules.js`): a reader's email reaches another newsletter only through their own tick on the "You might also like" step, as a normal signup with that newsletter's confirmation email, and the dashboard is at `/recommending` because `/recommendations` is the public directory. Recommendations stay inside a region. diff --git a/app/(app)/newsletters/actions.js b/app/(app)/newsletters/actions.js index 653d23d8..6d50996e 100644 --- a/app/(app)/newsletters/actions.js +++ b/app/(app)/newsletters/actions.js @@ -58,6 +58,7 @@ function readSettings(formData) { preferencesLink: formData.get("preferencesLink") === "on", utmLinks: formData.get("utmLinks") === "on", shareButtons: formData.get("shareButtons") === "on", + requireClaimCheck: formData.get("requireClaimCheck") === "on", ...(formData.has("customHeaders") && { customHeaders: readHeaderLines(String(formData.get("customHeaders") ?? "")) }), ...(formData.has("sendUndoSeconds") && { sendUndoSeconds: Number(readText(formData, "sendUndoSeconds")) }), author: { bio: readText(formData, "authorBio"), imageUrl: readText(formData, "authorImageUrl") }, diff --git a/app/(site)/docs/briefings-bot/page.js b/app/(site)/docs/briefings-bot/page.js index a55d1b4b..d0abd2f3 100644 --- a/app/(site)/docs/briefings-bot/page.js +++ b/app/(site)/docs/briefings-bot/page.js @@ -26,11 +26,16 @@ export default function BriefingsBotPage() {

- Only pages that a FlockStack user's briefing found: a result of a search for the briefing's topic, a story in a feed the + Pages that a FlockStack user's briefing found: a result of a search for the briefing's topic, a story in a feed the briefing reads, or a page linked from one of those. It does not crawl a site, follow its links beyond one step, or keep a copy of a page after the briefing's run: it keeps the facts it needs for a short summary, at most 25 words of quotation, and the link to the page.

+

+ It also opens the pages a FlockStack issue links to when its writer checks the issue before sending it: it looks for the + issue's figures on the pages it cites and checks that its links load, at most 25 links and 12 pages per check, and keeps + nothing of the pages it read. +

It reads at most 1 MB of a page, waits at most 8 seconds, makes at most 2 requests to one host at a time and at most 300 requests in one run, and only to public addresses over https. diff --git a/app/api/cron/hourly/route.js b/app/api/cron/hourly/route.js index 42a0ace2..0a29caaf 100644 --- a/app/api/cron/hourly/route.js +++ b/app/api/cron/hourly/route.js @@ -4,6 +4,7 @@ import { runAgentHousekeeping } from "@/lib/agent-housekeeping"; import { announceTelegram } from "@/lib/alerts"; import { sendCommentDigests } from "@/lib/comment-notify"; import { sendConfirmationReminders } from "@/lib/confirmation-reminders"; +import { purgeOldClaimChecks } from "@/lib/claim-check"; import { withCronSecret } from "@/lib/cron"; import { purgeCustomRequests } from "@/lib/custom-requests"; import { followDedicatedIps } from "@/lib/dedicated-ip"; @@ -44,10 +45,10 @@ async function runBriefingsInCron() { // exports whose links ran out, the owners' digests of new comments and reports, the extra emails Polar hasn't been told // about yet, dedicated IPs that no longer match their add-on, the day of every IP warm-up and the stock of spare IPs, the reminders to signups that haven't confirmed, and the // Custom plan requests past their two years, the RSS feeds' kept copies nobody has read for a day, comps and gifts that ran out, and -// pledge charging whose job went quiet, and the agent approvals nobody decided and the agent activity past its 90 days, the held sends whose release job never ran, and the link checks nobody looked at for a week. +// pledge charging whose job went quiet, and the agent approvals nobody decided and the agent activity past its 90 days, the held sends whose release job never ran, and the link and claim checks nobody looked at for a week. // On a self-hosted server with BRIEFINGS_IN_CRON=true, also a tick of the briefings engine. export const GET = withCronSecret(async () => { - const [postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, editorSnapshots, briefings] = await Promise.all([ + const [postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, claimChecks, editorSnapshots, briefings] = await Promise.all([ checkPostshibaStatus(), announceTelegram(), expireAccountExports(), @@ -66,9 +67,10 @@ export const GET = withCronSecret(async () => { runAgentHousekeeping(), sweepHeldSends(), purgeOldLinkChecks(), + purgeOldClaimChecks(), expireCollaborationSnapshots(), runBriefingsInCron(), ]); - console.info(`[hourly] ${JSON.stringify({ postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, editorSnapshots, briefings })}`); + console.info(`[hourly] ${JSON.stringify({ postshiba, telegram, expiredExports, commentDigests, extraEmails, dedicatedIps, ipWarmups, ipInventory, reminders, customRequests, feedCopies, grants, pledges, giftsSent, giftsExpired, agentRecords, heldSends, linkChecks, claimChecks, editorSnapshots, briefings })}`); return { postshiba, telegram }; }); diff --git a/app/api/jobs/check-claims/route.js b/app/api/jobs/check-claims/route.js new file mode 100644 index 00000000..2dfe12cb --- /dev/null +++ b/app/api/jobs/check-claims/route.js @@ -0,0 +1,18 @@ +import { NextResponse } from "next/server"; +import { withQStashSignature } from "@/lib/qstash-route"; + +export const dynamic = "force-dynamic"; +export const maxDuration = 60; + +// Handled outcomes answer 200 so QStash doesn't retry them; a thrown error (database down) is retried. The checker and the +// fetcher load only when a job comes in. +async function handleCheck(request) { + const body = await request.json().catch(() => null); + const job = { accountId: body?.accountId, campaignId: body?.campaignId, hash: body?.hash }; + const { runClaimCheck } = await import("@/lib/claim-check-run"); + const result = await runClaimCheck(job); + console.info(`[claim-check] ${JSON.stringify({ ...job, ...result })}`); + return NextResponse.json(result); +} + +export const POST = withQStashSignature(handleCheck); diff --git a/components/FormNewsletter.jsx b/components/FormNewsletter.jsx index 2647badd..ea4a3020 100644 --- a/components/FormNewsletter.jsx +++ b/components/FormNewsletter.jsx @@ -429,6 +429,12 @@ export default function FormNewsletter({ In the email header + + + set("isClaimCheckRequired", isChecked)} aria-label="Require the claim check to pass before a send" /> + Required before a send + +