diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 75791ba2d..0bcaeb6b7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -58,6 +58,9 @@ jobs: - name: Test the E2E stale-stack pruning run: node --test scripts/e2e/stale-stacks.test.mjs + - name: Test the agent release scripts + run: node --test scripts/ops/merge-when-green.test.mjs scripts/ops/board.test.mjs scripts/ops/promote.test.mjs scripts/ops/verify-live.test.mjs + - name: Replay rate migrations on PostgreSQL 17 (Docker) run: node scripts/test-migration-replay.mjs diff --git a/AGENTS.md b/AGENTS.md index da45f22a8..894ffb537 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,6 +39,11 @@ Tài liệu này tổng hợp đầy đủ **User-Level Hard Rules (`C:\Users\JO - **TUYỆT ĐỐI KHÔNG** chạy `vercel deploy --prod`, `vercel --prod`, `--prebuilt --prod`, `--skip-domain` rồi `promote`, hay promote một bản Preview của `dev` lên production, kể cả khi JOY đã cho phép "deploy". Lời cho phép deploy nghĩa là merge PR promote `dev` -> `main`, không phải deploy từ máy local. - Chốt kỹ thuật: `scripts/vercel-production-guard.mjs` (chạy trong `buildCommand` của `vercel.json`) làm build production thất bại nếu không phải GitHub build từ `DOS/Rate@main`; workflow `Validate PR Base` chặn PR vào `main` không từ `dev`; workflow `Branch Divergence` báo lỗi khi `main` có commit mà `dev` không có. Không được sửa hay vô hiệu hóa các chốt này để deploy cho nhanh. - Ngoại lệ duy nhất: **rollback** về một bản production cũ đã build từ Git (`vercel rollback`), vì không build lại. Sau đó sửa tiếp qua PR hotfix. + - **Script vận hành có sẵn trong repo (`scripts/ops/`, dùng thay cho lệnh gõ tay):** + - `node scripts/ops/merge-when-green.mjs `: chờ commit head xanh rồi merge, ghim theo SHA head. Xanh nghĩa là mọi check run và mọi commit status (Vercel) đều đạt, và các check mà ruleset bắt buộc đã báo kết quả. PR vào `main` luôn dùng merge commit, PR khác mặc định squash. `--ignore ""` chỉ dùng khi một check không bắt buộc đang hỏng có lý do đã ghi trong PR. + - `node scripts/ops/promote.mjs`: mở hoặc dùng lại PR promote `dev` -> `main`, liệt kê các PR được phát hành, merge bằng merge commit khi CI xanh. + - `node scripts/ops/verify-live.mjs --wait 900`: kiểm tra rate.vn chạy đúng commit của `main` (`/api/health` trả `commit`), database đọc được và các route chính trả 200. Script đọc lại `main` mỗi vòng chờ. Chỉ khi bản live chậm hơn `main`, nó mới in lệnh `vercel promote --scope metados`; nếu bản live đi trước hoặc lệch nhánh so với `main`, nó dặn không làm gì. Không bao giờ redeploy hay promote một deployment build từ commit khác (`promote` không build lại, và bật lại tự gán domain sau một lần `vercel rollback`). + - `node scripts/ops/board.mjs --status in-progress|in-review|done --agent `: cập nhật Project #9 và nhãn (xem mục 1). Script từ chối issue đang mang nhãn `agent:*` của agent khác; `--take-over` chỉ dùng khi agent đó đã bàn giao hoặc JOY chỉ định. 7. **Chủ động thực thi liên tục & đề xuất việc tiếp theo (Proactive Next-Task):** - **Khi đang ở chế độ `/goal` hoặc được giao làm hết backlog:** Tự động tạo/nhận và làm liên tục từng task đến khi sạch toàn bộ backlog mà **KHÔNG dừng lại hỏi `Bắt đầu chứ?`**. @@ -75,6 +80,8 @@ gh issue list -R DOS/Rate-Services --state open ``` - **Chuyển trạng thái sang `In Progress` & Ghi danh Agent ngay khi bắt đầu (BẮT BUỘC đồng bộ cả Issue lẫn Board Project #9, comment bằng Tiếng Anh):** + Cách nhanh, một lệnh làm cả nhãn lẫn board (tự thêm issue vào Project #9 nếu chưa có, chỉ giữ một nhãn `status:*`): `node scripts/ops/board.mjs --status in-progress --agent $agentNick`. Các trạng thái khác: `ready`, `in-review`, `done`, `backlog`. Các lệnh `gh` dưới đây là cách làm tương đương bằng tay. + ```powershell $agentNick = "Crawl" # Nickname của Agent hiện tại $agentLabel = "agent:$($agentNick.ToLower())" diff --git a/Docs/Resources/RELEASE.md b/Docs/Resources/RELEASE.md index 525ad4eee..ebad64b04 100644 --- a/Docs/Resources/RELEASE.md +++ b/Docs/Resources/RELEASE.md @@ -17,8 +17,9 @@ This is the release workflow for `rate.vn`. It follows the same model as DOS/DOS `gh pr create --repo DOS/Rate --base main --head dev --title "chore(release): promote dev to main"` List the PRs being released in the body. 4. Wait for the required checks on the promotion PR: `Typecheck & Build Monorepo`, `PR into main comes from dev`, `main must not lead dev`. + `node scripts/ops/promote.mjs` does steps 3 to 5 in one command: it opens the promotion PR from `dev` (or reuses the open one) with the released PRs listed, then merges it with a merge commit once its checks pass. `--no-merge` leaves it open. 5. Promotions are batched and need no per-release question: the agent promotes at the first checkpoint reached: the backlog runs out, 10 or more PRs are waiting on `dev`, the oldest unpromoted change is more than 24 hours old, or the agent is ending a long working session. Promote only when `dev` head CI is green and no PR is mid-way through fixing a production bug. The agent merges it with a merge commit: `gh pr merge --repo DOS/Rate --merge`. JOY does not merge PRs; never ask JOY to click Merge. **Never squash or rebase a promotion.** Squashing creates a commit on `main` that can never exist on `dev`, and the next promotion then diffs from a stale merge base. -6. Vercel builds `main` and assigns `rate.vn`. Verify the live revision (the deployment's Git SHA equals `main`), `/api/health`, the main routes and a browser check before calling the release done. To check a build before it takes traffic, create it from Git with `autoAssignCustomDomains: false` (Vercel API `POST /v13/deployments` with `gitSource` ref `main` and `target: "production"`), verify it with `vercel curl --deployment `, then `vercel promote `. Promoting an existing production deployment does not rebuild. +6. Vercel builds `main` and assigns `rate.vn`. Verify the live revision (the deployment's Git SHA equals `main`), `/api/health`, the main routes and a browser check before calling the release done. `node scripts/ops/verify-live.mjs --wait 900` checks the first three: `/api/health` reports the deployment's `commit`, which must equal `main`, and `checks.database` must be `ok`. It re-reads `main` on every poll. Only when the live commit is behind `main` does it print `vercel promote --scope metados`; when live is ahead of or diverged from `main` it says to do nothing. Never redeploy or promote a deployment built from another commit: that rolls the release back. `promote` does not rebuild, and it turns production domain auto-assignment back on after a `vercel rollback`. To check a build before it takes traffic, create it from Git with `autoAssignCustomDomains: false` (Vercel API `POST /v13/deployments` with `gitSource` ref `main` and `target: "production"`), verify it with `vercel curl --deployment `, then `vercel promote `. Promoting an existing production deployment does not rebuild. ## Hotfix diff --git a/apps/extension/src/extractors/ecommerce-detector.ts b/apps/extension/src/extractors/ecommerce-detector.ts index d738d68a3..a54de710f 100644 --- a/apps/extension/src/extractors/ecommerce-detector.ts +++ b/apps/extension/src/extractors/ecommerce-detector.ts @@ -7,11 +7,17 @@ import type { * Strip all tracking query strings (sp_atk, xptdk, utm_*, fbclid, etc.) and hashes. */ export function sanitizeProductUrl(rawUrl: string): string | null { + if (typeof rawUrl !== 'string') return null; + const cleaned = rawUrl.replace(/\p{Cc}+/gu, '').trim(); + if (!cleaned || cleaned.length > 2048) return null; try { - const parsed = new URL(rawUrl); + const parsed = new URL(cleaned); if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { return null; } + if (parsed.username || parsed.password) { + return null; + } return `${parsed.origin}${parsed.pathname}`; } catch { return null; @@ -25,12 +31,18 @@ function isDomainOrSubdomain(host: string, domain: string): boolean { export function detectEcommercePlatform( url: string ): SupportedEcommercePlatform | null { + if (typeof url !== 'string') return null; + const cleaned = url.replace(/\p{Cc}+/gu, '').trim(); + if (!cleaned || cleaned.length > 2048) return null; let host: string; try { - const parsed = new URL(url); + const parsed = new URL(cleaned); if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { return null; } + if (parsed.username || parsed.password) { + return null; + } host = parsed.hostname.toLowerCase(); } catch { return null; @@ -114,10 +126,10 @@ export function extractIdsFromUrl( const m = canonicalUrl.match(/(?:a-)?i\.(\d+)\.(\d+)/i) || canonicalUrl.match(/\/(?:product|opaanlp)\/(\d+)\/(\d+)/i); - if (m) { + if (m && m[1] && m[2] && m[1].length <= 24 && m[2].length <= 24) { return { - externalShopId: m[1] || null, - externalItemId: m[2] || null, + externalShopId: m[1], + externalItemId: m[2], }; } return { externalItemId: null, externalShopId: null }; @@ -125,9 +137,9 @@ export function extractIdsFromUrl( if (platform === 'lazada') { const m = canonicalUrl.match(/-i(\d+)(?:-s(\d+))?\.html/i); - if (m) { + if (m && m[1] && m[1].length <= 24 && (!m[2] || m[2].length <= 24)) { return { - externalItemId: m[1] || null, + externalItemId: m[1], externalShopId: m[2] || null, }; } @@ -136,9 +148,9 @@ export function extractIdsFromUrl( if (platform === 'tiki') { const m = canonicalUrl.match(/-p(\d+)\.html/i); - if (m) { + if (m && m[1] && m[1].length <= 24) { return { - externalItemId: m[1] || null, + externalItemId: m[1], externalShopId: null, }; } @@ -153,9 +165,13 @@ export function extractIdsFromUrl( canonicalUrl.match(/\/product\/(\d+)/i) || canonicalUrl.match(/\/view\/product\/(\d+)/i); const shopMatch = canonicalUrl.match(/\/@([a-zA-Z0-9_.-]+)/); + const itemId = + prodMatch?.[1] && prodMatch[1].length <= 24 ? prodMatch[1] : null; + const shopId = + shopMatch?.[1] && shopMatch[1].length <= 64 ? shopMatch[1] : null; return { - externalItemId: prodMatch?.[1] || null, - externalShopId: shopMatch?.[1] || null, + externalItemId: itemId, + externalShopId: shopId, }; } @@ -179,6 +195,8 @@ export function extractIdsFromUrl( const isValidItemSlug = Boolean(lastSegment) && lastSegment!.length >= 2 && + lastSegment!.length <= 120 && + /^[a-zA-Z0-9_.-]+$/.test(lastSegment!) && !RETAIL_NON_PRODUCT_FIRST_SEGMENTS.has(firstSegLower) && !RETAIL_NON_PRODUCT_FIRST_SEGMENTS.has(lastSegLower) && !(segments.length === 1 && RETAIL_CATEGORY_ROOT_SLUGS.has(firstSegLower)) && @@ -225,6 +243,9 @@ const JSON_LD_PRODUCT_TYPES = new Set([ 'productmodel', ]); +const MAX_JSON_LD_ENTITIES = 100; +const MAX_JSON_LD_OFFERS = 100; + export function flattenJsonLdEntities( raw: unknown, depth = 0 @@ -233,7 +254,12 @@ export function flattenJsonLdEntities( if (Array.isArray(raw)) { const out: Record[] = []; for (const entry of raw) { - out.push(...flattenJsonLdEntities(entry, depth + 1)); + if (out.length >= MAX_JSON_LD_ENTITIES) break; + const nested = flattenJsonLdEntities(entry, depth + 1); + for (const item of nested) { + if (out.length >= MAX_JSON_LD_ENTITIES) break; + out.push(item); + } } return out; } @@ -241,7 +267,11 @@ export function flattenJsonLdEntities( const obj = raw as Record; const out: Record[] = [obj]; if (Array.isArray(obj['@graph'])) { - out.push(...flattenJsonLdEntities(obj['@graph'], depth + 1)); + const nested = flattenJsonLdEntities(obj['@graph'], depth + 1); + for (const item of nested) { + if (out.length >= MAX_JSON_LD_ENTITIES) break; + out.push(item); + } } return out; } @@ -270,11 +300,18 @@ export function extractPriceFromJsonLdOffers(offersRaw: unknown): { } const candidates: Record[] = []; const pushOffer = (entry: unknown) => { - if (!entry || typeof entry !== 'object') return; + if ( + !entry || + typeof entry !== 'object' || + candidates.length >= MAX_JSON_LD_OFFERS + ) { + return; + } const obj = entry as Record; candidates.push(obj); if (Array.isArray(obj.offers)) { for (const nested of obj.offers) { + if (candidates.length >= MAX_JSON_LD_OFFERS) break; if (nested && typeof nested === 'object') { candidates.push(nested as Record); } @@ -283,7 +320,10 @@ export function extractPriceFromJsonLdOffers(offersRaw: unknown): { }; if (Array.isArray(offersRaw)) { - for (const entry of offersRaw) pushOffer(entry); + for (const entry of offersRaw) { + if (candidates.length >= MAX_JSON_LD_OFFERS) break; + pushOffer(entry); + } } else { pushOffer(offersRaw); } diff --git a/apps/extension/src/security-guards.ts b/apps/extension/src/security-guards.ts index 7fdf7b8d0..9179ad356 100644 --- a/apps/extension/src/security-guards.ts +++ b/apps/extension/src/security-guards.ts @@ -122,6 +122,35 @@ export function selectFreshestSessionSignal( return tabSignal ?? latestSignal ?? null; } +const SAFE_EXTERNAL_ID_REGEX = /^[a-zA-Z0-9_.-]{1,120}$/; + +function sanitizeExternalIdentifier(val: unknown): string | null { + if (typeof val !== 'string') return null; + const cleaned = val.replace(/\p{Cc}+/gu, '').trim(); + if (!cleaned || !SAFE_EXTERNAL_ID_REGEX.test(cleaned)) { + return null; + } + return cleaned; +} + +function sanitizeCanonicalSignalUrl(rawUrl: unknown): string { + if (typeof rawUrl !== 'string') return ''; + const cleaned = rawUrl.replace(/\p{Cc}+/gu, '').trim(); + if (!cleaned || cleaned.length > 2048) return ''; + try { + const parsed = new URL(cleaned); + if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { + return ''; + } + if (parsed.username || parsed.password) { + return ''; + } + return `${parsed.origin}${parsed.pathname}`; + } catch { + return ''; + } +} + export function sanitizeExtensionProductSignal( signal: ExtractedPageProductSignal ): ExtractedPageProductSignal { @@ -159,14 +188,27 @@ export function sanitizeExtensionProductSignal( : null; const trimmedTitle = - typeof signal.title === 'string' ? signal.title.trim().slice(0, 500) : ''; + typeof signal.title === 'string' + ? signal.title + .replace(/\p{Cc}+/gu, ' ') + .replace(/\s+/g, ' ') + .trim() + .slice(0, 500) + : ''; const trimmedVariant = - typeof signal.selectedVariant === 'string' && signal.selectedVariant.trim() - ? signal.selectedVariant.trim().slice(0, 200) + typeof signal.selectedVariant === 'string' + ? signal.selectedVariant + .replace(/\p{Cc}+/gu, ' ') + .replace(/\s+/g, ' ') + .trim() + .slice(0, 200) || null : null; return { ...signal, + canonicalUrl: sanitizeCanonicalSignalUrl(signal.canonicalUrl), + externalItemId: sanitizeExternalIdentifier(signal.externalItemId), + externalShopId: sanitizeExternalIdentifier(signal.externalShopId), title: trimmedTitle, selectedVariant: trimmedVariant, price: validPrice, @@ -181,13 +223,16 @@ export function escapeSafeExternalHref( fallbackUrl = 'https://rate.vn' ): string { if (typeof rawUrl !== 'string') return fallbackUrl; - const trimmed = rawUrl.trim(); - if (!trimmed) return fallbackUrl; + const trimmed = rawUrl.replace(/\p{Cc}+/gu, '').trim(); + if (!trimmed || trimmed.length > 2048) return fallbackUrl; try { const parsed = new URL(trimmed, 'https://rate.vn'); if (parsed.protocol !== 'https:' && parsed.protocol !== 'http:') { return fallbackUrl; } + if (parsed.username || parsed.password) { + return fallbackUrl; + } return parsed.href .replace(/&/g, '&') .replace(/ | null | undefined ): number[] { if (!Array.isArray(points)) return []; - return points + const valid = points .map((p) => Number(p?.price)) .filter( (p) => Number.isFinite(p) && p > 0 && p <= MAX_VALID_EXTENSION_VND_PRICE ); + return valid.length > MAX_SPARKLINE_POINTS + ? valid.slice(-MAX_SPARKLINE_POINTS) + : valid; } export function shouldRenderTrustPill( diff --git a/apps/extension/src/test-extension-pipeline.ts b/apps/extension/src/test-extension-pipeline.ts index 1905f0d32..ecb751762 100644 --- a/apps/extension/src/test-extension-pipeline.ts +++ b/apps/extension/src/test-extension-pipeline.ts @@ -16,6 +16,7 @@ import { } from './extractors/ecommerce-detector.js'; import { DEFAULT_API_BASE_URL, + MAX_SPARKLINE_POINTS, escapeSafeExternalHref, isPrivilegedExtensionSender, normalizeAllowedApiBaseUrl, @@ -2442,6 +2443,126 @@ async function runTests() { ); } + // --- DOS/Rate#846: security-guards signal/URL sanitization & ecommerce-detector JSON-LD/ID bounds --- + { + const sanitizedSig = sanitizeExtensionProductSignal({ + platform: 'shopee', + canonicalUrl: + 'https://shopee.vn/iPhone-16-i.88201679.29512841920?utm_source=ads#frag', + externalItemId: ' 29512841920\u0000 ', + externalShopId: 'bad'], + '

Tổng kết đánh giá

' + ); + assert.deepEqual( + summary834.pros, + ['Pin trâu'], + 'computeJReviewsSummary must strip HTML tags and deduplicate pros' + ); + assert.deepEqual(summary834.cons, ['Nóng máy']); + assert.equal(summary834.ai_summary, 'Tổng kết đánh giá'); + + await assert.rejects( + () => + analyzer834.castReviewVote(pricesOnlyStub!, { + reviewId: ' ', + voterId: 'voter-1', + vote: 'up', + }), + /reviewId and voterId must be non-empty strings/, + 'castReviewVote must reject blank reviewId or voterId' + ); + + // 16l. DOS/Rate#836: ProductNormalizer integer RAM/storage bounds, TB formatting, and asset/color sanitization + const normalizer836 = new ProductNormalizer(); + const baseRaw836 = { + platform: 'shopee' as const, + itemId: 'item-836', + shopId: 'shop-836', + url: 'https://shopee.vn/product/836/836', + priceMin: 12_990_000, + priceMax: 12_990_000, + currency: 'VND', + historicalSold: 10, + ratingStar: 4.8, + ratingCount: 5, + attributes: [], + shop: { shopId: 'shop-836', name: 'Shop 836', isMall: false }, + reviews: [], + }; + const fractionalRom836 = normalizer836.normalize({ + ...baseRaw836, + title: 'Điện thoại Samsung Galaxy S24 FE 8.5GB RAM ROM 12.5GB', + images: [ + 'https://cf.shopee.vn/file/valid.jpg', + 'javascript:alert(1)', + 'ftp://files.example.com/img.jpg', + ' ', + ], + models: [ + { modelId: 'm-1', name: ' Xanh Mint\u0007 ', price: 12_990_000, stock: 5 }, + { modelId: 'm-2', name: 'Xanh Mint', price: 12_990_000, stock: 3 }, + { modelId: 'm-3', name: 'Đ'.repeat(120), price: 12_990_000, stock: 2 }, + ], + description: ' Máy chính hãng\u0000 bảo hành 12 tháng ', + }); + assert.equal( + fractionalRom836.storage_gb, + null, + 'ProductNormalizer must reject non-integer GB storage values like 12.5GB' + ); + assert.equal( + fractionalRom836.ram_gb, + null, + 'ProductNormalizer must reject non-integer GB RAM values like 8.5GB' + ); + assert.deepEqual( + fractionalRom836.images, + ['https://cf.shopee.vn/file/valid.jpg'], + 'ProductNormalizer must keep only valid http/https asset URLs in images' + ); + assert.equal( + fractionalRom836.colors[0], + 'Xanh Mint', + 'ProductNormalizer must strip control chars, collapse whitespace, and deduplicate colors' + ); + assert.equal( + fractionalRom836.colors.length, + 2, + 'ProductNormalizer must deduplicate collapsed color names' + ); + assert.equal( + fractionalRom836.colors[1]?.length, + 80, + 'ProductNormalizer must cap color name length at 80 characters' + ); + assert.equal( + fractionalRom836.description, + 'Máy chính hãng bảo hành 12 tháng', + 'ProductNormalizer must strip control characters and trim description' + ); + + const nonMultipleTb836 = normalizer836.normalize({ + ...baseRaw836, + title: 'Laptop Asus Vivobook 15 16GB RAM ROM 1280GB', + images: [], + models: [], + }); + assert.equal(nonMultipleTb836.storage_gb, 1280); + assert.equal( + nonMultipleTb836.variant_name, + '1280GB', + 'ProductNormalizer must format non-512-multiple storage >= 1024 as GB rather than fractional TB' + ); + assert.equal(nonMultipleTb836.key_specs.storage, '1280GB'); + + const binaryTb836 = normalizer836.normalize({ + ...baseRaw836, + title: 'Laptop Asus Vivobook 15 16GB RAM 1.5TB SSD', + images: [], + models: [], + }); + assert.equal(binaryTb836.storage_gb, 1536); + assert.equal( + binaryTb836.variant_name, + '1.5TB', + 'ProductNormalizer must preserve 1.5TB (1536GB) as 1.5TB' + ); + console.log( - `✅ [P2/P3 Trust Model & #292/#352/#356/#359/#379/#382/#401/#497/#514/#522/#528/#557] Voter weight, Wilson lower bound (50/52=${wilson50of52} > 1/1=${wilson1of1}), shingle copy seeding (sim=${shingleSim}), affiliate shortlinks, peer floor, ArenaSpecSyncer, Importer & ReviewAnalyzer hardening (#352, #356, #359, #379, #382, #401, #497, #514, #522, #528, #557) verified` + `✅ [P2/P3 Trust Model & #292/#352/#356/#359/#379/#382/#401/#497/#514/#522/#528/#557/#834/#836] Voter weight, Wilson lower bound (50/52=${wilson50of52} > 1/1=${wilson1of1}), shingle copy seeding (sim=${shingleSim}), affiliate shortlinks, peer floor, ArenaSpecSyncer, Importer & ReviewAnalyzer hardening (#352, #356, #359, #379, #382, #401, #497, #514, #522, #528, #557, #834, #836) verified` ); console.log('\n🎉 ALL AI REVIEW ANALYZER TESTS PASSED!'); diff --git a/apps/web/src/app/api/health/route.ts b/apps/web/src/app/api/health/route.ts index c5295376a..666a2bf40 100644 --- a/apps/web/src/app/api/health/route.ts +++ b/apps/web/src/app/api/health/route.ts @@ -14,6 +14,9 @@ export async function GET() { service: 'rate-web', version: '2.0.0', timestamp: new Date().toISOString(), + // The Git commit this deployment was built from, so a release check can + // compare it with main (scripts/ops/verify-live.mjs). Null off Vercel. + commit: process.env.VERCEL_GIT_COMMIT_SHA || null, checks: { database }, }, { diff --git a/apps/web/src/app/api/reviews/[id]/vote/route.ts b/apps/web/src/app/api/reviews/[id]/vote/route.ts index f3606ad67..527cf1d5e 100644 --- a/apps/web/src/app/api/reviews/[id]/vote/route.ts +++ b/apps/web/src/app/api/reviews/[id]/vote/route.ts @@ -31,6 +31,14 @@ export async function OPTIONS() { const NOT_ELIGIBLE_AGE_REASON = 'Account must be at least 7 days old to vote or report reviews.'; +function isValidConfirmedTimestamp(raw: unknown): boolean { + if (typeof raw !== 'string') return false; + const trimmed = raw.trim(); + if (!trimmed || trimmed.length > 64) return false; + const ms = Date.parse(trimmed); + return Number.isFinite(ms) && ms > 0; +} + /** * Review Trust Model Section 9: Voter Eligibility Gate. The database applies * the same rules in rate.review_votes_guard (DOS/Rate#613, #771): @@ -40,42 +48,66 @@ const NOT_ELIGIBLE_AGE_REASON = * - Account must be at least 7 days old; a missing or unparseable * `created_at` fails closed. */ -export function evaluateVoterEligibility(user: { - created_at?: string | null; - email_confirmed_at?: string | null; - phone_confirmed_at?: string | null; - is_anonymous?: boolean | null; -}): { eligible: boolean; reason?: string } { +export function evaluateVoterEligibility( + user: + | { + created_at?: string | null; + email_confirmed_at?: string | null; + phone_confirmed_at?: string | null; + is_anonymous?: boolean | null; + } + | null + | undefined +): { eligible: boolean; reason?: string } { + if (!user || typeof user !== 'object') { + return { eligible: false, reason: NOT_ELIGIBLE_AGE_REASON }; + } if (user.is_anonymous === true) { return { eligible: false, reason: 'Anonymous accounts cannot vote on reviews.', }; } - if (!user.email_confirmed_at && !user.phone_confirmed_at) { + if ( + !isValidConfirmedTimestamp(user.email_confirmed_at) && + !isValidConfirmedTimestamp(user.phone_confirmed_at) + ) { return { eligible: false, reason: 'Verified email or phone number is required to vote on reviews.', }; } const createdMs = - typeof user.created_at === 'string' ? Date.parse(user.created_at) : NaN; - if (!Number.isFinite(createdMs)) { + typeof user.created_at === 'string' && user.created_at.trim().length <= 64 + ? Date.parse(user.created_at.trim()) + : NaN; + if (!Number.isFinite(createdMs) || createdMs <= 0) { return { eligible: false, reason: NOT_ELIGIBLE_AGE_REASON }; } const ageDays = (Date.now() - createdMs) / 86_400_000; - if (ageDays < 7) { + if (!Number.isFinite(ageDays) || ageDays < 7) { return { eligible: false, reason: NOT_ELIGIBLE_AGE_REASON }; } return { eligible: true }; } function toSafeVoteCount(value: unknown): number { + if (typeof value === 'boolean' || value === null || value === undefined) { + return 0; + } const n = Number(value); - return Number.isFinite(n) && n > 0 ? Math.floor(n) : 0; + return Number.isFinite(n) && n > 0 ? Math.min(100_000_000, Math.floor(n)) : 0; } function toSafeHelpfulnessWeight(value: unknown, fallback = 1): number { + if ( + typeof value === 'boolean' || + value === null || + value === undefined || + (typeof value === 'string' && !value.trim()) + ) { + return fallback; + } const n = Number(value); return Number.isFinite(n) ? Math.max(0, Math.min(10, n)) : fallback; } diff --git a/apps/web/src/app/api/v1/rate/deals/route.ts b/apps/web/src/app/api/v1/rate/deals/route.ts index da1d7d3ee..8046d21e4 100644 --- a/apps/web/src/app/api/v1/rate/deals/route.ts +++ b/apps/web/src/app/api/v1/rate/deals/route.ts @@ -5,6 +5,7 @@ import { PUBLIC_API_CORS_HEADERS, jsonApiError, parseBoundedIntParam, + parseBoundedStringParam, } from '@/server/rate/api-handler'; const VALID_CATEGORIES = new Set([ @@ -39,9 +40,10 @@ export async function GET(request: Request) { try { const { searchParams } = new URL(request.url); const rawCat = ( - searchParams.get('category') || - searchParams.get('cat') || - 'all' + parseBoundedStringParam( + searchParams.get('category') || searchParams.get('cat'), + 32 + ) || 'all' ) .trim() .toLowerCase(); @@ -51,7 +53,9 @@ export async function GET(request: Request) { ? (rawCat as CanonicalCategorySlug) : 'all'; - const rawPlatform = (searchParams.get('platform') || 'all') + const rawPlatform = ( + parseBoundedStringParam(searchParams.get('platform'), 32) || 'all' + ) .trim() .toLowerCase(); const platform = VALID_PLATFORMS.has(rawPlatform) ? rawPlatform : 'all'; diff --git a/apps/web/src/app/api/v1/rate/extension/ingest/route.ts b/apps/web/src/app/api/v1/rate/extension/ingest/route.ts index dbdc0f66b..ac4b6873b 100644 --- a/apps/web/src/app/api/v1/rate/extension/ingest/route.ts +++ b/apps/web/src/app/api/v1/rate/extension/ingest/route.ts @@ -7,6 +7,7 @@ import { import { BoundedSlidingWindowRateLimiter, extractClientRateLimitKey, + isSafeCrowdIngestHttpUrl, jsonApiError, } from '@/server/rate/api-handler'; import { parseEcommerceUrlOrQuery } from '@/lib/rate-api'; @@ -43,8 +44,8 @@ const CrowdIngestSchema = z.object({ .url() .max(1024) .refine( - (u) => /^https?:\/\//i.test(u), - 'Product URL must use http:// or https://' + (u) => isSafeCrowdIngestHttpUrl(u), + 'Product URL must use http:// or https:// without embedded credentials' ), title: z.string().trim().min(3).max(320), selectedVariant: z.string().trim().max(160).nullish(), diff --git a/apps/web/src/app/api/v1/rate/inspect/route.ts b/apps/web/src/app/api/v1/rate/inspect/route.ts index cf1181ad9..26443b058 100644 --- a/apps/web/src/app/api/v1/rate/inspect/route.ts +++ b/apps/web/src/app/api/v1/rate/inspect/route.ts @@ -12,6 +12,7 @@ import { jsonApiError, parseBoundedStringParam, parseOptionalVndPriceParam, + resolveInspectObservedPrices, } from '@/server/rate/api-handler'; export async function OPTIONS() { @@ -51,32 +52,15 @@ export async function GET(request: NextRequest) { const inspection = await getCachedInspectProductOrShopUrl(rawQuery); const canonicalUrl = canonicalizeProductUrl(rawQuery); - const rawObservedPrice = Number( - queryPrice ?? - inspection.matchedOffer?.price_current ?? - inspection.masterProduct?.min_price ?? - 0 - ); - const observedPrice = - Number.isFinite(rawObservedPrice) && - rawObservedPrice > 0 && - rawObservedPrice <= MAX_VALID_VND_PRICE - ? Math.round(rawObservedPrice) - : 0; - const rawObservedOriginalPrice = Number( - queryOriginalPrice ?? - (queryPrice === null - ? inspection.matchedOffer?.price_original - : null) ?? - 0 - ); - const observedOriginalPrice = - Number.isFinite(rawObservedOriginalPrice) && - rawObservedOriginalPrice > 0 && - rawObservedOriginalPrice >= observedPrice && - rawObservedOriginalPrice <= MAX_VALID_VND_PRICE - ? Math.round(rawObservedOriginalPrice) - : 0; + const { observedPrice, observedOriginalPrice } = + resolveInspectObservedPrices({ + queryPrice, + queryOriginalPrice, + matchedOfferPriceCurrent: inspection.matchedOffer?.price_current, + matchedOfferPriceOriginal: inspection.matchedOffer?.price_original, + masterMinPrice: inspection.masterProduct?.min_price, + maxVnd: MAX_VALID_VND_PRICE, + }); const gatekeeper = detectAccessoryOrJunk({ title: diff --git a/apps/web/src/app/api/v1/rate/leaderboard/route.ts b/apps/web/src/app/api/v1/rate/leaderboard/route.ts index dbc379c1c..f167ba98d 100644 --- a/apps/web/src/app/api/v1/rate/leaderboard/route.ts +++ b/apps/web/src/app/api/v1/rate/leaderboard/route.ts @@ -31,9 +31,10 @@ export async function GET(req: NextRequest) { try { const { searchParams } = new URL(req.url); const rawCat = ( - searchParams.get('category') || - searchParams.get('cat') || - 'smartphones' + parseBoundedStringParam( + searchParams.get('category') || searchParams.get('cat'), + 32 + ) || 'smartphones' ) .toLowerCase() .trim(); diff --git a/apps/web/src/app/api/v1/rate/revalidate/route.ts b/apps/web/src/app/api/v1/rate/revalidate/route.ts index 0780dd710..e969e9546 100644 --- a/apps/web/src/app/api/v1/rate/revalidate/route.ts +++ b/apps/web/src/app/api/v1/rate/revalidate/route.ts @@ -62,7 +62,10 @@ export function verifyRevalidateSecret( !cleanExpected || cleanProvided.length > MAX_REVALIDATE_TOKEN_LENGTH || cleanExpected.length > MAX_REVALIDATE_TOKEN_LENGTH || - /\s/.test(cleanProvided) + /\s/.test(cleanProvided) || + /\s/.test(cleanExpected) || + /\p{Cc}/u.test(cleanProvided) || + /\p{Cc}/u.test(cleanExpected) ) { return false; } @@ -75,7 +78,9 @@ export function extractProvidedToken(request: NextRequest): string | null { const headerToken = request.headers.get('x-rate-revalidate-token'); if (headerToken && headerToken.trim()) { const trimmed = headerToken.trim(); - return trimmed.length <= MAX_REVALIDATE_TOKEN_LENGTH && !/\s/.test(trimmed) + return trimmed.length <= MAX_REVALIDATE_TOKEN_LENGTH && + !/\s/.test(trimmed) && + !/\p{Cc}/u.test(trimmed) ? trimmed : null; } @@ -84,7 +89,8 @@ export function extractProvidedToken(request: NextRequest): string | null { const trimmed = authHeader.replace(/^Bearer\s+/i, '').trim(); return trimmed.length > 0 && trimmed.length <= MAX_REVALIDATE_TOKEN_LENGTH && - !/\s/.test(trimmed) + !/\s/.test(trimmed) && + !/\p{Cc}/u.test(trimmed) ? trimmed : null; } diff --git a/apps/web/src/app/api/v1/rate/reviews/route.ts b/apps/web/src/app/api/v1/rate/reviews/route.ts index 5c9f82297..4eef71c53 100644 --- a/apps/web/src/app/api/v1/rate/reviews/route.ts +++ b/apps/web/src/app/api/v1/rate/reviews/route.ts @@ -125,7 +125,13 @@ const SubmitReviewSchema = z.object({ .trim() .uuid('Mã sản phẩm chuẩn hóa (masterProductId) không hợp lệ.'), authorName: z.string().optional(), - ratingStar: z.coerce.number().optional(), + ratingStar: z.preprocess( + (val) => + val === null || (typeof val === 'string' && !val.trim()) + ? undefined + : val, + z.coerce.number().finite().optional() + ), comment: z .string() .trim() @@ -180,8 +186,9 @@ const RPC_REJECTIONS: Record = { function sanitizeCriteria(raw: Record | undefined) { const out: Record = {}; + if (!raw || typeof raw !== 'object') return out; for (const key of CRITERIA_KEYS) { - const val = Number(raw?.[key]); + const val = asNumber(raw[key], NaN); if (Number.isFinite(val) && val >= 1 && val <= 5) { out[key] = Math.round(val * 10) / 10; } @@ -190,10 +197,23 @@ function sanitizeCriteria(raw: Record | undefined) { } function asNumber(value: unknown, fallback: number): number { - const n = Number(value); + if ( + value === null || + value === undefined || + typeof value === 'boolean' || + (typeof value === 'string' && !value.trim()) + ) { + return fallback; + } + const n = typeof value === 'number' ? value : Number(String(value).trim()); return Number.isFinite(n) ? n : fallback; } +function toSafeCount(value: unknown): number { + const n = asNumber(value, 0); + return n > 0 ? Math.min(100_000_000, Math.floor(n)) : 0; +} + /** * Builds the response item from what rate.submit_community_review stored, so the * verdict and weight shown to the user are the ones the database applied. @@ -202,41 +222,75 @@ export function toReviewItem( record: Record, criteria: Record ): ReviewItem { - const verdict = record.verdict; - const flag = record.flag_category as ReviewFlagCategory; - const rawStar = asNumber(record.rating_star, 5); + const safeRecord = + record && typeof record === 'object' && !Array.isArray(record) + ? record + : {}; + const safeFallbackCriteria = + criteria && typeof criteria === 'object' && !Array.isArray(criteria) + ? sanitizeCriteria(criteria) + : {}; + const verdict = safeRecord.verdict; + const flag = safeRecord.flag_category as ReviewFlagCategory; + const rawStar = asNumber(safeRecord.rating_star, 5); const clampedStar = rawStar > 0 ? Math.max(1, Math.min(5, Math.round(rawStar))) : 0; const clampedWeight = Math.max( 0, - Math.min(10, asNumber(record.helpfulness_weight, 0)) + Math.min(10, asNumber(safeRecord.helpfulness_weight, 0)) ); const sanitizedRecordCriteria = - record.criteria_ratings && - typeof record.criteria_ratings === 'object' && - !Array.isArray(record.criteria_ratings) - ? sanitizeCriteria(record.criteria_ratings as Record) + safeRecord.criteria_ratings && + typeof safeRecord.criteria_ratings === 'object' && + !Array.isArray(safeRecord.criteria_ratings) + ? sanitizeCriteria(safeRecord.criteria_ratings as Record) : {}; const effectiveCriteria = Object.keys(sanitizedRecordCriteria).length > 0 ? sanitizedRecordCriteria - : criteria; + : safeFallbackCriteria; + const rawCreatedAt = + typeof safeRecord.created_at === 'string' + ? safeRecord.created_at.trim() + : ''; + const createdMs = + rawCreatedAt && rawCreatedAt.length <= 64 ? Date.parse(rawCreatedAt) : NaN; + const createdAt = + Number.isFinite(createdMs) && createdMs > 0 + ? rawCreatedAt + : new Date().toISOString(); + const cleanComment = + typeof safeRecord.comment === 'string' + ? safeRecord.comment.trim().slice(0, 3000) + : ''; + const cleanVariation = + typeof safeRecord.variation === 'string' + ? safeRecord.variation.trim().slice(0, 120) || null + : null; + const cleanReason = + typeof safeRecord.reason === 'string' + ? safeRecord.reason.trim().slice(0, 500) || null + : null; + const rawFakeProb = asNumber(safeRecord.fake_probability, NaN); return { - id: String(record.id), - author_name: String(record.author_name ?? ''), + id: + typeof safeRecord.id === 'string' || typeof safeRecord.id === 'number' + ? String(safeRecord.id).trim().slice(0, 120) + : '', + author_name: + typeof safeRecord.author_name === 'string' + ? safeRecord.author_name.trim().slice(0, 80) + : '', rating_star: clampedStar, - comment: String(record.comment ?? ''), - variation: typeof record.variation === 'string' ? record.variation : null, - created_at: - typeof record.created_at === 'string' - ? record.created_at - : new Date().toISOString(), - reviewer_type: record.reviewer_type === 'expert' ? 'expert' : 'user', - upvotes: 0, - downvotes: 0, - report_count: 0, + comment: cleanComment, + variation: cleanVariation, + created_at: createdAt, + reviewer_type: safeRecord.reviewer_type === 'expert' ? 'expert' : 'user', + upvotes: toSafeCount(safeRecord.upvotes), + downvotes: toSafeCount(safeRecord.downvotes), + report_count: toSafeCount(safeRecord.report_count), helpfulness_weight: clampedWeight, - include_in_rating: record.include_in_rating === true, + include_in_rating: safeRecord.include_in_rating === true, criteria_ratings: effectiveCriteria, verdict: verdict === 'authentic' || @@ -245,16 +299,15 @@ export function toReviewItem( verdict === 'unanalyzed' ? verdict : null, - fake_probability: - record.fake_probability == null - ? null - : Math.max(0, Math.min(1, asNumber(record.fake_probability, 0))), + fake_probability: Number.isFinite(rawFakeProb) + ? Math.max(0, Math.min(1, rawFakeProb)) + : null, flag_category: FLAG_CATEGORIES.has(flag) ? flag : 'authentic', - reason: typeof record.reason === 'string' ? record.reason : null, + reason: cleanReason, signals: { source: 'rate_community_jreviews', criteria_count: Object.keys(effectiveCriteria).length, - comment_length: String(record.comment ?? '').length, + comment_length: cleanComment.length, }, }; } diff --git a/apps/web/src/app/api/v1/rate/watchlist/route.ts b/apps/web/src/app/api/v1/rate/watchlist/route.ts index 4191b655b..9ef597ab7 100644 --- a/apps/web/src/app/api/v1/rate/watchlist/route.ts +++ b/apps/web/src/app/api/v1/rate/watchlist/route.ts @@ -1,10 +1,6 @@ import { NextRequest, NextResponse } from 'next/server'; -import { plausibleMinPrice, plausibleOffers } from '@/lib/offer-plausibility'; import { z } from 'zod'; -import { - MAX_VALID_VND_PRICE, - normalizeOfferOriginalPrice, -} from '@/lib/rate-api'; +import { MAX_VALID_VND_PRICE } from '@/lib/rate-api'; import { computeProductRadarSignal, getCachedMasterProductBySlug, @@ -17,6 +13,8 @@ import { jsonApiError, parseWatchlistSlugsParam, resolveWatchlistMasterPriceBounds, + selectBestOfferFromListings, + withPlausibleMinPrice, } from '@/server/rate/api-handler'; import { authenticateRateRequest, @@ -64,83 +62,6 @@ const RPC_REJECTIONS: Record = { }, }; -export interface WatchlistBestOffer { - id: string; - title: string; - price: number; - original_price: number; - url: string; - platform: string; - merchant_name: string; - is_official_mall: boolean; -} - -// The stored min_price can be a bait-price listing; target checks use the -// lowest plausible offer instead (DOS/Rate#797). -function withPlausibleMinPrice( - master: T, - offers: readonly { price_current: number }[] -): T { - return { - ...master, - min_price: plausibleMinPrice(Number(master.min_price || 0), offers), - }; -} - -function selectBestOfferFromListings( - offers: Awaited> -): { bestOffer: WatchlistBestOffer | null; offerPlatforms: string[] } { - // A bait-price listing is never the best offer (DOS/Rate#797). - const valid = plausibleOffers(offers).filter((o) => { - const price = Number(o.price_current); - return ( - Number(o.stock ?? 1) > 0 && - Number.isFinite(price) && - price > 0 && - price <= MAX_VALID_VND_PRICE - ); - }); - const platforms = Array.from( - new Set( - valid - .map((o) => - String(o.merchant?.platform || o.platform || 'shopee').toLowerCase() - ) - .filter(Boolean) - ) - ); - if (valid.length === 0) { - return { bestOffer: null, offerPlatforms: platforms }; - } - - const sorted = [...valid].sort((a, b) => { - const diff = Number(a.price_current) - Number(b.price_current); - if (diff !== 0) return diff; - const mallA = a.merchant?.is_official ? 1 : 0; - const mallB = b.merchant?.is_official ? 1 : 0; - return mallB - mallA; - }); - - const best = sorted[0]!; - const bestPrice = Number(best.price_current); - const validOrig = normalizeOfferOriginalPrice(bestPrice, best.price_original); - return { - bestOffer: { - id: best.id, - title: best.title, - price: bestPrice, - original_price: validOrig ?? bestPrice, - url: best.url, - platform: String( - best.merchant?.platform || best.platform || 'shopee' - ).toLowerCase(), - merchant_name: String(best.merchant?.name || 'Official Store'), - is_official_mall: Boolean(best.merchant?.is_official), - }, - offerPlatforms: platforms, - }; -} - /** * GET /api/v1/rate/watchlist?slugs=apple-iphone-16-128gb,apple-iphone-16-pro-max-256gb * Evaluates live deal status across all 10 platforms for a list of watched product slugs, @@ -343,12 +264,14 @@ export async function POST(request: NextRequest) { } if (!rpcError && rpcData) { + const safeRpcData = + typeof rpcData === 'object' && !Array.isArray(rpcData) ? rpcData : {}; return NextResponse.json( { ok: true, persistedInCloud: true, data: { - ...(typeof rpcData === 'object' ? rpcData : {}), + ...safeRpcData, best_offer: bestOffer, offer_platforms: offerPlatforms, deal_radar: radarExportFields(radarSignal), diff --git a/apps/web/src/lib/__tests__/analyzer-copy.test.ts b/apps/web/src/lib/__tests__/analyzer-copy.test.ts index 64d3326d2..534053f35 100644 --- a/apps/web/src/lib/__tests__/analyzer-copy.test.ts +++ b/apps/web/src/lib/__tests__/analyzer-copy.test.ts @@ -196,6 +196,12 @@ function main() { } } + // Non-string & oversized input bounds (#854) + assert.deepEqual(analyzerCopy(null as unknown as string), { text: '' }); + const oversized = analyzerCopy('Đ' + 'a'.repeat(5_000)); + assert.ok('text' in oversized && oversized.text.length === 4_000); + assert.equal('text' in oversized ? oversized.lang : undefined, 'vi'); + console.warn('✅ Analyzer copy suite passed'); } diff --git a/apps/web/src/lib/__tests__/catalog-listing.test.ts b/apps/web/src/lib/__tests__/catalog-listing.test.ts index c40a6bb95..ccf4e56fb 100644 --- a/apps/web/src/lib/__tests__/catalog-listing.test.ts +++ b/apps/web/src/lib/__tests__/catalog-listing.test.ts @@ -50,4 +50,25 @@ for (const filter of [ ); } +// Invalid/non-object params or out-of-bounds limit/offset are rejected (#854) +assert.equal(isPlainListing(null as never), false); +assert.equal(isPlainListing([] as never), false); +assert.equal(isPlainListing({ categorySlug: 'smartphones', limit: 0 }), false); +assert.equal( + isPlainListing({ categorySlug: 'smartphones', limit: 501 }), + false +); +assert.equal( + isPlainListing({ categorySlug: 'smartphones', limit: Number.NaN }), + false +); +assert.equal( + isPlainListing({ categorySlug: 'smartphones', offset: -1 }), + false +); +assert.equal( + isPlainListing({ categorySlug: 'smartphones', offset: 100_001 }), + false +); + console.warn('✅ Catalogue plain-listing suite passed'); diff --git a/apps/web/src/lib/__tests__/client-messages.test.ts b/apps/web/src/lib/__tests__/client-messages.test.ts index b725788f8..872885bf2 100644 --- a/apps/web/src/lib/__tests__/client-messages.test.ts +++ b/apps/web/src/lib/__tests__/client-messages.test.ts @@ -3,6 +3,21 @@ import { readdirSync, readFileSync, statSync } from 'node:fs'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; import { CLIENT_MESSAGE_SCOPES, pickClientMessages } from '../client-messages'; +import { + buildRadarIndex, + getRadarSignal, + radarExportFields, +} from '../deal-radar'; +import { + formatStorageLabel, + getCategoryFilterSchema, + localizeCategoryFilterSchema, +} from '../category-filters'; +import { + rankMasterProductsForLeaderboard, + scoreMasterProductForLeaderboard, +} from '../leaderboard-engine'; +import type { MasterProduct } from '../rate-api'; const srcDir = fileURLToPath(new URL('../../', import.meta.url)); const en = JSON.parse( @@ -70,6 +85,122 @@ function main() { } assert.deepEqual([...missing], [], 'client namespaces without a scope'); + // Prototype-key & invalid input guards (#852) + assert.deepEqual( + pickClientMessages(null as unknown as Record), + {} + ); + const protoFallback = pickClientMessages(en, 'constructor' as never); + assert.deepEqual(Object.keys(protoFallback).sort(), [ + 'compare', + 'deals', + 'errors', + 'rate', + 'watchlist', + ]); + + // deal-radar bounds: negative percentages clamp to 0 & control chars stripped from slugs (#852) + const negExport = radarExportFields({ + verdict: 'none', + savingsPct: -15, + fakeMarkupPct: -30, + spreadPct: -8, + genuineDeal: null, + fakeSaleTrap: null, + priceSpread: null, + }); + assert.equal(negExport.savingsPct, 0); + assert.equal(negExport.fakeMarkupPct, 0); + assert.equal(negExport.spreadPct, 0); + + const radarIdx = buildRadarIndex({ + generatedAt: '2026-06-01T00:00:00Z', + categoryFilter: 'all', + totalClustersAnalyzed: 1, + totalOffersAnalyzed: 2, + genuineDeals: [ + { + id: 'gd-ctrl', + verdict: 'best_market_deal', + verdictBadgeVi: 'Giá tốt', + productSlug: ' iphone-16\u0000-pro ', + productName: 'iPhone 16 Pro', + brand: 'Apple', + categorySlug: 'smartphones', + productHref: '/smartphones/iphone-16-pro', + imageUrl: null, + platform: 'shopee', + merchantId: 'm-1', + merchantName: 'Shop A', + isOfficialMall: true, + merchantRiskScore: 5, + offerUrl: 'https://shopee.vn/a', + priceCurrent: 24_000_000, + priceOriginal: 28_000_000, + marketMedianPrice: 26_000_000, + marketMinPrice: 24_000_000, + marketMaxPrice: 28_000_000, + savingsVsMedianVnd: 2_000_000, + savingsVsMedianPct: 7.7, + claimedDiscountPct: 14.3, + fakeMarkupPct: 0, + jreviewsScore10: 9.0, + explanationVi: 'Tốt', + }, + ], + fakeSaleTraps: [], + priceSpreadLeaders: [], + }); + assert.equal(getRadarSignal(radarIdx, 'iphone-16-pro').savingsPct, 7.7); + + // category-filters prototype-key & storage guards (#852) + const protoSchema = getCategoryFilterSchema('constructor', [ + ' Sony\u0007 ', + ]); + assert.equal(protoSchema.categorySlug, 'smartphones'); + assert.ok(protoSchema.defaultBrands.includes('Sony')); + const localizedProto = localizeCategoryFilterSchema( + { ...protoSchema, categorySlug: '__proto__' }, + (k) => k + ); + assert.equal(localizedProto.categoryTitle, 'schemas.smartphones.title'); + assert.equal(formatStorageLabel(Number.NaN), '0GB'); + assert.equal(formatStorageLabel(-128), '0GB'); + assert.equal(formatStorageLabel(1024), '1TB'); + + // leaderboard-engine count ceiling & limit cap (#852) + const scored = scoreMasterProductForLeaderboard({ + id: 'p1', + slug: 'iphone\u0000-16', + name: 'iPhone\u0007 16', + brand: 'Apple', + model: 'iPhone 16', + user_rating_weighted: 4.8, + total_reviews: 999_999_999, + authentic_review_count: 888_888_888, + merchant_count: 777_777_777, + min_price: 20_000_000, + } as unknown as MasterProduct); + assert.equal(scored.slug, 'iphone-16'); + assert.equal(scored.name, 'iPhone 16'); + assert.equal(scored.authenticReviewCount, 10_000_000); + assert.equal(scored.merchantCount, 10_000_000); + const ranked = rankMasterProductsForLeaderboard( + [ + { + id: 'p1', + slug: 'iphone-16', + name: 'iPhone 16', + brand: 'Apple', + model: 'iPhone 16', + user_rating_weighted: 4.8, + min_price: 20_000_000, + } as unknown as MasterProduct, + ], + { limit: 999_999, query: 'iphone\u0000' } + ); + assert.equal(ranked.length, 1); + console.warn('✅ Client messages suite passed'); } diff --git a/apps/web/src/lib/__tests__/content-lang.test.ts b/apps/web/src/lib/__tests__/content-lang.test.ts index fe3ca35df..35c461b89 100644 --- a/apps/web/src/lib/__tests__/content-lang.test.ts +++ b/apps/web/src/lib/__tests__/content-lang.test.ts @@ -28,6 +28,11 @@ function main() { assert.equal(contentLang(null, 'en'), undefined); assert.equal(contentLang(5000, 'en'), undefined); assert.equal(contentLang(true, 'en'), undefined); + assert.equal(isVietnamese(''), false); + assert.equal(isVietnamese(null as unknown as string), false); + // Oversized input (> 4_000 chars) slices before NFC normalization (#852) + assert.equal(isVietnamese('a'.repeat(5_000) + 'đ'), false); + assert.equal(isVietnamese('Đ' + 'a'.repeat(5_000)), true); console.warn('✅ Content language suite passed'); } diff --git a/apps/web/src/lib/__tests__/discount-verdict-copy.test.ts b/apps/web/src/lib/__tests__/discount-verdict-copy.test.ts index b85c52b97..7f6189564 100644 --- a/apps/web/src/lib/__tests__/discount-verdict-copy.test.ts +++ b/apps/web/src/lib/__tests__/discount-verdict-copy.test.ts @@ -134,6 +134,13 @@ function main() { const suspicious = discountVerdictCopy(en, cases[1]![1]); assert.ok(suspicious.explanation.includes('(-0%)'), suspicious.explanation); + // Nullish analysis input falls back safely to normal_price (#854) + const fallbackNull = discountVerdictCopy( + en, + null as unknown as OfferDiscountAnalysis + ); + assert.equal(fallbackNull.label, messages.en.labelNormal); + console.warn('✅ Discount verdict copy suite passed'); } diff --git a/apps/web/src/lib/__tests__/logger.test.ts b/apps/web/src/lib/__tests__/logger.test.ts index 5d5d79933..95e76d79d 100644 --- a/apps/web/src/lib/__tests__/logger.test.ts +++ b/apps/web/src/lib/__tests__/logger.test.ts @@ -85,6 +85,26 @@ function main() { } ); + // Circular references, shared DAG sub-objects, BigInt, NaN, and boundary-straddling emails/tokens (#844). + const sharedDag = { ok: true }; + const cyclic: Record = { + first: sharedDag, + second: sharedDag, + big: BigInt('9007199254740993'), + nanVal: Number.NaN, + bearerMsg: `Authorization: Bearer ${'s'.repeat(16)}`, + boundaryEmail: `${'a'.repeat(1995)}victim@example.com`, + }; + cyclic.self = cyclic; + const redactedCyclic = redact(cyclic) as Record; + assert.equal(redactedCyclic.self, '[Circular]'); + assert.deepEqual(redactedCyclic.first, { ok: true }); + assert.deepEqual(redactedCyclic.second, { ok: true }); + assert.equal(redactedCyclic.big, '9007199254740993'); + assert.equal(redactedCyclic.nanVal, null); + assert.equal(redactedCyclic.bearerMsg, 'Authorization: Bearer [redacted]'); + assert.equal(String(redactedCyclic.boundaryEmail).includes('victim'), false); + console.warn('✅ Structured logger suite passed'); } diff --git a/apps/web/src/lib/__tests__/offer-plausibility.test.ts b/apps/web/src/lib/__tests__/offer-plausibility.test.ts index 7216bab42..279c8a9f6 100644 --- a/apps/web/src/lib/__tests__/offer-plausibility.test.ts +++ b/apps/web/src/lib/__tests__/offer-plausibility.test.ts @@ -93,6 +93,16 @@ function main() { assert.equal(plausibleMinPrice(5000, []), 5000); assert.equal(plausibleMinPrice(5000, [{ price_current: 0 }]), 5000); + // DOS/Rate#850: Overflowed >1B VND prices are ignored and invalid storedMinPrice falls back to 0. + assert.equal( + implausiblyLowIndexes([20_000_000, 22_000_000, 9_000_000_000]).size, + 0, + 'Overflowed >1B VND price must be ignored and not pull median up (#850)' + ); + assert.equal(plausibleMinPrice(Number.NaN, []), 0); + assert.equal(plausibleMinPrice(-5000, []), 0); + assert.equal(plausibleMinPrice(5_000_000_000, []), 0); + console.warn('✅ Offer plausibility suite passed'); } diff --git a/apps/web/src/lib/__tests__/product-competitors.test.ts b/apps/web/src/lib/__tests__/product-competitors.test.ts index dd4450ede..023244717 100644 --- a/apps/web/src/lib/__tests__/product-competitors.test.ts +++ b/apps/web/src/lib/__tests__/product-competitors.test.ts @@ -109,6 +109,25 @@ function main() { 4.5 ); + // DOS/Rate#850: Non-finite/overflowed prices and control characters in buildKeySpecLine. + assert.equal(peerPriceDelta(Number.NaN, 20_000_000), 0); + assert.equal(peerPriceDelta(2_000_000_000, 20_000_000), 0); + assert.equal( + buildKeySpecLine({ + key_specs: { + chip: ' A18\u0000\r\nPro ', + ram: '8GB', + storage: '256GB', + battery: '4685 mAh', + }, + specifications: {}, + ram_gb: null, + storage_gb: null, + battery_mah: null, + } as never), + 'A18 Pro • 8GB • 256GB • 4685 mAh' + ); + console.warn('✅ Product competitors suite passed'); } diff --git a/apps/web/src/lib/__tests__/product-detail-copy.test.ts b/apps/web/src/lib/__tests__/product-detail-copy.test.ts index c5376d558..42db350dd 100644 --- a/apps/web/src/lib/__tests__/product-detail-copy.test.ts +++ b/apps/web/src/lib/__tests__/product-detail-copy.test.ts @@ -2,6 +2,7 @@ import assert from 'node:assert/strict'; import { readFileSync } from 'node:fs'; import { computeProductRadarSignal } from '../../server/rate/deals'; import { bayesianTierKey, radarExplanation } from '../product-detail-copy'; +import { localizeSpecValue } from '../spec-localizer'; import { computeBayesianRatingSummary } from '../rate-api'; type Messages = Record>; @@ -130,6 +131,38 @@ function main() { } } + // Non-finite / negative radar values & control-char sanitization (#854) + assert.equal( + bayesianTierKey({ + confidenceTier: 'none' as never, + effectiveWeightSum: Number.NaN, + }), + 'bayesianTierPrior' + ); + const sanitizedDealExp = radarExplanation({ + verdict: 'genuine_deal', + genuineDeal: { + ...signal.genuineDeal!, + savingsVsMedianVnd: -500_000, + savingsVsMedianPct: Number.NaN, + merchantName: ' Official\u0000 Store ', + platform: ' shopee\u0007 ', + }, + fakeSaleTrap: null, + }); + assert.equal(sanitizedDealExp?.key, 'radarExplanationGenuineDeal'); + if (sanitizedDealExp?.key === 'radarExplanationGenuineDeal') { + assert.equal(sanitizedDealExp.values.savings, '0'); + assert.equal(sanitizedDealExp.values.pct, '0'); + assert.equal(sanitizedDealExp.values.merchant, 'Official Store'); + assert.equal(sanitizedDealExp.values.platform, 'SHOPEE'); + } + + // spec-localizer control-char stripping & length cap (#854) + assert.equal(localizeSpecValue('\u0000\u0007', 'vi'), '-'); + assert.equal(localizeSpecValue('Quad\u0000Camera', 'vi'), 'Cụm 4 camera'); + assert.equal(localizeSpecValue('x'.repeat(3_000), 'en').length, 2_000); + console.warn('✅ Product detail copy suite passed'); } diff --git a/apps/web/src/lib/__tests__/review-display.test.ts b/apps/web/src/lib/__tests__/review-display.test.ts index d22d029b7..944ccb40a 100644 --- a/apps/web/src/lib/__tests__/review-display.test.ts +++ b/apps/web/src/lib/__tests__/review-display.test.ts @@ -262,6 +262,32 @@ function main() { }), { text: cached.explanationVi } ); + // DOS/Rate#850: NaN weight, prototype keys, and control chars in explanationVi + assert.equal( + isFilteredOutReview({ ...counted, helpfulness_weight: Number.NaN }), + true + ); + assert.equal(sponsorshipBadgeKey('constructor' as never), null); + assert.equal( + blendNoteKey('constructor' as never), + 'blendNoteInsufficientData' + ); + assert.deepEqual( + weightExplanationCopy({ + weight: Number.NaN, + explanationVi: ' Trọng số\u0000\r\n0.00x ', + explanationKind: 'seeding', + }), + { key: 'weightExplanationSeeding', values: { weight: '0.00' } } + ); + assert.deepEqual( + weightExplanationCopy({ + weight: 1, + explanationVi: ' A=1\u0000\r\nB=1 ', + explanationKind: 'formula', + }), + { text: 'A=1 B=1' } + ); const blendCases: Array< [string, Parameters[0]] diff --git a/apps/web/src/lib/__tests__/seo.test.ts b/apps/web/src/lib/__tests__/seo.test.ts index 4bbcb8c57..a4e701537 100644 --- a/apps/web/src/lib/__tests__/seo.test.ts +++ b/apps/web/src/lib/__tests__/seo.test.ts @@ -2,11 +2,17 @@ import assert from 'node:assert/strict'; import { MAX_DESCRIPTION_LENGTH, MAX_TITLE_LENGTH, + SITE_URL, absoluteUrl, fitDescription, fitTitle, siteJsonLd, } from '../seo'; +import { buildSafeCsvContent, escapeCsvCell } from '../csv-safe'; +import { + pageMerchantDirectory, + parseMerchantDirectoryFilters, +} from '../merchant-directory'; function main() { console.warn('🧪 SEO helpers suite'); @@ -77,6 +83,41 @@ function main() { assert.ok(fittedDesc.endsWith('…')); assert.equal(fitDescription('Ngắn gọn.'), 'Ngắn gọn.'); + // DOS/Rate#848: absoluteUrl protocol-relative/credential/invalid guards, fitTitle NaN/control chars, CSV & merchant-directory bounds + assert.equal(absoluteUrl('//evil.example.com/phish'), SITE_URL); + assert.equal(absoluteUrl('/\\evil.example.com/phish'), SITE_URL); + assert.equal( + absoluteUrl('https://user:pass@cdn.example.com/a.jpg'), + SITE_URL + ); + assert.equal(absoluteUrl('javascript:alert(1)'), SITE_URL); + assert.equal(absoluteUrl('http://[invalid-url'), SITE_URL); + assert.equal( + fitTitle([' iPhone 16\u0000\r\nPro Max '], Number.NaN), + 'iPhone 16 Pro Max' + ); + + assert.equal( + escapeCsvCell('\u0000=cmd|"/c calc"!A0'), + `"'=cmd|""/c calc""!A0"` + ); + assert.equal(escapeCsvCell('\u000b+SUM(A1:A2)'), "'+SUM(A1:A2)"); + assert.equal(escapeCsvCell('-15.4%'), '-15.4%'); + assert.equal(escapeCsvCell('x'.repeat(12_000)).length, 10_000); + const csvOut = buildSafeCsvContent(['col1', 'col2'], [['\u0000=1+1', 42]]); + assert.ok(csvOut.includes("'=1+1,42")); + + const params848 = new URLSearchParams( + 'q= Apple\u0000\r\nStore &platform=bad'), '#'); + assert.equal(formatHref('//evil.example/phish'), '#'); + assert.equal(formatHref('/\\evil.example/phish'), '#'); + assert.equal(formatHref('https://user:pass@rate.vn/deals'), '#'); + assert.equal(formatHref('httpfoo:bar'), '#'); + assert.equal(formatHref(`https://rate.vn/${'a'.repeat(2100)}`), '#'); + assert.equal(formatHref('products/iphone-16\u0000'), '/products/iphone-16'); + assert.equal(formatHref('#reviews\u0007'), '#reviews'); + assert.equal( + formatHref('mailto:support@rate.vn'), + 'mailto:support@rate.vn' + ); + assert.equal(formatHref('mailto:bad address@rate.vn'), '#'); + + assert.equal(isAppLink('https://rate.vn/smartphones'), true); + assert.equal(isAppLink('/smartphones/iphone-16'), true); + assert.equal(isAppLink('//evil.example/phish'), false); + assert.equal(isAppLink('javascript:alert(1)'), false); + assert.equal(isAppLink('https://user:pass@rate.vn/smartphones'), false); + + const summary856 = summarizeMerchantDirectory([ + { + id: 'm1', + name: 'Shop 1', + platform: 'constructor', + platform_shop_id: 's1', + is_official: true, + trust_score: 90, + dosafe_risk_score: 10, + } as never, + { + id: 'm2', + name: 'Shop 2', + platform: ' Shopee\u0000 ', + platform_shop_id: 's2', + is_official: false, + trust_score: 70, + dosafe_risk_score: 20, + } as never, + ]); + assert.equal(summary856.platformCounts.shopee, 1); + assert.equal( + Object.hasOwn(summary856.platformCounts, 'constructor'), + false, + 'summarizeMerchantDirectory must not collide with Object.prototype.constructor' + ); + + const filterParams856 = merchantDirectoryFilterParams({ + query: ` apple\u0000 store ${'x'.repeat(200)}`, + platform: 'bad/platform', + officialOnly: true, + riskTier: 'safe', + seedingTier: 'low', + sortBy: 'products_desc', + }); + assert.equal(filterParams856.get('q')?.length, 160); + assert.equal(filterParams856.get('platform'), null); + assert.equal(filterParams856.get('official'), 'true'); + + const heroMerchants856 = toHeroMerchantSuggestions( + Array.from({ length: MAX_HERO_MERCHANTS + 10 }, (_, idx) => ({ + id: `m-${idx}\u0000`, + name: ` Store ${idx}\u0007 `, + platform: 'shopee', + platform_shop_id: `shop-${idx}`, + is_official: true, + trust_score: 150, + dosafe_risk_score: -10, + })) + ); + assert.equal(heroMerchants856.length, MAX_HERO_MERCHANTS); + assert.equal(heroMerchants856[0]?.id, 'm-0'); + assert.equal(heroMerchants856[0]?.name, 'Store 0'); + assert.equal(heroMerchants856[0]?.trust_score, 100); + assert.equal(heroMerchants856[0]?.dosafe_risk_score, 0); + + const heroProducts856 = toHeroProductSuggestions( + Array.from({ length: MAX_HERO_PRODUCTS + 5 }, (_, idx) => ({ + id: `p-${idx}`, + slug: `slug-${idx}`, + name: `Product ${idx}`, + brand: 'Apple', + model: 'iPhone', + model_key: `apple:iphone:${idx}`, + variant_name: '128GB', + key_specs: { + chip: ' A18\u0000 Pro ', + screen: '\u0007 ', + }, + storage_gb: 128, + images: ['img1.jpg', 'img2.jpg'], + thumbnail: 'thumb.jpg', + min_price: 20_000_000, + merchant_count: 3, + user_rating_weighted: 4.8, + average_rating: 4.7, + category_id: 'cat-1', + metadata: { category_slug: ' smartphones\u0000 ', extra: 1 } as never, + specifications: { category_slug: '\u0000 ' } as never, + })) + ); + assert.equal(heroProducts856.length, MAX_HERO_PRODUCTS); + assert.equal(heroProducts856[0]?.key_specs?.chip, 'A18 Pro'); + assert.equal(heroProducts856[0]?.key_specs?.screen, undefined); + assert.deepEqual(heroProducts856[0]?.metadata, { + category_slug: 'smartphones', + }); + assert.deepEqual(heroProducts856[0]?.specifications, {}); + + assert.equal(normalizeRateLocale(' EN-US '), 'en'); + assert.equal(normalizeRateLocale('cs_CZ\u0000'), 'cs'); + assert.equal(normalizeRateLocale('vi-VN'), 'vi'); + assert.equal(normalizeRateLocale('fr-FR'), 'vi'); + console.log( + ' ✅ [97/98] Navigation, MerchantDirectory, Hero Suggestions & Rate i18n Bounds (#856) passed' + ); + } + + // ========================================================================= + // 98. Server Rate Deals Radar & Cached Query Key Bounds (#858) + // ========================================================================= + { + const { + MAX_DEAL_RADAR_CLUSTERS, + getCachedMasterProductOffers, + getCachedMasterProductReviews, + getCachedMerchantProfileById, + normalizeCachedFacetCategory, + normalizeDealRadarOptions, + sanitizeCachedCompareSlugsCsv, + sanitizeCachedEntityId, + sanitizeCachedInspectQuery, + sanitizeCachedProductSlug, + } = await import('../index'); + + assert.equal(normalizeCachedFacetCategory(undefined), 'smartphones'); + assert.equal(normalizeCachedFacetCategory(' '), 'smartphones'); + assert.equal( + normalizeCachedFacetCategory(' SMARTPHONES\u0000 '), + 'smartphones' + ); + assert.equal(normalizeCachedFacetCategory('tablets'), 'tablets'); + assert.equal(normalizeCachedFacetCategory('arbitrary-category'), 'all'); + + assert.equal(await getCachedMerchantProfileById(' \u0000 '), null); + assert.deepEqual(await getCachedMasterProductOffers(' \u0000 '), []); + assert.deepEqual(await getCachedMasterProductReviews(' \u0000 '), []); + + assert.deepEqual( + normalizeDealRadarOptions({ + categorySlug: ' SMARTPHONES\u0000 ', + platform: ' Shopee\u0007 ', + limit: 999, + }), + { + categoryFilter: 'smartphones', + platformFilter: 'shopee', + limit: 50, + } + ); + assert.deepEqual( + normalizeDealRadarOptions({ + categorySlug: 'arbitrary-invalid-category', + platform: 'bad/platform!@#', + limit: -5, + }), + { + categoryFilter: 'all', + platformFilter: 'all', + limit: 12, + } + ); + assert.equal(MAX_DEAL_RADAR_CLUSTERS, 500); + + // Every offer of a cluster counts: with the priciest of 250 offers last, + // the spread still reaches it (DOS/Rate#867; a 200-offer slice made it 0). + { + const { computeCrossPlatformDealRadarFromData: radar867 } = + await import('../index'); + const offers867 = Array.from({ length: 250 }, (_, i) => ({ + id: 'o867-' + i, + platform: i % 2 ? 'shopee' : 'lazada', + url: 'https://example.test/867/' + i, + price_current: i < 249 ? 10_000_000 : 12_000_000, + price_original: null, + merchant: { + id: 'm867-' + i, + name: 'Shop ' + i, + risk_score: 5, + is_official: true, + }, + })); + const feed867 = radar867( + [ + { + master: { + id: 'id-867', + slug: 'phone-867', + name: 'Phone 867', + brand: 'Brand', + category: 'smartphones', + min_price: 10_000_000, + max_price: 12_000_000, + }, + offers: offers867, + }, + ] as never, + { categorySlug: 'all', limit: 10 } + ); + assert.equal(feed867.totalOffersAnalyzed, 250); + const spread867 = feed867.priceSpreadLeaders.find( + (s) => s.productSlug === 'phone-867' + ); + assert.ok(spread867, 'a 20% spread across 250 offers is reported'); + assert.equal(spread867.maxPrice, 12_000_000); + } + + assert.equal(sanitizeCachedEntityId(' m-123\u0000\u0007 '), 'm-123'); + assert.equal( + sanitizeCachedProductSlug(' iPhone-16-Pro\u0000 '), + 'iphone-16-pro' + ); + assert.equal(sanitizeCachedProductSlug('bad/slug/../../etc'), ''); + assert.equal( + sanitizeCachedCompareSlugsCsv( + ' iPhone-16\u0000 , bad/slug , GALAXY-S24 , iphone-16 , pixel-9 , xiaomi-15 , extra-6 ' + ), + 'iphone-16,galaxy-s24,pixel-9,xiaomi-15' + ); + assert.equal( + sanitizeCachedInspectQuery(' https://shopee.vn/item\u0000\n123 '), + 'https://shopee.vn/item 123' + ); + console.log( + ' ✅ [98/99] Server Rate Deals Radar & Cached Query Key Bounds (#858) passed' + ); + } + + // ========================================================================= + // 99. Review Vote Eligibility, Community Review Mapper & OpenAPI Origin Bounds (#860) + // ========================================================================= + { + const { evaluateVoterEligibility } = + await import('../../../app/api/reviews/[id]/vote/route'); + const { toReviewItem } = + await import('../../../app/api/v1/rate/reviews/route'); + const { normalizeOpenApiServerOrigin } = await import('../openapi'); + + assert.equal(evaluateVoterEligibility(null).eligible, false); + assert.equal(evaluateVoterEligibility(undefined).eligible, false); + assert.equal( + evaluateVoterEligibility({ + created_at: '2026-01-01T00:00:00Z', + email_confirmed_at: ' ', + phone_confirmed_at: 'not-a-timestamp', + }).eligible, + false + ); + assert.equal( + evaluateVoterEligibility({ + created_at: '1970-01-01T00:00:00Z', + email_confirmed_at: '2026-01-01T00:00:00Z', + }).eligible, + false + ); + assert.equal( + evaluateVoterEligibility({ + created_at: new Date(Date.now() - 8 * 86_400_000).toISOString(), + email_confirmed_at: '2026-01-01T00:00:00Z', + }).eligible, + true + ); + + const mappedNullStar = toReviewItem( + { + id: ' rev-1 ', + author_name: ' Nguyen Van A ', + rating_star: null, + comment: ' Đánh giá chi tiết sản phẩm ', + variation: ' ', + created_at: 'invalid-date', + upvotes: '12', + downvotes: -3, + report_count: '2', + helpfulness_weight: null, + fake_probability: '', + reason: ' ', + }, + { screen: 4.56, battery: 99 } + ); + assert.equal(mappedNullStar.id, 'rev-1'); + assert.equal(mappedNullStar.author_name, 'Nguyen Van A'); + assert.equal( + mappedNullStar.rating_star, + 5, + 'null rating_star must fall back to default 5 instead of coercing to 0' + ); + assert.equal(mappedNullStar.variation, null); + assert.ok(Number.isFinite(Date.parse(mappedNullStar.created_at))); + assert.equal(mappedNullStar.upvotes, 12); + assert.equal(mappedNullStar.downvotes, 0); + assert.equal(mappedNullStar.report_count, 2); + assert.equal(mappedNullStar.fake_probability, null); + assert.equal(mappedNullStar.reason, null); + assert.deepEqual(mappedNullStar.criteria_ratings, { screen: 4.6 }); + + assert.equal( + normalizeOpenApiServerOrigin(123 as unknown as string), + 'https://rate.vn' + ); + assert.equal( + normalizeOpenApiServerOrigin('https://user:pass@rate.vn/api'), + 'https://rate.vn' + ); + assert.equal( + normalizeOpenApiServerOrigin('https://staging.rate.vn/path'), + 'https://staging.rate.vn' + ); + console.log( + ' ✅ [99/99] Review Vote Eligibility, Community Review Mapper & OpenAPI Origin Bounds (#860) passed' + ); + } + + // 100. Issue #864: API Handler Price/IP Bounds, Watchlist Offer Selection, and Crowd Ingest URL Checks + { + const { + extractClientRateLimitKey, + isSafeCrowdIngestHttpUrl, + parseOptionalVndPriceParam, + resolveWatchlistMasterPriceBounds, + selectBestOfferFromListings, + withPlausibleMinPrice, + } = await import('../api-handler'); + + assert.equal( + parseOptionalVndPriceParam('15000000', Number.NaN), + 15_000_000, + 'parseOptionalVndPriceParam must fall back to 500M cap when maxVnd is NaN' + ); + assert.deepEqual( + resolveWatchlistMasterPriceBounds(null, 24_990_000, Number.NaN), + { + currentMinPrice: 24_990_000, + currentMaxPrice: 24_990_000, + }, + 'null master must fall back to bestOfferPrice without throwing' + ); + assert.deepEqual( + resolveWatchlistMasterPriceBounds( + { min_price: true, max_price: true }, + 24_990_000 + ), + { + currentMinPrice: 24_990_000, + currentMaxPrice: 24_990_000, + }, + 'boolean min_price/max_price must not coerce to 1 VND' + ); + + assert.equal(extractClientRateLimitKey(null), 'anon'); + const spoofedHeaders = { + get(name: string) { + return name.toLowerCase() === 'x-forwarded-for' + ? ' 203.0.113.10\r\n\u0000 , 10.0.0.1 ' + : null; + }, + } as unknown as Headers; + assert.equal(extractClientRateLimitKey(spoofedHeaders), '203.0.113.10'); + const wsHeaders = new Headers({ + 'x-forwarded-for': ' 203.0.113.10\t99 , 10.0.0.1 ', + }); + assert.equal(extractClientRateLimitKey(wsHeaders), '203.0.113.1099'); + + const plausibleBoolMaster = withPlausibleMinPrice( + { min_price: true as unknown as number }, + [{ price_current: 28_990_000 }] + ); + assert.equal( + plausibleBoolMaster.min_price, + 28_990_000, + 'withPlausibleMinPrice must ignore boolean min_price and use plausible offer price' + ); + + const bestOfferRes = selectBestOfferFromListings([ + { + id: 'off-bool', + title: 'Boolean Price Offer', + price_current: true as unknown as number, + price_original: 0, + stock: 10, + url: 'https://shopee.vn/product/1/2', + platform: 'shopee', + merchant: { name: 'Bad Shop', platform: 'shopee', is_official: true }, + }, + { + id: ' off-valid ', + title: ' Apple\u0000 iPhone 16 128GB ', + price_current: 21_990_000, + price_original: 22_990_000, + stock: 5, + url: ' https://shopee.vn/product/88201679/27110898667 ', + platform: 'shopee', + merchant: { + name: ' Apple\u0007 Flagship Store ', + platform: 'Shopee\u0000', + is_official: true, + }, + }, + ] as unknown as Parameters[0]); + assert.equal(bestOfferRes.bestOffer?.id, 'off-valid'); + assert.equal(bestOfferRes.bestOffer?.title, 'Apple iPhone 16 128GB'); + assert.equal( + bestOfferRes.bestOffer?.merchant_name, + 'Apple Flagship Store' + ); + assert.equal(bestOfferRes.bestOffer?.platform, 'shopee'); + assert.deepEqual(bestOfferRes.offerPlatforms, ['shopee']); + + assert.equal( + isSafeCrowdIngestHttpUrl( + 'https://shopee.vn/product-i.88201679.27110898667' + ), + true + ); + assert.equal( + isSafeCrowdIngestHttpUrl( + 'https://user:pass@shopee.vn/product-i.88201679.27110898667' + ), + false, + 'isSafeCrowdIngestHttpUrl must reject embedded URL credentials' + ); + assert.equal( + isSafeCrowdIngestHttpUrl('ftp://shopee.vn/product-i.1.2'), + false + ); + + console.log( + ' ✅ [100/100] API Handler Price/IP Bounds, Watchlist Offer Selection & Crowd Ingest URL Checks (#864) passed' + ); + } + + // 101. Issue #868: Inspect Observed Price Bounds, Health TTL Clock Guard & Revalidate Token Checks + { + const { NextRequest } = await import('next/server'); + const { resolveInspectObservedPrices } = await import('../api-handler'); + const { extractProvidedToken, verifyRevalidateSecret } = + await import('../../../app/api/v1/rate/revalidate/route'); + + assert.deepEqual( + resolveInspectObservedPrices({ + queryPrice: null, + queryOriginalPrice: null, + matchedOfferPriceCurrent: true, + matchedOfferPriceOriginal: true, + masterMinPrice: 18_990_000, + }), + { + observedPrice: 18_990_000, + observedOriginalPrice: 0, + }, + 'resolveInspectObservedPrices must reject boolean price_current/price_original and fall back to masterMinPrice' + ); + assert.deepEqual( + resolveInspectObservedPrices({ + queryPrice: null, + queryOriginalPrice: null, + matchedOfferPriceCurrent: 22_990_000.4, + matchedOfferPriceOriginal: 25_990_000.4, + masterMinPrice: 20_000_000, + }), + { + observedPrice: 22_990_000, + observedOriginalPrice: 25_990_000, + } + ); + assert.deepEqual( + resolveInspectObservedPrices({ + queryPrice: 24_990_000, + queryOriginalPrice: null, + matchedOfferPriceCurrent: 22_990_000, + matchedOfferPriceOriginal: 25_990_000, + masterMinPrice: 20_000_000, + }), + { + observedPrice: 24_990_000, + observedOriginalPrice: 0, + }, + 'when queryPrice is provided without queryOriginalPrice, matchedOfferPriceOriginal must not be attached' + ); + + assert.equal( + verifyRevalidateSecret('secret\u0000123', 'secret\u0000123'), + false + ); + assert.equal(verifyRevalidateSecret('secret 123', 'secret 123'), false); + assert.equal( + verifyRevalidateSecret('aaaa-aaaa-aaaa', 'aaaa-aaaa-aaaa'), + true + ); + assert.equal( + extractProvidedToken( + new NextRequest('https://rate.vn/api/v1/rate/revalidate', { + headers: { 'x-rate-revalidate-token': 'tok\u0007en' }, + }) + ), + null + ); + + let probes868 = 0; + await withHealthDatabase( + async () => { + probes868++; + return new Response(JSON.stringify([{ id: 'cat-1' }]), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + }, + async () => { + assert.equal(await healthMod.checkDatabase(100_000), 'ok'); + assert.equal(probes868, 1); + // Same window: cached + assert.equal(await healthMod.checkDatabase(105_000), 'ok'); + assert.equal(probes868, 1); + // Backward clock jump (now < cached.at): must not blindly trust a future-stamped cache + assert.equal(await healthMod.checkDatabase(50_000), 'ok'); + assert.equal(probes868, 2); + } + ); + + console.log( + ' ✅ [101/101] Inspect Observed Price Bounds, Health TTL Clock Guard & Revalidate Token Checks (#868) passed' + ); + } + console.log('🎉 All Rate.vn Core Architecture tests passed!'); } diff --git a/apps/web/src/server/rate/api-handler.ts b/apps/web/src/server/rate/api-handler.ts index 6230f17eb..16b4ef417 100644 --- a/apps/web/src/server/rate/api-handler.ts +++ b/apps/web/src/server/rate/api-handler.ts @@ -1,6 +1,11 @@ import 'server-only'; import { NextResponse } from 'next/server'; import { z } from 'zod'; +import { plausibleMinPrice, plausibleOffers } from '@/lib/offer-plausibility'; +import { + MAX_VALID_VND_PRICE, + normalizeOfferOriginalPrice, +} from '@/lib/rate-api'; export const PUBLIC_API_CORS_HEADERS: Record = { 'Access-Control-Allow-Origin': '*', @@ -91,9 +96,15 @@ export function parseBoundedStringParam( maxLength: number ): string | null { if (raw == null) return null; - const trimmed = raw.trim(); - if (!trimmed) return null; - return trimmed.length > maxLength ? trimmed.slice(0, maxLength) : trimmed; + const safeMaxLength = + Number.isFinite(maxLength) && maxLength >= 1 ? Math.floor(maxLength) : 256; + const cleaned = raw.replace(/\p{Cc}+/gu, ' ').trim(); + if (!cleaned) return null; + const sliced = + cleaned.length > safeMaxLength + ? cleaned.slice(0, safeMaxLength).trim() + : cleaned; + return sliced || null; } /** @@ -104,12 +115,16 @@ export function parseOptionalVndPriceParam( raw: string | null | undefined, maxVnd = 500_000_000 ): number | null { - if (raw == null) return null; + if (raw == null || typeof raw !== 'string') return null; const trimmed = raw.trim(); if (!trimmed) return null; const n = Number(trimmed); if (!Number.isFinite(n) || n <= 0) return null; - return Math.min(Math.round(n), maxVnd); + const safeMaxVnd = + typeof maxVnd === 'number' && Number.isFinite(maxVnd) && maxVnd > 0 + ? Math.round(maxVnd) + : 500_000_000; + return Math.min(Math.round(n), safeMaxVnd); } /** @@ -146,33 +161,247 @@ export function parseBoundedPositiveIntArrayParam( * when `master.min_price` is missing, non-positive, or exceeds `maxVnd` (1B VND). */ export function resolveWatchlistMasterPriceBounds( - master: { min_price?: unknown; max_price?: unknown }, + master: { min_price?: unknown; max_price?: unknown } | null | undefined, bestOfferPrice?: number | null, maxVnd = 1_000_000_000 ): { currentMinPrice: number; currentMaxPrice: number } { - const rawMin = Number(master.min_price); + const safeMaxVnd = + typeof maxVnd === 'number' && Number.isFinite(maxVnd) && maxVnd > 0 + ? Math.round(maxVnd) + : 1_000_000_000; + const rawMin = + !master || + typeof master !== 'object' || + typeof master.min_price === 'boolean' || + master.min_price === null || + master.min_price === undefined + ? Number.NaN + : Number(master.min_price); const safeBestOffer = typeof bestOfferPrice === 'number' && Number.isFinite(bestOfferPrice) && bestOfferPrice > 0 && - bestOfferPrice <= maxVnd + bestOfferPrice <= safeMaxVnd ? bestOfferPrice : 0; const currentMinPrice = - Number.isFinite(rawMin) && rawMin > 0 && rawMin <= maxVnd + Number.isFinite(rawMin) && rawMin > 0 && rawMin <= safeMaxVnd ? rawMin : safeBestOffer; - const rawMax = Number(master.max_price); + const rawMax = + !master || + typeof master !== 'object' || + typeof master.max_price === 'boolean' || + master.max_price === null || + master.max_price === undefined + ? Number.NaN + : Number(master.max_price); const currentMaxPrice = currentMinPrice > 0 && Number.isFinite(rawMax) && rawMax >= currentMinPrice && - rawMax <= maxVnd + rawMax <= safeMaxVnd ? rawMax : currentMinPrice; return { currentMinPrice, currentMaxPrice }; } +export interface WatchlistBestOffer { + id: string; + title: string; + price: number; + original_price: number; + url: string; + platform: string; + merchant_name: string; + is_official_mall: boolean; +} + +export interface WatchlistCandidateOffer { + id?: unknown; + title?: unknown; + price_current: number; + price_original?: unknown; + stock?: unknown; + url?: unknown; + platform?: unknown; + merchant?: { + name?: unknown; + platform?: unknown; + is_official?: unknown; + } | null; +} + +// The stored min_price can be a bait-price listing; target checks use the +// lowest plausible offer instead (DOS/Rate#797). +export function withPlausibleMinPrice( + master: T, + offers: readonly { price_current: number }[] +): T { + const rawMin = + !master || + typeof master !== 'object' || + typeof master.min_price === 'boolean' || + master.min_price === null || + master.min_price === undefined + ? 0 + : Number(master.min_price || 0); + return { + ...master, + min_price: plausibleMinPrice( + Number.isFinite(rawMin) ? rawMin : 0, + Array.isArray(offers) ? offers : [] + ), + }; +} + +export function selectBestOfferFromListings( + offers: readonly WatchlistCandidateOffer[] +): { bestOffer: WatchlistBestOffer | null; offerPlatforms: string[] } { + const safeOffers = Array.isArray(offers) ? offers : []; + // A bait-price listing is never the best offer (DOS/Rate#797). + const valid = plausibleOffers(safeOffers).filter((o) => { + if (!o || typeof o !== 'object') return false; + if (typeof o.price_current === 'boolean' || typeof o.stock === 'boolean') { + return false; + } + const price = Number(o.price_current); + const stock = o.stock == null ? 1 : Number(o.stock); + return ( + Number.isFinite(stock) && + stock > 0 && + Number.isFinite(price) && + price > 0 && + price <= MAX_VALID_VND_PRICE + ); + }); + const platforms = Array.from( + new Set( + valid + .map((o) => + String(o.merchant?.platform || o.platform || 'shopee') + .replace(/\p{Cc}+/gu, '') + .trim() + .toLowerCase() + ) + .filter(Boolean) + ) + ); + if (valid.length === 0) { + return { bestOffer: null, offerPlatforms: platforms }; + } + + const sorted = [...valid].sort((a, b) => { + const diff = Number(a.price_current) - Number(b.price_current); + if (diff !== 0) return diff; + const mallA = a.merchant?.is_official ? 1 : 0; + const mallB = b.merchant?.is_official ? 1 : 0; + return mallB - mallA; + }); + + const best = sorted[0]!; + const bestPrice = Number(best.price_current); + const validOrig = normalizeOfferOriginalPrice( + bestPrice, + typeof best.price_original === 'boolean' + ? 0 + : Number(best.price_original ?? 0) + ); + const cleanTitle = String(best.title || '') + .replace(/\p{Cc}+/gu, ' ') + .trim() + .slice(0, 320); + const cleanMerchantName = String(best.merchant?.name || 'Official Store') + .replace(/\p{Cc}+/gu, ' ') + .trim() + .slice(0, 120); + return { + bestOffer: { + id: String(best.id || '').trim(), + title: cleanTitle, + price: bestPrice, + original_price: validOrig ?? bestPrice, + url: typeof best.url === 'string' ? best.url.trim().slice(0, 2048) : '', + platform: String(best.merchant?.platform || best.platform || 'shopee') + .replace(/\p{Cc}+/gu, '') + .trim() + .toLowerCase(), + merchant_name: cleanMerchantName || 'Official Store', + is_official_mall: Boolean(best.merchant?.is_official), + }, + offerPlatforms: platforms, + }; +} + +export function isSafeCrowdIngestHttpUrl(rawUrl: string): boolean { + if (typeof rawUrl !== 'string') return false; + const trimmed = rawUrl.trim(); + if ( + !trimmed || + trimmed.length > 1024 || + /\s/.test(trimmed) || + /\p{Cc}/u.test(trimmed) + ) { + return false; + } + try { + const parsed = new URL(trimmed); + if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { + return false; + } + if (parsed.username || parsed.password) return false; + if (!parsed.hostname) return false; + return true; + } catch { + return false; + } +} + +export function resolveInspectObservedPrices(input: { + queryPrice?: number | null; + queryOriginalPrice?: number | null; + matchedOfferPriceCurrent?: unknown; + matchedOfferPriceOriginal?: unknown; + masterMinPrice?: unknown; + maxVnd?: number; +}): { observedPrice: number; observedOriginalPrice: number } { + const safeMaxVnd = + typeof input.maxVnd === 'number' && + Number.isFinite(input.maxVnd) && + input.maxVnd > 0 + ? Math.round(input.maxVnd) + : MAX_VALID_VND_PRICE; + const toPositiveBoundedPrice = (val: unknown): number => { + if ( + val === null || + val === undefined || + typeof val === 'boolean' || + (typeof val === 'string' && !val.trim()) + ) { + return 0; + } + const n = Number(val); + return Number.isFinite(n) && n > 0 && n <= safeMaxVnd ? Math.round(n) : 0; + }; + + const observedPrice = + toPositiveBoundedPrice(input.queryPrice) || + toPositiveBoundedPrice(input.matchedOfferPriceCurrent) || + toPositiveBoundedPrice(input.masterMinPrice) || + 0; + + const rawOrigCandidate = + input.queryOriginalPrice ?? + (input.queryPrice === null || input.queryPrice === undefined + ? input.matchedOfferPriceOriginal + : null); + const candidateOrig = toPositiveBoundedPrice(rawOrigCandidate); + const observedOriginalPrice = + observedPrice > 0 && candidateOrig >= observedPrice ? candidateOrig : 0; + + return { observedPrice, observedOriginalPrice }; +} + const CANONICAL_PRODUCT_SLUG_REGEX = /^[a-zA-Z0-9._-]+$/; /** @@ -245,9 +474,19 @@ export class BoundedSlidingWindowRateLimiter { private readonly maxBuckets: number; constructor(options: SlidingWindowRateLimiterOptions) { - this.windowMs = Math.max(1_000, options.windowMs); - this.maxRequests = Math.max(1, options.maxRequests); - this.maxBuckets = Math.max(1, options.maxBuckets ?? 2_000); + this.windowMs = + Number.isFinite(options.windowMs) && options.windowMs >= 1_000 + ? Math.min(86_400_000, Math.floor(options.windowMs)) + : 60_000; + this.maxRequests = + Number.isFinite(options.maxRequests) && options.maxRequests >= 1 + ? Math.min(100_000, Math.floor(options.maxRequests)) + : 30; + const rawMaxBuckets = options.maxBuckets ?? 2_000; + this.maxBuckets = + Number.isFinite(rawMaxBuckets) && rawMaxBuckets >= 1 + ? Math.min(50_000, Math.floor(rawMaxBuckets)) + : 2_000; } public get size(): number { @@ -297,15 +536,23 @@ export class BoundedSlidingWindowRateLimiter { } export function extractClientRateLimitKey( - headers: Pick + headers: Pick | null | undefined ): string { + if (!headers || typeof headers.get !== 'function') return 'anon'; + const sanitizeIpCandidate = (raw: string | null | undefined): string => { + if (typeof raw !== 'string') return ''; + return raw + .replace(/\p{Cc}+/gu, '') + .replace(/\s+/g, '') + .slice(0, 128); + }; const forwardedFor = headers.get('x-forwarded-for'); if (forwardedFor) { - const firstIp = forwardedFor.split(',')[0]?.trim(); - if (firstIp) return firstIp.slice(0, 128); + const firstIp = sanitizeIpCandidate(forwardedFor.split(',')[0]); + if (firstIp) return firstIp; } - const realIp = headers.get('x-real-ip')?.trim(); - if (realIp) return realIp.slice(0, 128); + const realIp = sanitizeIpCandidate(headers.get('x-real-ip')); + if (realIp) return realIp; return 'anon'; } diff --git a/apps/web/src/server/rate/cache-tags.ts b/apps/web/src/server/rate/cache-tags.ts index bd0295ecd..2719d709a 100644 --- a/apps/web/src/server/rate/cache-tags.ts +++ b/apps/web/src/server/rate/cache-tags.ts @@ -40,24 +40,51 @@ export interface RevalidationTargetsInput { paths?: readonly string[]; } -export function canonicalizeCacheParams(value: unknown): string { +const SAFE_CACHE_TAG_REGEX = /^rate:[a-z0-9:._-]{1,150}$/i; +const SAFE_ENTITY_TOKEN_REGEX = /^[a-z0-9._-]+$/i; +const SAFE_MERCHANT_IDENTIFIER_REGEX = /^[a-z0-9:._-]+$/i; + +export function canonicalizeCacheParams( + value: unknown, + depth = 0, + seen: WeakSet = new WeakSet() +): string { if (value === null || value === undefined) { - return '{}'; + return depth === 0 ? '{}' : 'null'; + } + if (typeof value === 'number') { + return Number.isFinite(value) ? JSON.stringify(value) : 'null'; + } + if (typeof value !== 'object') { + return JSON.stringify(value); } - if (Array.isArray(value)) { - return `[${value.map((item) => canonicalizeCacheParams(item)).join(',')}]`; + if (seen.has(value)) { + return '"[Circular]"'; } - if (typeof value === 'object') { + if (depth > 6) { + return '"[MaxDepth]"'; + } + seen.add(value); + try { + if (Array.isArray(value)) { + return `[${value + .slice(0, 100) + .map((item) => canonicalizeCacheParams(item, depth + 1, seen)) + .join(',')}]`; + } const entries = Object.entries(value as Record) .filter(([, val]) => val !== undefined) - .sort(([keyA], [keyB]) => keyA.localeCompare(keyB)); + .sort(([keyA], [keyB]) => keyA.localeCompare(keyB)) + .slice(0, 60); return `{${entries .map( - ([key, val]) => `${JSON.stringify(key)}:${canonicalizeCacheParams(val)}` + ([key, val]) => + `${JSON.stringify(key)}:${canonicalizeCacheParams(val, depth + 1, seen)}` ) .join(',')}}`; + } finally { + seen.delete(value); } - return JSON.stringify(value); } export function isSafeRevalidationPath(rawPath: string): boolean { @@ -80,13 +107,19 @@ export function resolveRevalidationTargets(input: RevalidationTargetsInput): { const tagSet = new Set(); const pathSet = new Set(); - for (const tag of input.tags ?? []) { - const clean = tag.trim(); - if (clean) tagSet.add(clean); + for (const tag of (input.tags ?? []).slice(0, 50)) { + if (typeof tag !== 'string') continue; + const clean = tag.trim().toLowerCase(); + if (clean && SAFE_CACHE_TAG_REGEX.test(clean)) { + tagSet.add(clean); + } } - for (const slug of input.slugs ?? []) { + for (const slug of (input.slugs ?? []).slice(0, 50)) { + if (typeof slug !== 'string') continue; const clean = slug.trim(); - if (!clean) continue; + if (!clean || clean.length > 120 || !SAFE_ENTITY_TOKEN_REGEX.test(clean)) { + continue; + } tagSet.add(productCacheTag(clean)); tagSet.add(RATE_CACHE_TAGS.PRODUCTS); tagSet.add(RATE_CACHE_TAGS.OFFERS); @@ -95,18 +128,28 @@ export function resolveRevalidationTargets(input: RevalidationTargetsInput): { tagSet.add(RATE_CACHE_TAGS.DEALS); tagSet.add(RATE_CACHE_TAGS.STATS); } - for (const cat of input.categories ?? []) { + for (const cat of (input.categories ?? []).slice(0, 20)) { + if (typeof cat !== 'string') continue; const clean = cat.trim(); - if (!clean) continue; + if (!clean || clean.length > 60 || !SAFE_ENTITY_TOKEN_REGEX.test(clean)) { + continue; + } tagSet.add(categoryCacheTag(clean)); tagSet.add(RATE_CACHE_TAGS.CATEGORIES); tagSet.add(RATE_CACHE_TAGS.PRODUCTS); tagSet.add(RATE_CACHE_TAGS.DEALS); tagSet.add(RATE_CACHE_TAGS.STATS); } - for (const merchantId of input.merchants ?? []) { + for (const merchantId of (input.merchants ?? []).slice(0, 50)) { + if (typeof merchantId !== 'string') continue; const clean = merchantId.trim(); - if (!clean) continue; + if ( + !clean || + clean.length > 120 || + !SAFE_MERCHANT_IDENTIFIER_REGEX.test(clean) + ) { + continue; + } tagSet.add(merchantCacheTag(clean)); tagSet.add(RATE_CACHE_TAGS.MERCHANTS); tagSet.add(RATE_CACHE_TAGS.OFFERS); @@ -114,7 +157,8 @@ export function resolveRevalidationTargets(input: RevalidationTargetsInput): { tagSet.add(RATE_CACHE_TAGS.DEALS); tagSet.add(RATE_CACHE_TAGS.STATS); } - for (const p of input.paths ?? []) { + for (const p of (input.paths ?? []).slice(0, 50)) { + if (typeof p !== 'string') continue; const clean = p.trim(); if (isSafeRevalidationPath(clean)) { pathSet.add(clean); diff --git a/apps/web/src/server/rate/deals.ts b/apps/web/src/server/rate/deals.ts index a25ad98c1..162ed438b 100644 --- a/apps/web/src/server/rate/deals.ts +++ b/apps/web/src/server/rate/deals.ts @@ -5,6 +5,7 @@ import { getMasterProductCategorySlug, getMasterProductOffersBatch, getMasterProducts, + isCanonicalCategorySlug, normalizeOfferOriginalPrice, resolveProductImageUrl, type CanonicalCategorySlug, @@ -129,6 +130,49 @@ function resolveEffectiveMerchantRisk( * 3. `priceSpreadLeaders`: Products with the largest absolute price spread (`max_price - min_price`) * across 10 retailers so buyers can avoid overpriced listings and buy at the lowest verified retailer. */ +const SAFE_PLATFORM_FILTER_REGEX = /^[a-z0-9_-]{1,40}$/; +export const MAX_DEAL_RADAR_CLUSTERS = 500; + +export function normalizeDealRadarOptions(options?: { + categorySlug?: string | null; + platform?: string | null; + limit?: number; +}): { + categoryFilter: CanonicalCategorySlug | 'all'; + platformFilter: string; + limit: number; +} { + const rawCat = + typeof options?.categorySlug === 'string' + ? options.categorySlug + .replace(/\p{Cc}+/gu, '') + .trim() + .toLowerCase() + : ''; + const categoryFilter: CanonicalCategorySlug | 'all' = + rawCat && isCanonicalCategorySlug(rawCat) ? rawCat : 'all'; + + const rawPlat = + typeof options?.platform === 'string' + ? options.platform + .replace(/\p{Cc}+/gu, '') + .trim() + .toLowerCase() + : ''; + const platformFilter = + rawPlat && rawPlat !== 'all' && SAFE_PLATFORM_FILTER_REGEX.test(rawPlat) + ? rawPlat + : 'all'; + + const rawLimit = Number(options?.limit); + const limit = + Number.isFinite(rawLimit) && rawLimit >= 1 + ? Math.min(Math.floor(rawLimit), 50) + : 12; + + return { categoryFilter, platformFilter, limit }; +} + export function computeCrossPlatformDealRadarFromData( clusters: Array<{ master: MasterProduct; @@ -140,14 +184,8 @@ export function computeCrossPlatformDealRadarFromData( limit?: number; } ): CrossPlatformDealRadarResponse { - const categoryFilter = options?.categorySlug || 'all'; - const platformFilter = - (options?.platform || 'all').trim().toLowerCase() || 'all'; - const rawLimit = Number(options?.limit); - const limit = - Number.isFinite(rawLimit) && rawLimit >= 1 - ? Math.min(Math.floor(rawLimit), 50) - : 12; + const { categoryFilter, platformFilter, limit } = + normalizeDealRadarOptions(options); const genuineDeals: CrossPlatformDealRadarItem[] = []; const fakeSaleTraps: CrossPlatformDealRadarItem[] = []; @@ -155,16 +193,24 @@ export function computeCrossPlatformDealRadarFromData( let totalClustersAnalyzed = 0; let totalOffersAnalyzed = 0; - for (const { master, offers } of clusters) { + const boundedClusters = Array.isArray(clusters) + ? clusters.slice(0, MAX_DEAL_RADAR_CLUSTERS) + : []; + + for (const { master, offers } of boundedClusters) { const categorySlug = getMasterProductCategorySlug(master); if (categoryFilter !== 'all' && categorySlug !== categoryFilter) { continue; } + // Every offer counts: offers arrive sorted by price, so a slice would keep + // only the cheapest and understate the spread (DOS/Rate#867, #333). + const clusterOffers = Array.isArray(offers) ? offers : []; + // Bait-price listings (far below every other store) set no market floor, // spread, deal or trap (DOS/Rate#797). const validOffers = plausibleOffers( - offers.filter( + clusterOffers.filter( (o) => Number.isFinite(o.price_current) && o.price_current > 0 && @@ -472,13 +518,7 @@ export function getCrossPlatformDealRadar(options?: { platform?: string; limit?: number; }): Promise { - const categorySlug = options?.categorySlug || 'all'; - const platformFilter = - (options?.platform || 'all').trim().toLowerCase() || 'all'; - const rawLimit = Number(options?.limit); - const limit = - Number.isFinite(rawLimit) && rawLimit >= 1 - ? Math.min(Math.floor(rawLimit), 50) - : 12; - return _getCrossPlatformDealRadarByKey(categorySlug, platformFilter, limit); + const { categoryFilter, platformFilter, limit } = + normalizeDealRadarOptions(options); + return _getCrossPlatformDealRadarByKey(categoryFilter, platformFilter, limit); } diff --git a/apps/web/src/server/rate/extension.ts b/apps/web/src/server/rate/extension.ts index d83e8f40b..853c84719 100644 --- a/apps/web/src/server/rate/extension.ts +++ b/apps/web/src/server/rate/extension.ts @@ -278,19 +278,34 @@ function isDomainOrSubdomain(host: string, domain: string): boolean { return host === domain || host.endsWith(`.${domain}`); } +const MAX_EXTENSION_URL_LENGTH = 2048; + +function sanitizeExtensionField(raw: unknown, maxLen: number): string { + if (typeof raw !== 'string') return ''; + return raw + .replace(/\p{Cc}+/gu, ' ') + .replace(/\s+/g, ' ') + .trim() + .slice(0, maxLen) + .trim(); +} + /** * Strip all tracking & session parameters (sp_atk, xptdk, utm_*, fbclid, etc.) * to comply with Chrome Web Store User Data Privacy policies. */ export function canonicalizeProductUrl(rawUrl?: string | null): string | null { if (!rawUrl || typeof rawUrl !== 'string') return null; - const trimmed = rawUrl.trim(); - if (!trimmed) return null; + const trimmed = rawUrl.replace(/\p{Cc}+/gu, '').trim(); + if (!trimmed || trimmed.length > MAX_EXTENSION_URL_LENGTH) return null; try { const parsed = new URL(trimmed); if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { return null; } + if (parsed.username !== '' || parsed.password !== '') { + return null; + } return `${parsed.origin}${parsed.pathname}`; } catch { return null; @@ -321,12 +336,17 @@ export function detectPlatformFromInput( return norm; } if (!rawUrl || typeof rawUrl !== 'string') return null; + const trimmedUrl = rawUrl.replace(/\p{Cc}+/gu, '').trim(); + if (!trimmedUrl || trimmedUrl.length > MAX_EXTENSION_URL_LENGTH) return null; let host: string; try { - const parsed = new URL(rawUrl.trim()); + const parsed = new URL(trimmedUrl); if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { return null; } + if (parsed.username !== '' || parsed.password !== '') { + return null; + } host = parsed.hostname.toLowerCase(); } catch { return null; @@ -1321,9 +1341,20 @@ export async function ingestExtensionCrowdSignal( rpcClient = getRateClient(); } - const normalizedItemId = input.externalShopId - ? `${input.externalShopId}_${input.externalItemId}` - : input.externalItemId; + const cleanItemId = sanitizeExtensionField(input.externalItemId, 128); + const cleanShopId = sanitizeExtensionField(input.externalShopId, 128); + if (!cleanItemId) { + return { + accepted: false, + quarantinedForWorker: false, + jobState: 'rejected_invalid_url', + jobId: null, + reason: 'Mã định danh sản phẩm không hợp lệ.', + }; + } + const normalizedItemId = cleanShopId + ? `${cleanShopId}_${cleanItemId}` + : cleanItemId; const validOriginalPrice = typeof input.originalPrice === 'number' && Number.isFinite(input.originalPrice) && @@ -1346,16 +1377,20 @@ export async function ingestExtensionCrowdSignal( ? Math.round(input.ratingCount) : null; + const cleanObservedTitle = sanitizeExtensionField(input.title, 280); + const cleanSelectedVariant = + sanitizeExtensionField(input.selectedVariant, 120) || null; + const { data, error } = await rpcClient.rpc('enqueue_crowd_verify_signal', { p_platform: input.platform, p_target_url: canonicalUrl, p_platform_item_id: normalizedItemId, - p_platform_shop_id: input.externalShopId || undefined, + p_platform_shop_id: cleanShopId || undefined, p_payload: { source: 'extension_mv3_quarantine', canonical_url: canonicalUrl, - observed_title: input.title.slice(0, 280), - selected_variant: input.selectedVariant?.slice(0, 120) || null, + observed_title: cleanObservedTitle, + selected_variant: cleanSelectedVariant, observed_price: safeObservedPrice, observed_original_price: validOriginalPrice, observed_rating_star: validRatingStar, diff --git a/apps/web/src/server/rate/index.ts b/apps/web/src/server/rate/index.ts index 021f5811f..65e31807a 100644 --- a/apps/web/src/server/rate/index.ts +++ b/apps/web/src/server/rate/index.ts @@ -138,30 +138,101 @@ const _getCachedMasterProductsByKey = cache( )(serializedParams) ); +const SAFE_PRODUCT_SLUG_REGEX = /^[a-z0-9._-]{1,120}$/; + +export function sanitizeCachedEntityId(value: unknown, maxLen = 128): string { + if (typeof value !== 'string') return ''; + return value + .replace(/\p{Cc}+/gu, '') + .trim() + .slice(0, maxLen); +} + +export function sanitizeCachedProductSlug(value: unknown): string { + if (typeof value !== 'string') return ''; + const cleaned = value + .replace(/\p{Cc}+/gu, '') + .trim() + .toLowerCase() + .slice(0, 120); + return SAFE_PRODUCT_SLUG_REGEX.test(cleaned) ? cleaned : ''; +} + +export function sanitizeCachedCompareSlugsCsv(slugsCsv: unknown): string { + if (typeof slugsCsv !== 'string') return ''; + return Array.from( + new Set( + slugsCsv + .split(',') + .map((s) => sanitizeCachedProductSlug(s)) + .filter(Boolean) + ) + ) + .slice(0, 4) + .join(','); +} + +export function sanitizeCachedInspectQuery(rawQuery: unknown): string { + if (typeof rawQuery !== 'string') return ''; + return rawQuery + .replace(/\p{Cc}+/gu, ' ') + .replace(/\s+/g, ' ') + .trim() + .slice(0, 2048); +} + +export function normalizeCachedFacetCategory( + categorySlug?: string | null +): import('../../lib/rate-api').CanonicalCategorySlug | 'all' { + if (categorySlug === undefined || categorySlug === null) { + return 'smartphones'; + } + const cleaned = + typeof categorySlug === 'string' + ? categorySlug + .replace(/\p{Cc}+/gu, '') + .trim() + .toLowerCase() + : ''; + if (!cleaned) return 'smartphones'; + return isCanonicalCategorySlug(cleaned) ? cleaned : 'all'; +} + export function getCachedMasterProducts(params?: MasterProductFilterParams) { - const cleanCat = params?.categorySlug?.trim().toLowerCase() || 'all'; + const rawCat = + typeof params?.categorySlug === 'string' + ? params.categorySlug + .replace(/\p{Cc}+/gu, '') + .trim() + .toLowerCase() + : ''; + const cleanCat = rawCat && isCanonicalCategorySlug(rawCat) ? rawCat : 'all'; return _getCachedMasterProductsByKey( canonicalizeCacheParams(params ?? {}), cleanCat ); } -export const getCachedMasterProductBySlug = cache((slug: string) => { - const cleanSlug = slug.trim().toLowerCase(); - return withRateDataCache( +const _getCachedMasterProductBySlugKey = cache((cleanSlug: string) => + withRateDataCache( _getMasterProductBySlug, ['rate-master-product-by-slug', cleanSlug], { revalidate: 300, tags: [RATE_CACHE_TAGS.PRODUCTS, productCacheTag(cleanSlug)], } - )(cleanSlug); -}); + )(cleanSlug) +); -export const getCachedCategoryDirectoryFacets = cache( - (categorySlug?: string) => { - const cleanCat = categorySlug?.trim().toLowerCase() || 'all'; - return withRateDataCache( +export function getCachedMasterProductBySlug(slug: string) { + const cleanSlug = sanitizeCachedProductSlug(slug); + if (!cleanSlug) return Promise.resolve(null); + return _getCachedMasterProductBySlugKey(cleanSlug); +} + +const _getCachedCategoryDirectoryFacetsByKey = cache( + (cleanCat: import('../../lib/rate-api').CanonicalCategorySlug | 'all') => + withRateDataCache( _getCategoryDirectoryFacets, ['rate-category-facets', cleanCat], { @@ -172,47 +243,61 @@ export const getCachedCategoryDirectoryFacets = cache( ...(cleanCat !== 'all' ? [categoryCacheTag(cleanCat)] : []), ], } - )(categorySlug); - } + )(cleanCat) ); -export const getCachedMasterProductOffers = cache((masterProductId: string) => { - const cleanId = masterProductId.trim(); - return withRateDataCache( +export function getCachedCategoryDirectoryFacets(categorySlug?: string) { + return _getCachedCategoryDirectoryFacetsByKey( + normalizeCachedFacetCategory(categorySlug) + ); +} + +const _getCachedMasterProductOffersByKey = cache((cleanId: string) => + withRateDataCache( _getMasterProductOffers, ['rate-master-product-offers', cleanId], { revalidate: 180, tags: [RATE_CACHE_TAGS.OFFERS, RATE_CACHE_TAGS.DEALS], } - )(cleanId); -}); + )(cleanId) +); -export const getCachedMasterProductReviews = cache( - (masterProductId: string) => { - const cleanId = masterProductId.trim(); - return withRateDataCache( - _getMasterProductReviews, - ['rate-master-product-reviews', cleanId], - { - revalidate: 300, - tags: [RATE_CACHE_TAGS.REVIEWS], - } - )(cleanId); - } +export function getCachedMasterProductOffers(masterProductId: string) { + const cleanId = sanitizeCachedEntityId(masterProductId); + if (!cleanId) return Promise.resolve([]); + return _getCachedMasterProductOffersByKey(cleanId); +} + +const _getCachedMasterProductReviewsByKey = cache((cleanId: string) => + withRateDataCache( + _getMasterProductReviews, + ['rate-master-product-reviews', cleanId], + { + revalidate: 300, + tags: [RATE_CACHE_TAGS.REVIEWS], + } + )(cleanId) ); +export function getCachedMasterProductReviews(masterProductId: string) { + const cleanId = sanitizeCachedEntityId(masterProductId); + if (!cleanId) return Promise.resolve([]); + return _getCachedMasterProductReviewsByKey(cleanId); +} + export const getCachedMasterProductReviewSummary = cache( ( masterProductId: string, existingReviews?: import('../../lib/rate-api').ReviewItem[] ) => { - const cleanId = masterProductId.trim(); + const cleanId = sanitizeCachedEntityId(masterProductId); const reviewsKey = existingReviews ? `reviews:${existingReviews.length}:${existingReviews .slice(0, 20) .map( - (r) => `${r.id}:${r.verdict ?? ''}:${r.include_in_rating ? 1 : 0}` + (r) => + `${sanitizeCachedEntityId(r.id)}:${sanitizeCachedEntityId(r.verdict ?? '', 32)}:${r.include_in_rating ? 1 : 0}` ) .join('|')}` : 'reviews:auto'; @@ -236,11 +321,14 @@ export const getCachedMasterProductPriceHistory = cache( masterProductId: string, existingOffers?: import('../../lib/rate-api').MerchantOffer[] ) => { - const cleanId = masterProductId.trim(); + const cleanId = sanitizeCachedEntityId(masterProductId); const offersKey = existingOffers ? `offers:${existingOffers.length}:${existingOffers .slice(0, 20) - .map((o) => `${o.id}:${o.price_current}:${o.price_original ?? ''}`) + .map( + (o) => + `${sanitizeCachedEntityId(o.id)}:${o.price_current}:${o.price_original ?? ''}` + ) .join('|')}` : 'offers:auto'; return withRateDataCache( @@ -277,21 +365,13 @@ const _getCachedCompareMasterProductsByKey = cache( ); export function getCachedCompareMasterProducts(slugsCsv: string) { - const canonicalSlugsCsv = Array.from( - new Set( - (slugsCsv || '') - .split(',') - .map((s) => s.trim().toLowerCase()) - .filter(Boolean) - ) - ) - .slice(0, 4) - .join(','); + const canonicalSlugsCsv = sanitizeCachedCompareSlugsCsv(slugsCsv); return _getCachedCompareMasterProductsByKey(canonicalSlugsCsv); } async function inspectProductOrShopUrlWithRadar(rawQuery: string) { - const res = await _inspectProductOrShopUrl(rawQuery); + const safeQuery = sanitizeCachedInspectQuery(rawQuery); + const res = await _inspectProductOrShopUrl(safeQuery); return { ...res, radarSignal: res.masterProduct @@ -351,17 +431,22 @@ export async function getCachedMerchantsList( return all; } -export const getCachedMerchantProfileById = cache((merchantId: string) => { - const cleanId = merchantId.trim(); - return withRateDataCache( +const _getCachedMerchantProfileByIdKey = cache((cleanId: string) => + withRateDataCache( _getMerchantProfileById, ['rate-merchant-profile', cleanId], { revalidate: 600, tags: [RATE_CACHE_TAGS.MERCHANTS, merchantCacheTag(cleanId)], } - )(cleanId); -}); + )(cleanId) +); + +export function getCachedMerchantProfileById(merchantId: string) { + const cleanId = sanitizeCachedEntityId(merchantId); + if (!cleanId) return Promise.resolve(null); + return _getCachedMerchantProfileByIdKey(cleanId); +} export const getCachedRatePlatformStats = cache(() => withRateDataCache(_getRatePlatformStats, ['rate-platform-stats'], { @@ -380,10 +465,16 @@ export const getCachedInspectProductOrShopUrl = cache( // JSON, so the category map travels as entries. const SITEMAP_CACHE_WARN_BYTES = 1_500_000; -const categorySlugOnly = (value: Record | null | undefined) => - typeof value?.category_slug === 'string' - ? { category_slug: value.category_slug } - : null; +const categorySlugOnly = ( + value: Record | null | undefined +) => { + if (typeof value?.category_slug !== 'string') return null; + const slug = value.category_slug + .replace(/\p{Cc}+/gu, '') + .trim() + .slice(0, 64); + return slug ? { category_slug: slug } : null; +}; const _getCachedSitemapRowsData = withRateDataCache( async () => { @@ -466,9 +557,11 @@ export { } from './openapi'; export { + MAX_DEAL_RADAR_CLUSTERS, computeCrossPlatformDealRadarFromData, computeProductRadarSignal, getCrossPlatformDealRadar, + normalizeDealRadarOptions, type CrossPlatformDealRadarItem, type CrossPlatformDealRadarResponse, type CrossPlatformPriceSpreadItem, diff --git a/apps/web/src/server/rate/openapi.ts b/apps/web/src/server/rate/openapi.ts index 6b4ddfa2f..8bfbb172b 100644 --- a/apps/web/src/server/rate/openapi.ts +++ b/apps/web/src/server/rate/openapi.ts @@ -76,13 +76,22 @@ export interface RateOpenApiDocument { export function normalizeOpenApiServerOrigin( rawBaseUrl?: string | null ): string { - const trimmed = (rawBaseUrl ?? '').trim(); - if (!trimmed || trimmed.length > 256) return 'https://rate.vn'; + if (typeof rawBaseUrl !== 'string') return 'https://rate.vn'; + const trimmed = rawBaseUrl.trim(); + if (!trimmed || trimmed.length > 256 || /[\s\p{Cc}]/u.test(trimmed)) { + return 'https://rate.vn'; + } try { const parsed = new URL(trimmed); if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { return 'https://rate.vn'; } + if (parsed.username || parsed.password) { + return 'https://rate.vn'; + } + if (!parsed.origin || parsed.origin === 'null') { + return 'https://rate.vn'; + } return parsed.origin; } catch { return 'https://rate.vn'; @@ -1470,7 +1479,7 @@ export function buildRateOpenApiSpec( operationId: 'getHealthStatus', summary: 'Liveness & readiness health check probe', description: - 'Returns service metadata (`status`, `service: "rate-web"`, `version`, ISO-8601 `timestamp`) and `checks.database`, the result of a small read with the app\'s own database client (3 s timeout, reused for up to 15 s per instance), with `Cache-Control: no-store, max-age=0` for load balancers and uptime monitors.', + 'Returns service metadata (`status`, `service: "rate-web"`, `version`, ISO-8601 `timestamp`, `commit`: the Git SHA the deployment was built from, or null outside Vercel) and `checks.database`, the result of a small read with the app\'s own database client (3 s timeout, reused for up to 15 s per instance), with `Cache-Control: no-store, max-age=0` for load balancers and uptime monitors.', tags: ['Platform Telemetry'], responses: { '200': { diff --git a/apps/web/src/server/rate/request-auth.ts b/apps/web/src/server/rate/request-auth.ts index a13c7438f..688cffbbc 100644 --- a/apps/web/src/server/rate/request-auth.ts +++ b/apps/web/src/server/rate/request-auth.ts @@ -44,7 +44,11 @@ export function readBearerToken(request: Request): string { const match = /^Bearer\s+(.+)$/i.exec(header); if (!match || !match[1]) return ''; const token = match[1].trim(); - if (!token || token.length > MAX_BEARER_TOKEN_LENGTH || /\s/.test(token)) { + if ( + !token || + token.length > MAX_BEARER_TOKEN_LENGTH || + /[\s\p{Cc}]/u.test(token) + ) { return ''; } return token; @@ -59,6 +63,7 @@ function readCookies(request: Request): CookieRecord[] { const AUTH_COOKIE_RE = /^(sb-[^.]+-auth-token)(?:\.\d+)?$/; const BASE64_PREFIX = 'base64-'; +const MAX_COOKIE_SESSION_LENGTH = 16_384; /** * Reads the access token from the @supabase/ssr session cookie @@ -74,16 +79,23 @@ export async function readCookieAccessToken( .find((k): k is string => Boolean(k)); if (!key) return null; const raw = await combineChunks(key, (name) => byName.get(name)); - if (!raw) return null; + if (!raw || raw.length > MAX_COOKIE_SESSION_LENGTH) return null; try { const json = raw.startsWith(BASE64_PREFIX) ? stringFromBase64URL(raw.slice(BASE64_PREFIX.length)) : raw; + if (!json || json.length > MAX_COOKIE_SESSION_LENGTH) return null; const session = JSON.parse(json) as { access_token?: unknown }; - return typeof session.access_token === 'string' && - session.access_token.length > 0 - ? session.access_token - : null; + if (typeof session.access_token !== 'string') return null; + const token = session.access_token.trim(); + if ( + !token || + token.length > MAX_BEARER_TOKEN_LENGTH || + /[\s\p{Cc}]/u.test(token) + ) { + return null; + } + return token; } catch { return null; } diff --git a/scripts/ops/board.mjs b/scripts/ops/board.mjs new file mode 100644 index 000000000..65c12fd9d --- /dev/null +++ b/scripts/ops/board.mjs @@ -0,0 +1,210 @@ +#!/usr/bin/env node +// Sets an issue's status and agent on GitHub Project #9 and in its labels, as +// AGENTS.md section 1 requires, in one command: +// +// node scripts/ops/board.mjs ... --status in-progress --agent Guard +// [--repo DOS/Rate] +// +// The status label replaces any other status:* label, the agent label +// (agent:) is created when missing, and an issue not yet on the board +// is added to it. An issue another agent has claimed is refused unless +// --take-over is passed (a hand-over or JOY's assignment). + +import { parseArgs } from 'node:util'; +import { DEFAULT_REPO, fail, gh, ghJson, isMain } from './lib.mjs'; + +export const PROJECT = { + owner: 'DOS', + number: 9, + id: 'PVT_kwDOA388Vc4Bm5Ml', + statusField: 'PVTSSF_lADOA388Vc4Bm5MlzhlgSt8', + agentField: 'PVTF_lADOA388Vc4Bm5MlzhlqBH8', +}; + +export const STATUSES = { + backlog: { option: 'f75ad846', label: null }, + ready: { option: 'e18bf179', label: 'status:ready' }, + 'in-progress': { option: '47fc9ee4', label: 'status:in-progress' }, + 'in-review': { option: 'aba860b9', label: 'status:in-review' }, + done: { option: '98236657', label: 'status:done' }, +}; + +export const agentLabel = (nick) => `agent:${nick.trim().toLowerCase()}`; + +/** + * Label edits that leave exactly one status label and one agent label. + * An issue claimed by another agent is never taken silently (AGENTS.md + * section 1): that needs `takeOver`, and its claim label is then removed. + */ +export function labelChanges(current, status, nick, { takeOver = false } = {}) { + const spec = STATUSES[status]; + if (!spec) throw new Error(`unknown status ${status}`); + const mine = agentLabel(nick); + const others = current.filter((l) => l.startsWith('agent:') && l !== mine); + if (others.length > 0 && !takeOver) { + throw new Error( + `claimed by ${others.join(', ')}: pass --take-over only when that agent handed it over or JOY assigned it` + ); + } + const want = [spec.label, mine].filter(Boolean); + return { + add: want.filter((l) => !current.includes(l)), + remove: current.filter( + (l) => + (l.startsWith('status:') && l !== spec.label) || + (l.startsWith('agent:') && l !== mine) + ), + }; +} + +const ISSUE_QUERY = `query($owner: String!, $name: String!, $n: Int!) { + repository(owner: $owner, name: $name) { + issue(number: $n) { + url + labels(first: 50) { nodes { name } } + projectItems(first: 20) { nodes { id project { number } } } + } + } +}`; + +function readIssue(repo, number) { + const [owner, name] = repo.split('/'); + const data = ghJson([ + 'api', + 'graphql', + '-f', + `query=${ISSUE_QUERY}`, + '-F', + `owner=${owner}`, + '-F', + `name=${name}`, + '-F', + `n=${number}`, + ]); + const issue = data.data.repository.issue; + if (!issue) throw new Error(`${repo}#${number} not found`); + return { + url: issue.url, + labels: issue.labels.nodes.map((l) => l.name), + itemId: + issue.projectItems.nodes.find((i) => i.project.number === PROJECT.number) + ?.id ?? null, + }; +} + +export function setBoard({ + repo = DEFAULT_REPO, + number, + status, + nick, + takeOver = false, +}) { + const spec = STATUSES[status]; + if (!spec) throw new Error(`unknown status ${status}`); + const issue = readIssue(repo, number); + + // Labels first: the required task-claim check reads them, so a failure + // later on the board leaves the claim intact. + const { add, remove } = labelChanges(issue.labels, status, nick, { + takeOver, + }); + const taken = remove.filter((l) => l.startsWith('agent:')); + if (taken.length > 0) { + console.log(`${repo}#${number}: taking over from ${taken.join(', ')}`); + } + if (add.includes(agentLabel(nick))) { + gh( + [ + 'label', + 'create', + agentLabel(nick), + '-R', + repo, + '--color', + 'D93F0B', + '--description', + `Claimed by Agent: ${nick}`, + ], + { allowFailure: true } // already exists; a real failure resurfaces below + ); + } + gh([ + 'issue', + 'edit', + String(number), + '-R', + repo, + '--add-assignee', + '@me', + ...add.flatMap((l) => ['--add-label', l]), + ...remove.flatMap((l) => ['--remove-label', l]), + ]); + + let itemId = issue.itemId; + if (!itemId) { + itemId = ghJson([ + 'project', + 'item-add', + String(PROJECT.number), + '--owner', + PROJECT.owner, + '--url', + issue.url, + '--format', + 'json', + ]).id; + } + const edit = (field, value) => + gh([ + 'project', + 'item-edit', + '--project-id', + PROJECT.id, + '--id', + itemId, + '--field-id', + field, + ...value, + ]); + edit(PROJECT.statusField, ['--single-select-option-id', spec.option]); + edit(PROJECT.agentField, ['--text', nick]); + return `${repo}#${number} -> ${status} (${nick})`; +} + +if (isMain(import.meta.url)) { + const { values, positionals } = parseArgs({ + allowPositionals: true, + options: { + repo: { type: 'string', default: DEFAULT_REPO }, + status: { type: 'string' }, + agent: { type: 'string' }, + 'take-over': { type: 'boolean', default: false }, + }, + }); + const numbers = positionals.map(Number); + if ( + numbers.length === 0 || + numbers.some((n) => !Number.isInteger(n) || n < 1) || + !STATUSES[values.status] || + !values.agent?.trim() + ) { + fail( + `usage: board.mjs ... --status ${Object.keys(STATUSES).join('|')} --agent [--take-over] [--repo owner/name]` + ); + } + for (const number of numbers) { + try { + console.log( + setBoard({ + repo: values.repo, + number, + status: values.status, + nick: values.agent.trim(), + takeOver: values['take-over'], + }) + ); + } catch (err) { + fail(`${values.repo}#${number}: ${err.message}`); + } + } +} diff --git a/scripts/ops/board.test.mjs b/scripts/ops/board.test.mjs new file mode 100644 index 000000000..6b6aac46a --- /dev/null +++ b/scripts/ops/board.test.mjs @@ -0,0 +1,80 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { STATUSES, agentLabel, labelChanges } from './board.mjs'; + +test('agent labels are lowercase agent:', () => { + assert.equal(agentLabel(' Guard '), 'agent:guard'); +}); + +test('moves an issue to a new status and claims it', () => { + assert.deepEqual( + labelChanges(['bug', 'status:ready'], 'in-progress', 'Guard'), + { + add: ['status:in-progress', 'agent:guard'], + remove: ['status:ready'], + } + ); +}); + +test('leaves exactly one status label', () => { + assert.deepEqual( + labelChanges( + ['status:in-progress', 'status:in-review', 'agent:guard'], + 'done', + 'Guard' + ), + { add: ['status:done'], remove: ['status:in-progress', 'status:in-review'] } + ); +}); + +test('never takes an issue another agent claimed without --take-over', () => { + assert.throws( + () => + labelChanges( + ['status:in-progress', 'agent:crawl'], + 'in-progress', + 'Guard' + ), + /claimed by agent:crawl: pass --take-over/ + ); +}); + +test('a deliberate hand-over replaces the previous agent label', () => { + assert.deepEqual( + labelChanges(['status:in-review', 'agent:crawl'], 'in-progress', 'Guard', { + takeOver: true, + }), + { + add: ['status:in-progress', 'agent:guard'], + remove: ['status:in-review', 'agent:crawl'], + } + ); +}); + +test('changes nothing when the labels already match', () => { + assert.deepEqual( + labelChanges(['status:in-review', 'agent:guard'], 'in-review', 'guard'), + { + add: [], + remove: [], + } + ); +}); + +test('backlog has no status label and removes any', () => { + assert.equal(STATUSES.backlog.label, null); + assert.deepEqual(labelChanges(['status:ready'], 'backlog', 'Guard'), { + add: ['agent:guard'], + remove: ['status:ready'], + }); +}); + +test('rejects an unknown status', () => { + assert.throws(() => labelChanges([], 'blocked', 'Guard'), /unknown status/); +}); + +test('every status has a Project #9 option id', () => { + for (const spec of Object.values(STATUSES)) + assert.match(spec.option, /^[0-9a-f]{8}$/); +}); diff --git a/scripts/ops/lib.mjs b/scripts/ops/lib.mjs new file mode 100644 index 000000000..297cd4b39 --- /dev/null +++ b/scripts/ops/lib.mjs @@ -0,0 +1,53 @@ +// Shared helpers for the agent release scripts in scripts/ops (DOS/Rate#865). +// They drive the GitHub CLI with argument arrays, never a shell string, so no +// value is ever interpreted by a shell. + +import { spawnSync } from 'node:child_process'; +import { pathToFileURL } from 'node:url'; + +export const DEFAULT_REPO = 'DOS/Rate'; + +export function gh(args, { allowFailure = false } = {}) { + const r = spawnSync('gh', args, { + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, + }); + if (r.error) throw r.error; + if (r.status !== 0 && !allowFailure) { + throw new Error( + `gh ${args[0]} ${args[1] ?? ''} failed: ${r.stderr.trim()}` + ); + } + return r; +} + +export function ghJson(args) { + const out = gh(args).stdout.trim(); + return out ? JSON.parse(out) : null; +} + +/** + * Every item of a paginated REST list: `jq` selects the items of one page, + * each printed as one JSON line. + */ +export function ghPaginated(path, jq) { + return gh(['api', '--paginate', path, '--jq', jq]) + .stdout.split('\n') + .filter((line) => line.trim()) + .map((line) => JSON.parse(line)); +} + +/** A path segment per branch-name part: `fix#12` must not become `fix`. */ +export const encodeRef = (ref) => + ref.split('/').map(encodeURIComponent).join('/'); + +export const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +export const isMain = (url) => + url === pathToFileURL(process.argv[1] ?? '').href; + +/** Prints the message and exits non-zero: the scripts' one failure path. */ +export function fail(message) { + console.error(message); + process.exit(1); +} diff --git a/scripts/ops/merge-when-green.mjs b/scripts/ops/merge-when-green.mjs new file mode 100644 index 000000000..b6d491432 --- /dev/null +++ b/scripts/ops/merge-when-green.mjs @@ -0,0 +1,312 @@ +#!/usr/bin/env node +// Merges a PR once its head commit is green, pinned to that head so a commit +// pushed meanwhile is never merged unchecked. Green means every check run and +// every commit status (Vercel posts statuses, not check runs) passed, and the +// checks the branch's rulesets require have reported. +// +// node scripts/ops/merge-when-green.mjs [--repo DOS/Rate] +// [--method squash|merge] [--require ""]... [--ignore ""]... +// +// PRs into main always use a merge commit (Docs/Resources/RELEASE.md); other +// PRs default to squash, and a squashed head branch is deleted afterwards. +// --ignore lets a known-broken, non-required check through; name it in the PR. +// Prints one line and exits 0 when merged, 1 when it cannot merge. + +import { parseArgs } from 'node:util'; +import { + DEFAULT_REPO, + encodeRef, + fail, + gh, + ghJson, + ghPaginated, + isMain, + sleep, +} from './lib.mjs'; + +/** Bot reviews that report late or never finish and gate nothing. */ +export const IGNORED_CHECKS = ['DOS Code Review', 'CodeRabbit']; +const PASSING = new Set(['success', 'skipped', 'neutral']); +const PROTECTED_BRANCHES = new Set(['main', 'dev', 'master']); +/** Polls a green but `blocked` PR gets before it is reported as blocked. */ +export const BLOCKED_POLLS = 6; +/** Consecutive GitHub API errors tolerated before giving up. */ +const MAX_API_ERRORS = 3; +/** Merge refusals retried after a fresh read of the head. */ +const MAX_MERGE_REFUSALS = 3; + +/** Required checks an --ignore would hide: that can never turn green. */ +export const ignoredRequired = (required, ignore) => + required.filter((name) => ignore.includes(name)); + +/** A re-run adds a check run with the same name: keep the newest of each. */ +export function latestRuns(checkRuns) { + const byName = new Map(); + for (const run of checkRuns) { + const seen = byName.get(run.name); + if (!seen || run.id > seen.id) byName.set(run.name, run); + } + return [...byName.values()]; +} + +/** + * `checkRuns` from the check-runs API, `statuses` from the combined status + * API (already the latest per context). + */ +export function evaluateChecks( + checkRuns, + statuses = [], + { ignore = IGNORED_CHECKS, required = [] } = {} +) { + const runs = latestRuns(checkRuns).filter((r) => !ignore.includes(r.name)); + const contexts = statuses.filter((s) => !ignore.includes(s.context)); + const failed = [ + ...runs + .filter((r) => r.status === 'completed' && !PASSING.has(r.conclusion)) + .map((r) => r.name), + ...contexts + .filter((s) => s.state === 'failure' || s.state === 'error') + .map((s) => s.context), + ]; + if (failed.length > 0) return { state: 'failed', failed }; + const reported = new Set([ + ...runs.map((r) => r.name), + ...contexts.map((s) => s.context), + ]); + const pending = [ + ...runs.filter((r) => r.status !== 'completed').map((r) => r.name), + ...contexts.filter((s) => s.state === 'pending').map((s) => s.context), + ...required.filter((name) => !reported.has(name)), + ]; + // No check run yet means CI has not started, not that it passed. + if (runs.length === 0 || pending.length > 0) { + return { state: 'pending', pending }; + } + return { state: 'passed' }; +} + +/** Check names the branch's rulesets require, from the rules API. */ +export function requiredChecksFromRules(rules) { + return [ + ...new Set( + (rules ?? []) + .filter((r) => r.type === 'required_status_checks') + .flatMap((r) => r.parameters?.required_status_checks ?? []) + .map((c) => c.context) + ), + ]; +} + +export function mergeMethodFor(baseRef, requested) { + if (baseRef === 'main') { + if (requested && requested !== 'merge') { + throw new Error( + `PRs into main merge with a merge commit, not ${requested} (RELEASE.md)` + ); + } + return 'merge'; + } + if (requested && !['squash', 'merge'].includes(requested)) { + throw new Error(`unknown merge method ${requested}`); + } + return requested ?? 'squash'; +} + +export function shouldDeleteBranch({ method, headRef, headRepo, baseRepo }) { + return ( + method === 'squash' && + headRepo === baseRepo && + !PROTECTED_BRANCHES.has(headRef) + ); +} + +/** + * What to do with the PR now: 'merge', 'wait', or a final 'done' / 'fail' + * with a one-line message. + */ +export function decide(pr, checks, blockedPolls) { + if (pr.merged) + return { action: 'done', message: `PR ${pr.number} already merged` }; + if (pr.state !== 'open') + return { action: 'fail', message: `PR ${pr.number} is ${pr.state}` }; + if (pr.mergeable_state === 'dirty') { + return { action: 'fail', message: `PR ${pr.number} has merge conflicts` }; + } + if (checks.state === 'failed') { + return { + action: 'fail', + message: `PR ${pr.number} failing at ${pr.head.sha.slice(0, 8)}: ${checks.failed.join(', ')}`, + }; + } + if (checks.state === 'pending') return { action: 'wait' }; + // GitHub recomputes mergeability a little after the last check finishes. + if (pr.mergeable_state === 'blocked' || pr.mergeable_state === 'unknown') { + return blockedPolls + 1 >= BLOCKED_POLLS + ? { + action: 'fail', + message: `PR ${pr.number} is blocked at ${pr.head.sha.slice(0, 8)} although its checks passed (a required review or check is missing)`, + } + : { action: 'wait', blocked: true }; + } + return { action: 'merge' }; +} + +function readHead(repo, number) { + const pr = ghJson(['api', `repos/${repo}/pulls/${number}`]); + if (pr.state !== 'open') return { pr, runs: [], statuses: [], required: [] }; + const runs = ghPaginated( + `repos/${repo}/commits/${pr.head.sha}/check-runs?per_page=100`, + '.check_runs[] | {id, name, status, conclusion}' + ); + const statuses = + ghJson(['api', `repos/${repo}/commits/${pr.head.sha}/status`])?.statuses ?? + []; + const rules = ghJson([ + 'api', + `repos/${repo}/rules/branches/${encodeRef(pr.base.ref)}`, + ]); + return { pr, runs, statuses, required: requiredChecksFromRules(rules) }; +} + +export async function mergeWhenGreen({ + repo = DEFAULT_REPO, + number, + method: requested, + required: extraRequired = [], + ignore = [], + intervalMs = 30_000, + timeoutMs = 60 * 60_000, +} = {}) { + const deadline = Date.now() + timeoutMs; + let blockedPolls = 0; + let apiErrors = 0; + let mergeRefusals = 0; + for (;;) { + let head; + try { + head = readHead(repo, number); + apiErrors = 0; + } catch (err) { + if (++apiErrors >= MAX_API_ERRORS) { + return { ok: false, message: `PR ${number}: ${err.message}` }; + } + await sleep(intervalMs); + continue; + } + const { pr } = head; + // Refuse a wrong method or an ignored required check before waiting, not + // after the checks pass; both are re-read each poll (a PR can be + // retargeted). + let method = null; + if (pr.state === 'open') { + try { + method = mergeMethodFor(pr.base.ref, requested); + } catch (err) { + return { ok: false, message: `PR ${number}: ${err.message}` }; + } + const conflict = ignoredRequired(head.required, ignore); + if (conflict.length > 0) { + return { + ok: false, + message: `PR ${number}: cannot --ignore required check(s) ${conflict.join(', ')}`, + }; + } + } + const checks = evaluateChecks(head.runs, head.statuses, { + ignore: [...IGNORED_CHECKS, ...ignore], + required: [...head.required, ...extraRequired], + }); + const next = decide(pr, checks, blockedPolls); + if (next.action === 'done') return { ok: true, message: next.message }; + if (next.action === 'fail') return { ok: false, message: next.message }; + if (next.action === 'merge') { + const merged = gh( + [ + 'api', + '-X', + 'PUT', + `repos/${repo}/pulls/${number}/merge`, + '-f', + `merge_method=${method}`, + '-f', + `sha=${pr.head.sha}`, + '--jq', + '.sha', + ], + { allowFailure: true } + ); + if (merged.status !== 0) { + // A check re-queued after the last read (for example by a PR edit) + // makes GitHub refuse briefly: read the head again before giving up. + if (++mergeRefusals < MAX_MERGE_REFUSALS) { + await sleep(intervalMs); + continue; + } + return { + ok: false, + message: `PR ${number} merge refused: ${merged.stderr.trim()}`, + }; + } + if ( + shouldDeleteBranch({ + method, + headRef: pr.head.ref, + headRepo: pr.head.repo?.full_name, + baseRepo: pr.base.repo.full_name, + }) + ) { + gh( + [ + 'api', + '-X', + 'DELETE', + `repos/${repo}/git/refs/heads/${encodeRef(pr.head.ref)}`, + ], + { allowFailure: true } + ); + } + return { + ok: true, + message: `PR ${number} MERGED ${merged.stdout.trim()}`, + }; + } + blockedPolls = next.blocked ? blockedPolls + 1 : 0; + if (Date.now() > deadline) { + const waiting = checks.pending?.length + ? ` (waiting on ${checks.pending.join(', ')})` + : ''; + return { + ok: false, + message: `PR ${number} still not green after ${timeoutMs / 60_000} min${waiting}`, + }; + } + await sleep(intervalMs); + } +} + +if (isMain(import.meta.url)) { + const { values, positionals } = parseArgs({ + allowPositionals: true, + options: { + repo: { type: 'string', default: DEFAULT_REPO }, + method: { type: 'string' }, + require: { type: 'string', multiple: true }, + ignore: { type: 'string', multiple: true }, + }, + }); + const number = Number(positionals[0]); + if (!Number.isInteger(number) || number < 1) { + fail( + 'usage: merge-when-green.mjs [--repo owner/name] [--method squash|merge] [--require ]... [--ignore ]...' + ); + } + const result = await mergeWhenGreen({ + repo: values.repo, + number, + method: values.method, + required: values.require ?? [], + ignore: values.ignore ?? [], + }); + if (!result.ok) fail(result.message); + console.log(result.message); +} diff --git a/scripts/ops/merge-when-green.test.mjs b/scripts/ops/merge-when-green.test.mjs new file mode 100644 index 000000000..1c8733696 --- /dev/null +++ b/scripts/ops/merge-when-green.test.mjs @@ -0,0 +1,212 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { + BLOCKED_POLLS, + decide, + evaluateChecks, + ignoredRequired, + latestRuns, + mergeMethodFor, + requiredChecksFromRules, + shouldDeleteBranch, +} from './merge-when-green.mjs'; +import { encodeRef } from './lib.mjs'; + +const run = (name, status, conclusion = null, id = 1) => ({ + id, + name, + status, + conclusion, +}); +const status = (context, state) => ({ context, state }); +const pr = (overrides = {}) => ({ + number: 12, + merged: false, + state: 'open', + mergeable_state: 'clean', + head: { sha: 'abcdef1234567890' }, + ...overrides, +}); +const green = [run('CI', 'completed', 'success')]; + +test('keeps only the newest run of a re-run check', () => { + const runs = latestRuns([ + run('CI', 'completed', 'failure', 1), + run('CI', 'completed', 'success', 2), + ]); + assert.deepEqual( + runs.map((r) => r.conclusion), + ['success'] + ); +}); + +test('passes when every check succeeded or was skipped', () => { + assert.deepEqual( + evaluateChecks([ + run('CI', 'completed', 'success'), + run('Lint', 'completed', 'skipped'), + ]), + { state: 'passed' } + ); +}); + +test('fails on a failed or cancelled check, even while others run', () => { + const r = evaluateChecks([ + run('CI', 'completed', 'cancelled'), + run('E2E', 'in_progress'), + ]); + assert.equal(r.state, 'failed'); + assert.deepEqual(r.failed, ['CI']); +}); + +test('waits while a check runs, before any check exists, or while a required one is missing', () => { + assert.equal(evaluateChecks([run('CI', 'queued')]).state, 'pending'); + assert.equal(evaluateChecks([]).state, 'pending'); + const r = evaluateChecks([run('Lint', 'completed', 'success')], [], { + required: ['Typecheck & Build Monorepo'], + }); + assert.equal(r.state, 'pending'); + assert.deepEqual(r.pending, ['Typecheck & Build Monorepo']); +}); + +test('a commit status counts like a check: Vercel failing fails, pending waits', () => { + const failing = evaluateChecks(green, [status('Vercel', 'failure')]); + assert.equal(failing.state, 'failed'); + assert.deepEqual(failing.failed, ['Vercel']); + assert.equal( + evaluateChecks(green, [status('Vercel', 'error')]).state, + 'failed' + ); + assert.equal( + evaluateChecks(green, [status('Vercel', 'pending')]).state, + 'pending' + ); + assert.equal( + evaluateChecks(green, [status('Vercel', 'success')]).state, + 'passed' + ); +}); + +test('a required check may report as a commit status', () => { + assert.equal( + evaluateChecks(green, [status('Vercel', 'success')], { + required: ['Vercel'], + }).state, + 'passed' + ); +}); + +test('ignores bot reviews that never finish, as checks or statuses', () => { + assert.equal( + evaluateChecks( + [...green, run('DOS Code Review', 'in_progress')], + [status('CodeRabbit', 'pending')] + ).state, + 'passed' + ); +}); + +test('--ignore lets a named broken check through, nothing else', () => { + const runs = [...green, run('E2E (Playwright)', 'completed', 'failure')]; + assert.equal(evaluateChecks(runs).state, 'failed'); + assert.equal( + evaluateChecks(runs, [], { ignore: ['E2E (Playwright)'] }).state, + 'passed' + ); +}); + +test('an --ignore that hides a required check is refused', () => { + assert.deepEqual( + ignoredRequired( + ['Typecheck & Build Monorepo', 'task-claim'], + ['E2E (Playwright)', 'task-claim'] + ), + ['task-claim'] + ); + assert.deepEqual(ignoredRequired(['task-claim'], ['E2E (Playwright)']), []); +}); + +test('reads the required check names from the branch rules', () => { + assert.deepEqual( + requiredChecksFromRules([ + { type: 'deletion' }, + { + type: 'required_status_checks', + parameters: { + required_status_checks: [ + { context: 'Typecheck & Build Monorepo' }, + { context: 'task-claim' }, + ], + }, + }, + { + type: 'required_status_checks', + parameters: { required_status_checks: [{ context: 'task-claim' }] }, + }, + ]), + ['Typecheck & Build Monorepo', 'task-claim'] + ); + assert.deepEqual(requiredChecksFromRules(null), []); +}); + +test('PRs into main always merge with a merge commit', () => { + assert.equal(mergeMethodFor('main'), 'merge'); + assert.equal(mergeMethodFor('main', 'merge'), 'merge'); + assert.throws(() => mergeMethodFor('main', 'squash'), /merge commit/); + assert.equal(mergeMethodFor('dev'), 'squash'); + assert.equal(mergeMethodFor('dev', 'merge'), 'merge'); + assert.throws(() => mergeMethodFor('dev', 'rebase'), /unknown merge method/); +}); + +test('deletes only a squashed branch of the same repository that is not dev or main', () => { + const base = { + method: 'squash', + headRef: 'fix/x', + headRepo: 'DOS/Rate', + baseRepo: 'DOS/Rate', + }; + assert.equal(shouldDeleteBranch(base), true); + assert.equal(shouldDeleteBranch({ ...base, method: 'merge' }), false); + assert.equal(shouldDeleteBranch({ ...base, headRef: 'dev' }), false); + assert.equal( + shouldDeleteBranch({ ...base, headRepo: 'someone/Rate' }), + false + ); + assert.equal(shouldDeleteBranch({ ...base, headRepo: undefined }), false); +}); + +test('branch names are encoded per path segment', () => { + assert.equal(encodeRef('fix#12'), 'fix%2312'); + assert.equal(encodeRef('feat/a b'), 'feat/a%20b'); +}); + +test('decides from the PR state and its checks', () => { + const passed = { state: 'passed' }; + assert.equal(decide(pr({ merged: true }), passed, 0).action, 'done'); + assert.equal(decide(pr({ state: 'closed' }), passed, 0).action, 'fail'); + assert.match( + decide(pr({ mergeable_state: 'dirty' }), passed, 0).message, + /conflicts/ + ); + assert.match( + decide(pr(), { state: 'failed', failed: ['CI'] }, 0).message, + /failing at abcdef12: CI/ + ); + assert.equal( + decide(pr(), { state: 'pending', pending: ['CI'] }, 0).action, + 'wait' + ); + assert.equal(decide(pr(), passed, 0).action, 'merge'); +}); + +test('gives a green but blocked PR a few polls before reporting it', () => { + const blocked = pr({ mergeable_state: 'blocked' }); + assert.deepEqual(decide(blocked, { state: 'passed' }, 0), { + action: 'wait', + blocked: true, + }); + const last = decide(blocked, { state: 'passed' }, BLOCKED_POLLS - 1); + assert.equal(last.action, 'fail'); + assert.match(last.message, /required review or check/); +}); diff --git a/scripts/ops/promote.mjs b/scripts/ops/promote.mjs new file mode 100644 index 000000000..1200d5b5b --- /dev/null +++ b/scripts/ops/promote.mjs @@ -0,0 +1,142 @@ +#!/usr/bin/env node +// Promotes dev to main as Docs/Resources/RELEASE.md describes: opens the +// promotion PR from dev itself (or reuses the open one), lists the PRs being +// released, and merges it with a merge commit once its checks pass. +// +// node scripts/ops/promote.mjs [--repo DOS/Rate] [--no-merge] +// +// Then check the release: node scripts/ops/verify-live.mjs --wait 900 + +import { parseArgs } from 'node:util'; +import { DEFAULT_REPO, fail, gh, ghJson, isMain } from './lib.mjs'; +import { mergeWhenGreen } from './merge-when-green.mjs'; + +export const PROMOTION_TITLE = 'chore(release): promote dev to main'; +/** The compare API lists at most this many commits. */ +const COMPARE_COMMIT_LIMIT = 250; + +/** + * The squashed PRs on dev ("title (#123)"), oldest first and once each, and + * how many single-parent commits carry no PR number. Merge commits are + * back-merges, earlier promotions or merge-commit PRs, not separate work. + */ +export function releasedPrs(commits) { + const seen = new Set(); + const prs = []; + let unnumbered = 0; + for (const c of commits) { + if (c.parents.length > 1) continue; + const subject = c.commit.message.split('\n')[0]; + const match = subject.match(/\(#(\d+)\)\s*$/); + if (!match) { + unnumbered += 1; + continue; + } + if (seen.has(match[1])) continue; + seen.add(match[1]); + prs.push({ + number: Number(match[1]), + title: subject.replace(/\s*\(#\d+\)\s*$/, ''), + }); + } + return { prs, unnumbered }; +} + +export function promotionBody({ prs, unnumbered }, { truncated = false } = {}) { + const lines = ['Promote dev to main.', '']; + lines.push(...prs.map((p) => `- #${p.number} ${p.title}`)); + if (unnumbered > 0) { + lines.push(`- ${unnumbered} other commit(s) without a PR number`); + } + if (truncated) { + lines.push( + `- more than ${COMPARE_COMMIT_LIMIT} commits: the list above is incomplete` + ); + } + lines.push( + '', + 'Merge with a merge commit, never squash (Docs/Resources/RELEASE.md).' + ); + return lines.join('\n'); +} + +/** The open promotion PR: head dev of this repository, never a fork's dev. */ +export const ownPromotionPr = (open) => + (open ?? []).find((p) => !p.isCrossRepository) ?? null; + +export async function promote({ repo = DEFAULT_REPO, merge = true } = {}) { + const compare = ghJson(['api', `repos/${repo}/compare/main...dev`]); + if (compare.ahead_by === 0) { + return { + ok: true, + message: 'dev has nothing that main lacks: no promotion needed', + }; + } + const body = promotionBody(releasedPrs(compare.commits), { + truncated: compare.ahead_by > compare.commits.length, + }); + + const existing = ownPromotionPr( + ghJson([ + 'pr', + 'list', + '--repo', + repo, + '--base', + 'main', + '--head', + 'dev', + '--state', + 'open', + '--json', + 'number,url,isCrossRepository,body', + ]) + ); + let number = existing?.number; + if (number) { + // dev may have moved since the PR was opened: list what it releases now. + // Only on a real change, since an edit re-runs the PR-base check. + if ((existing.body ?? '').trim() !== body.trim()) { + gh(['pr', 'edit', String(number), '--repo', repo, '--body', body]); + } + console.log(`reusing open promotion PR #${number}`); + } else { + const url = gh([ + 'pr', + 'create', + '--repo', + repo, + '--base', + 'main', + '--head', + 'dev', + '--title', + PROMOTION_TITLE, + '--body', + body, + ]).stdout.trim(); + number = Number(url.split('/').pop()); + console.log(`opened ${url} (${compare.ahead_by} commits)`); + } + if (!merge) + return { + ok: true, + message: `promotion PR #${number} left open (--no-merge)`, + }; + return mergeWhenGreen({ repo, number, method: 'merge' }); +} + +if (isMain(import.meta.url)) { + const { values } = parseArgs({ + options: { + repo: { type: 'string', default: DEFAULT_REPO }, + 'no-merge': { type: 'boolean', default: false }, + }, + }); + const result = await promote({ + repo: values.repo, + merge: !values['no-merge'], + }); + if (!result.ok) fail(result.message); + console.log(result.message); +} diff --git a/scripts/ops/promote.test.mjs b/scripts/ops/promote.test.mjs new file mode 100644 index 000000000..0d8918e6e --- /dev/null +++ b/scripts/ops/promote.test.mjs @@ -0,0 +1,60 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { ownPromotionPr, promotionBody, releasedPrs } from './promote.mjs'; + +const commit = (message, parents = 1) => ({ + commit: { message }, + parents: Array.from({ length: parents }, (_, i) => ({ sha: String(i) })), +}); + +test('lists the squashed PRs on dev, oldest first and once each', () => { + const released = releasedPrs([ + commit('fix(web): bait prices (#800)\n\nbody'), + commit('Merge pull request #806 from DOS/dev', 2), + commit('feat(web): health (#830)'), + commit('fix(web): bait prices (#800)'), + commit('chore: tidy without a number'), + ]); + assert.deepEqual(released, { + prs: [ + { number: 800, title: 'fix(web): bait prices' }, + { number: 830, title: 'feat(web): health' }, + ], + unnumbered: 1, + }); +}); + +test('the PR number is the trailing one; an issue number stays in the title', () => { + assert.deepEqual(releasedPrs([commit('fix(x): y (#834) (#835)')]).prs, [ + { number: 835, title: 'fix(x): y (#834)' }, + ]); + assert.deepEqual(releasedPrs([commit('docs: mention (#12) in the text')]), { + prs: [], + unnumbered: 1, + }); +}); + +test('the body lists the PRs, the rest and the merge rule', () => { + const body = promotionBody( + { prs: [{ number: 830, title: 'feat(web): health' }], unnumbered: 2 }, + { truncated: true } + ); + assert.match(body, /^Promote dev to main\./); + assert.match(body, /- #830 feat\(web\): health/); + assert.match(body, /2 other commit\(s\) without a PR number/); + assert.match(body, /more than 250 commits/); + assert.match(body, /never squash/); +}); + +test('a fork PR from its own dev is never taken as the promotion', () => { + assert.equal(ownPromotionPr([{ number: 9, isCrossRepository: true }]), null); + assert.deepEqual( + ownPromotionPr([ + { number: 9, isCrossRepository: true }, + { number: 10, isCrossRepository: false }, + ]), + { number: 10, isCrossRepository: false } + ); + assert.equal(ownPromotionPr([]), null); +}); diff --git a/scripts/ops/verify-live.mjs b/scripts/ops/verify-live.mjs new file mode 100644 index 000000000..ba3141a5a --- /dev/null +++ b/scripts/ops/verify-live.mjs @@ -0,0 +1,231 @@ +#!/usr/bin/env node +// Checks that rate.vn serves main: the live deployment's commit (reported by +// /api/health) equals main, the database is readable, and the main routes +// answer 200. Docs/Resources/RELEASE.md step 6. +// +// node scripts/ops/verify-live.mjs [--wait ] [--base-url https://rate.vn] +// [--repo DOS/Rate] [--route /path]... +// +// --wait keeps polling while the only problem is the commit (the build of main +// is still deploying); main is re-read on every poll, so a newer promotion is +// followed, never undone. When the live commit is behind main and main's +// production deployment is built, it prints that deployment as the one to +// promote: on 2026-10-24 a redeploy of an older production deployment rolled +// back a release (DOS/Rate#814). It never suggests a deployment of any other +// commit. + +import { parseArgs } from 'node:util'; +import { DEFAULT_REPO, fail, ghJson, isMain, sleep } from './lib.mjs'; + +export const DEFAULT_ROUTES = [ + '/', + '/smartphones', + '/earbuds', + '/deals', + '/api/v1/rate/stats', +]; +const REQUEST_TIMEOUT_MS = 30_000; +const POLL_MS = 20_000; + +/** Problems with the live site; an empty list means it serves main. */ +export function evaluateLive({ mainSha, health, routes }) { + const problems = []; + if (health.status !== 200) { + problems.push({ + kind: 'health', + message: `/api/health returned ${health.status}`, + }); + } + if (health.body?.checks?.database !== 'ok') { + problems.push({ + kind: 'database', + message: `database check is ${health.body?.checks?.database ?? 'missing'}`, + }); + } + const commit = health.body?.commit; + if (!commit) { + problems.push({ + kind: 'commit', + message: + '/api/health reports no commit (a deployment from before DOS/Rate#865, or the site is down)', + }); + } else if (commit !== mainSha) { + problems.push({ + kind: 'commit', + message: `live commit ${commit.slice(0, 8)} is not main ${mainSha.slice(0, 8)}`, + }); + } + for (const r of routes) { + if (r.status !== 200) + problems.push({ + kind: 'route', + message: `${r.path} returned ${r.status}`, + }); + } + return problems; +} + +/** Still worth waiting: only the commit differs, so a build may be deploying. */ +export const onlyCommitMismatch = (problems) => + problems.length > 0 && problems.every((p) => p.kind === 'commit'); + +/** The newest successful Production deployment status of main's commit. */ +export function redeploySource(statuses) { + const ok = statuses.find((s) => s.state === 'success' && s.environment_url); + return ok?.environment_url ?? null; +} + +/** + * What to do about a live commit that is not main. `relation` is the GitHub + * compare status of live...main: 'ahead' means main has commits live lacks, + * i.e. live is behind main. Only then is main's own deployment the fix. + */ +export function restoreAdvice({ relation, source }) { + if (relation === 'behind' || relation === 'diverged') { + return 'Live is not behind main (main moved, or live has commits main lacks). Do not redeploy or promote anything: re-run verify-live.'; + } + if (!source) { + return 'No successful Production deployment of main yet: wait for the Vercel build of main. Never redeploy or promote an older deployment.'; + } + const how = `vercel promote ${source} --scope metados`; + return relation === 'ahead' + ? `Live is behind main. Once main's build is Ready, put it on the domains with: ${how} (a promote does not rebuild and re-enables auto-assignment after a rollback). Never promote or redeploy a deployment built from another commit.` + : `Live commit unknown. If rate.vn does not serve main's build, the only deployment to promote is main's: ${how}`; +} + +// One retry after a network error: a single dropped connection is not an +// outage, while an HTTP error status is reported as it is. +async function request(url, attempt = 1) { + try { + const res = await fetch(url, { + headers: { 'cache-control': 'no-cache' }, + signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS), + }); + let body = null; + if ((res.headers.get('content-type') ?? '').includes('application/json')) { + body = await res.json().catch(() => null); + } else { + await res.arrayBuffer(); + } + return { status: res.status, body }; + } catch (err) { + if (attempt < 2) { + await sleep(2_000); + return request(url, attempt + 1); + } + const cause = + err instanceof Error ? (err.cause?.code ?? err.name) : 'unknown'; + return { status: `error (${cause})`, body: null }; + } +} + +async function readLive(baseUrl, paths) { + const health = await request(`${baseUrl}/api/health`); + const routes = []; + for (const path of paths) { + routes.push({ path, status: (await request(`${baseUrl}${path}`)).status }); + } + return { health, routes }; +} + +function findRedeploySource(repo, sha) { + const deployments = ghJson([ + 'api', + `repos/${repo}/deployments?sha=${sha}&environment=Production&per_page=5`, + ]); + for (const d of deployments ?? []) { + const url = redeploySource( + ghJson([ + 'api', + `repos/${repo}/deployments/${d.id}/statuses?per_page=10`, + ]) ?? [] + ); + if (url) return url; + } + return null; +} + +function adviceFor(repo, liveSha, mainSha) { + try { + const relation = liveSha + ? ghJson(['api', `repos/${repo}/compare/${liveSha}...${mainSha}`])?.status + : null; + return restoreAdvice({ + relation, + source: findRedeploySource(repo, mainSha), + }); + } catch (err) { + return `Could not look up main's deployment (${err.message}). Do not redeploy or promote anything until it is known.`; + } +} + +export async function verifyLive({ + repo = DEFAULT_REPO, + baseUrl = 'https://rate.vn', + routes = DEFAULT_ROUTES, + waitMs = 0, +} = {}) { + const deadline = Date.now() + waitMs; + let apiErrors = 0; + for (;;) { + // Re-read every poll: a promotion during the wait moves main forward. + let mainSha; + try { + mainSha = ghJson(['api', `repos/${repo}/commits/main`]).sha; + apiErrors = 0; + } catch (err) { + if (++apiErrors >= 3 || Date.now() >= deadline) { + return { + ok: false, + message: `Could not read main (${err.message}). Do not redeploy or promote anything until it is known.`, + }; + } + await sleep(POLL_MS); + continue; + } + const live = await readLive(baseUrl, routes); + const problems = evaluateLive({ mainSha, ...live }); + if (problems.length === 0) { + return { + ok: true, + message: `${baseUrl} serves main ${mainSha.slice(0, 8)}: database ok, ${routes.length} routes 200`, + }; + } + if (!onlyCommitMismatch(problems) || Date.now() >= deadline) { + const lines = problems.map((p) => `- ${p.message}`); + if (problems.some((p) => p.kind === 'commit')) { + lines.push(adviceFor(repo, live.health.body?.commit, mainSha)); + } + return { + ok: false, + message: [ + `${baseUrl} does not serve main ${mainSha.slice(0, 8)}:`, + ...lines, + ].join('\n'), + }; + } + await sleep(POLL_MS); + } +} + +if (isMain(import.meta.url)) { + const { values } = parseArgs({ + options: { + repo: { type: 'string', default: DEFAULT_REPO }, + 'base-url': { type: 'string', default: 'https://rate.vn' }, + route: { type: 'string', multiple: true }, + wait: { type: 'string', default: '0' }, + }, + }); + const waitSeconds = Number(values.wait); + if (!Number.isFinite(waitSeconds) || waitSeconds < 0) + fail('--wait takes seconds'); + const result = await verifyLive({ + repo: values.repo, + baseUrl: values['base-url'].replace(/\/$/, ''), + routes: values.route ?? DEFAULT_ROUTES, + waitMs: waitSeconds * 1000, + }); + if (!result.ok) fail(result.message); + console.log(result.message); +} diff --git a/scripts/ops/verify-live.test.mjs b/scripts/ops/verify-live.test.mjs new file mode 100644 index 000000000..d34935d12 --- /dev/null +++ b/scripts/ops/verify-live.test.mjs @@ -0,0 +1,122 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { + evaluateLive, + onlyCommitMismatch, + redeploySource, + restoreAdvice, +} from './verify-live.mjs'; + +const MAIN = 'c20a9ef1b196e26b9a0d5498344ce6dc59f5cb31'; +const SOURCE = 'https://rate-main-metados.vercel.app'; +const healthy = (commit = MAIN) => ({ + status: 200, + body: { status: 'ok', commit, checks: { database: 'ok' } }, +}); +const ok = [{ path: '/', status: 200 }]; + +test('a site that serves main with a readable database passes', () => { + assert.deepEqual( + evaluateLive({ mainSha: MAIN, health: healthy(), routes: ok }), + [] + ); +}); + +test('an older deployment is a commit mismatch worth waiting for', () => { + const problems = evaluateLive({ + mainSha: MAIN, + health: healthy('d24f3bf5a6bb'), + routes: ok, + }); + assert.equal(problems.length, 1); + assert.match( + problems[0].message, + /live commit d24f3bf5 is not main c20a9ef1/ + ); + assert.equal(onlyCommitMismatch(problems), true); +}); + +test('a deployment that reports no commit fails and says why', () => { + const problems = evaluateLive({ + mainSha: MAIN, + health: healthy(null), + routes: ok, + }); + assert.match(problems[0].message, /reports no commit .*#865/); +}); + +test('a database outage or a failing route is not worth waiting for', () => { + const outage = evaluateLive({ + mainSha: MAIN, + health: { + status: 503, + body: { status: 'degraded', commit: MAIN, checks: { database: 'error' } }, + }, + routes: [{ path: '/laptops', status: 500 }], + }); + assert.deepEqual( + outage.map((p) => p.kind), + ['health', 'database', 'route'] + ); + assert.equal(onlyCommitMismatch(outage), false); +}); + +test('an unreachable health endpoint fails every health check', () => { + const problems = evaluateLive({ + mainSha: MAIN, + health: { status: 'error (ETIMEDOUT)', body: null }, + routes: ok, + }); + assert.deepEqual( + problems.map((p) => p.kind), + ['health', 'database', 'commit'] + ); +}); + +test('the redeploy source is the newest successful deployment of main', () => { + assert.equal( + redeploySource([ + { state: 'success', environment_url: SOURCE }, + { state: 'pending', environment_url: '' }, + ]), + SOURCE + ); + assert.equal( + redeploySource([{ state: 'failure', environment_url: 'https://x' }]), + null + ); + assert.equal(redeploySource([]), null); +}); + +test('only a live commit behind main gets main promoted', () => { + const behind = restoreAdvice({ relation: 'ahead', source: SOURCE }); + assert.match(behind, /Live is behind main/); + assert.match( + behind, + /vercel promote https:\/\/rate-main-metados\.vercel\.app --scope metados/ + ); + assert.doesNotMatch(behind, /vercel redeploy/); +}); + +test('a live commit ahead of or diverged from main is never rolled back', () => { + for (const relation of ['behind', 'diverged']) { + const advice = restoreAdvice({ relation, source: SOURCE }); + assert.match(advice, /Do not redeploy or promote anything/); + assert.doesNotMatch(advice, /vercel promote/); + } +}); + +test('without a built deployment of main, it says to wait', () => { + assert.match( + restoreAdvice({ relation: 'ahead', source: null }), + /wait for the Vercel build of main/ + ); +}); + +test('an unknown live commit names only main deployment', () => { + assert.match( + restoreAdvice({ relation: null, source: SOURCE }), + /only deployment to promote is main's: vercel promote/ + ); +});