From 50829030f6ea4b48556bfb50fff2761700560dcd Mon Sep 17 00:00:00 2001 From: Remi Antczak <140408771+remigiusz-antczak@users.noreply.github.com> Date: Sat, 31 Oct 2026 05:29:37 +0100 Subject: [PATCH] Semgrep: flag body caps checked after a whole read and forwarded-header limiter keys templates/semgrep-taint/js.yml gains two INFO rules for resource exhaustion. perun.js.body-cap-after-buffer reports `await .formData()`, `.json()`, `.text()` or `.arrayBuffer()` on a receiver named like req or request (req, request, c.req, ctx.request, event.request) when the same block later compares the result's .size, .length or .byteLength, directly or on an entry read with .get(...), against a cap, or when only a content-length comparison precedes the read. The whole body is already in memory when such a cap runs, and a declared length does not bound a chunked body; the message points to reading the stream with getReader() and stopping at a running byte total. A comparison with the literal 0 is an emptiness check and is not reported, and neither is a read on res, response or file. perun.js.limiter-key-forwarded-header reports an x-forwarded-for or x-real-ip read (headers.get, headers.header or headers[...]) inside a function named like clientKey, limiterKey, rateKey, deriveKey or getIp. The message asks the reviewer to check that only entries added by trusted proxies are used, that a missing header does not map every caller onto one shared key, and that a missing header does not fail open. Both rules skip the test, fixture and mock paths the sibling rules skip. Their ceilings are ponytail: lines in the js.yml header: receiver and function names are heuristics; only a > comparison with the result on the left is recognised; a destructured, inline, optional-chained or non-null-asserted result is not tied to a later check; a read and its check in different blocks are not tied; and a cap enforced in middleware or a framework option is invisible. New test scripts/test_semgrep_taint_ext.py, registered as scripts/ci-tests.d/semgrep-taint-ext.sh. The live part runs when semgrep is installed: in annotated fixtures a `// ruleid:` line must be reported and an `// ok:` or unannotated line must stay clean, copies of the fixtures under test paths must stay clean, and a copy of the rules with an invalid severity must be rejected, which checks the rule schema without contacting semgrep.dev. The offline part always runs: severity, languages, message keywords and skipped paths of each rule, the header ceiling lines, the whole pattern skeleton of each rule compared with the rule text, and replays of the rules' regexes on positive and negative values. CI installs no semgrep, so CI runs the offline part. Tests: test_semgrep_taint_ext ok live (semgrep 1.182.0) and offline (semgrep removed from PATH), test_semgrep_taint ok live; 9 of 9 sections passed (run-ci-tests.sh --affected e91de6971, privacy and trigger lint included); size ratchet, fragments gate, privacy and identity gates clean. Co-Authored-By: Claude Sonnet 5.5 Co-Authored-By: Claude Opus 5.5 --- .../templates/semgrep-taint/js.yml | 84 +++++ changelog.d/semgrep-taint-extended-rules.md | 5 + scripts/ci-tests.d/semgrep-taint-ext.sh | 2 + scripts/test_semgrep_taint_ext.py | 292 ++++++++++++++++++ 4 files changed, 383 insertions(+) create mode 100644 changelog.d/semgrep-taint-extended-rules.md create mode 100644 scripts/ci-tests.d/semgrep-taint-ext.sh create mode 100644 scripts/test_semgrep_taint_ext.py diff --git a/.claude/skills/deep-code-review/templates/semgrep-taint/js.yml b/.claude/skills/deep-code-review/templates/semgrep-taint/js.yml index a92a1a133..fcb78ffea 100644 --- a/.claude/skills/deep-code-review/templates/semgrep-taint/js.yml +++ b/.claude/skills/deep-code-review/templates/semgrep-taint/js.yml @@ -2,6 +2,8 @@ # Sinks: classic injection calls on well-known receivers. A guard `if (!ALLOW.has(x)) return|throw` before the sink suppresses it for x. # Also hardcoded-credential-compare (a pass/secret/token/api-key name vs a non-empty string literal, WARNING) and hardcoded-basic-auth (a base64-decoded value vs a literal, ERROR); env reads, `typeof` checks, JWT header fields (alg, typ, kid), scheme/sentinel literals (bearer, none, true, ...), single-character literals and test/fixture paths are skipped. ponytail: template literals are not matched. # Also date-parse-rollover (INFO: a date accepted only because Number.isFinite/isNaN of an inline Date.parse(x) or new Date(x) passed, with no .toISOString() round trip in the enclosing function; V8 reads 2026-02-30 as 2 March; test paths are skipped) and gcm-no-auth-tag-length (WARNING: createDecipheriv with an *-gcm cipher and no authTagLength option, so a truncated tag is accepted, a forgery risk only when the tag length is attacker-controlled). ponytail: any .toISOString() in the enclosing function counts as the round trip, component compares such as getUTCDate are not recognised, a numeric-timestamp new Date(ts) check can report, and a check on a variable bound first (const d = new Date(x); isNaN(d)) is not flagged because those hits were mostly noise; a cipher name that is not a string literal (or a const bound to one) and an options argument that is not an object literal are skipped. +# Also body-cap-after-buffer (INFO: await req.formData()/json()/text()/arrayBuffer() on a receiver named like req or request (req, request, c.req, event.request) followed later in the same block by a .size/.length/.byteLength > cap on the result or on a .get(...) entry of it, or preceded only by a content-length > cap check; a comparison with the literal 0 is an emptiness check, not a cap, and a counted getReader() read passes; test, fixture and mock paths are skipped). ponytail: the receiver is matched by name (one ending in req or request), so a request bound to another name (r) or read through a call (req.clone().json()) is missed, only a > comparison with the result on the left is recognised, a result bound by destructuring or read inline (await req.formData()).get(...) is not tied to a later check, a ?. or ! on the result or on its .get(...) entry (body?.length, body!.length, form.get("f")!.size) is not matched, a deeper field path (body.items.length) is not the result, a read and its check in different blocks (the read inside a try, the check after it) are not tied, and a cap that lives in a middleware or framework option is invisible. +# Also limiter-key-forwarded-header (INFO: an x-forwarded-for / x-real-ip read inside a function named like clientKey, limiterKey, rateKey, deriveKey or getIp; test, fixture and mock paths are skipped). ponytail: the function name is a heuristic, so a key built in an anonymous or differently named function (getClientIp, for one) is missed and any function with a matching name is reported. rules: - id: perun.js.sqli mode: taint @@ -734,3 +736,85 @@ rules: - metavariable-regex: metavariable: $ALG regex: (?i).*-gcm$ +- id: perun.js.body-cap-after-buffer + languages: [javascript, typescript] + severity: INFO + message: The request body is read whole (formData, json, text or arrayBuffer) and its size is only checked afterwards, or only a content-length header is compared before reading. The whole body is already in memory when the cap runs, and a declared length does not bound a chunked body. Read the stream with getReader() and stop at a running byte total instead. + patterns: + - pattern: await $REQ.$M() + - metavariable-regex: + metavariable: $REQ + regex: (?i).*req(?:uest)?$ + - metavariable-regex: + metavariable: $M + regex: ^(?:formData|json|text|arrayBuffer)$ + - metavariable-regex: + metavariable: $N + regex: ^(?!0$) + - pattern-either: + - patterns: + - pattern-either: + - pattern-inside: | + $V = await $REQ.$M(); + ... + <... $V.$P > $N ...> + - pattern-inside: | + $V = await $REQ.$M(); + ... + $F = $V.get(...); + ... + <... $F.$P > $N ...> + - pattern-inside: | + $V = await $REQ.$M(); + ... + <... $V.get(...).$P > $N ...> + - metavariable-regex: + metavariable: $P + regex: ^(?:size|length|byteLength)$ + - patterns: + - pattern-either: + - pattern-inside: | + <... (<... $H.get("$CL") ...>) > $N ...>; + ... + <... await $REQ.$M() ...> + - pattern-inside: | + $L = <... $H.get("$CL") ...>; + ... + <... (<... $L ...>) > $N ...>; + ... + <... await $REQ.$M() ...> + - metavariable-regex: + metavariable: $CL + regex: (?i)^content-length$ + paths: + exclude: ["test", "tests", "__tests__", "spec", "fixtures", "mocks", "__mocks__", "e2e", "*.test.*", "*.spec.*"] +- id: perun.js.limiter-key-forwarded-header + languages: [javascript, typescript] + severity: INFO + message: A client or rate-limit key is derived from x-forwarded-for or x-real-ip. Check that only entries added by trusted proxies are used (a client controls the leftmost ones), that a missing header does not map every caller to one shared constant key, and that a missing header does not fail open and skip limiting. + patterns: + - pattern-either: + - pattern: $H.get("$HDR") + - pattern: $H.header("$HDR") + - pattern: $H["$HDR"] + - metavariable-regex: + metavariable: $HDR + regex: (?i)^(?:x-forwarded-for|x-real-ip)$ + - pattern-either: + - pattern-inside: | + function $F(...) { + ... + } + - pattern-inside: | + $F = function (...) { + ... + } + - pattern-inside: | + $F = ($...P) => { + ... + } + - metavariable-regex: + metavariable: $F + regex: (?i).*(?:(?:client|limiter|rate)key|derivekey|getip) + paths: + exclude: ["test", "tests", "__tests__", "spec", "fixtures", "mocks", "__mocks__", "e2e", "*.test.*", "*.spec.*"] diff --git a/changelog.d/semgrep-taint-extended-rules.md b/changelog.d/semgrep-taint-extended-rules.md new file mode 100644 index 000000000..0565bfcdc --- /dev/null +++ b/changelog.d/semgrep-taint-extended-rules.md @@ -0,0 +1,5 @@ +### Added +- Bundled semgrep rule `perun.js.body-cap-after-buffer` (INFO, in `templates/semgrep-taint/js.yml`): a request body read whole with `await req.formData()`, `.json()`, `.text()` or `.arrayBuffer()` on a receiver named like `req` or `request` (`req`, `request`, `c.req`, `event.request`) and followed later in the same block by a `.size`, `.length` or `.byteLength` comparison against a cap on the result (directly, or on an entry read from it with `.get(...)`), or preceded only by a `content-length` comparison. The body is already in memory when the cap runs, and a declared length does not bound a chunked body. A comparison with the literal `0` is an emptiness check, a read on `res`, `response` or `file` is not a request body, and a counted `getReader()` read is a bounded read: none of them is reported. Test, fixture and mock paths are skipped. +- Bundled semgrep rule `perun.js.limiter-key-forwarded-header` (INFO): an `x-forwarded-for` or `x-real-ip` read (`headers.get(...)`, `headers.header(...)` or `headers[...]`) inside a function named like `clientKey`, `limiterKey`, `rateKey`, `deriveKey` or `getIp`. The message asks the reviewer to check the trusted-hop count, whether a missing header maps every caller onto one shared key, and whether it fails open. Test, fixture and mock paths are skipped. +- `scripts/test_semgrep_taint_ext.py`, registered in `scripts/ci-tests.d/semgrep-taint-ext.sh`, checks these rules with annotated fixtures: a `// ruleid:` comment marks the next line as a required finding, an `// ok:` comment marks it as one that must stay clean, and every unannotated line must stay clean. The live part needs semgrep and prints SKIP without it; it also scans a copy of each fixture under a test path and expects no finding, and loads a copy of the rules with an invalid severity and expects semgrep to reject it, so the schema check of the rule file does not depend on the network. The offline part always runs: severity, languages and message keywords, the skipped paths compared with a sibling rule's list, a `ponytail:` ceiling line in the file header for each rule, the whole pattern skeleton of each rule (every shape and the tags that join them) compared with the rule text, and replays of the rules' own regexes on positive and negative values. +- Known ceilings, also recorded as `ponytail:` lines in the `js.yml` header. `perun.js.limiter-key-forwarded-header` follows the function name, so it misses a differently named function (`getClientIp`, for one) and reports any function whose name matches. `perun.js.body-cap-after-buffer` matches the receiver by name, so it misses a request bound to another name (`r`) or read through a call (`req.clone().json()`). It recognises only `>` comparisons with the result on the left, does not tie a body bound by destructuring or read inline to a later check, does not match a `?.` or `!` on the result or on its `.get(...)` entry (`body?.length`, `body!.length`, `form.get("file")!.size`), does not count a deeper field path such as `body.items.length` or a read and a check in different blocks, and cannot see a cap enforced in middleware or a framework option. diff --git a/scripts/ci-tests.d/semgrep-taint-ext.sh b/scripts/ci-tests.d/semgrep-taint-ext.sh new file mode 100644 index 000000000..4ef7e2ad5 --- /dev/null +++ b/scripts/ci-tests.d/semgrep-taint-ext.sh @@ -0,0 +1,2 @@ +# Bundled semgrep rules perun.js.body-cap-after-buffer and perun.js.limiter-key-forwarded-header (templates/semgrep-taint/js.yml): annotated ruleid/ok fixtures (live when semgrep is installed, SKIP otherwise) plus offline structural pins and regex replays that still run where semgrep is absent. +python3 scripts/test_semgrep_taint_ext.py diff --git a/scripts/test_semgrep_taint_ext.py b/scripts/test_semgrep_taint_ext.py new file mode 100644 index 000000000..f752a4f25 --- /dev/null +++ b/scripts/test_semgrep_taint_ext.py @@ -0,0 +1,292 @@ +#!/usr/bin/env python3 +"""Tests for the bundled semgrep resource-exhaustion rules in templates/semgrep-taint/js.yml: perun.js.body-cap-after-buffer (request +receiver, non-zero cap, each pattern shape) and perun.js.limiter-key-forwarded-header, both skipping test, fixture and mock paths. + +Fixtures are annotated like semgrep's own tests: a `// ruleid: ` comment marks the NEXT line as a required finding, an +`// ok: ` comment marks the next line as one that must stay clean, and every unannotated line must stay clean too, whichever +rule would report it. All names and values are fictional. RULES= checks another copy of the rules (used to show the cases +fail without them). The live check needs semgrep and prints SKIP without it (CI installs none); the offline checks always run. +`semgrep --validate` needs the network (it fetches a lint pack), so the live part loads the rules instead: a schema error makes +semgrep report an error and exit non-zero, which a mutated copy below proves. Plain asserts.""" +import json, os, re, shutil, subprocess, tempfile + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +RULES = os.environ.get("RULES") or os.path.join(ROOT, ".claude", "skills", "deep-code-review", "templates", "semgrep-taint") + +# Body caps: a size check after the whole body is buffered, or only a declared content-length, is reported. +CAP_TS = r''' +export async function formCap(req: Request) { + // ruleid: perun.js.body-cap-after-buffer + const form = await req.formData(); + const file = form.get("file") as File; + if (file.size > MAX_BYTES) return new Response("too big", { status: 413 }); + return Response.json({ ok: true }); +} +export async function inlineFileCap(req: Request) { + // ruleid: perun.js.body-cap-after-buffer + const form = await req.formData(); + if ((form.get("file") as File).size > MAX_BYTES) return fail(); + return ok(form); +} +export async function textCap(req: Request) { + // ruleid: perun.js.body-cap-after-buffer + const text = await req.text(); + if (text.length > MAX_CHARS) return fail(); + return ok(text); +} +export async function bufferCap(req: Request) { + // ruleid: perun.js.body-cap-after-buffer + const buf = await req.arrayBuffer(); + return buf.byteLength > MAX_BYTES ? fail() : ok(buf); +} +export async function jsonCap(req: Request) { + // ruleid: perun.js.body-cap-after-buffer + const items = await req.json(); + if (items.length > MAX_ITEMS) return fail(); + return ok(items); +} +export async function declaredLength(req: Request) { + if (Number(req.headers.get("content-length")) > MAX_BYTES) return fail(); + // ruleid: perun.js.body-cap-after-buffer + const form = await req.formData(); + return ok(form); +} +export async function declaredLengthVar(req: Request) { + const len = Number(req.headers.get("Content-Length") ?? "0"); + if (len > MAX_BYTES) return fail(); + // ruleid: perun.js.body-cap-after-buffer + const body = await req.json(); + return ok(body); +} +export async function countedStream(req: Request) { + const reader = req.body!.getReader(); + let total = 0; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + total += value.length; + if (total > MAX_BYTES) return fail(); + } + return ok(); +} +export async function declaredLengthThenStream(req: Request) { + if (Number(req.headers.get("content-length")) > MAX_BYTES) return fail(); + const reader = req.body!.getReader(); + let total = 0; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + total += value.length; + if (total > MAX_BYTES) return fail(); + } + return ok(); +} +export async function noCap(req: Request) { + // ok: perun.js.body-cap-after-buffer + const body = await req.json(); + return ok(body); +} +export async function fieldCheck(req: Request) { + // ok: perun.js.body-cap-after-buffer + const body = await req.json(); + if (body.name.length > MAX_NAME) return fail(); + return ok(body); +} +export async function otherField(req: Request) { + // ok: perun.js.body-cap-after-buffer + const body = await req.json(); + if (body.limit > MAX_LIMIT) return fail(); + return ok(body); +} +export async function otherSize(req: Request) { + // ok: perun.js.body-cap-after-buffer + const body = await req.json(); + if (cache.size > MAX_CACHE) cache.clear(); + return ok(body); +} +export async function otherHeader(req: Request) { + const agent = req.headers.get("user-agent") ?? ""; + if (agent.length > MAX_AGENT) return fail(); + // ok: perun.js.body-cap-after-buffer + const body = await req.json(); + return ok(body); +} +export async function lengthOnlyLogged(req: Request) { + log(Number(req.headers.get("content-length"))); + // ok: perun.js.body-cap-after-buffer + const body = await req.json(); + return ok(body); +} +export async function emptinessCheck(req: Request) { + // ok: perun.js.body-cap-after-buffer + const rows = await req.json(); + if (rows.length > 0) return ok(rows); + return fail(); +} +export async function declaredLengthNonZero(req: Request) { + if (Number(req.headers.get("content-length")) > 0) return ok(); + // ok: perun.js.body-cap-after-buffer + const body = await req.json(); + return ok(body); +} +export async function responseBody(res: Response) { + // ok: perun.js.body-cap-after-buffer + const items = await res.json(); + if (items.length > 0) return ok(items); + return fail(); +} +export async function upstreamCap(res: Response) { + // ok: perun.js.body-cap-after-buffer + const items = await res.json(); + if (items.length > MAX_ITEMS) return fail(); + return ok(items); +} +export async function uploadedText(file: File) { + // ok: perun.js.body-cap-after-buffer + const text = await file.text(); + if (text.length > MAX_CHARS) return fail(); + return ok(text); +} +export async function receiverNames(c: Context, ctx: Ctx, event: Ev, nextReq: Request) { + // ruleid: perun.js.body-cap-after-buffer + const a = await c.req.json(); + // ruleid: perun.js.body-cap-after-buffer + const b = await ctx.request.text(); + // ruleid: perun.js.body-cap-after-buffer + const d = await event.request.json(); + // ruleid: perun.js.body-cap-after-buffer + const e = await nextReq.json(); + if (a.length > MAX_ITEMS || b.length > MAX_CHARS || d.length > MAX_ITEMS || e.length > MAX_ITEMS) return fail(); + return ok(); +} +''' + +# Rate-limit keys: a forwarded-header read inside a key-deriving function is reported. +LIMIT_TS = r''' +export function getClientKey(req: Request) { + // ruleid: perun.js.limiter-key-forwarded-header + return req.headers.get("x-forwarded-for") ?? "unknown"; +} +// ruleid: perun.js.limiter-key-forwarded-header +export const rateKey = (req: Request) => req.headers.get("x-real-ip"); +export async function getIp(req: Request): Promise { + // ruleid: perun.js.limiter-key-forwarded-header + const fwd = req.headers.get("X-Forwarded-For"); + return fwd ? fwd.split(",")[0].trim() : "0.0.0.0"; +} +export const limiterKey = async (headers: Record) => { + // ruleid: perun.js.limiter-key-forwarded-header + return headers["x-forwarded-for"]; +}; +class Limiter { + deriveKey(req: Request) { + // ruleid: perun.js.limiter-key-forwarded-header + return req.header("x-real-ip"); + } +} +export function logRequest(req: Request) { + // ok: perun.js.limiter-key-forwarded-header + console.log(req.headers.get("x-forwarded-for")); +} +export function getClientKeyFromAuth(req: Request) { + // ok: perun.js.limiter-key-forwarded-header + return req.headers.get("authorization"); +} +export function rateKeyFromProto(req: Request) { + // ok: perun.js.limiter-key-forwarded-header + return req.headers.get("x-forwarded-proto"); +} +''' + +FILES = {"app/api/uploads/route.ts": CAP_TS, "lib/limit.ts": LIMIT_TS} +SKIPPED = {"tests/uploads.ts": CAP_TS, "lib/limit.test.ts": LIMIT_TS} # test paths are excluded from both rules: nothing in these copies may be reported +ANN = re.compile(r"^\s*//\s*(ruleid|ok):\s*(\S+)\s*$") +META = {"perun.js.body-cap-after-buffer": ("INFO", "getReader()"), "perun.js.limiter-key-forwarded-header": ("INFO", "trusted proxies")} + + +def annotated(body, kind): + """(line, rule id) of the line each `// : ` comment points at (the next one).""" + return {(i + 1, m.group(2)) for i, l in enumerate(body.split("\n"), 1) if (m := ANN.match(l)) and m.group(1) == kind} + + +ids = lambda kind: {r for b in FILES.values() for _, r in annotated(b, kind)} +assert all(annotated(b, "ok") | annotated(b, "ruleid") for b in FILES.values()) # every fixture is annotated +assert set(META) <= ids("ruleid") & ids("ok"), "a flag and a pass fixture per rule" + +if shutil.which("semgrep"): + with tempfile.TemporaryDirectory() as d: + for p, body in {**FILES, **SKIPPED}.items(): + os.makedirs(os.path.dirname(os.path.join(d, p)), exist_ok=True) + open(os.path.join(d, p), "w").write(body) + env = dict(os.environ, HOME=d, SEMGREP_USER_HOME=d) + scan = lambda cfg: subprocess.run(["semgrep", "--config", cfg, "--metrics=off", "--disable-version-check", "--json", "--quiet", "--", *FILES, *SKIPPED], + capture_output=True, text=True, env=env, cwd=d) + run = scan(RULES) + r = json.loads(run.stdout) + # Stand-in for `semgrep --validate` (it needs the network): loading the rules schema-checks them. Prove that check can fail. + assert run.returncode == 0 and not r["errors"], (run.returncode, r["errors"]) + os.makedirs(os.path.join(d, "bad")) + js = open(os.path.join(RULES, "js.yml")).read() + head = "- id: perun.js.body-cap-after-buffer\n languages: [javascript, typescript]\n severity: " + bad = js.replace(head + "INFO\n", head + "LOUD\n", 1) + assert bad != js, "the body-cap-after-buffer header changed: update the mutation" + open(os.path.join(d, "bad", "js.yml"), "w").write(bad) + broken = scan(os.path.join(d, "bad")) + assert broken.returncode != 0 and json.loads(broken.stdout)["errors"], "a schema error must surface" + got = set() # semgrep prefixes a rule id with each local config's directory: keep the perun.. tail + for x in r["results"]: + got.add((x["path"], x["start"]["line"], "perun." + x["check_id"].rsplit("perun.", 1)[-1])) + want = {(p, ln, rid) for p, body in FILES.items() for ln, rid in annotated(body, "ruleid")} + assert got == want, ("unexpected", sorted(got - want), "missed", sorted(want - got)) +else: + print("SKIP: semgrep not installed; the live fixture check did not run, the offline checks below do") + +# Offline: severity, languages, message and test-path skip of each rule, the ponytail ceiling in the header, every pattern shape, and the +# name, value and receiver regexes each rule carries (semgrep anchors metavariable-regex at the start, like Python's re.match). CI installs +# no semgrep, so every guard the live fixtures prove is pinned here too: dropping one fails this file even when the live part is skipped. +JS = open(os.path.join(RULES, "js.yml")).read() +block = lambda src, rid: src.split(f"- id: {rid}\n")[1].split("\n- id:")[0] +rx = lambda b, mv: re.search(r"metavariable: " + re.escape(mv) + r"\n\s+regex: (.*)", b).group(1) +norm = lambda s: re.sub(r"\s+", " ", s) +excl = lambda b: (re.search(r"paths:\n\s+exclude: (\[.*\])", b) or [0, 0])[1] +for rid, (sev, word) in META.items(): + b = block(JS, rid) + assert f" severity: {sev}\n" in b and word in re.search(r" message: (.*)", b).group(1), rid + assert " languages: [javascript, typescript]\n" in b, rid + assert excl(b) and excl(b) == excl(block(JS, "perun.js.hardcoded-credential-compare")), f"{rid} skips the sibling rules' test, fixture and mock paths" + assert any(l.startswith("#") and rid.split(".")[-1] in l and "ponytail:" in l for l in JS.split("\n")[:20]), f"header ponytail line for {rid}" + +# The pattern skeleton of a rule, whitespace-normalised and with the regex values dropped (those are replayed below): every shape and every +# tag that joins them, so deleting or editing any of them fails here even though only the live part runs semgrep. +skel = lambda b: re.sub(r"(? $N ...>", # the result compared in the same block + "- pattern-inside: | $V = await $REQ.$M(); ... $F = $V.get(...); ... <... $F.$P > $N ...>", # an entry read from it with get + "- pattern-inside: | $V = await $REQ.$M(); ... <... $V.get(...).$P > $N ...>", # an entry compared inline + "- metavariable-regex: metavariable: $P regex:", + "- patterns: - pattern-either:", + '- pattern-inside: | <... (<... $H.get("$CL") ...>) > $N ...>; ... <... await $REQ.$M() ...>', # a declared length compared first + '- pattern-inside: | $L = <... $H.get("$CL") ...>; ... <... (<... $L ...>) > $N ...>; ... <... await $REQ.$M() ...>', # ... through a variable + "- metavariable-regex: metavariable: $CL regex:")), skel(c) +assert all(re.match(rx(c, "$M"), v) for v in ("formData", "json", "text", "arrayBuffer")) and not any(re.match(rx(c, "$M"), v) for v in ("read", "blob", "jsonp", "")) +assert all(re.match(rx(c, "$P"), v) for v in ("size", "length", "byteLength")) and not any(re.match(rx(c, "$P"), v) for v in ("limit", "count", "name", "sizes", "")) +assert all(re.match(rx(c, "$CL"), v) for v in ("content-length", "Content-Length")) and not any(re.match(rx(c, "$CL"), v) for v in ("content-type", "x-content-length", "content-length-x")) +assert all(re.match(rx(c, "$REQ"), v) for v in ("req", "request", "c.req", "ctx.request", "event.request", "nextReq", "REQ")) and not any(re.match(rx(c, "$REQ"), v) for v in ("res", "response", "file", "form", "r", "seq", "req.clone()", "requestBody")), "the receiver is a request" +assert all(re.match(rx(c, "$N"), v) for v in ("MAX", "1", "10", "1024 * 1024", "0.5")) and not re.match(rx(c, "$N"), "0"), "a comparison with 0 is not a cap" +l = block(JS, "perun.js.limiter-key-forwarded-header") +assert skel(l) == " ".join(( + '- pattern-either: - pattern: $H.get("$HDR") - pattern: $H.header("$HDR") - pattern: $H["$HDR"]', + "- metavariable-regex: metavariable: $HDR regex:", + "- pattern-either: - pattern-inside: | function $F(...) { ... }", + "- pattern-inside: | $F = function (...) { ... }", + "- pattern-inside: | $F = ($...P) => { ... }", + "- metavariable-regex: metavariable: $F regex:")), skel(l) +assert all(re.match(rx(l, "$HDR"), v) for v in ("x-forwarded-for", "X-Forwarded-For", "x-real-ip")) and not any(re.match(rx(l, "$HDR"), v) for v in ("x-forwarded-proto", "x-forwarded-for-x", "user-agent", "")) +assert all(re.match(rx(l, "$F"), v) for v in ("getClientKey", "rateKey", "limiterKey", "deriveKey", "getIp", "this.deriveKey")) and not any(re.match(rx(l, "$F"), v) for v in ("handler", "logRequest", "setCookie", "getClientIp", "cacheKey", "createKey", "setIp", "getItem")) +print("ok")