Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@
- [x] Unaudited-lockfile coverage warning when a repo ships both a lockfile and open version floors (2026/08/21)
- [x] SQL-identifier rule cluster + mutable-action-tag downgraded to low confidence (2026/08/21) - 49% of historical Semgrep output
- [x] Gitleaks confidence keyed on rule id and placeholder shape (2026/08/21) - 93% of historical hits leave the `high` tier
- [ ] Rule applicability check: suppress a framework-specific rule when the framework is not a declared dependency (`sqlalchemy-execute-raw-query` fired 157 times on a project with no SQLAlchemy)
- [ ] Rule applicability check: suppress a framework-specific rule when the framework is not a declared dependency (`sqlalchemy-execute-raw-query` fired 157 times on a project with no SQLAlchemy; `django-no-csrf-token` went 45 for 45 false on FastAPI templates in fossasia/voxbento, 2026-09-28)
- [ ] Gitleaks: fingerprint hits by secret hash and report one finding with N locations (fossasia/voxbento: one CI-only key quoted across `docs/plans/` surfaced as 49 high-severity findings)
- [ ] Semgrep partial-coverage advice keyed on error type: recommend a higher `--timeout` only when rules timed out; parse errors never succeed on retry (flagged 2026-09-13, again 2026-09-28 with 0 timeouts in 40 errors)
- [ ] Report lockfile and open-floor dependency sets as separate rows naming the install path, rather than one merged verdict
- [ ] Timeouts on the semgrep, trivy and gitleaks runners (pip-audit done; the others share the same hang risk)
- [ ] Exempt meta findings from `--ignore-file` suppression as well as `--min-severity`
Expand Down
368 changes: 368 additions & 0 deletions corpus/verdicts/fossasia-voxbento.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/fixed.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: "The 26 scans in the AI PatchLab series where the maintainer shippe

# Findings a maintainer fixed

Of 111 scans, **26** ended with a maintainer shipping a fix. This page is the
Of 112 scans, **26** ended with a maintainer shipping a fix. This page is the
short version of the argument: a report is only worth writing if someone can act on it.

The fastest turnaround in the series was about six hours from filing to a merged pull
Expand Down
7 changes: 4 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
layout: default
title: AI PatchLab Scans
description: "111 curated security scans of open-source AI agents, MCP servers and LLM apps - 26 confirmed fixes, run local-first with Semgrep, Gitleaks, Trivy and pip-audit."
description: "112 curated security scans of open-source AI agents, MCP servers and LLM apps - 26 confirmed fixes, run local-first with Semgrep, Gitleaks, Trivy and pip-audit."
---

# AI PatchLab Scans
Expand All @@ -20,7 +20,7 @@ remediation and confidence rules to normalize the findings.

> **Want this run privately against your own codebase?** I do independent
> security review of AI agents, MCP servers, and LLM apps —
> [**work with me →**]({{ '/work-with-me' | relative_url }}). 111 scans, 26 confirmed fixes, methodology in the open.
> [**work with me →**]({{ '/work-with-me' | relative_url }}). 112 scans, 26 confirmed fixes, methodology in the open.

> **OpenAI just launched [Daybreak](https://openai.com/index/daybreak-securing-the-world/) and Patch the Planet.**
> Same remediation loop, opposite trade-off: their path is a cloud frontier model;
Expand Down Expand Up @@ -103,7 +103,7 @@ login and static assets. Fifty-two flagged, none reported.

## All scans

111 scans, newest first. **Findings** is the raw count the tools produced;
112 scans, newest first. **Findings** is the raw count the tools produced;
**Real** is what survived curation. The gap between those two columns is the
entire job.

Expand All @@ -118,6 +118,7 @@ filed, which is the usual outcome of a clean scan.

| Date | Repository | Findings | Real | Outcome |
| --- | --- | ---: | --- | --- |
| 2026-09-28 | [fossasia/voxbento](scans/fossasia-voxbento.html) | 168 | 0 first-party — dependency | — |
| 2026-09-24 | [superdesigndev/treg](scans/superdesigndev-treg.html) | 96 | 1 real — withheld | private |
| 2026-09-23 | [can4hou6joeng4/boss-agent-cli](scans/can4hou6joeng4-boss-agent-cli.html) | 26 | 1 real — withheld | private |
| 2026-09-22 | [overwirehq/claude-code-telegram](scans/overwirehq-claude-code-telegram.html) | 54 | 0 first-party — dependency | — |
Expand Down
3 changes: 2 additions & 1 deletion docs/scan-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ description: "The complete AI PatchLab scan log: every public repository scanned

# Full scan log

Every scan in the series, newest first, with the summary written on the day of the scan. 111 scans. For the compact index, see the [scan log home]({{ '/' | relative_url }}).
Every scan in the series, newest first, with the summary written on the day of the scan. 112 scans. For the compact index, see the [scan log home]({{ '/' | relative_url }}).

- **2026-09-28** — [fossasia/voxbento](scans/fossasia-voxbento.html) — 168 findings at `medium+` (83 high, 82 medium, 3 meta), **0 first-party defects — dependency currency, post-only** — FOSSASIA's **real-time interpretation platform for live events** (1.5k★, Apache-2.0): interpreters broadcast translated audio from a browser tab, attendees listen from theirs, and a FastAPI portal handles accounts and JWT sessions, the admin console, WebSocket booth coordination, transcription and translation workers, and a developer platform with OAuth2 clients and API keys. One of the most active targets in the series — 167 merged PRs from 23 human authors and 77 closed issues in 60 days; no `SECURITY.md`, private reporting off, not strict-norm. **Two families explain most of the 168.** Semgrep's `django-no-csrf-token` fired 45 times on a FastAPI app's Jinja templates, where the auth cookies are `HttpOnly` + `SameSite=Lax`, the admin router's writes are POST or DELETE, and the developer dashboard adds a double-submit token; Gitleaks counted **one** CI-only encryption key 49 times because 17 files under `docs/plans/` quote the test workflow. **The defaults fail closed:** the compose file's `SECRET_KEY=change-me` fallback cannot boot a production portal, because the FastAPI lifespan runs `validate_production_secrets()` and refuses known-weak values outside debug (tested, and debug defaults off), while third-party provider keys sit under a separate rotating `MultiFernet` key. The one actionable item is **Starlette 1.1.0 → 1.3.1** (CVE-2026-54283, form limits ignored for urlencoded bodies): transitive via FastAPI, reachable on the unauthenticated `/register` and `/login` routes, fixed by `uv lock --upgrade-package starlette`, with no open Dependabot PR at scan time. 39 of the 45 Trivy advisories are the Docusaurus docs site's build chain — static output the portal never imports, in the one ecosystem (`npm`) the repo's Dependabot does not cover. **Scope stated in the post:** every tool finding is accounted for, but the OAuth2 provider, the API-key platform and the WebSocket layer were not swept by hand this time. Coverage `partial` and stated so — Semgrep's 40 errors are all template and Dockerfile parse errors (zero timeouts), and pip-audit left `uv.lock` unaudited while Trivy read it. Quality gate not met · post-only, nothing filed
- **2026-09-24** — [superdesigndev/treg](scans/superdesigndev-treg.html) — 96 findings at `medium+` (1 critical, 23 high, 69 medium, 3 meta), **1 real — withheld, reported privately by email** — **"OpenRouter for agent tools"** (3k★, source-available; hosted at treg.to and self-hostable), a multi-tenant server holding a team's provider credentials Fernet-encrypted and proxying calls to 3,000+ upstream endpoints with the credential injected server-side so it never reaches the caller. Its `SECURITY.md` makes two load-bearing promises: the proxy never hands a key to the caller, and outbound requests are *"re-resolved at call time and internal/metadata addresses are refused."* **Zero of the 96 scanner findings is the vulnerability** — the real item came from tabulating the server's outbound request paths and asking which the call-time host guard actually covers. It does have a proper guard, `host_is_public` in `infra/upstream/ssrf.py`, and its address classification is genuinely thorough (every loopback/link-local/CGNAT/NAT64 and decimal/hex/octal/short-form IPv4 literal I tried was refused). The finding is that the guard is wired into the main proxy path and **not** into a sibling outbound path a plain member can drive; the registration-time check that does run there is, by its own docstring, a static check that allows any DNS name, so a name resolving to an internal address passes it and nothing re-resolves before the request goes out — a member-authenticated, semi-blind [SSRF on the unguarded sibling of a guarded transport](scans/liaohch3-claude-tap.html). Confirmed with the [run-the-primitive](scans/observal-observal.html) discipline: treg's own two guard functions run side by side on a host that resolves internally return "allowed" (registration) and "refused" (call-time), and the sibling path calls only the first — one differential, using the project's code not my reasoning. Graded **Medium**: the injected credential on the affected path is the member's own, so it is internal + cloud-metadata reach, not cross-tenant theft; the org boundary itself held everywhere ([the probe could return "no" on isolation and did](scans/tracecathq-tracecat.html)). **The fix is one guard applied at the outbound path** and changes nothing for a legitimate public target. The 96: seventeen `sqlalchemy-text` are the [#1 identifier FP](scans/aurelio-labs-semantic-router.html) (Alembic `SET`/`CREATE INDEX CONCURRENTLY` from module constants + a test-reset `TRUNCATE` over dialect-quoted table names, no value interpolated); eighteen `v-html` render the project's own static tutorial/snippet HTML with an `esc(&<>)` helper on every interpolated field; four `insecure-file-permissions` are the project [tightening to `0o700`](scans/realiti4-claude-swap.html) (the rule's suggestion would loosen them); one Trivy critical is `anyio` 4.14.1→4.14.2 IDNA hostname-confusion, a worthwhile lockfile bump reachable only by a network-positioned attacker against a non-ASCII upstream host. Coverage `partial` and stated so — Semgrep hit a Windows per-file temp error on five large modules (recovered by a targeted re-run of just those five), and pip-audit left `uv.lock` unaudited while Trivy read it; neither touches the finding. **Strict-norm** (`SECURITY.md` forbids public vuln issues, PVR disabled — checked `{"enabled":false}` — names an email) · reported **privately by email to the maintainer**; post-only here, finding withheld at class level · sent by email 2026-09-24, ten minutes after this entry was published
- **2026-09-23** — [can4hou6joeng4/boss-agent-cli](scans/can4hou6joeng4-boss-agent-cli.html) — 26 findings at `medium+` (7 high, 16 medium, 3 meta), **1 real — withheld, filed privately** — a **CLI for the BOSS Zhipin recruitment platform** (2k★, MIT; a human drives it from a terminal wizard, an agent drives the same workflows over a JSON-envelope API, an ~77-tool MCP server, or a Python SDK) that authenticates by reusing the operator's own logged-in browser session. Responsive maintainer — 67 merged PRs from 7 human authors in 60 days — which is why it cleared the pre-check. **Zero of the 26 scanner findings is the vulnerability**, which is the whole point of the entry: the real item is the *absence* of a check on an otherwise-working local channel, the shape no rule can match, and it is the [desktop-loopback-inversion](scans/liaohch3-claude-tap.html) class — a `127.0.0.1` control surface that binds neither *who* may call it nor which host the request claims, driving privileged browser actions against a site where the operator is authenticated. Confirmed with a **server-side exploit primitive against the shipped daemon** (a foreign `Host` is served, a cross-origin "simple" request reaches the command channel with no preflight, a hostile `Origin` is not rejected) plus a read of the extension source for the browser half — the [run-the-primitive discipline](scans/observal-observal.html), because reading alone could not separate "the channel is drivable" from "the channel is drivable *and* blind does not blunt it." Graded **High not Critical** on an honest precondition (the bridge component must be running, and the user has to start it deliberately; once started it stays up while the browser side is connected — *corrected 2026-09-24: first published as "a bounded window", the one claim reasoned about rather than checked, and wrong both ways; the private report was corrected the same day*) and the sensitive scope is the one site the tool is built around; **the fix was written before it was named**, narrow and verified not to break the tool's own two legitimate clients ([write the patch first](scans/roflcoopter-viseron.html)). The 26: eight SQL hits (five high) are the [#1 identifier FP](scans/aurelio-labs-semantic-router.html) — every interpolation is a hardcoded table/column name with values bound as `?`, settled by reading the call sites; eight of sixteen mediums are one `github-actions-mutable-action-tag` flood; the rest are two `sha1()[:16]` dedup-id digests, two `urllib` calls to a local DevTools endpoint and a build script, a React rule on a loopback health-check `fetch`, and a Python-3.7 compatibility rule that is not a security check. Coverage `partial` and stated so — Semgrep did not cover every file and the dependency scan left `uv.lock` unaudited (Trivy read it, the [luck that hides the bug](scans/roflcoopter-viseron.html)), neither touching the finding. **Strict-norm** (`SECURITY.md` forbids public vulnerability issues, gives an email and a PVR link) · PVR-enabled, filed **privately, accepted into triage first try** ([GHSA-xp54-2hxc-w78q](https://github.com/can4hou6joeng4/boss-agent-cli/security/advisories/GHSA-xp54-2hxc-w78q)) **with the `vulnerabilities` array** — the [viseron rule](scans/roflcoopter-viseron.html) holds again · post-only, finding withheld at class level under embargo
- **2026-09-22** — [overwirehq/claude-code-telegram](scans/overwirehq-claude-code-telegram.html) — 54 findings at `medium+` (1 critical, 21 high, 30 medium), **0 first-party defects — dependency currency, post-only** — a **Telegram bot that gives authorised users remote access to Claude Code**, i.e. it runs file and Bash tools on a host machine by design, so the whole security surface *is* the boundary around that execution (2k★, MIT, org-backed; active — 15 merged PRs from 7 authors and 9 closed issues in 60 days; strict-norm: real `SECURITY.md`, PVR enabled). Picked with the manual queue at zero. **The story is what precise security documentation looks like.** The real boundary is the `can_use_tool` callback that stops Claude — when steered off-course by content it reads mid-task — from acting outside the approved directory, and the maintainers know and *state* exactly what it covers: path validation is scoped in the docs to precisely six tools (`Read`, `Write`, `Edit`, `MultiEdit`, `NotebookEdit`, `NotebookRead`), while `Grep`/`Glob`/`LS` are deliberately left to the OS sandbox. Reading that as "Read is guarded but Grep is not, so the boundary is incomplete" is the plausible-but-wrong finding this site exists to resist — the boundary is [advertised, not accidental](scans/tracecathq-tracecat.html), and `ROADMAP-v2.md` even records the SDK's own `CanUseToolShadowedWarning` naming every tool the callback will never see, filed as tracking item #221. Two more choices earn credit: guarded tools are deliberately *stripped from* the SDK `allowed_tools` list (a pre-approved tool never produces a `can_use_tool` request, so leaving them in would render the checks silently inert — [issue #219](https://github.com/overwirehq/claude-code-telegram/issues/219)), and `autoAllowBashIfSandboxed` is disabled whenever the boundary checks must run, closing a second bypass. That is a maintainer who traced how the framework resolves permissions rather than trusting that a config allow-list is an enforcement boundary — it isn't, and the code says so. **When the machine is built and documented this carefully, the finding moves to the dependency manifest.** The 30 Trivy advisories in `poetry.lock` split cleanly by reachability: **17 are opt-in-only and not reachable on a default install** — `starlette` (6) and `python-multipart` (3) need the FastAPI webhook server (`ENABLE_API_SERVER=false` default); `pyjwt` (5) needs token auth (`ENABLE_TOKEN_AUTH=false` default, and the SECURITY.md documents token auth as non-functional, [#58](https://github.com/overwirehq/claude-code-telegram/issues/58)); the MCP SDK highs (3) need MCP enabled — while **13 are unconditionally installed** (`anyio` incl. the critical IDNA/TLS advisory, the `cryptography`/OpenSSL cluster, `urllib3`, `idna`, `requests`, `pydantic-settings`, `python-dotenv`) and are the genuine currency gap. The direct deps are current; the drift is transitive, which the repo's existing `.github/dependabot.yml` (configured for *version* updates, not *security* updates) will not raise on its own. **No advisory filed** — there is no first-party vulnerability. Of the other 24: 16 `github-actions-mutable-action-tag` (SHA-pin hardening); one `pull_request_target` checkout wrapped in a 40-line threat-model header (read-only tool allowlist, secrets scrubbed, "worst case is a prompt-injected review comment" — [the trigger decides severity](scans/lightseekorg-tokenspeed.html)); two `sqlalchemy-execute-raw-query` [identifier FPs](scans/aurelio-labs-semantic-router.html) (only a generated run of `?` placeholders is interpolated, values bound via `execute(query, params)`); and three gitleaks hits that are a `your-api-secret` placeholder plus two fake tokens in a test asserting the project's own `_redact_secrets()` helper scrubs them ([credited defence](scans/realiti4-claude-swap.html)). Coverage `partial` and honestly so: pip-audit resolved **no dependencies** from a `pyproject.toml` declaring `dynamic = ["dependencies"]` — reading identically to "clean" — and Trivy's `poetry.lock` read carried the run, the [same dependency blind spot](scans/harnessrouter-harnessrouter.html) as yesterday reached by a different manifest shape.
Expand Down
Loading
Loading