Repository navigation
Conversation
fetchWithRateLimit returned a 403 as soon as x-ratelimit-remaining was above zero, before looking at retry-after. That is the shape of a GitHub secondary rate limit (too many concurrent requests or too many points per minute): 403 or 429, a retry-after header, and a primary budget that is still positive. Those answers were returned to the caller as if they were permission errors, so a burst of imports or a bulk extract run failed on the first secondary limit instead of waiting the seconds GitHub asked for. The function's own docstring already promised '403 + retry-after'. A 403 is now treated as a permission answer only when it has budget left and no retry-after. The wait follows GitHub's documented order: retry-after when present, the reset time when remaining is 0, otherwise one minute. Checked with a stubbed fetch: a 403 with retry-after: 1 and remaining 4999 followed by 200 now takes two attempts and returns 200 (one attempt and 403 before); a 403 with remaining 4999 and no retry-after still returns at once; a 403 with remaining 0 still waits for the reset.
…egex
parseGitHubPlugin built the per-directory matchers with
new RegExp(`^${prefix}rules/...`) where prefix is a sub-plugin dir taken
from marketplace.json plugins[].source or from the tree. Any regex
metacharacter in the dir name changed the pattern; a name like
plugins/c++ made RegExp throw "Nothing to repeat", a plain SyntaxError
that escaped parseGitHubPlugin instead of a GitHubParseError. A dot in a
dir name matched any character, and a leading ./ in source only worked
because . matched the literal dot.
pathUnder() escapes the prefix before building the RegExp, and
normalizeSubPluginDir() strips a leading ./ so that case keeps working
on purpose rather than by accident.
The scan capped the compressed download at 100 MB to bound what it materializes on the function's /tmp, but the tar it wrote and the tree it extracted were unbounded. gzip turns a few hundred KB of zeros into hundreds of MB, so a repository with large low-entropy files passed the download cap and filled the shared 500 MB tmpfs, and the ENOSPC counted as retryable, so the queue redelivered the same scan up to five times. The download is now decompressed in the pipeline under a second budget of 100 MB, and the file written to disk is the plain tar, so extraction cannot grow past it. Exceeding either budget is an UnscannableRepoError, which already maps to a suspicious verdict for manual review instead of a retry. tar 7.5.15, the version in bun.lock, has no decompression guard of its own; the ratio guard in later 7.5.x releases is relative and lets a 2 MB download unpack to 300 MB at 144:1, so an absolute cap is still needed.
…an tool getStuckScans() treated a pending or scanning plugin as stuck once its created_at was older than 15 minutes. The recover-stuck-scans cron, added to use "the same 15-min staleness threshold getStuckScans uses", and the MCP rescan_plugin guard both measure from updated_at, which the plugins_set_updated_at trigger bumps on every status write. So a re-scan of any plugin older than 15 minutes showed up in the admin Stuck tab and in list_moderation_queue scan_issues the moment it was queued, while the cron left it alone, and the rescan tool refused to re-queue it as in progress. The query now uses updated_at, so the three agree.
fetchGitHubFile and the relative logo URL interpolated tree paths into the raw URL as they came from the GitHub tree API, unencoded. A path with a # stopped at the fragment, a ? at the query, and a % was read as a broken escape, so a rule such as rules/c#.mdc fetched rules/c, got a 404, and was dropped from the import without a message. A sub-plugin directory with such a character lost its plugin.json the same way. Each path segment is now percent-encoded and the slashes between them kept, through one rawGitHubUrl helper used by both call sites. Plain paths produce the same URL as before.
…m every entry point
…queueing it again
…untrusted, and neutralise delimiters inside values
…gins instead of dropping the later one
|
@deanrie is attempting to deploy a commit to the Anysphere Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One branch with the nine scanner and GitHub-import fixes currently open as separate PRs, cherry-picked in order, plus one formatting commit. Each change keeps its own commit, so this can be reviewed as nine small diffs or landed as one. If a single PR is easier to review, this replaces #466, #467, #469, #470, #472, #473, #474, #475 and #476; if the separate PRs are preferred, close this one.
Included
fetchWithRateLimitretries GitHub secondary rate limits, not only primary onescloneRepocaps what the archive unpacks to (100 MB) before it reaches tmpfsgetStuckScansjudges a stuck scan byupdated_at, like the recover cron and the rescan tool; Stuck card shows last change#,?,%)rescanPluginrefuses while a scan is queued or running, from every entry pointrecover-stuck-scansresets a stalescanningrow without enqueueing a second messagebiome formaton two filter lines from #467The reasoning, probes and merge checks for each are in the original PR descriptions.
Checked on the combined branch
maine816606 without conflicts.src: no errors (the fivenoImgElementwarnings are pre-existing onmain).tsc --noEmit: the same single pre-existing error inscan.ts(platformonAgentOptions) as onmain.rescanPluginstate table and three clicks give one message; the recover cron tick resets thescanningrow and enqueues thependingone, second tick recovers; the monorepo import keeps six components; all seven injected strings sit inside UNTRUSTED blocks with balanced delimiters.git merge-treeclean againstmain.Not included: #449, #450, #457, #458 (features and the lint chore), which stay as they are.