Stop traffic jobs from exhausting the GitHub token budget - #2
Merged
Conversation
Production piled up 657 failed SnapshotTraffic jobs, all with "github: rate limited", each burning its 3 attempts within seconds. Three compounding causes: - withTraffic enqueued a SnapshotTraffic job on every page view, with no dedup. Each job snapshots every cached repo (307 user + 114 org repos, two API calls each, ~842 requests), so a handful of page views spent the whole 5000/hour token budget. - SnapshotLogin kept walking the remaining ~420 repos after the first rate-limit error, spending ~45s per attempt firing requests that could only fail. GitHub warns this can get an integration banned. - The worker backs off 2s/4s/8s while GitHub's window resets in up to an hour, so the job was marked failed for good before the quota returned. Traffic refresh already runs on the hourly SnapshotTraffic cron, so the per-view enqueue was redundant: the request path now only fills repos that have no snapshot yet. A rate limit ends the run without marking it failed, leaving the next scheduled run to refresh it. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problema
Em produção (github.gestaobem.com) havia 657 jobs
SnapshotTrafficfailed, todos comgithub: rate limitedeattempts = 3/3— ou seja, falha definitiva. No dia 07/09 foram 576 jobs e zero concluídos.Causa raiz
withTrafficenfileirava um jobSnapshotTraffic{login}a cada requisição, sem dedup. Cada job percorre todos os repos em cache: 307 do usuário + 114 de 9 orgs = 421 repos × 2 chamadas (/traffic/views+/traffic/clones) ≈ 842 requests. Com 5.000/h, ~6 jobs esgotam a hora inteira.SnapshotLogincontinuava varrendo os ~420 repos depois do primeiro erro de rate limit (só guardava ofirst). Cada tentativa gastava ~45s disparando centenas de requests que só podiam falhar. A doc do GitHub avisa que insistir com a integração rate-limited pode resultar em ban.faileddefinitivo muito antes da quota voltar.Correção
SnapshotTraffic), que já existia. O path de request agora só preenche repos sem snapshot (fillMissingTraffic), o que já fazia.SnapshotLoginpara no primeiro erro em vez de continuar varrendo.Testes
Quatro testes, escritos antes da implementação (TDD):
TestLoader_User_DoesNotEnqueueTrafficPerView— 3 page views não enfileiram nada (antes: 3 jobs)TestLoader_SnapshotLogin_StopsOnFirstRateLimit— 25 repos rate-limited → 1 chamada (antes: 25)TestPerformSnapshotTraffic_RateLimitedIsSkipped— rate limit não falha o jobTestClient_*existentes seguem cobrindo o mapeamento 403/remaining=0 →ErrRateLimitedmake cilocal: race tests, golangci-lint (0 issues) e Prettier verdes.Observações
PerformRefreshCatalogtem o mesmo formato (erro → 3 tentativas em segundos), mas hoje não acumula falhas em produção; ficou fora do escopo.failedforam limpos manualmente em produção após o deploy.