[schmitch]

Fixes #709

Every instance counts the requests it answers by day, repository and client network. Once a minute it writes the counts to Forge's bucket as a Parquet file of its own. Admin → Traffic reads every instance's files back, so it shows all servers (or one) with no calls between them and no Redis. Clients are located with MaxMind databases (country and autonomous system), so you can compare where each repository's requests come from.

What is recorded

• One file per instance and day: traffic/v1/days/<date>/<instance>.parquet. Each row is a repository and client network, with country, ASN, AS organisation, requests and bytes.
• The client address is the one the login lockout resolves (the socket peer, or a trusted proxy's X-Forwarded-For). IPv6 is counted by its /64.
• The repository is noted by the package dispatcher and the image registry once the repository has been found. Requests to no repository (UI, APIs, sign-in) are counted under none.
• Filling them in. The outermost track middleware opens a request-scoped slot (forge_traffic::Slot), and the principal layer and the dispatchers fill it in. The request path does no I/O.
• Bounds. At most 100k keys per minute and 500k rows per day per instance; past that, new clients are counted as other.
• Client addresses off. The address is looked up in memory and dropped; rows keep repository, country and AS only.
• Shutdown. A graceful shutdown writes the last minute.

Geolocation (MaxMind)

• Where the databases live. traffic/v1/geoip/country.mmdb and asn.mmdb in the bucket. Every instance loads them, again when their ETag changes (checked every ten minutes). The maxminddb crate reads them: Country or City editions for countries, ASN or ISP editions for autonomous systems. A file of another edition is refused.
• Download.
  • An admin enters a MaxMind account ID and license key in the UI. The key is sealed as a secret forge_setting (traffic_maxmind_license_key, added to SECRET_SETTINGS, so DEK rotation reseals it).
  • One instance claims the download with a compare-and-swap on the state setting. It fetches GeoLite2-Country and GeoLite2-ASN (.tar.gz), checks them, and stores them in the bucket.
  • It downloads every 3 days, and waits 6 hours after an attempt, so failures are not retried hourly against MaxMind's daily limit.
  • traffic.maxmind_url can point at a mirror.
• By hand. Operators can also put the .mmdb files in the bucket themselves (air-gapped installs, GeoIP2 editions).
• Credit. The GeoLite credit is shown next to the countries and networks.

API and UI

• API:
  • GET /admin/traffic?range=today|7d|30d|90d|365d&instance=&repository=&network=&country=&asn=, or from and to. It answers the days, totals, top repositories, countries, ASes and clients. repository=- and country=- select none or unknown.
  • GET/PUT /admin/settings/traffic: enabled, client addresses, retention days (default 90).
  • GET/PUT/DELETE /admin/traffic/geoip and POST /admin/traffic/geoip/update. The license key is never answered back.
• Admin → Traffic (/manage/traffic):
  • range and instance filters;
  • stats: requests, data, clients, countries;
  • requests, data or clients by day, as an SVG chart or a table;
  • repositories, countries, networks (ASes) and top clients. Every row narrows the whole view; filters are removable chips kept in the URL.
• Settings (/manage/traffic/settings): recording, retention, and geolocation (account, download now, turn off after confirming).

Docs

• docs/deployment.md → Traffic statistics and Countries and networks (MaxMind).
• forge.toml.example → [traffic].
• Changelog fragment: docs/changes/2.2.0/709.feature.md.

Tests

• cargo test -p forge-traffic (14):
  • counting and flushing over days;
  • rows without addresses;
  • pruning;
  • Parquet round trip;
  • geo lookups from MMDB databases built by the crate's test writer (testing feature); wrong editions refused; refresh from the bucket;
  • overview totals and every filter over two instances.
• cargo test -p forge-server --test traffic (3, Postgres, versitygw and a wiremock MaxMind):
  • real requests are counted, flushed and read back with repository, client and filters; admins only;
  • settings validation;
  • the MaxMind account: a wrong key's 401 is reported, the key is sealed at rest, the right key downloads both .tar.gz databases, clients are located, and turning off removes everything.
• Server unit tests (254), forge-db --test encryption (13), and the server suites admin, dispatch, npm, raw, registry, telemetry and otlp_export all pass.
• pnpm tsc, pnpm test (529, including 6 for the traffic screens) and pnpm build.
• cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo xtask changelog --check and cargo xtask docs --check.
• Rendered in Chromium against mocked data in light, dark and phone widths, with no horizontal overflow.

🤖 Generated with Claude Code

https://claude.ai/code/session_012e6DJbpgZmQnhz5Jjjhug6

[schmitch]

CI: test failed on raw::groups_fall_through_their_members_by_priority (stack overflow). Fixed in 96b85cc.

The test's thread has 2 MiB of stack. In a debug build, the request that fetches a file through the group's proxy and stores it already needed about 1.95 MiB on main, measured with RUST_MIN_STACK (it passes at 2048 KiB and overflows at 1900 KiB). This PR adds a few KiB per request: the traffic slot in track. That was enough to overflow on the CI runner.

gdb at the overflow showed where the stack goes. The AWS SDK's upload future was held inline by every future up that request: put_if_absent, the CAS commit, the file store, the proxy fill, the group walk and the plugin dispatch. Each of their poll frames carried a copy (frames of 45–190 KB).

96b85cc makes BlobStore::put_if_absent and the conditional PUT return boxed futures. The SDK future now lives in one heap allocation, and the same test needs about 1.55 MiB: it passes at 1600 KiB and overflows at 1500 KiB. Nothing else changes.

The branch also has main merged in. On the merged code, raw, registry, npm, dispatch, admin, import, telemetry and traffic, the storage tests, clippy and the web tests all pass locally.

---

Generated by Claude Code