Skip to content

feat: Log redaction, FIFO QOS for user vs API HTTP requests and http traffic accounting enrichment - #46

Merged
Zacgoose merged 4 commits into
mainfrom
dev
Oct 2, 2026
Merged

Zacgoose merged 4 commits into
mainfrom
dev

Conversation

@Zacgoose

@Zacgoose Zacgoose commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

This pull request introduces detailed endpoint-level accounting and interactive user traffic tracking to the egress logging and metering system. It also adds configurable log redaction for sensitive information in logs and console output. The changes enhance observability, enable more granular reporting, and improve privacy controls.

Egress metering enhancements:

  • Endpoint-level tracking is now included in all egress accounting: each response is attributed to its endpoint, and statistics are maintained per endpoint for both API clients and interactive users. [1] [2] [3] [4]
  • Interactive (signed-in user) traffic is tracked and reported separately, never counting against the API client cap, with its own partition in the accounting system. [1] [2] [3] [4] [5]
  • The egress ledger file format is migrated to version 3 to persist endpoint and interactive user breakdowns. [1] [2]

Log redaction improvements:

  • Added configuration options to FileLoggingSettings for masking UPNs and customer domains in logs, with an allow list for readable domains and environment variable override. [1] [2]
  • Integrated a redacting console logger that applies the same redaction logic to console output, ensuring sensitive information is masked unless explicitly disabled.

Other improvements:

  • Log lines are revealed (unredacted) when read via LogBridge, ensuring authorized access sees the original data.
  • Egress accounting now records the endpoint label for shed (429) requests, improving cap-hit diagnostics. [1] [2]

These changes provide more accurate and privacy-aware logging and metering, supporting better reporting and diagnostics for both API and interactive user traffic.

- HTTP worker checkout waits in two FIFO tiers; API clients and background cache refreshes only get a worker when no interactive request is waiting
- Queue timeout unchanged and shared by both tiers
- Keep both ends of each label and the public suffix readable; the hidden middle is encrypted deterministically (short HMAC tag seeding an AES-CTR keystream), so a value always masks to the same short token
- Instance key lives in the CraftInstanceKeys table, created on first start
- LogBridge reveals masked values before filtering, so search, exclude and regex work on the originals
- Platform hosts, tenant ids and GUIDs are left as is; CRAFT_LOG_REDACTION=false turns it off
…ser traffic

- record each response against its endpoint (bytes, requests, max size, cache hits, errors, shed) on daily client rows and instance 15-minute buckets
- handlers can sub-label traffic with an X-Craft-Endpoint response header, which is stripped before the response is sent
- signed-in users are tracked in an 'interactive' partition that never counts toward the cap
- instance daily request count now includes clients idle since an earlier bucket
@Zacgoose Zacgoose changed the title feat: Log redaction and http traffic accounting enrichment feat: Log redaction, FIFO QOS for user vs API HTTP requests and http traffic accounting enrichment Oct 2, 2026
@Zacgoose
Zacgoose merged commit ed26ff4 into main Oct 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants