Skip to content

feat: Exporter 12x faster hopefully#5691

Open
another-rex wants to merge 5 commits into
google:masterfrom
another-rex:exporter-zoom
Open

feat: Exporter 12x faster hopefully#5691
another-rex wants to merge 5 commits into
google:masterfrom
another-rex:exporter-zoom

Conversation

@another-rex

Copy link
Copy Markdown
Contributor

Made a few changes to vastly increase the speed of the exporter.

  • Increased each channel buffer up to 100 from 0.
  • 5x ed the worker count from 600 to 3000
  • Added a fast object list GCS library (mostly copied over from vulnfeeds), and split up the initial object listing loop at various places.
    • This is mostly guesswork at the moment, ideally we should do a review of where to roughly evenly split the work every now and then. But good enough for now.
    • This does mean the exporter will slow down near the end.
  • Refactored message receiving to use the for syntax. This should be a slight behavior change where we won't cancel based on the context, but only when the channel is closed. (Or when the context is cancelled on the sending side, just never on the receiving side)
  • Fixed README typos
  • Use logger.ErrorContext instead of logger.Error

@another-rex
another-rex requested a review from michaelkedar July 20, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant