ref(spans): Remove the process-segments consumer - #122725
Merged
untitaker merged 6 commits intoAug 27, 2026
Merged
Conversation
Segments have been processed by the `process_segment` task in production for a while, and the rollout option has defaulted to 1.0 since #122555. This removes the now-dead Arroyo consumer and everything that only existed to feed it: - the `process-segments` consumer and its strategy factory - `spans.buffer.process-segments-task-rollout-rate`, and the flusher branch that produced to Kafka instead of spawning the task - `MultiProducer` and the `buffered-segments` DLQ topic - the span dedupe options (`dedupe-ttl`, `dedupe-filter-enable`) and `SENTRY_SPAN_DEDUPE_CLUSTER`, which only ran inside the consumer - `skip_produce`, which was only set by the consumer's `--skip-produce` flag The `buffered-segments` topic definition stays: its schema still validates the payload the flusher hands to the task. ref STREAM-1590
Nothing produces to or consumes that topic anymore. The schema is still used to validate the payload the flusher hands to process_segment_task, so that one call site now looks the schema up by name instead.
The dedupe code lived in process_segments/factory.py, but that module held both the Arroyo consumer and the shared segment processing helpers that tasks.py imported. Removing the consumer dropped the `_check_span_duplicates` call from `_process_segment_bytes` along with it, which would have disabled dedupe on the task path -- the only path left. `dedupe-filter-enable` is true in all regions with a ttl of 120-240s, and the duplicate_span metric shows it filtering ~1.25B spans a week, so this would have let all of those through to EAP. Restores `_check_span_duplicates`, `get_dedupe_redis_client`, the two dedupe options and SENTRY_SPAN_DEDUPE_CLUSTER, now living in tasks.py next to their only caller. The dedupe tests only covered `_check_span_duplicates` directly, so they would not have caught this. test_tasks.py now also asserts that process_segment_task actually runs spans through dedupe before producing.
enochtangg
approved these changes
Aug 26, 2026
…e-rollout-option-and-remove-old-consumer-code
github-actions
Bot
added
the
Scope: Backend
getsentry's test_topic_definition asserts that every Topic enum member has a KAFKA_TOPIC_TO_CLUSTER entry, so dropping the enum broke its CI with KeyError: 'buffered-segments'. Nothing produces to or consumes the topic anymore, but it still needs to be registered, and the schema is still what test_buffer.py validates the flusher's payload against. Reverts the enum removal and restores the cluster mapping.
…cleanup-the-rollout-option-and-remove-old-consumer-code' into markusunterwaditzer/stream-1590-cleanup-the-rollout-option-and-remove-old-consumer-code
untitaker
deleted the
markusunterwaditzer/stream-1590-cleanup-the-rollout-option-and-remove-old-consumer-code
branch
August 27, 2026 15:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Segments have been processed by the
process_segmenttask in production for a while, andspans.buffer.process-segments-task-rollout-ratehas defaulted to 1.0 since #122555. This removes the now-dead Arroyo consumer and everything that only existed to feed it:process-segmentsconsumer and its strategy factoryspans.buffer.process-segments-task-rollout-rate, and the flusher branch that produced to Kafka instead of spawning the taskMultiProducerand thebuffered-segmentstopicsskip_produce, which was only set by the consumer's--skip-produceflag, andstart_new_transaction, which the task always passed asFalseThe segment processing helpers that
factory.pyshared with the task move totasks.py, alongside their only remaining caller.test_buffer.pystill validates the flushed payload against thebuffered-segmentsschema, since that schema describes what the flusher hands to the task. It looks the codec up by name now that the topic has no enum entry.The ops-side shutdown is getsentry/ops#22790.
ref STREAM-1590