Skip to content

ref(spans): Run process-segments as a task in devenv - #122555

Merged
untitaker merged 1 commit into
masterfrom
markusunterwaditzer/stream-1568-deploy-process-segments-tasks-in-devenv-and-self-hosted
Aug 25, 2026
Merged

ref(spans): Run process-segments as a task in devenv#122555
untitaker merged 1 commit into
masterfrom
markusunterwaditzer/stream-1568-deploy-process-segments-tasks-in-devenv-and-self-hosted

Conversation

@untitaker

@untitaker untitaker commented Aug 25, 2026

Copy link
Copy Markdown
Member

Segments are already processed by the process_segment task in production.
This flips the default of spans.buffer.process-segments-task-rollout-rate to
1.0 and drops the dedicated process-segments Kafka consumer from devservices
and the devserver, so devenv matches production.

The flusher spawns the task directly, so no raw-mode taskbroker topic is
involved.

Self-hosted counterpart: getsentry/self-hosted#4488

ref STREAM-1568

Segments are already processed by the `process_segment` task in production.
Flip the default of `spans.buffer.process-segments-task-rollout-rate` to 1.0
and drop the dedicated `process-segments` Kafka consumer from devservices and
the devserver, so devenv matches production.

ref STREAM-1568
@linear-code

linear-code Bot commented Aug 25, 2026

Copy link
Copy Markdown

STREAM-1568

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 25, 2026
@untitaker
untitaker marked this pull request as ready for review August 25, 2026 08:43
@untitaker
untitaker requested review from a team as code owners August 25, 2026 08:43
"spans.buffer.process-segments-task-rollout-rate",
type=Float,
default=0.0,
default=1.0,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: If apply_async_with_future returns None, span segments are silently dropped because there is no fallback to the Kafka-based processing path.
Severity: MEDIUM

Suggested Fix

When task_produce_future is None after calling apply_async_with_future, either log an error to make the failure visible or implement a fallback to the Kafka-based produce call. This ensures segments are not silently lost when the task cannot be queued.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/sentry/options/defaults.py#L3308

Potential issue: In `src/sentry/spans/consumers/process/flusher.py`, when processing
span segments, the code attempts to use a task-based approach if the
`spans.buffer.process-segments-task-rollout-rate` option is enabled. If the call to
`apply_async_with_future` returns `None`, the corresponding segment is silently dropped.
There is no fallback to the Kafka-based processing mechanism and no error is logged,
leading to silent data loss. This change makes the task-based approach the default in
development environments, increasing the risk of this issue occurring if the task broker
is misconfigured or unavailable.

Also affects:

  • devservices/config.yml:157~162
  • src/sentry/runner/commands/devserver.py:336~336

Did we get this right? 👍 / 👎 to inform future reviews.

@untitaker
untitaker merged commit 171fed1 into master Aug 25, 2026
90 of 91 checks passed
@untitaker
untitaker deleted the markusunterwaditzer/stream-1568-deploy-process-segments-tasks-in-devenv-and-self-hosted branch August 25, 2026 13:43
untitaker added a commit that referenced this pull request Aug 27, 2026
Segments have been processed by the `process_segment` task in production
for a while, and `spans.buffer.process-segments-task-rollout-rate` has
defaulted to 1.0 since #122555. This removes the now-dead Arroyo
consumer and everything that only existed to feed it:

- the `process-segments` consumer and its strategy factory
- `spans.buffer.process-segments-task-rollout-rate`, and the flusher
branch that produced to Kafka instead of spawning the task
- `MultiProducer` and the `buffered-segments` topics
- `skip_produce`, which was only set by the consumer's `--skip-produce`
flag, and `start_new_transaction`, which the task always passed as
`False`

The segment processing helpers that `factory.py` shared with the task
move to `tasks.py`, alongside their only remaining caller.

`test_buffer.py` still validates the flushed payload against the
`buffered-segments` schema, since that schema describes what the flusher
hands to the task. It looks the codec up by name now that the topic has
no enum entry.

The ops-side shutdown is getsentry/ops#22790.

ref STREAM-1590
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants