minor: Emit taskStatus on task/run/time from the statsd and prometheus emitters - #20122
Open
amaechler wants to merge 1 commit into
Open
minor: Emit taskStatus on task/run/time from the statsd and prometheus emitters#20122amaechler wants to merge 1 commit into
taskStatus on task/run/time from the statsd and prometheus emitters#20122amaechler wants to merge 1 commit into
Conversation
task/run/time is emitted on every task completion with a taskStatus dimension of SUCCESS or FAILED, set by IndexTaskUtils.setTaskStatusDimensions from both call sites (TaskQueue.emitTaskCompletionLogsAndMetrics and UnusedSegmentsKiller). docs/operations/metrics.md documents it. Both emitters filter user dimensions through a per-metric allow-list, and neither listed taskStatus, so the dimension was dropped on the way out and the metric could not distinguish a successful task from a failed one. This is the dimension that makes "count task/run/time grouped by taskType and taskStatus" work, which is the usual recommendation for alerting on failures of one task type while ignoring another. taskId, groupId and description are documented on this metric too and stay filtered out: the first two are unbounded and description is a truncated error message. PrometheusEmitterTest.testEmitterMetric asserted the exact label set for this metric and is updated for the added label.
amaechler
changed the title
taskStatus on task/run/time from the statsd and prometheus emitterstaskStatus on task/run/time from the statsd and prometheus emitters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds
taskStatustotask/run/timein the Prometheus emitter, which will allow to easily group run times by success status (and alert on outliers).taskStatustakes at most two values here (SUCCESS,FAILED), so this at most doubles the cardinality.Release note
The statsd and prometheus emitters now emit the
taskStatusdimension ontask/run/timeThis makes it possible to count task successes and failures per task type through these emitters.This is a breaking change for existing dashboards and alerts on
task/run/time:druid.overlord.task.run.time.<dataSource>.<taskType>becomesdruid.overlord.task.run.time.<dataSource>.<taskStatus>.<taskType>. Queries against the old name stop receiving data.taskStatustag is added; existing queries keep working, aggregating over it.taskStatuslabel is added. Queries selecting on an exact label set, and recording rules assuming the old set, need updating.Operators wanting the previous behaviour can pin the old dimension list with
druid.emitter.statsd.dimensionMapPathordruid.emitter.prometheus.dimensionMapPath.This PR has: