Skip to content

Add topic reader received messages metric - #900

Open
NastyaKa wants to merge 1 commit into
ydb-platform:mainfrom
NastyaKa:base_reader_metrics
Open

NastyaKa wants to merge 1 commit into
ydb-platform:mainfrom
NastyaKa:base_reader_metrics

Conversation

@NastyaKa

Copy link
Copy Markdown

Pull request type

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

What is the current behavior?

The SDK does not expose a metric showing whether topic messages are being received into the local reader buffer. It is therefore difficult to distinguish a lack of incoming progress from application-side consumption or decoding problems.

Topic readers also do not expose a public option for assigning a reader name that can be used to distinguish metric series.

Issue Number: N/A

What is the new behavior?

This PR adds the ydb.topic.reader.received.messages counter.

The counter is incremented by the number of messages in a batch after the batch has been accepted into the SDK's local reader buffer and before message decoding. This allows decoding or application-consumption problems to be diagnosed by comparing the received-message rate with a delivered-message metric.

The metric has the following attributes:

  • endpoint
  • database
  • topic
  • consumer
  • reader.name

The partition attribute is intentionally excluded to keep the metric series stable when partition assignments change. For reads without a consumer, the consumer attribute is set to an empty string.

An optional reader_name argument is added to both synchronous and asynchronous topic reader APIs. A user-provided value is also forwarded in the topic read initialization request. If the value is omitted or empty, the SDK generates a process-local name in the form reader-N. The resolved name remains stable across reader reconnects.

The counter is registered in the OpenTelemetry metrics provider with the {message} unit. The metrics build-info token is updated to ydb-sdk-metrics/0.2.0.

Unit tests and observability documentation are updated accordingly.

Other information

The metric context belongs to the logical topic reader and is shared by all underlying stream readers created during reconnects.

No new dependencies are introduced.

@codecov

codecov Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.33333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.63%. Comparing base (aa3e16b) to head (f907321).

Files with missing lines Patch % Lines
ydb/_topic_reader/topic_reader.py 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #900      +/-   ##
==========================================
+ Coverage   82.61%   82.63%   +0.02%     
==========================================
  Files          99       99              
  Lines       12939    12968      +29     
  Branches     1268     1272       +4     
==========================================
+ Hits        10689    10716      +27     
- Misses       1798     1800       +2     
  Partials      452      452              
Flag Coverage Δ
integration 80.47% <83.33%> (+0.03%) ⬆️
unit 49.04% <76.66%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
ydb/_grpc/grpcwrapper/ydb_topic.py 86.37% <100.00%> (+0.04%) ⬆️
ydb/_topic_reader/topic_reader_asyncio.py 87.08% <100.00%> (+0.18%) ⬆️
ydb/observability/metrics.py 96.50% <100.00%> (+0.10%) ⬆️
ydb/opentelemetry/metrics_plugin.py 96.00% <ø> (ø)
ydb/topic.py 86.01% <ø> (ø)
ydb/_topic_reader/topic_reader.py 89.02% <50.00%> (-2.01%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant