feat(libsy): expose algorithm metrics through driver by nachiketb-nvidia · Pull Request #307 · NVIDIA-NeMo/Switchyard

nachiketb-nvidia · 2026-08-05T17:45:03Z

What

Add typed Driver::record_counter and Driver::record_histogram APIs for algorithm-defined metrics.
Emit each metric once to OpenTelemetry and as RunObservation::AlgorithmMetric for host-side aggregation.
Move the existing classifier fail-open counter onto the Driver path.

Why

Algorithms need to publish domain-specific measurements without receiving or depending on a server accumulator. This keeps algorithms server-independent while allowing both exported telemetry and local consumers to derive the same rollups.

Implements SWITCH-1227.

How

The Driver captures the algorithm name once per run, adds it to every custom metric, records the metric through the global OpenTelemetry provider, and forwards the same owned event to the existing RunObserver. Metric names and attribute keys are static; attribute values must remain bounded and low-cardinality. Non-finite histogram samples are dropped from both outputs.

The Rust server intentionally ignores the new observation variant for now. Endpoint integration remains separate work.

Aggregate proof

One comprehensive integration test emits two batches through the same observer and reconstructs the accumulator-style summary from raw events:

counter total: 5
histogram: count 4, total 100, min 10, max 40, average 25, p50 30, p99 40
OpenTelemetry independently reports the same counter total and histogram count/sum
non-finite samples appear in neither output

This demonstrates that a host can derive the Python accumulator rollups without coupling algorithms to accumulator storage.

How tested

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
No live network calls

Checklist

New public Rust symbols are exported from switchyard-libsy
One comprehensive integration test covers the new contract
Commit is DCO signed off

Notes for reviewers

Please focus on the typed metric API, the guarantee that OpenTelemetry and RunObserver see the same event, and the intentional absence of server accumulator integration.

Summary by CodeRabbit

New Features
- Added support for algorithm-defined counters and histograms.
- Algorithm metrics now include bounded attributes and are available through observers and OpenTelemetry.
- Added public metric types and streaming support for integrations.
Bug Fixes
- Improved fail-open metric reporting for judge transport, response, and parsing failures.
- Invalid histogram samples no longer affect recorded aggregates.

Signed-off-by: nachiketb <nachiketb@nvidia.com>

github-actions · 2026-08-05T17:46:19Z

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-307/
Built to branch `gh-pages` at 2026-08-05 18:19 UTC. Preview will be ready when the GitHub Pages deployment is complete.

coderabbitai · 2026-08-05T17:48:26Z

Walkthrough

The PR adds algorithm-defined counter and histogram metrics. Drivers emit observations to OpenTelemetry and run observers. Judge fail-open reporting uses the new driver API. Tests validate aggregation, finite samples, attributes, and server handling.

Changes

Algorithm metrics

Layer / File(s)	Summary
Metric contracts and driver recording `crates/libsy/src/core/algorithm.rs`, `crates/libsy/src/lib.rs`	Defines metric observations and values. `Driver` records counters and finite histogram samples with the algorithm name. Public re-exports expose the metric types and `StepStream`.
Observability metric recording `crates/libsy/src/observability.rs`	Adds global-meter recording for algorithm counters and histograms. The algorithm and bounded metric attributes are included.
Judge fail-open metric migration `crates/libsy/src/algorithms/util/llm_judge.rs`	Routes judge failures through `Driver::record_counter` with `judge_model` and bounded `reason` attributes.
Metric validation and observation consumers `crates/libsy/tests/observability.rs`, `crates/switchyard-server/src/lib.rs`	Tests observer and OpenTelemetry aggregation, finite histogram handling, and metric attributes. The server statistics observer ignores algorithm metric events.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Poem

A rabbit counts hops in a metric stream,
Counters tick softly beside histograms’ dream.
NaN takes a nap, while valid values flow,
Observers and meters record what they know.
Judge failures leave bounded tracks in the snow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: exposing algorithm metrics through the libsy Driver API.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/libsy/src/core/algorithm.rs`:
- Around line 272-283: Update record_algorithm_metric to reserve the "algorithm"
attribute key by filtering or rejecting any caller-provided MetricAttribute with
that key before constructing AlgorithmMetricObservation, preserving the
self.algorithm value as the sole algorithm attribute. Add a regression test
covering a caller-supplied "algorithm" attribute and verifying it cannot produce
duplicate observer or sink attributes.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 264915bb-e85b-4361-842d-28f0244f0375

📥 Commits

Reviewing files that changed from the base of the PR and between 5e83e2b and 4d63173.

📒 Files selected for processing (6)

crates/libsy/src/algorithms/util/llm_judge.rs
crates/libsy/src/core/algorithm.rs
crates/libsy/src/lib.rs
crates/libsy/src/observability.rs
crates/libsy/tests/observability.rs
crates/switchyard-server/src/lib.rs

Signed-off-by: nachiketb <nachiketb@nvidia.com>

sabhatinas · 2026-08-06T15:32:14Z

+        ("exploring", dimensions.exploring),
+        ("production_intensity", dimensions.production_intensity),
+    ] {
+        driver.record_histogram(


This is great! We can stick to reporting raw scores. can you update the routing algorithms page for stage router where Im talking about decision sources ?

sabhatinas · 2026-08-06T15:32:48Z

+    driver.record_histogram(CONFIDENCE_METRIC, confidence, []);
+
+    let dimensions = dimensions_from_signal(signal);
+    for (name, value) in [


can we generalize this a bit? if we add a new signal, Ill have to manually update the strings here.

that's probably best practice since that would be a meaningful update, can probably pull this up to a const variable would that be preferable?

feat(libsy): expose algorithm metrics through driver

4d63173

Signed-off-by: nachiketb <nachiketb@nvidia.com>

nachiketb-nvidia requested a review from a team as a code owner August 5, 2026 17:45

coderabbitai Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread crates/libsy/src/core/algorithm.rs

nachiketb-nvidia added 2 commits August 5, 2026 11:13

feat(libsy): record stage router metrics

8bfccbe

Signed-off-by: nachiketb <nachiketb@nvidia.com>

fix(libsy): reserve algorithm metric attribute

e6b5018

Signed-off-by: nachiketb <nachiketb@nvidia.com>

sabhatinas reviewed Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(libsy): expose algorithm metrics through driver - #307

feat(libsy): expose algorithm metrics through driver#307
nachiketb-nvidia wants to merge 3 commits into
mainfrom
nachiketb/switch-1227-driver-algorithm-metrics

nachiketb-nvidia commented Aug 5, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

github-actions Bot commented Aug 5, 2026 •

edited

Loading

Built to branch `gh-pages` at 2026-08-05 18:19 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

sabhatinas Aug 6, 2026

Uh oh!

sabhatinas Aug 6, 2026

Uh oh!

nachiketb-nvidia Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nachiketb-nvidia commented Aug 5, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Aggregate proof

How tested

Checklist

Notes for reviewers

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Built to branch gh-pages at 2026-08-05 18:19 UTC. Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sabhatinas Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

sabhatinas Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

nachiketb-nvidia Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nachiketb-nvidia commented Aug 5, 2026 •

edited by coderabbitai Bot

Loading

github-actions Bot commented Aug 5, 2026 •

edited

Loading

Built to branch `gh-pages` at 2026-08-05 18:19 UTC.
Preview will be ready when the GitHub Pages deployment is complete.