test: add configurable NeMo Relay latency benchmark - #722
Conversation
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
github-actions
Bot
added
size:XL
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds an opt-in latency benchmark with validated configuration, deterministic local fixtures, Relay process measurements, exporter verification, JSON and HTML reports, command integration, documentation, and unit tests. ChangesLatency benchmark
Estimated code review effort: 4 (Complex) | ~60 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/maintain-coding-agent-benchmark/SKILL.md:
- Around line 13-15: Add complete lead-in sentences before each affected list:
introduce the layout list at
.agents/skills/maintain-coding-agent-benchmark/SKILL.md lines 13-15, the metrics
list at lines 135-137, and the integrity list at lines 154-156; replace “To add
a test suite:” with a complete procedural lead-in at lines 169-173; and
introduce the troubleshooting list in scripts/latency_benchmark/README.md lines
268-270. Ensure every sentence grammatically leads into its following list.
In `@docs/reference/performance.mdx`:
- Around line 39-41: Add a complete introductory sentence immediately before the
latency-benchmark command block, clearly explaining that the command runs the
benchmark. Keep the existing command unchanged.
In `@justfile`:
- Around line 1121-1131: Update the latency-benchmark recipe’s benchmark_args
handling to use the Bash + expansion guard when expanding the array, including
both the help-check loop and Python invocation, so empty arguments work under
set -u on Bash 3.2. Also quote the just interpolation used to initialize
benchmark_args so arguments containing whitespace remain single array elements.
In `@scripts/latency_benchmark/data/mock-codex.cmd`:
- Around line 1-12: Normalize scripts/latency_benchmark/data/mock-codex.cmd to
CRLF and add a .gitattributes rule for *.cmd text with CRLF checkout. In
scripts/latency_benchmark/src/fixtures.py, update the fixture-writing logic to
use newline="\r\n" for the Windows .cmd target and newline="\n" for the POSIX
target, ensuring materialized scripts have platform-correct line endings
regardless of checkout configuration.
In `@scripts/latency_benchmark/data/mock-codex.sh`:
- Line 9: Update scripts/latency_benchmark/data/mock-codex.sh lines 9-9 to write
NEMO_RELAY_GATEWAY_URL to a temporary file and atomically move it onto
BENCHMARK_GATEWAY_FILE. Update scripts/latency_benchmark/src/processes.py lines
169-177 to poll until the gateway file’s stripped content is non-empty, then
assign that stripped value to self.url.
In `@scripts/latency_benchmark/src/benchmarks.py`:
- Around line 120-147: Update the worker function’s exception handling around
warmup and barrier synchronization so any worker exception calls barrier.abort()
before cleanup and re-raises the original error. Ensure other workers blocked at
barrier.wait() receive BrokenBarrierError while preserving connection closure in
finally and allowing future.result() to surface the failure.
- Around line 299-310: Update the relay variant loop around RelayProcess to use
its context-manager interface, entering the process before startup timing and
ensuring cleanup through __exit__ even when startup raises or is interrupted.
Preserve the existing startup_ns measurement and cycle population behavior.
In `@scripts/latency_benchmark/src/fixtures.py`:
- Around line 90-103: Update isolated_environment to construct a minimal
environment containing only the variables required by the benchmark, rather than
copying os.environ. Preserve the HOME, XDG_CONFIG_HOME, XDG_DATA_HOME, and
NO_COLOR entries, and exclude inherited Relay, provider, proxy, and logging
variables such as NEMO_RELAY_*, API keys, HTTP(S)_PROXY, and RUST_LOG.
In `@scripts/latency_benchmark/src/report/report.js`:
- Around line 426-438: Remove the unused comparisons parameter from summaryRows
and update its call site to stop passing true. Keep summaryRows’s existing
row-generation behavior unchanged.
In `@scripts/latency_benchmark/src/report/styles.css`:
- Line 16: Add the required blank line before the font-family declaration in the
affected CSS rule, preserving the existing declaration and satisfying
stylelint’s declaration-empty-line-before rule.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 5fedce9c-9127-4903-ab37-4cb0ff5046ca
📒 Files selected for processing (33)
.agents/skills/maintain-coding-agent-benchmark/SKILL.md.agents/skills/maintain-coding-agent-benchmark/agents/openai.yamldocs/reference/performance.mdxjustfilescripts/README.mdscripts/latency_benchmark/README.mdscripts/latency_benchmark/config/agent-config.tomlscripts/latency_benchmark/config/default.tomlscripts/latency_benchmark/config/plugins-file.tomlscripts/latency_benchmark/config/plugins-minimal.tomlscripts/latency_benchmark/config/plugins-otlp.tomlscripts/latency_benchmark/config/plugins-pii-redaction.tomlscripts/latency_benchmark/config/relay-config.tomlscripts/latency_benchmark/data/mock-codex.cmdscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/report/styles.cssscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_processes.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (40)
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
**/*: Use release tags in raw Rust-compatible SemVer without a leadingv; tags such asv0.1.0are prohibited.
Use branch prefixesfeat/,fix/,docs/,test/, orrefactor/according to the change purpose.
Every commit in a pull request must include a DCOSigned-off-by:sign-off.
Before submitting a pull request, ensure pre-commit hooks, relevant tests, target-specific builds, documentation updates, and a rebase on the latestmainare complete.
Use commit messages in the formtype: short description, with a valid type and a first line under 72 characters.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/config/plugins-minimal.tomlscripts/latency_benchmark/config/agent-config.tomlscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/config/relay-config.tomlscripts/latency_benchmark/config/plugins-pii-redaction.tomlscripts/latency_benchmark/config/plugins-file.tomlscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/data/mock-codex.cmdscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/src/html_report.pyscripts/README.mdscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/config/default.tomlscripts/latency_benchmark/config/plugins-otlp.tomlscripts/latency_benchmark/src/__main__.pyjustfilescripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pydocs/reference/performance.mdxscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/styles.cssscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolveheader_envvalues at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests andjust test-rustwhen event fields change; runjust test-python,just test-go, andjust test-nodewhen binding-native configuration or lifecycle changes.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{md,mdx,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Examples and documentation must use each exporter's documented flush/deregister order before shutdown.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/README.mdscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pydocs/reference/performance.mdxscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{rs,py,js,jsx,ts,tsx,go,c,h,cc,cpp,md,toml,yml,yaml,sh}
📄 CodeRabbit inference engine (AGENTS.md)
Keep SPDX headers on source, documentation, scripts, and configuration files; the project is Apache-2.0.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/config/plugins-minimal.tomlscripts/latency_benchmark/config/agent-config.tomlscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/config/relay-config.tomlscripts/latency_benchmark/config/plugins-pii-redaction.tomlscripts/latency_benchmark/config/plugins-file.tomlscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/README.mdscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/config/default.tomlscripts/latency_benchmark/config/plugins-otlp.tomlscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Use
snake_casenaming in Rust and Python.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{rs,py,js,mjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Preserve the existing Tokio-based asynchronous model and callback/future lifetimes; do not unexpectedly block or hide async work in bindings.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Lint Python with Ruff using rule setsE,F,W, andI.
Format Python with the Ruff formatter, using a 120-character line length and double quotes.
Runtyfor Python type checking.
Use Pythonsnake_casenaming conventions.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}: Run tests for every language affected by a change; changes to the core Rust crate require tests across all bindings.
UseSONAR_IGNORE_START/SONAR_IGNORE_ENDonly for documented false positives, keep ignored blocks minimal, explain them with a comment, and obtain reviewer sign-off.
Preserve the layered architecture in which Rust provides the core runtime and C FFI, PyO3, and NAPI provide bindings that mirror the full API surface.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/{test,tests}/**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When adding functionality, include tests in the appropriate test files for each affected language binding.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.py
**/*.{rs,py,go,js,jsx,ts,tsx,c,h,html,md,mdx,toml}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Include the appropriate SPDX copyright and Apache-2.0 license header in every source file.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/config/plugins-minimal.tomlscripts/latency_benchmark/config/agent-config.tomlscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/config/relay-config.tomlscripts/latency_benchmark/config/plugins-pii-redaction.tomlscripts/latency_benchmark/config/plugins-file.tomlscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/src/html_report.pyscripts/README.mdscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/config/default.tomlscripts/latency_benchmark/config/plugins-otlp.tomlscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pydocs/reference/performance.mdxscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
scripts/latency_benchmark/**/*.py
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
scripts/latency_benchmark/**/*.py: Keep benchmark runtime Python modules underscripts/latency_benchmark/src/; place configuration parsing and validation insrc/config.py, protocol payloads insrc/protocol.py, server behavior insrc/servers.py, process lifecycle insrc/processes.py, measurements insrc/benchmarks.py, orchestration insrc/cli.py, reporting insrc/reporting.py, and HTML assembly insrc/html_report.py.
Keep providers loopback and deterministic; do not add model-service or Internet latency to the core benchmark fixture.
Compare benchmark variants within the same measurement cycle, retain rotated or randomized execution order, and warm persistent connections before recording gateway samples.
Keep the three default benchmark variants enabled when adding custom middleware, and include custom gateway variants in selected hook and startup suites.
Interpret hook absolute values as completehook-forwardsubprocess wall time, with paired deltas subtracting anemo-relay --versionprocess measurement from the same cycle; interpret startup values as launch through healthy gateway readiness with the same baseline subtraction.
When adding a test suite, register it inAVAILABLE_TESTS, implement measurement logic, conditionally dispatch it, update terminal and HTML reporting, and add configuration and selection tests.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
scripts/latency_benchmark/tests/**/*.py
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
scripts/latency_benchmark/tests/**/*.py: When adding a test suite, provider, mode, or matrix axis, update configuration validation, fixtures, orchestration, result parameters, and corresponding tests together.
Run focused configuration, HTML-report, and process lifecycle tests after relevant benchmark changes.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.py
{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}
⚙️ CodeRabbit configuration file
{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}: Review automation changes for reproducibility, pinned versions where appropriate, secret handling, and consistency with the documented validation matrix.
Pay attention to commands that need generated native artifacts, FFI libraries, or platform-specific environment variables.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/config/plugins-minimal.tomlscripts/latency_benchmark/config/agent-config.tomlscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/config/relay-config.tomlscripts/latency_benchmark/config/plugins-pii-redaction.tomlscripts/latency_benchmark/config/plugins-file.tomlscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/data/mock-codex.cmdscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/src/html_report.pyscripts/README.mdscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/config/default.tomlscripts/latency_benchmark/config/plugins-otlp.tomlscripts/latency_benchmark/src/__main__.pyjustfilescripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/report/styles.cssscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
**/*.{yaml,yml,toml,json}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Maintain valid YAML, TOML, and JSON files; pre-commit hooks validate these formats.
Files:
scripts/latency_benchmark/config/plugins-minimal.tomlscripts/latency_benchmark/config/agent-config.tomlscripts/latency_benchmark/config/relay-config.tomlscripts/latency_benchmark/config/plugins-pii-redaction.tomlscripts/latency_benchmark/config/plugins-file.tomlscripts/latency_benchmark/config/default.tomlscripts/latency_benchmark/config/plugins-otlp.toml
scripts/latency_benchmark/config/**/*.toml
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
Store TOML benchmark and plugin assets under
config/, and load or render them throughsrc/fixtures.pyrather than embedding them in executable modules.
Files:
scripts/latency_benchmark/config/plugins-minimal.tomlscripts/latency_benchmark/config/agent-config.tomlscripts/latency_benchmark/config/relay-config.tomlscripts/latency_benchmark/config/plugins-pii-redaction.tomlscripts/latency_benchmark/config/plugins-file.tomlscripts/latency_benchmark/config/default.tomlscripts/latency_benchmark/config/plugins-otlp.toml
scripts/latency_benchmark/src/**/*.py
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
scripts/latency_benchmark/src/**/*.py: Keep temporary benchmark state isolated from the developer's home directory and Relay configuration.
Usepython -m scripts.latency_benchmark.srcas the direct benchmark entry point, withjust latency-benchmarkas the normal wrapper.
For paired latency results, use p50 as the median, p95/p99 as tail percentiles, andmedian_ci95_msas a bootstrap uncertainty interval for the median paired delta rather than an interval containing 95% of observations.
Files:
scripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/html_report.pyscripts/latency_benchmark/src/servers.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/protocol.pyscripts/latency_benchmark/src/config.pyscripts/latency_benchmark/src/benchmarks.py
scripts/latency_benchmark/data/**/*
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
Store platform scripts and static lifecycle fixtures under
data/; treatdata/mock-codex.*as the transparent-mode lifecycle stub, not as the Codex hook implementation.
Files:
scripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/data/mock-codex.cmd
**/*.{md,rst,html,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
**/*.{md,rst,html,txt}: Always spellNVIDIAin all caps. Do not useNvidia,nvidia,nVidia,nVIDIA, orNV.
Usean NVIDIAbefore a noun because the name starts with an 'en' sound.
Do not add a registered trademark symbol afterNVIDIAwhen referring to the company.
Use trademark symbols with product names only when the document type or legal guidance requires them.
Verify official capitalization, spacing, and hyphenation for product names.
Precede NVIDIA product names withNVIDIAon first mention when it is natural and accurate.
Do not rewrite product names for grammar or title-case rules.
Preserve third-party product names according to the owner's spelling.
Include the company name and full model qualifier on first use when it helps identify the model.
Preserve the official capitalization and punctuation of model names.
Use shorter family names only after the full name is established.
Spell out a term on first use and put the acronym in parentheses unless the acronym is widely understood by the intended audience.
Use the acronym on later mentions after it has been defined.
For long documents, reintroduce the full term if readers might lose context.
Form plurals of acronyms withs, not an apostrophe, such asGPUs.
In headings, common acronyms can remain abbreviated. Spell out the term in the first or second sentence of the body.
Common terms such asCPU,GPU,PC,API, andUIusually do not need to be spelled out for developer audiences.
Files:
scripts/latency_benchmark/src/report/template.htmlscripts/README.mdscripts/latency_benchmark/README.md
**/*.{md,rst,html}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
Link the first mention of a product name when the destination helps the reader.
Files:
scripts/latency_benchmark/src/report/template.htmlscripts/README.mdscripts/latency_benchmark/README.md
scripts/latency_benchmark/src/report/**/*
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
Keep the HTML benchmark report self-contained and offline, with static template, CSS, and JavaScript files under
scripts/latency_benchmark/src/report/.
Files:
scripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/src/report/styles.cssscripts/latency_benchmark/src/report/report.js
**/*.{md,rst,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
Spell
NVIDIAin all caps. Do not useNvidia,nvidia, orNV.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/*.{md,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
**/*.{md,rst}: Format commands, code elements, expressions, package names, file names, and paths as inline code.
Use descriptive link text. Avoid raw URLs and weak anchors such as "here" or "read more."
Use title case consistently for technical documentation headings.
Introduce code blocks, lists, tables, and images with complete sentences.
Write procedures as imperative steps. Keep steps parallel and split long procedures into smaller tasks.
Prefer active voice, present tense, short sentences, contractions, and plain English.
Usecanfor possibility and reservemayfor permission.
Useafterfor temporal relationships instead ofonce.
Preferrefer tooverseewhen the wording points readers to another resource.
Avoid culture-specific idioms, unnecessary Latinisms, jokes, and marketing exaggeration in technical docs.
Spell out months in body text, avoid ordinal dates, and use clear time zones.
Spell out whole numbers from zero through nine unless they are technical values, parameters, versions, or UI values.
Use numerals for 10 or greater and include commas in thousands.
Do not add trademark symbols to learning-oriented docs unless the source, platform, or legal guidance explicitly requires them.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/*.{md,mdx}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If links in documentation change, run
just docs-linkcheck.Use documented public APIs and stable wrapper commands in examples and user-facing documentation; do not rely on internal helpers.
**/*.{md,mdx}: Prefer the documented public API over internal shortcuts in documentation and examples.
Keep package names, repository references, and build commands current.
Contribution workflow documentation must require an issue before external contribution pull requests and note that NVIDIA contributors may use a GitHub or Linear issue.
Update entry-point documentation when examples or reading paths change.
Keep release-process and release-notes guidance in maintainer documentation such asRELEASING.md, rather than user-facing documentation pages orCHANGELOG.md.
Use stable user-facing wrappers at thescripts/root in documentation and examples; reference namespaced helper paths only for internal maintenance documentation.
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pages.
Dynamic plugin manifests in documentation and examples should usecompat.relay = ">=0.5,<1.0"unless deliberately narrower.
Render images, diagrams, tables, and other visual content at representative page widths, ensuring legibility and complete access without clipping; use responsive scaling, reflow, or overflow as appropriate and scope visual styling narrowly.
Dynamic plugin entry pages should link to native, worker, Rust example, Python example, and protocol pages when those pages exist.
Images, diagrams, tables, and custom visual content must remain legible and fully accessible at representative desktop and narrow page widths.
Release-policy documentation must point to GitHub Releases as the only release-history source of truth.
Runjust docswhen the documentation site changes; retain./scripts/build-docs.sh htmlas the compatibility wrapper.
Files:
scripts/README.mdscripts/latency_benchmark/README.mddocs/reference/performance.mdx
scripts/**/*.md
📄 CodeRabbit inference engine (AGENTS.md)
Use stable public wrappers at the
scripts/root in documentation and examples; reference namespaced helper paths only for internal maintenance documentation.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/README.md
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update the relevant crate or package README when that package surface changes.
Update relevant package or crate
README.mdfiles when examples or binding guidance changes.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/{README.md,docs/index.md}
📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)
Update
README.mdordocs/index.mdwhen documentation entry points change.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/*.{md,mdx,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-technical-docs.md)
**/*.{md,mdx,rst}: Use title case consistently for technical documentation headings and table headers; avoid quotation marks, ampersands, and exclamation marks in headings, while preserving official product, event, research, and whitepaper title case.
Format code elements, commands, parameters, package names, expressions, directories, file names, and paths in monospace; represent path placeholders with angle brackets inside monospace.
Format UI buttons, menus, fields, and labels in bold, and separate consecutive UI navigation labels with>.
Use quotation marks for error messages and strings when appropriate, italics for newly introduced terms and publication titles, and plain text for keyboard shortcuts.
Represent GitHub repositories with owner/repository link text, such as[NVIDIA/NeMo](link), rather than generic repository wording.
Introduce every code block with a complete sentence; do not let a code block complete or interrupt the grammar of surrounding prose; use syntax highlighting when supported.
Keep inline method, function, and class references consistent with nearby documentation; omit empty parentheses in prose when no call is shown.
Use descriptive link text matching the destination title when possible; avoid raw URLs, generic anchors, long-sentence links, and unnecessary links that distract from procedures.
Ensure lists have a complete lead-in sentence, more than one item, no more than two levels, parallel construction, one idea or action per item, and appropriate punctuation; use bullets for unordered items and numbers for ordered tasks.
Format definition lists with a bold term followed by a complete, parallel, punctuated definition.
Use tables for reference information, decision support, compatibility matrices, and comparable choices; flag one-row tables, missing captions or lead-ins, sentence-case headers where title case is expected, unexplained empty cells, and code or links that would be clearer as prose.
Write procedure steps as imperative ...
Files:
scripts/README.mdscripts/latency_benchmark/README.mddocs/reference/performance.mdx
{justfile,codecov.yml,codecov.yaml,.github/workflows/**/*.yml,.github/workflows/**/*.yaml}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
justfile, Codecov, and CI package/test workflows must include new plugin crates and packages.
Files:
justfile
justfile
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Keep
justfilebuild, test, clean, version, and package recipes for plugin crates and packages aligned with the current packaging layout.
Files:
justfile
scripts/latency_benchmark/src/reporting.py
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
Do not hard-code custom middleware names in reporting; discover and report configured middleware variants dynamically.
Files:
scripts/latency_benchmark/src/reporting.py
scripts/latency_benchmark/README.md
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
Keep the human benchmark run guide current with CLI and configuration changes.
Files:
scripts/latency_benchmark/README.md
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
**/*.mdx: In MDX files, top-of-file comments must use JSX comment delimiters ({/*and*/}); do not use HTML comments for MDX SPDX headers.
New or regenerated MDX files must use{/* ... */}for top-of-file SPDX comments.
Files:
docs/reference/performance.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/reference/performance.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/reference/performance.mdx
docs/**/*.{md,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update relevant reference documentation when public behavior or APIs change.
Files:
docs/reference/performance.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/reference/performance.mdx
**/*.{js,jsx,ts,tsx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use Node.js
camelCasenaming conventions.
Files:
scripts/latency_benchmark/src/report/report.js
scripts/latency_benchmark/src/config.py
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
Validate custom middleware names and plugin paths before starting subprocesses; resolve TOML plugin paths relative to the benchmark configuration.
Files:
scripts/latency_benchmark/src/config.py
scripts/latency_benchmark/src/benchmarks.py
📄 CodeRabbit inference engine (.agents/skills/maintain-coding-agent-benchmark/SKILL.md)
scripts/latency_benchmark/src/benchmarks.py: Keep streaming time-to-first-content separate from total stream time; gatewaytotalmust cover request start through buffered-body completion or streaming end-of-stream.
Preserve exporter-delivery checks when measuring gateway or hook traffic, and treat exporter-delivery bytes and request counts as correctness checks rather than latency metrics.
Files:
scripts/latency_benchmark/src/benchmarks.py
🪛 ast-grep (0.45.0)
scripts/latency_benchmark/src/html_report.py
[info] 20-20: use jsonify instead of json.dumps for JSON output
Context: json.dumps(results, separators=(",", ":"), sort_keys=True)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
scripts/latency_benchmark/src/servers.py
[info] 55-55: use jsonify instead of json.dumps for JSON output
Context: json.dumps(event, separators=(',', ':'))
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 69-69: use jsonify instead of json.dumps for JSON output
Context: json.dumps(event, separators=(',', ':'))
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 86-86: use jsonify instead of json.dumps for JSON output
Context: json.dumps(value, separators=(",", ":"))
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
scripts/latency_benchmark/tests/test_html_report.py
[info] 64-64: Do not hardcode temporary file or directory names
Context: "/tmp/plugins.toml"
Note: [CWE-377] Insecure Temporary File.
(hardcoded-tmp-file)
scripts/latency_benchmark/src/reporting.py
[error] 17-17: Command coming from incoming request
Context: subprocess.run(["git", *args], cwd=ROOT, capture_output=True, text=True, check=False)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
[error] 22-22: Command coming from incoming request
Context: subprocess.run([str(binary), "--version"], capture_output=True, text=True, check=True)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
scripts/latency_benchmark/src/fixtures.py
[info] 33-33: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(value))
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
scripts/latency_benchmark/src/cli.py
[info] 123-123: use jsonify instead of json.dumps for JSON output
Context: json.dumps(results, indent=2, sort_keys=True)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
scripts/latency_benchmark/src/processes.py
[error] 60-66: Use of unsanitized data to create processes
Context: subprocess.Popen(
command,
cwd=self.root,
env=isolated_environment(self.root),
stdout=subprocess.DEVNULL,
stderr=self.log_handle,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(os-system-unsanitized-data)
[error] 160-167: Use of unsanitized data to create processes
Context: subprocess.Popen(
command,
cwd=self.root,
env=environment,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=self.log_handle,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(os-system-unsanitized-data)
[error] 60-66: Command coming from incoming request
Context: subprocess.Popen(
command,
cwd=self.root,
env=isolated_environment(self.root),
stdout=subprocess.DEVNULL,
stderr=self.log_handle,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
[error] 160-167: Command coming from incoming request
Context: subprocess.Popen(
command,
cwd=self.root,
env=environment,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=self.log_handle,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
scripts/latency_benchmark/src/protocol.py
[info] 214-214: use jsonify instead of json.dumps for JSON output
Context: json.dumps(value, separators=(",", ":"))
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
scripts/latency_benchmark/src/benchmarks.py
[info] 239-244: use jsonify instead of json.dumps for JSON output
Context: json.dumps(
{
"session_id": f"benchmark-{agent}-{variant_name}-{index}",
"hook_event_name": event_name,
}
)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 52-52: use secrets package over random package
Context: random.Random(seed)
Note: [CWE-330] Use of Insufficiently Random Values.
(avoid-random-python)
[info] 266-266: use secrets package over random package
Context: random.Random(index)
Note: [CWE-330] Use of Insufficiently Random Values.
(avoid-random-python)
[error] 191-199: Use of unsanitized data to create processes
Context: subprocess.run(
command,
cwd=root,
env=isolated_environment(root),
input=input_bytes,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(os-system-unsanitized-data)
[error] 191-199: Command coming from incoming request
Context: subprocess.run(
command,
cwd=root,
env=isolated_environment(root),
input=input_bytes,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🪛 Blinter (1.0.113)
scripts/latency_benchmark/data/mock-codex.cmd
[error] 1-1: Unix line endings detected. Explanation: Batch file uses Unix line endings (LF-only) which can cause GOTO/CALL label parsing failures and script malfunction due to Windows batch parser 512-byte boundary bugs. Recommendation: Convert file to Windows line endings (CRLF). Use tools like dos2unix, notepad++, or configure git with 'git config core.autocrlf true'. Context: File uses Unix line endings (LF-only) - 12 LF sequences found
(E018)
[warning] 8-8: Potentially undefined variable reference. Explanation: Script references variables that were never set in this script. This may be intentional if using environment variables, but could cause runtime errors if the variable is not set by parent process or system. Recommendation: If this is an environment variable, this warning can be ignored. Otherwise, define the variable using SET before referencing it, or add IF DEFINED checks to handle undefined cases. Context: Variable 'BENCHMARK_GATEWAY_FILE' is used but never defined. Variable 'NEMO_RELAY_GATEWAY_URL' is used but never defined. Variable 'BENCHMARK_STOP_FILE' is used but never defined
(E006)
[warning] 3-3: Redundant command echoing suppression. Explanation: Multiple @ECHO OFF commands or redundant @ prefixes waste processing. Recommendation: Use single @ECHO OFF at script start, avoid @ on subsequent commands. Context: @ECHO OFF should only appear once at script start
(P020)
🪛 LanguageTool
.agents/skills/maintain-coding-agent-benchmark/SKILL.md
[grammar] ~147-~147: Use a hyphen to join words.
Context: ...ugh healthy gateway readiness. Startup paired deltas subtract the same process ...
(QB_NEW_EN_HYPHEN)
scripts/latency_benchmark/README.md
[style] ~219-~219: ‘in proportion to’ might be wordy. Consider a shorter alternative.
Context: ...mal`. Custom variants increase runtime in proportion to the number of variants. A custom plugin...
(EN_WORDINESS_PREMIUM_IN_PROPORTION_TO)
docs/reference/performance.mdx
[style] ~49-~49: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... - Relay with a local ATOF file sink. - Relay with a local OpenTelemetry HTTP receive...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🪛 Ruff (0.16.1)
scripts/latency_benchmark/src/servers.py
[warning] 36-36: Dynamically typed expressions (typing.Any) are disallowed in *args
(ANN401)
[warning] 43-43: Do not call getattr with a constant attribute value. It is not any safer than normal property access.
Replace getattr with attribute access
(B009)
[warning] 43-43: Do not call getattr with a constant attribute value. It is not any safer than normal property access.
Replace getattr with attribute access
(B009)
[warning] 44-44: Do not call getattr with a constant attribute value. It is not any safer than normal property access.
Replace getattr with attribute access
(B009)
[warning] 53-53: Do not call getattr with a constant attribute value. It is not any safer than normal property access.
Replace getattr with attribute access
(B009)
[warning] 65-65: Do not call getattr with a constant attribute value. It is not any safer than normal property access.
Replace getattr with attribute access
(B009)
[warning] 116-116: Dynamically typed expressions (typing.Any) are disallowed in *args
(ANN401)
[warning] 135-135: Dynamically typed expressions (typing.Any) are disallowed in **attributes
(ANN401)
[warning] 152-152: Avoid specifying long messages outside the exception class
(TRY003)
scripts/latency_benchmark/tests/test_html_report.py
[error] 65-65: Probable insecure usage of temporary file or directory: "/tmp/plugins.toml"
(S108)
scripts/latency_benchmark/src/reporting.py
[error] 18-18: subprocess call: check for execution of untrusted input
(S603)
[error] 18-18: Starting a process with a partial executable path
(S607)
[error] 23-23: subprocess call: check for execution of untrusted input
(S603)
scripts/latency_benchmark/src/fixtures.py
[warning] 27-27: Avoid specifying long messages outside the exception class
(TRY003)
scripts/latency_benchmark/src/cli.py
[warning] 110-110: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 112-112: Avoid specifying long messages outside the exception class
(TRY003)
scripts/latency_benchmark/src/processes.py
[error] 61-61: subprocess call: check for execution of untrusted input
(S603)
[warning] 74-74: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 85-85: Consider moving this statement to an else block
(TRY300)
[warning] 108-108: Avoid specifying long messages outside the exception class
(TRY003)
[error] 161-161: subprocess call: check for execution of untrusted input
(S603)
[warning] 175-175: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 200-200: Avoid specifying long messages outside the exception class
(TRY003)
scripts/latency_benchmark/src/protocol.py
[warning] 102-112: Use list.extend to create a transformed list
(PERF401)
[warning] 167-173: Use list.extend to create a transformed list
(PERF401)
[warning] 190-190: Boolean-typed positional argument in function definition
(FBT001)
scripts/latency_benchmark/tests/test_config.py
[warning] 167-167: Use pytest.raises instead of unittest-style assertRaisesRegex
Replace assertRaisesRegex with pytest.raises
(PT027)
[warning] 175-175: Use pytest.raises instead of unittest-style assertRaisesRegex
Replace assertRaisesRegex with pytest.raises
(PT027)
[warning] 186-186: Use pytest.raises instead of unittest-style assertRaisesRegex
Replace assertRaisesRegex with pytest.raises
(PT027)
scripts/latency_benchmark/src/config.py
[warning] 130-130: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 135-135: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 138-138: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 145-145: Dynamically typed expressions (typing.Any) are disallowed in value
(ANN401)
[warning] 147-147: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 151-151: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 153-153: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 157-157: Dynamically typed expressions (typing.Any) are disallowed in value
(ANN401)
[warning] 159-159: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 163-163: Dynamically typed expressions (typing.Any) are disallowed in value
(ANN401)
[warning] 165-165: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 168-168: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 172-172: Dynamically typed expressions (typing.Any) are disallowed in value
(ANN401)
[warning] 174-174: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 178-178: Dynamically typed expressions (typing.Any) are disallowed in value
(ANN401)
[warning] 180-180: Prefer TypeError exception for invalid type
(TRY004)
[warning] 180-180: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 185-185: Prefer TypeError exception for invalid type
(TRY004)
[warning] 185-185: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 189-189: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 192-192: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 194-194: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 196-196: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 204-204: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 212-212: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 216-216: Prefer TypeError exception for invalid type
(TRY004)
[warning] 216-216: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 219-219: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 223-223: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 243-243: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 263-263: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 271-271: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 273-273: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 281-281: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 288-288: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 290-290: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 297-297: Avoid specifying long messages outside the exception class
(TRY003)
scripts/latency_benchmark/src/benchmarks.py
[error] 53-53: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
[warning] 68-68: Boolean-typed positional argument in function definition
(FBT001)
[warning] 75-75: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 93-93: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 192-200: Prefer capture_output over sending stdout and stderr to PIPE
Replace with capture_output keyword argument
(UP022)
[error] 192-192: subprocess call: check for execution of untrusted input
(S603)
[warning] 204-204: Avoid specifying long messages outside the exception class
(TRY003)
[warning] 265-265: Function definition does not bind loop variable index
(B023)
[error] 267-267: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
[warning] 281-281: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
[warning] 317-317: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
🪛 Stylelint (17.14.0)
scripts/latency_benchmark/src/report/styles.css
[error] 16-16: Expected empty line before declaration (declaration-empty-line-before)
(declaration-empty-line-before)
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/run-latency-benchmark/SKILL.md:
- Around line 54-62: Update the “Report the Outcome” instructions to require
reporting the resolved output paths whenever --output, --report, or the
result-directory override is supplied. Otherwise, report the default HTML and
JSON paths, ensuring the agent does not claim default files were written when
per-file overrides changed their locations.
In `@justfile`:
- Line 1124: Update the benchmark recipe commands in justfile to enforce tracked
lockfiles: add --locked to both uv run invocations and to the cargo build
command, preserving their existing arguments and behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: cbb66bbc-16cc-4714-b5d6-bef482551974
📒 Files selected for processing (16)
.agents/skills/run-latency-benchmark/SKILL.md.agents/skills/run-latency-benchmark/agents/openai.yaml.gitattributesdocs/reference/performance.mdxjustfilescripts/latency_benchmark/README.mdscripts/latency_benchmark/data/mock-codex.cmdscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/src/processes.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/report/styles.cssscripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/tests/test_processes.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (29)
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
**/*: Use release tags in raw Rust-compatible SemVer without a leadingv; tags such asv0.1.0are prohibited.
Use branch prefixesfeat/,fix/,docs/,test/, orrefactor/according to the change purpose.
Every commit in a pull request must include a DCOSigned-off-by:sign-off.
Before submitting a pull request, ensure pre-commit hooks, relevant tests, target-specific builds, documentation updates, and a rebase on the latestmainare complete.
Use commit messages in the formtype: short description, with a valid type and a first line under 72 characters.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/data/mock-codex.cmdscripts/latency_benchmark/README.mddocs/reference/performance.mdxscripts/latency_benchmark/src/report/report.jsjustfilescripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/report/styles.cssscripts/latency_benchmark/src/processes.py
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolveheader_envvalues at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests andjust test-rustwhen event fields change; runjust test-python,just test-go, andjust test-nodewhen binding-native configuration or lifecycle changes.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/*.{md,mdx,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Examples and documentation must use each exporter's documented flush/deregister order before shutdown.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/README.mddocs/reference/performance.mdxscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/*.{rs,py,js,jsx,ts,tsx,go,c,h,cc,cpp,md,toml,yml,yaml,sh}
📄 CodeRabbit inference engine (AGENTS.md)
Keep SPDX headers on source, documentation, scripts, and configuration files; the project is Apache-2.0.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Use
snake_casenaming in Rust and Python.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/*.{rs,py,js,mjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Preserve the existing Tokio-based asynchronous model and callback/future lifetimes; do not unexpectedly block or hide async work in bindings.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Lint Python with Ruff using rule setsE,F,W, andI.
Format Python with the Ruff formatter, using a 120-character line length and double quotes.
Runtyfor Python type checking.
Use Pythonsnake_casenaming conventions.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}: Run tests for every language affected by a change; changes to the core Rust crate require tests across all bindings.
UseSONAR_IGNORE_START/SONAR_IGNORE_ENDonly for documented false positives, keep ignored blocks minimal, explain them with a comment, and obtain reviewer sign-off.
Preserve the layered architecture in which Rust provides the core runtime and C FFI, PyO3, and NAPI provide bindings that mirror the full API surface.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
**/{test,tests}/**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When adding functionality, include tests in the appropriate test files for each affected language binding.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/tests/test_processes.py
**/*.{rs,py,go,js,jsx,ts,tsx,c,h,html,md,mdx,toml}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Include the appropriate SPDX copyright and Apache-2.0 license header in every source file.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/README.mddocs/reference/performance.mdxscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/processes.py
{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}
⚙️ CodeRabbit configuration file
{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}: Review automation changes for reproducibility, pinned versions where appropriate, secret handling, and consistency with the documented validation matrix.
Pay attention to commands that need generated native artifacts, FFI libraries, or platform-specific environment variables.
Files:
scripts/latency_benchmark/tests/test_benchmarks.pyscripts/latency_benchmark/data/mock-codex.shscripts/latency_benchmark/data/mock-codex.cmdscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/report/report.jsjustfilescripts/latency_benchmark/src/benchmarks.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/fixtures.pyscripts/latency_benchmark/tests/test_processes.pyscripts/latency_benchmark/src/report/styles.cssscripts/latency_benchmark/src/processes.py
**/*.{md,rst,html,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
**/*.{md,rst,html,txt}: Always spellNVIDIAin all caps. Do not useNvidia,nvidia,nVidia,nVIDIA, orNV.
Usean NVIDIAbefore a noun because the name starts with an 'en' sound.
Do not add a registered trademark symbol afterNVIDIAwhen referring to the company.
Use trademark symbols with product names only when the document type or legal guidance requires them.
Verify official capitalization, spacing, and hyphenation for product names.
Precede NVIDIA product names withNVIDIAon first mention when it is natural and accurate.
Do not rewrite product names for grammar or title-case rules.
Preserve third-party product names according to the owner's spelling.
Include the company name and full model qualifier on first use when it helps identify the model.
Preserve the official capitalization and punctuation of model names.
Use shorter family names only after the full name is established.
Spell out a term on first use and put the acronym in parentheses unless the acronym is widely understood by the intended audience.
Use the acronym on later mentions after it has been defined.
For long documents, reintroduce the full term if readers might lose context.
Form plurals of acronyms withs, not an apostrophe, such asGPUs.
In headings, common acronyms can remain abbreviated. Spell out the term in the first or second sentence of the body.
Common terms such asCPU,GPU,PC,API, andUIusually do not need to be spelled out for developer audiences.
Files:
scripts/latency_benchmark/README.md
**/*.{md,rst,html}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
Link the first mention of a product name when the destination helps the reader.
Files:
scripts/latency_benchmark/README.md
**/*.{md,rst,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
Spell
NVIDIAin all caps. Do not useNvidia,nvidia, orNV.
Files:
scripts/latency_benchmark/README.md
**/*.{md,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
**/*.{md,rst}: Format commands, code elements, expressions, package names, file names, and paths as inline code.
Use descriptive link text. Avoid raw URLs and weak anchors such as "here" or "read more."
Use title case consistently for technical documentation headings.
Introduce code blocks, lists, tables, and images with complete sentences.
Write procedures as imperative steps. Keep steps parallel and split long procedures into smaller tasks.
Prefer active voice, present tense, short sentences, contractions, and plain English.
Usecanfor possibility and reservemayfor permission.
Useafterfor temporal relationships instead ofonce.
Preferrefer tooverseewhen the wording points readers to another resource.
Avoid culture-specific idioms, unnecessary Latinisms, jokes, and marketing exaggeration in technical docs.
Spell out months in body text, avoid ordinal dates, and use clear time zones.
Spell out whole numbers from zero through nine unless they are technical values, parameters, versions, or UI values.
Use numerals for 10 or greater and include commas in thousands.
Do not add trademark symbols to learning-oriented docs unless the source, platform, or legal guidance explicitly requires them.
Files:
scripts/latency_benchmark/README.md
**/*.{md,mdx}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If links in documentation change, run
just docs-linkcheck.Use documented public APIs and stable wrapper commands in examples and user-facing documentation; do not rely on internal helpers.
**/*.{md,mdx}: Prefer the documented public API over internal shortcuts in documentation and examples.
Keep package names, repository references, and build commands current.
Contribution workflow documentation must require an issue before external contribution pull requests and note that NVIDIA contributors may use a GitHub or Linear issue.
Update entry-point documentation when examples or reading paths change.
Keep release-process and release-notes guidance in maintainer documentation such asRELEASING.md, rather than user-facing documentation pages orCHANGELOG.md.
Use stable user-facing wrappers at thescripts/root in documentation and examples; reference namespaced helper paths only for internal maintenance documentation.
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pages.
Dynamic plugin manifests in documentation and examples should usecompat.relay = ">=0.5,<1.0"unless deliberately narrower.
Render images, diagrams, tables, and other visual content at representative page widths, ensuring legibility and complete access without clipping; use responsive scaling, reflow, or overflow as appropriate and scope visual styling narrowly.
Dynamic plugin entry pages should link to native, worker, Rust example, Python example, and protocol pages when those pages exist.
Images, diagrams, tables, and custom visual content must remain legible and fully accessible at representative desktop and narrow page widths.
Release-policy documentation must point to GitHub Releases as the only release-history source of truth.
Runjust docswhen the documentation site changes; retain./scripts/build-docs.sh htmlas the compatibility wrapper.
Files:
scripts/latency_benchmark/README.mddocs/reference/performance.mdx
scripts/**/*.md
📄 CodeRabbit inference engine (AGENTS.md)
Use stable public wrappers at the
scripts/root in documentation and examples; reference namespaced helper paths only for internal maintenance documentation.
Files:
scripts/latency_benchmark/README.md
**/README.md
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update the relevant crate or package README when that package surface changes.
Update relevant package or crate
README.mdfiles when examples or binding guidance changes.
Files:
scripts/latency_benchmark/README.md
**/{README.md,docs/index.md}
📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)
Update
README.mdordocs/index.mdwhen documentation entry points change.
Files:
scripts/latency_benchmark/README.md
**/*.{md,mdx,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-technical-docs.md)
**/*.{md,mdx,rst}: Use title case consistently for technical documentation headings and table headers; avoid quotation marks, ampersands, and exclamation marks in headings, while preserving official product, event, research, and whitepaper title case.
Format code elements, commands, parameters, package names, expressions, directories, file names, and paths in monospace; represent path placeholders with angle brackets inside monospace.
Format UI buttons, menus, fields, and labels in bold, and separate consecutive UI navigation labels with>.
Use quotation marks for error messages and strings when appropriate, italics for newly introduced terms and publication titles, and plain text for keyboard shortcuts.
Represent GitHub repositories with owner/repository link text, such as[NVIDIA/NeMo](link), rather than generic repository wording.
Introduce every code block with a complete sentence; do not let a code block complete or interrupt the grammar of surrounding prose; use syntax highlighting when supported.
Keep inline method, function, and class references consistent with nearby documentation; omit empty parentheses in prose when no call is shown.
Use descriptive link text matching the destination title when possible; avoid raw URLs, generic anchors, long-sentence links, and unnecessary links that distract from procedures.
Ensure lists have a complete lead-in sentence, more than one item, no more than two levels, parallel construction, one idea or action per item, and appropriate punctuation; use bullets for unordered items and numbers for ordered tasks.
Format definition lists with a bold term followed by a complete, parallel, punctuated definition.
Use tables for reference information, decision support, compatibility matrices, and comparable choices; flag one-row tables, missing captions or lead-ins, sentence-case headers where title case is expected, unexplained empty cells, and code or links that would be clearer as prose.
Write procedure steps as imperative ...
Files:
scripts/latency_benchmark/README.mddocs/reference/performance.mdx
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
**/*.mdx: In MDX files, top-of-file comments must use JSX comment delimiters ({/*and*/}); do not use HTML comments for MDX SPDX headers.
New or regenerated MDX files must use{/* ... */}for top-of-file SPDX comments.
Files:
docs/reference/performance.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/reference/performance.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/reference/performance.mdx
docs/**/*.{md,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update relevant reference documentation when public behavior or APIs change.
Files:
docs/reference/performance.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/reference/performance.mdx
**/*.{js,jsx,ts,tsx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use Node.js
camelCasenaming conventions.
Files:
scripts/latency_benchmark/src/report/report.js
{justfile,codecov.yml,codecov.yaml,.github/workflows/**/*.yml,.github/workflows/**/*.yaml}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
justfile, Codecov, and CI package/test workflows must include new plugin crates and packages.
Files:
justfile
justfile
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Keep
justfilebuild, test, clean, version, and package recipes for plugin crates and packages aligned with the current packaging layout.
Files:
justfile
🧠 Learnings (2)
📓 Common learnings
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Read `scripts/latency_benchmark/README.md` completely before running the NeMo Relay latency benchmark; treat it as the source of truth for prerequisites, commands, configuration, middleware, output paths, metrics, and troubleshooting.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Run benchmark commands from the repository root and use `just latency-benchmark --help` to confirm the current CLI without building Relay.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Do not modify benchmark code, fixtures, or configuration unless the user separately requests an implementation change.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Choose the smallest benchmark run that answers the user's question: use the smoke test for setup checks or unspecified requests, and use the default matrix only for statistically meaningful performance data.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Before running the default matrix, confirm that the operating system temporary directory has ample free space because ATOF output can reach tens of gigabytes.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Use `--config` for repeatable custom matrices and apply CLI overrides only for requested one-off changes; retain the minimal, ATOF file-exporter, and OTLP variants, adding custom middleware through the documented `--middleware NAME=PATH` or `[[middleware]]` workflow.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Report the exact command, completion status, selected suites and matrix, important warnings, and persistent output locations unless the result directory was overridden.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: Use the HTML report for graphs and explanations and the JSON result for machine-readable analysis; interpret paired deltas as added milliseconds over the named baseline, p50 as the median, p95/p99 as tail observations, and the median 95% confidence interval as uncertainty around the median paired delta rather than a range containing 95% of samples.
Learnt from: CR
Repo: NVIDIA/NeMo-Relay
Timestamp: 2026-08-05T23:14:09.459Z
Learning: For failures, follow the README troubleshooting section before changing the command and explicitly call out loopback permission errors, exporter-delivery failures, invalid matrix values, and stale temporary directories.
📚 Learning: 2026-04-15T18:16:52.951Z
Learnt from: bbednarski9
Repo: NVIDIA/NeMo-Flow PR: 1
File: docs/atof-event-format.md:381-381
Timestamp: 2026-04-15T18:16:52.951Z
Learning: In docs/atof-event-format.md (and when reviewing references to this format across the NeMo-Flow repo), treat `AtifExporter.events_to_steps()` as the intended public/API method name. Do not flag it as inconsistent with internal Rust symbol names (e.g., `event_to_steps` in `crates/core/src/atif.rs`)—the documentation’s public-facing naming may differ intentionally from internal implementation details.
Applied to files:
.gitattributes
🪛 ast-grep (0.45.0)
scripts/latency_benchmark/tests/test_config.py
[info] 228-228: Do not hardcode temporary file or directory names
Context: "/tmp/developer-config.toml"
Note: [CWE-377] Insecure Temporary File.
(hardcoded-tmp-file)
[warning] 227-227: Do not make http calls without encryption
Context: "http://proxy.example"
Note: [CWE-319] Cleartext Transmission of Sensitive Information.
(requests-http)
[warning] 230-230: Do not make http calls without encryption
Context: "http://collector.example"
Note: [CWE-319] Cleartext Transmission of Sensitive Information.
(requests-http)
[warning] 233-233: Do not make http calls without encryption
Context: "http://proxy.example"
Note: [CWE-319] Cleartext Transmission of Sensitive Information.
(requests-http)
🪛 Ruff (0.16.1)
scripts/latency_benchmark/tests/test_benchmarks.py
[warning] 17-23: Use a single with statement with multiple contexts instead of nested with statements
(SIM117)
[warning] 23-23: Use pytest.raises instead of unittest-style assertRaisesRegex
Replace assertRaisesRegex with pytest.raises
(PT027)
scripts/latency_benchmark/tests/test_config.py
[error] 229-229: Probable insecure usage of temporary file or directory: "/tmp/developer-config.toml"
(S108)
🔇 Additional comments (16)
scripts/latency_benchmark/tests/test_config.py (1)
9-20: LGTM!Also applies to: 215-246
scripts/latency_benchmark/tests/test_processes.py (1)
17-93: LGTM!scripts/latency_benchmark/src/fixtures.py (1)
14-27: LGTM!Also applies to: 89-90, 108-119
scripts/latency_benchmark/data/mock-codex.cmd (1)
8-9: LGTM!scripts/latency_benchmark/data/mock-codex.sh (1)
9-11: LGTM!.gitattributes (1)
5-5: LGTM!scripts/latency_benchmark/src/processes.py (1)
170-175: LGTM!scripts/latency_benchmark/src/benchmarks.py (1)
139-141: LGTM!Also applies to: 306-313
scripts/latency_benchmark/tests/test_benchmarks.py (1)
1-40: LGTM!justfile (1)
1116-1123: LGTM!Also applies to: 1125-1126, 1129-1131
scripts/latency_benchmark/src/report/report.js (1)
426-426: LGTM!Also applies to: 453-453
scripts/latency_benchmark/src/report/styles.css (1)
16-16: LGTM!docs/reference/performance.mdx (1)
32-61: LGTM!Also applies to: 63-99, 101-115, 117-133, 135-142
scripts/latency_benchmark/README.md (1)
6-6: LGTM!Also applies to: 270-271
.agents/skills/run-latency-benchmark/SKILL.md (1)
1-53: LGTM!Also applies to: 63-71
.agents/skills/run-latency-benchmark/agents/openai.yaml (1)
1-8: LGTM!
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/latency_benchmark/tests/test_reporting.py`:
- Around line 18-28: Update the mocks in the environment_record test to return
distinct Git outputs: keep the status response as “unknown” and provide a valid
commit value for the second _git_output call. Change the git_commit assertion to
match that commit value while preserving the git_dirty assertion.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d08486ce-b9ed-47ef-af1d-03e41de26ef3
📒 Files selected for processing (15)
docs/reference/performance.mdxjustfilescripts/README.mdscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/cli.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_reporting.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (29)
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
**/*: Use release tags in raw Rust-compatible SemVer without a leadingv; tags such asv0.1.0are prohibited.
Use branch prefixesfeat/,fix/,docs/,test/, orrefactor/according to the change purpose.
Every commit in a pull request must include a DCOSigned-off-by:sign-off.
Before submitting a pull request, ensure pre-commit hooks, relevant tests, target-specific builds, documentation updates, and a rebase on the latestmainare complete.
Use commit messages in the formtype: short description, with a valid type and a first line under 72 characters.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pydocs/reference/performance.mdxscripts/README.mdscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyjustfilescripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolveheader_envvalues at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests andjust test-rustwhen event fields change; runjust test-python,just test-go, andjust test-nodewhen binding-native configuration or lifecycle changes.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/*.{md,mdx,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Examples and documentation must use each exporter's documented flush/deregister order before shutdown.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pydocs/reference/performance.mdxscripts/README.mdscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/*.{rs,py,js,jsx,ts,tsx,go,c,h,cc,cpp,md,toml,yml,yaml,sh}
📄 CodeRabbit inference engine (AGENTS.md)
Keep SPDX headers on source, documentation, scripts, and configuration files; the project is Apache-2.0.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/README.mdscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Use
snake_casenaming in Rust and Python.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/cli.py
**/*.{rs,py,js,mjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Preserve the existing Tokio-based asynchronous model and callback/future lifetimes; do not unexpectedly block or hide async work in bindings.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Lint Python with Ruff using rule setsE,F,W, andI.
Format Python with the Ruff formatter, using a 120-character line length and double quotes.
Runtyfor Python type checking.
Use Pythonsnake_casenaming conventions.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/cli.py
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}: Run tests for every language affected by a change; changes to the core Rust crate require tests across all bindings.
UseSONAR_IGNORE_START/SONAR_IGNORE_ENDonly for documented false positives, keep ignored blocks minimal, explain them with a comment, and obtain reviewer sign-off.
Preserve the layered architecture in which Rust provides the core runtime and C FFI, PyO3, and NAPI provide bindings that mirror the full API surface.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/{test,tests}/**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When adding functionality, include tests in the appropriate test files for each affected language binding.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.py
**/*.{rs,py,go,js,jsx,ts,tsx,c,h,html,md,mdx,toml}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Include the appropriate SPDX copyright and Apache-2.0 license header in every source file.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pydocs/reference/performance.mdxscripts/README.mdscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyscripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}
⚙️ CodeRabbit configuration file
{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}: Review automation changes for reproducibility, pinned versions where appropriate, secret handling, and consistency with the documented validation matrix.
Pay attention to commands that need generated native artifacts, FFI libraries, or platform-specific environment variables.
Files:
scripts/latency_benchmark/tests/__init__.pyscripts/latency_benchmark/tests/test_reporting.pyscripts/README.mdscripts/latency_benchmark/src/reporting.pyscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/tests/test_cli.pyscripts/latency_benchmark/tests/test_html_report.pyjustfilescripts/latency_benchmark/tests/test_config.pyscripts/latency_benchmark/src/__init__.pyscripts/latency_benchmark/src/__main__.pyscripts/latency_benchmark/README.mdscripts/latency_benchmark/src/report/report.jsscripts/latency_benchmark/src/cli.py
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
**/*.mdx: In MDX files, top-of-file comments must use JSX comment delimiters ({/*and*/}); do not use HTML comments for MDX SPDX headers.
New or regenerated MDX files must use{/* ... */}for top-of-file SPDX comments.
Files:
docs/reference/performance.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/reference/performance.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/reference/performance.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If links in documentation change, run
just docs-linkcheck.Use documented public APIs and stable wrapper commands in examples and user-facing documentation; do not rely on internal helpers.
**/*.{md,mdx}: Prefer the documented public API over internal shortcuts in documentation and examples.
Keep package names, repository references, and build commands current.
Contribution workflow documentation must require an issue before external contribution pull requests and note that NVIDIA contributors may use a GitHub or Linear issue.
Update entry-point documentation when examples or reading paths change.
Keep release-process and release-notes guidance in maintainer documentation such asRELEASING.md, rather than user-facing documentation pages orCHANGELOG.md.
Use stable user-facing wrappers at thescripts/root in documentation and examples; reference namespaced helper paths only for internal maintenance documentation.
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pages.
Dynamic plugin manifests in documentation and examples should usecompat.relay = ">=0.5,<1.0"unless deliberately narrower.
Render images, diagrams, tables, and other visual content at representative page widths, ensuring legibility and complete access without clipping; use responsive scaling, reflow, or overflow as appropriate and scope visual styling narrowly.
Dynamic plugin entry pages should link to native, worker, Rust example, Python example, and protocol pages when those pages exist.
Images, diagrams, tables, and custom visual content must remain legible and fully accessible at representative desktop and narrow page widths.
Release-policy documentation must point to GitHub Releases as the only release-history source of truth.
Runjust docswhen the documentation site changes; retain./scripts/build-docs.sh htmlas the compatibility wrapper.
Files:
docs/reference/performance.mdxscripts/README.mdscripts/latency_benchmark/README.md
docs/**/*.{md,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update relevant reference documentation when public behavior or APIs change.
Files:
docs/reference/performance.mdx
**/*.{md,mdx,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-technical-docs.md)
**/*.{md,mdx,rst}: Use title case consistently for technical documentation headings and table headers; avoid quotation marks, ampersands, and exclamation marks in headings, while preserving official product, event, research, and whitepaper title case.
Format code elements, commands, parameters, package names, expressions, directories, file names, and paths in monospace; represent path placeholders with angle brackets inside monospace.
Format UI buttons, menus, fields, and labels in bold, and separate consecutive UI navigation labels with>.
Use quotation marks for error messages and strings when appropriate, italics for newly introduced terms and publication titles, and plain text for keyboard shortcuts.
Represent GitHub repositories with owner/repository link text, such as[NVIDIA/NeMo](link), rather than generic repository wording.
Introduce every code block with a complete sentence; do not let a code block complete or interrupt the grammar of surrounding prose; use syntax highlighting when supported.
Keep inline method, function, and class references consistent with nearby documentation; omit empty parentheses in prose when no call is shown.
Use descriptive link text matching the destination title when possible; avoid raw URLs, generic anchors, long-sentence links, and unnecessary links that distract from procedures.
Ensure lists have a complete lead-in sentence, more than one item, no more than two levels, parallel construction, one idea or action per item, and appropriate punctuation; use bullets for unordered items and numbers for ordered tasks.
Format definition lists with a bold term followed by a complete, parallel, punctuated definition.
Use tables for reference information, decision support, compatibility matrices, and comparable choices; flag one-row tables, missing captions or lead-ins, sentence-case headers where title case is expected, unexplained empty cells, and code or links that would be clearer as prose.
Write procedure steps as imperative ...
Files:
docs/reference/performance.mdxscripts/README.mdscripts/latency_benchmark/README.md
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/reference/performance.mdx
**/*.{md,rst,html,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
**/*.{md,rst,html,txt}: Always spellNVIDIAin all caps. Do not useNvidia,nvidia,nVidia,nVIDIA, orNV.
Usean NVIDIAbefore a noun because the name starts with an 'en' sound.
Do not add a registered trademark symbol afterNVIDIAwhen referring to the company.
Use trademark symbols with product names only when the document type or legal guidance requires them.
Verify official capitalization, spacing, and hyphenation for product names.
Precede NVIDIA product names withNVIDIAon first mention when it is natural and accurate.
Do not rewrite product names for grammar or title-case rules.
Preserve third-party product names according to the owner's spelling.
Include the company name and full model qualifier on first use when it helps identify the model.
Preserve the official capitalization and punctuation of model names.
Use shorter family names only after the full name is established.
Spell out a term on first use and put the acronym in parentheses unless the acronym is widely understood by the intended audience.
Use the acronym on later mentions after it has been defined.
For long documents, reintroduce the full term if readers might lose context.
Form plurals of acronyms withs, not an apostrophe, such asGPUs.
In headings, common acronyms can remain abbreviated. Spell out the term in the first or second sentence of the body.
Common terms such asCPU,GPU,PC,API, andUIusually do not need to be spelled out for developer audiences.
Files:
scripts/README.mdscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/README.md
**/*.{md,rst,html}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
Link the first mention of a product name when the destination helps the reader.
Files:
scripts/README.mdscripts/latency_benchmark/src/report/template.htmlscripts/latency_benchmark/README.md
**/*.{md,rst,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
Spell
NVIDIAin all caps. Do not useNvidia,nvidia, orNV.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/*.{md,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
**/*.{md,rst}: Format commands, code elements, expressions, package names, file names, and paths as inline code.
Use descriptive link text. Avoid raw URLs and weak anchors such as "here" or "read more."
Use title case consistently for technical documentation headings.
Introduce code blocks, lists, tables, and images with complete sentences.
Write procedures as imperative steps. Keep steps parallel and split long procedures into smaller tasks.
Prefer active voice, present tense, short sentences, contractions, and plain English.
Usecanfor possibility and reservemayfor permission.
Useafterfor temporal relationships instead ofonce.
Preferrefer tooverseewhen the wording points readers to another resource.
Avoid culture-specific idioms, unnecessary Latinisms, jokes, and marketing exaggeration in technical docs.
Spell out months in body text, avoid ordinal dates, and use clear time zones.
Spell out whole numbers from zero through nine unless they are technical values, parameters, versions, or UI values.
Use numerals for 10 or greater and include commas in thousands.
Do not add trademark symbols to learning-oriented docs unless the source, platform, or legal guidance explicitly requires them.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
scripts/**/*.md
📄 CodeRabbit inference engine (AGENTS.md)
Use stable public wrappers at the
scripts/root in documentation and examples; reference namespaced helper paths only for internal maintenance documentation.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/README.md
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update the relevant crate or package README when that package surface changes.
Update relevant package or crate
README.mdfiles when examples or binding guidance changes.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
**/{README.md,docs/index.md}
📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)
Update
README.mdordocs/index.mdwhen documentation entry points change.
Files:
scripts/README.mdscripts/latency_benchmark/README.md
{justfile,codecov.yml,codecov.yaml,.github/workflows/**/*.yml,.github/workflows/**/*.yaml}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
justfile, Codecov, and CI package/test workflows must include new plugin crates and packages.
Files:
justfile
justfile
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Keep
justfilebuild, test, clean, version, and package recipes for plugin crates and packages aligned with the current packaging layout.
Files:
justfile
**/*.{js,jsx,ts,tsx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use Node.js
camelCasenaming conventions.
Files:
scripts/latency_benchmark/src/report/report.js
🪛 LanguageTool
docs/reference/performance.mdx
[style] ~51-~51: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... - Relay with the ATOF file exporter. - Relay with the OTLP exporter sending to a loc...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🪛 Ruff (0.16.1)
scripts/latency_benchmark/tests/test_cli.py
[warning] 79-80: Use a single with statement with multiple contexts instead of nested with statements
(SIM117)
[warning] 105-110: Use a single with statement with multiple contexts instead of nested with statements
(SIM117)
[warning] 110-110: Use pytest.raises instead of unittest-style assertRaisesRegex
Replace assertRaisesRegex with pytest.raises
(PT027)
scripts/latency_benchmark/src/cli.py
[warning] 169-169: Avoid specifying long messages outside the exception class
(TRY003)
🔇 Additional comments (15)
scripts/latency_benchmark/src/__init__.py (1)
4-4: LGTM!scripts/latency_benchmark/src/cli.py (1)
4-10: LGTM!Also applies to: 30-30, 62-65, 68-87, 98-104, 122-129, 139-154, 167-169
scripts/latency_benchmark/src/__main__.py (1)
4-4: LGTM!scripts/latency_benchmark/src/report/report.js (1)
44-50: LGTM!scripts/latency_benchmark/tests/test_cli.py (1)
1-118: LGTM!scripts/latency_benchmark/tests/test_config.py (1)
4-4: LGTM!scripts/latency_benchmark/tests/test_html_report.py (1)
101-102: LGTM!justfile (1)
1122-1124: 🎯 Functional CorrectnessNo change required.
[positional-arguments]enables"$@"forlatency-benchmark, sobenchmark_argsare forwarded correctly.> Likely an incorrect or invalid review comment.scripts/latency_benchmark/src/reporting.py (1)
24-28: LGTM!Also applies to: 37-80
scripts/latency_benchmark/src/report/template.html (1)
82-120: LGTM!Also applies to: 187-198
scripts/latency_benchmark/tests/__init__.py (1)
1-4: LGTM!scripts/latency_benchmark/tests/test_reporting.py (1)
1-17: LGTM!Also applies to: 29-32
scripts/README.md (1)
29-64: LGTM!scripts/latency_benchmark/README.md (1)
18-20: LGTM!Also applies to: 35-43, 83-89, 100-101, 117-129, 177-237, 275-300
docs/reference/performance.mdx (1)
34-74: LGTM!
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
2 participants
Overview
Add an opt-in, configurable NeMo Relay latency benchmark with repeatable middleware variants and self-contained result reporting. This change is isolated to benchmark tooling and documentation; it does not change Relay runtime behavior.
Details
[x/y]progress after each completed gateway matrix scenario and selected process suite.nemo-relay-latency-report.jsonand a self-containednemo-relay-latency-report.htmlwith graphs, tables, environment metadata, and metric explanations.just test-latency-benchmarkfor the fast fixture test suite and document the benchmark workflow in the fixture README.Validation:
just test-latency-benchmark— 24 tests passed.uv run ruff check scripts/latency_benchmark— passed.uv run ruff format --check scripts/latency_benchmark— passed.node --check scripts/latency_benchmark/src/report/report.js— passed.just latency-benchmark --help— passed without building Relay.just docs-linkcheck— passed with the unauthenticated Fern redirects check skipped.SKIP=cargo-deny,go-fmt,go-vet,attributions-python uv run pre-commit run --all-files— passed. The skipped Cargo-deny and Go executables are not installed, and the Python-attribution metadata download timed out twice; no lockfile changed.Where should the reviewer start?
Start with
scripts/latency_benchmark/README.mdfor the intended behavior and workflow, then reviewscripts/latency_benchmark/src/cli.pyfor progress reporting and failure-artifact preservation andscripts/latency_benchmark/tests/test_cli.pyfor the regression coverage.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
New Features
Documentation
Tests