Skip to content

feat(benchsdk): add optional display manifest to BenchmarkConfig and upload it on upsert - #389

Open
dtice25 wants to merge 5 commits into
masterfrom
devin/1787085805-display-config-v2
Open

feat(benchsdk): add optional display manifest to BenchmarkConfig and upload it on upsert#389
dtice25 wants to merge 5 commits into
masterfrom
devin/1787085805-display-config-v2

Conversation

@dtice25

@dtice25 dtice25 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Makes the *.bench.ts file the source of truth for both display and scoring metadata. This PR adds an optional display manifest to BenchmarkConfig and also makes scoring.metrics unit metadata derive from display.metrics, removing the risk that the two config blocks drift.

What changed

  • packages/benchsdk-runner/src/bench-config.ts

    • Added display?: BenchmarkDisplayConfig to BenchmarkConfig.
    • BenchmarkDisplayConfig carries metrics, steps, and overview defaults.
    • defineBenchmarkConfig validates the manifest shape and rejects unknown metric display fields.
    • validateBenchmarkScoringConfig now accepts an optional display argument: scoring.metrics[i].unit is optional, and when both scoring and display declare the same key the units must match.
  • packages/benchsdk-runner/src/scoring.ts

    • BenchmarkScoringMetric.unit and MetricScoring.unit are now optional.
    • scoringConfigToSpec(config, dimensions, display) resolves each scoring metric's unit from the matching display.metrics entry, falling back to metric.unit, then defaulting to an empty string.
    • lowerIsBetter / higherIsBetter accept an optional unit so existing onScore callbacks keep working.
  • packages/benchsdk-runner/src/runner.ts

    • Passes config.display into scoringConfigToSpec so runtime scoring uses the canonical display unit.
  • packages/benchsdk-runner/src/__tests__/bench-config.test.ts and src/__tests__/scoring.test.ts

    • Added tests for unit derivation, fallback, and conflict validation.
  • benchmarks/browser/browser.bench.ts, benchmarks/sandbox/tti.bench.ts, packages/create-bench/src/index.ts

    • Removed unit from scoring.metrics and left it on display.metrics only.

Why

Previously the platform had to know about each benchmark's metrics in custom-metrics.ts and title-cased step names. Now the author declares labels, units, sort direction, and default overview layout in the same file that defines the workload, and scoring can reference those metrics without duplicating unit metadata.

Backwards compatibility

display is optional. Benchmarks without it continue to use existing platform-side fallbacks (METRIC_DISPLAY and title-cased step names). scoring.metrics[i].unit is still accepted when display is absent.

Link to Devin session: https://app.devin.ai/sessions/a45e3a08012c47e989ccfc4c055275ac
Open in Devin Desktop: https://app.devin.ai/desktop/session/a45e3a08012c47e989ccfc4c055275ac?variant=devin
Requested by: @dtice25


Devin Review

dtice25 and others added 2 commits September 3, 2026 15:01
…hmark

- Add optional BenchmarkConfig.display spec with metrics, steps, and overview defaults.
- Validate display manifest in defineBenchmarkConfig.
- Upload display config into benchmarks.config JSONB on upsert.
- Export display types from @benchsdk/runner.
- Add display manifest to browser-lifecycle benchmark.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…arks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Sep 3, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

dtice25 and others added 2 commits September 3, 2026 15:14
…ed metrics

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Make  optional in  and treat  as the
canonical source of metric metadata.  now resolves each
scoring metric's unit from the display manifest at runtime, falling back to
the scoring metric's own unit and then to an empty string.

 no longer requires ; when both
 and  declare the same key, it validates that their units do
not conflict. Updated , , and the
 template to remove duplicate  declarations from
and rely on  instead.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant