Plot newly added benchmarks in perf radar charts

[Amaury Chamayou (achamayou)]

Summary

• plot benchmarks absent from main instead of omitting them from the radar chart
• use the branch's earliest run as their baseline and derive their band from available branch runs
• keep long benchmark labels distinguishable and explain the branch-local baseline in the generated comparison

Split from #8158 so this display support can land before benchmarks which depend on it.

Testing

• exercised radar rendering with a synthetic metric absent from main
• python -m black scripts/perf_compare_radar.py
• python -m ruff check --fix scripts/perf_compare_radar.py
• python -m py_compile scripts/perf_compare_radar.py

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Updates the perf radar chart rendering so benchmarks introduced on a feature branch still appear in comparisons, using a branch-local baseline and adding label shortening for readability.

Changes:

• Add word-elision based label shortening to keep long radar axis labels distinguishable.
• Plot benchmarks missing on main by deriving their baseline/band from branch runs instead of omitting them.
• Document the branch-local baseline behavior in the generated comparison output.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cjen1-msft]

Looks fine. The benchmark against earliest on this branch for new benchmarks is somewhat reasonsable. But also that could just be two PRs...

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (5)

scripts/perf_compare_radar.py:403

• When a benchmark is new to the branch, baseline is taken from branch_values[0] and then axes are dropped if baseline <= 0. If the earliest branch run happens to be 0/negative (e.g., a failed/placeholder measurement) but later branch runs are valid, the benchmark will still be omitted. Consider selecting the earliest positive value from branch_values as the baseline (or otherwise falling back to a positive value) to avoid unintentionally hiding newly added benchmarks.

        # A benchmark added by this branch has no main runs to build a baseline
        # from. Rather than drop it, which would make a new benchmark invisible
        # on the very pull request which adds it, use this branch's own earliest
        # run as the reference, so the axis is normalised and plotted exactly
        # like every other one.
        if main_values:
            baseline = ewma(main_values)
            sigma = statistics.pstdev(main_values) if len(main_values) > 1 else 0.0
        else:
            branch_values = [
                value
                for data in branch_runs
                if (value := metric_value(data, benchmark, metric)) is not None
            ]
            if not branch_values:
                continue
            # load_runs orders branch_runs oldest first, so this is the earliest
            # available value for the metric.
            baseline = branch_values[0]
            # Spread is measured the same way as for a benchmark with main
            # history, from the runs available, so that such an axis carries a
            # band and a noise threshold like every other one rather than
            # collapsing to a point at the baseline.
            sigma = statistics.pstdev(branch_values) if len(branch_values) > 1 else 0.0

        if baseline <= 0:
            continue

scripts/perf_compare_radar.py:275

• For max_length == 1, returning elided[:1] can produce a misleading single character (it doesn’t indicate truncation). Returning ELLIPSIS when max_length == 1 would better signal that the label was shortened.

def shorten_label(label: str, max_length: int) -> str:
    """Shorten a label to fit by eliding the middles of its words.

    Words are elided from left to right, each keeping its first and last letter,
    until the label fits. Benchmarks measured at several settings differ only in
    their last word, for example the interval in "Basic Blocking Locust 100ms",
    so eliding from the left keeps the part which tells them apart readable for
    as long as possible.
    """
    if max_length <= 0:
        return ""

scripts/perf_compare_radar.py:296

• For max_length == 1, returning elided[:1] can produce a misleading single character (it doesn’t indicate truncation). Returning ELLIPSIS when max_length == 1 would better signal that the label was shortened.

    # Every word is elided and it still does not fit. Keep the end, which is
    # what distinguishes one setting of a benchmark from another.
    elided = " ".join(words)
    if max_length <= 1:
        return elided[:max_length]
    return ELLIPSIS + elided[len(elided) - (max_length - 1) :]

scripts/perf_compare_radar.py:390

• This PR generalizes the notion of a baseline (it can now be main EWMA or a branch-local baseline). However, related helper docstrings (e.g., format_delta_percent and within_noise_band) still describe behavior specifically in terms of the main baseline/stddev. Updating those docstrings to refer to a generic baseline (and baseline-derived sigma) would keep documentation consistent with the new behavior.

        # A benchmark added by this branch has no main runs to build a baseline
        # from. Rather than drop it, which would make a new benchmark invisible
        # on the very pull request which adds it, use this branch's own earliest
        # run as the reference, so the axis is normalised and plotted exactly
        # like every other one.
        if main_values:
            baseline = ewma(main_values)
            sigma = statistics.pstdev(main_values) if len(main_values) > 1 else 0.0
        else:
            branch_values = [
                value
                for data in branch_runs
                if (value := metric_value(data, benchmark, metric)) is not None
            ]

scripts/perf_compare_radar.py:407

• This PR generalizes the notion of a baseline (it can now be main EWMA or a branch-local baseline). However, related helper docstrings (e.g., format_delta_percent and within_noise_band) still describe behavior specifically in terms of the main baseline/stddev. Updating those docstrings to refer to a generic baseline (and baseline-derived sigma) would keep documentation consistent with the new behavior.

        branch_percent = normalized_percent(branch_value, baseline)
        sigma_percent = normalized_percent(sigma, baseline)
        within_noise = within_noise_band(branch_percent, sigma_percent)

[Amaury Chamayou (achamayou)]

Looks fine. The benchmark against earliest on this branch for new benchmarks is somewhat reasonsable. But also that could just be two PRs...

I am not sure I understand what could be two PRs? Do you mean two PRs with the same branch name, while there is nothing on main? Or this change could be made as two PRs?