Skip to content

Commit db42341

Browse files
committed
docs(bench): publish the Linux standard set — 696 samples, gaps marked -
`bench/results/standard-20260814-linux-x86_64/`,七格跑了五格,**696 个测量点**, 每格 3 轮。两份 README 都加了「这一次跑到底覆盖了什么 —— 以及没覆盖什么」。 `-` 表示**没测**,既不是「不适用」也不是 0。没被标出来的缺口会被读成结果 —— 这正是这套套件要消灭的东西,所以未跑的两格、macOS、Windows 全部显式列成 `-`。 停在第五格的理由写进去了:每点约 55 秒,剩下两格还要约 2 小时,而它们不改变任何 已发布结论(两种代码风格在 gcc 上已完整,clang 在 mcpp 工作负载上已覆盖), 并且 `--resume` 能补上、不重复任何已测的点。 ## 两处必须写清楚的东西 **唯一的失败不是缺口,是发现。** `clang/mcpp/xmake` 五个场景全红,手工复现到 `formatter<basic_string<char>, wchar_t>`(窄格式串),根因是 xmake 在 clang 上的 默认形状必须用 full BMI,而 full BMI 会让 clang 22.1.8 编错下游 TU。已开 #424。 不用 `allow_failed` 糊过去。 **一个离群点被声明而不是被替换。** `+schedule=on/touch-hub` 测到 `[1.77, 20.82, 1.79]`(离散度 1066%)。紧接着 8 轮重测是 `[1.79 ×6, 1.78, 1.81]`,离散度 1% —— 是机器噪声,不是级联抑制偶发失效。 **已发布的那一格原样保留**:把第二次跑的样本拼进第一次跑的报告,正是这套套件 绝不能做的事。重测报告作为独立文件存进同一目录,因为「关于一个已发布数字的论断」 本身也必须可核。 顺带删掉 `gcc-xlings-2026.8.13.1.log` —— 第六格起了个头就被停了,一个有日志、 没报告的文件读起来像「它跑过」,而覆盖表说的是「未测」。
1 parent a2407ce commit db42341

4 files changed

Lines changed: 145 additions & 7 deletions

File tree

bench/README.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,63 @@ performance effect.**
142142

143143
## The standard data
144144

145+
### What this run actually covers — and what it does not
146+
147+
`bench/results/standard-20260814-linux-x86_64/` · **696 measured samples** ·
148+
3 samples per cell · Linux x86_64 · i9-13900K.
149+
150+
`-` below means **not measured**. It never means "not applicable" and never
151+
means zero: a gap that is not marked is a gap that gets read as a result.
152+
153+
| toolchain | project | cells | outcome |
154+
|---|---|---|---|
155+
| gcc | `fixture` | 72 | 72 ok |
156+
| clang | `fixture` | 90 | 90 ok |
157+
| gcc | `mcpp-2026.8.11.3` | 25 | 25 ok |
158+
| clang | `mcpp-2026.8.11.3` | 25 | 20 ok, **5 failed** (xmake — see below) |
159+
| gcc | `xlings-2026.8.11.2` | 25 | 25 ok |
160+
| gcc | `xlings-2026.8.13.1` | - | **not measured** — the run was stopped here |
161+
| clang | `xlings-2026.8.11.2` | - | **not measured** — the run was stopped here |
162+
| macOS (all cells) | - | - | **not measured** |
163+
| Windows (all cells) | - | - | **not measured** |
164+
165+
The run was stopped after five of seven cells: at ~55s per sample the remaining
166+
two were another ~2 hours and neither changes a published conclusion — the
167+
two-code-style comparison is complete on gcc, and clang is covered on the mcpp
168+
workload. **Re-running `bash bench/run-standard.sh --resume` fills them in
169+
without repeating anything**, because every sample above is journaled.
170+
171+
#### The one failure, and why it is a finding rather than a gap
172+
173+
`clang / mcpp-2026.8.11.3 / xmake` failed all five scenarios with
174+
`seed build exited 255`. Reproduced by hand, the cause is exact:
175+
176+
```
177+
__format/format_functions.h:99:30: error: call to implicitly-deleted default
178+
constructor of 'formatter<basic_string<char>, wchar_t>'
179+
```
180+
181+
on a **narrow** format string. xmake's default shape on clang is
182+
`--precompile``.pcm``-c .pcm`, which requires clang's **full** BMI, and
183+
publishing a full BMI to importers makes clang 22.1.8 miscompile a downstream
184+
TU. mcpp is immune because it publishes reduced BMIs and recompiles the source
185+
for the object edge. This is upstream, filed as
186+
[#424](https://github.com/mcpp-community/mcpp/issues/424) — not something xmake
187+
got wrong, and not something this suite should paper over with `allow_failed`.
188+
189+
#### One declared outlier
190+
191+
`gcc / xlings-2026.8.11.2 / mcpp+schedule=on / touch-hub` was measured as
192+
`[1.77, 20.82, 1.79]` — a 1066% spread around a 1.79s median. Re-measured
193+
immediately afterwards at 8 samples it was
194+
`[1.79, 1.79, 1.79, 1.79, 1.78, 1.79, 1.81, 1.79]`, a 1% spread with no
195+
outlier, so the 20.82s was machine noise rather than an intermittent cascade
196+
failure. That probe is kept beside the run as
197+
[`probe-touch-hub-outlier-8-samples.json`](results/standard-20260814-linux-x86_64/probe-touch-hub-outlier-8-samples.json)
198+
— a claim about a published number has to be checkable too. **The published cell is left exactly as measured**: splicing a second
199+
run's samples into a first run's report is the one thing this suite must never
200+
do. The probe is evidence about the number, not a replacement for it.
201+
145202
### What is pinned, and why every one of these is pinned
146203

147204
A benchmark number is only worth the list of things that were held still while

bench/README.zh-CN.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,63 @@ BENCH=$(ls -t bench/target/*/*/bin/mbench | head -1)
134134

135135
---
136136

137+
## 标准数据
138+
139+
### 这一次跑到底覆盖了什么 —— 以及没覆盖什么
140+
141+
`bench/results/standard-20260814-linux-x86_64/` · **696 个测量点** ·
142+
每格 3 轮 · Linux x86_64 · i9-13900K。
143+
144+
下表里的 `-` 表示**没测**。它不表示「不适用」,更不表示 0 —— 没有被标出来的缺口,
145+
读者会当成结果。
146+
147+
| 工具链 | 工程 | 格数 | 结果 |
148+
|---|---|---|---|
149+
| gcc | `fixture` | 72 | 72 ok |
150+
| clang | `fixture` | 90 | 90 ok |
151+
| gcc | `mcpp-2026.8.11.3` | 25 | 25 ok |
152+
| clang | `mcpp-2026.8.11.3` | 25 | 20 ok,**5 failed**(xmake,见下) |
153+
| gcc | `xlings-2026.8.11.2` | 25 | 25 ok |
154+
| gcc | `xlings-2026.8.13.1` | - | **未测** —— 跑到这里停了 |
155+
| clang | `xlings-2026.8.11.2` | - | **未测** —— 跑到这里停了 |
156+
| macOS(全部格) | - | - | **未测** |
157+
| Windows(全部格) | - | - | **未测** |
158+
159+
七格跑了五格就停:按每个点约 55 秒算,剩下两格还要约 2 小时,而它们不改变任何
160+
已发布的结论 —— 两种代码风格的对比在 gcc 上已经完整,clang 在 mcpp 工作负载上
161+
也已覆盖。**重跑 `bash bench/run-standard.sh --resume` 就能补上,且不会重复任何
162+
已测的点**,因为上面每一个点都在 journal 里。
163+
164+
#### 唯一的失败,以及它为什么是「发现」而不是「缺口」
165+
166+
`clang / mcpp-2026.8.11.3 / xmake` 五个场景全部 `seed build exited 255`
167+
手工复现后根因是精确的:
168+
169+
```
170+
__format/format_functions.h:99:30: error: call to implicitly-deleted default
171+
constructor of 'formatter<basic_string<char>, wchar_t>'
172+
```
173+
174+
而那是一个****格式串。xmake 在 clang 上的默认形状是
175+
`--precompile``.pcm``-c .pcm`,这条链**必须**要 clang 的 **full** BMI,
176+
而把 full BMI 发布给 importer 会让 clang 22.1.8 编错一个下游 TU。mcpp 不受影响,
177+
因为它发布 reduced BMI、object 边重编源码。这是**上游**问题,已记为
178+
[#424](https://github.com/mcpp-community/mcpp/issues/424) —— 不是 xmake 写错了,
179+
也不该用 `allow_failed` 糊过去。
180+
181+
#### 一个已声明的离群点
182+
183+
`gcc / xlings-2026.8.11.2 / mcpp+schedule=on / touch-hub` 测到的是
184+
`[1.77, 20.82, 1.79]` —— 中位数 1.79s,离散度 1066%。紧接着用 8 轮重测得到
185+
`[1.79, 1.79, 1.79, 1.79, 1.78, 1.79, 1.81, 1.79]`,离散度 1%、没有离群点,
186+
所以那 20.82s 是机器噪声,不是级联抑制偶发失效。那次重测的报告就放在同一个目录:
187+
[`probe-touch-hub-outlier-8-samples.json`](results/standard-20260814-linux-x86_64/probe-touch-hub-outlier-8-samples.json)
188+
—— 关于一个已发布数字的论断,本身也必须可核。**已发布的那一格原样保留**:
189+
把第二次跑的样本拼进第一次跑的报告,正是这套套件绝不能做的事。那次重测是
190+
关于这个数字的**证据**,不是它的替代品。
191+
192+
---
193+
137194
## 0. 钉住了什么,以及为什么每一条都必须钉住
138195

139196
一个基准数字的价值,等于取它时被摁住不动的那张清单。下面每一行都曾经是松的,

bench/results/standard-20260814-linux-x86_64/gcc-xlings-2026.8.13.1.log

Lines changed: 0 additions & 7 deletions
This file was deleted.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
{
2+
"protocol_version": 1,
3+
"started_at": "2026-08-15T03:08:35Z",
4+
"host": {
5+
"os": "linux",
6+
"arch": "x86_64",
7+
"cpu_model": "13th Gen Intel(R) Core(TM) i9-13900K",
8+
"logical_cores": 32,
9+
"physical_cores": 24,
10+
"heterogeneous": true,
11+
"ram_bytes": 67147722752,
12+
"toolchain": "/home/speak/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/bin/g++"
13+
},
14+
"cells": [
15+
{
16+
"engine": "mcpp@2026.8.13.1+schedule=on",
17+
"compiler": "gcc",
18+
"profile": "release",
19+
"scenario": "touch-hub",
20+
"fixture": "xlings-2026.8.11.2",
21+
"variant": "modules",
22+
"status": "ok",
23+
"note": "mcpp 2026.8.13.1",
24+
"runs": 8,
25+
"median_s": 1.787,
26+
"min_s": 1.785,
27+
"max_s": 1.809,
28+
"samples": [1.790, 1.787, 1.787, 1.786, 1.785, 1.787, 1.809, 1.788]
29+
}
30+
]
31+
}

0 commit comments

Comments
 (0)