Skip to content

Commit ffbe0a6

Browse files
committed
docs: root README benchmark table from ONE run, both languages, n=3
原来那张表是**三次不同的跑拼起来的**(守卫只能硬编三个文件名),而且 n=1, 中英两份结构还不一样 —— 英文有 schedule 列,中文把它塞在脚注里。现在两份都由 `report.py --headline [--lang zh]` 从**同一份报告**生成: `bench/results/standard-20260814-linux-x86_64/gcc-mcpp-2026.8.11.3.json`,25/25 ok, 所有大于 1s 的中位数 min/max 都在 ±4% 内。 ## 数据本身换了个结论 新表有了 old-vs-new 列,于是最显眼的那个 200x **不再是「一直如此的默认行为」**: touch-hub 已发布 2026.8.11.3: 81.72s 本分支: 0.42s cmake: 83.21s edit-comment 已发布 2026.8.11.3: 79.11s 本分支: 0.40s cmake: 83.21s 也就是说级联抑制**以前并没有生效**,是这个分支让它真正工作的。旧文案写的是 「这是默认行为,不需要设任何键」,读起来像是它一直都在 —— 现在把话说准了。 `edit-body` 仍是对照组(1.1x 而不是 200x,级联是欠着的),`bmi_schedule` 明确写 「opt-in 且默认关闭」。 ## 列序 `engine_order` 改成:被测默认 → 它自己的 opt-in 臂 → 已发布参照 → 其他引擎。 原来按字符串排,把参照插在了默认和 opt-in 中间,两个要对比的列永远隔着一列。 ## 守卫 §5 重写:一份来源、两种语言、引擎名**从表头读**而不是在守卫里再列一遍 (加一条臂不会留下没被检查的列)。现在核 **50 个中位数**(5 行 × 5 引擎 × 2 语言), 原来是 20 个。另外 `-` 也被检查:标成「未测」的格子,数据里**必须真的没有** —— 否则 `-` 就成了藏数字的地方。
1 parent 1860c80 commit ffbe0a6

4 files changed

Lines changed: 163 additions & 142 deletions

File tree

README.md

Lines changed: 31 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -307,42 +307,38 @@ import mcpplibs.cmdline;
307307
## Benchmark
308308

309309
Building **mcpp itself** — 137 module interface units, 57k lines, every one of
310-
them `import std;` — with three engines given the **same compiler binary**.
311-
Each cell is the median wall-clock and how many times faster it is than cmake.
312-
313-
| scenario | what changed | **mcpp** `bmi_schedule=on` | mcpp default | cmake | xmake |
314-
|---|---|---|---|---|---|
315-
| `cold` | nothing built yet | **36.36s** · 2.5x | 79.54s · 1.2x | 92.33s · 1.0x | 90.30s · 1.0x |
316-
| `noop` | nothing at all | **0.16s** · 1.8x | 0.16s · 1.8x | 0.28s · 1.0x | 0.38s · 0.7x |
317-
| `touch-hub` | mtime on a widely-imported interface, content unchanged | 0.44s · 189.5x | **0.40s** · 207.9x | 83.39s · 1.0x | 82.08s · 1.0x |
318-
| `edit-body` | a real edit inside a function body | **30.48s** · 2.8x | 76.24s · 1.1x | 85.64s · 1.0x | 84.61s · 1.0x |
319-
| `edit-comment` | a comment added to a widely-imported interface | 0.44s · 188.5x | **0.38s** · 217.2x | 82.96s · 1.0x | 82.73s · 1.0x |
320-
321-
<sub>Linux x86_64 · i9-13900K · gcc 16.1.0 · n=1 · pinned workload `a749e9f`.
322-
**Both mcpp columns are shown because either alone misleads**: the default is
323-
what you get today, `bmi_schedule = "on"` is one opt-in manifest key that helps
324-
only where a cascade is genuinely owed — on the two rows where mcpp already
325-
skips the cascade it costs a little rather than saving any.
326-
`bmi_schedule` remains opt-in: it is new, and a scheduling change that is wrong
327-
is wrong silently.
328-
The `bmi_schedule=on` column was re-measured on 2026-08-14 after a defect was
329-
found in it — the earlier `touch-hub 0.22s` / `edit-comment 0.18s` were
330-
measuring a build that had not finished, because the object edge was being
331-
cleaned by the very restat that suppresses the cascade and ninja exited while
332-
the compiler was still running. See [`bench/README.md`](bench/README.md) §8b.</sub>
333-
334-
* **`touch-hub` and `edit-comment` are where the day goes.** cmake and xmake
335-
decide by timestamp and rebuild everything downstream; mcpp compares the BMI
336-
the compiler just produced against the previous one, and when the interface
337-
did not change it skips the cascade entirely. This is the DEFAULT behaviour —
338-
no key to set — and it is why those two rows read 200x.
310+
them `import std;` — with four engines handed the **same compiler binary**.
311+
Each cell is the median of **3 samples** and how many times faster it is than
312+
cmake. Every column comes from **one run**, [`bench/results/standard-20260814-linux-x86_64/gcc-mcpp-2026.8.11.3.json`](bench/results/standard-20260814-linux-x86_64/gcc-mcpp-2026.8.11.3.json).
313+
314+
| scenario | what changed | `mcpp@2026.8.13.1` | `mcpp@2026.8.13.1+schedule=on` | `mcpp@2026.8.11.3` | `cmake` | `xmake` |
315+
|---|---|---|---|---|---|---|
316+
| `cold` | nothing built yet | 86.69s · 1.1x | **35.73s · 2.6x** | 86.75s · 1.1x | 91.74s · 1.0x | 90.54s · 1.0x |
317+
| `noop` | nothing at all | **0.16s · 2.0x** | 0.18s · 1.8x | 0.24s · 1.3x | 0.32s · 1.0x | 0.38s · 0.8x |
318+
| `touch-hub` | mtime on a widely-imported interface, content unchanged | **0.42s · 197.7x** | 0.42s · 197.2x | 81.72s · 1.0x | 83.21s · 1.0x | 82.48s · 1.0x |
319+
| `edit-body` | a real edit inside a function body | 80.87s · 1.1x | **29.83s · 2.9x** | 81.19s · 1.1x | 85.30s · 1.0x | 84.33s · 1.0x |
320+
| `edit-comment` | a comment added to a widely-imported interface | **0.40s · 207.0x** | **0.40s · 207.0x** | 79.11s · 1.1x | 83.21s · 1.0x | 82.15s · 1.0x |
321+
322+
<sub>Linux x86_64 · i9-13900K · gcc 16.1.0 · n=3 · pinned workload `a749e9f` ·
323+
cmake 4.4.2 / xmake 3.1.0. `-` would mean not measured; there is none here.
324+
Min/max sit within 4% of every median above 1s.</sub>
325+
326+
* **`touch-hub` and `edit-comment` are where the day goes — and that is new.**
327+
cmake and xmake decide by timestamp and rebuild everything downstream. mcpp
328+
compares the BMI the compiler just produced against the previous one and, when
329+
the interface did not change, skips the cascade. The released **2026.8.11.3
330+
column shows this was not working before**: 81.72s, level with cmake. It is
331+
0.42s here. Default behaviour, no key to set.
339332
* **`edit-body` is the control.** There the interface really did change, so the
340-
cascade is owed — mcpp is 1.1x rather than 200x, and an engine that were
341-
faster would have skipped work it owed. `bmi_schedule=on` does not skip it
342-
either; it does the same owed work 2.5x faster.
343-
* **Cold builds** come down to one 26-deep chain of module interfaces. `mcpp`
344-
publishes each BMI as soon as it exists and moves code generation off the
345-
critical path; without that setting it is 79.5s, i.e. level with the others.
333+
cascade is owed — mcpp is 1.1x, not 200x. An engine that were faster on this
334+
row would be skipping work it owed. `bmi_schedule=on` does not skip it either;
335+
it does the same owed work 2.9x faster.
336+
* **`bmi_schedule` is opt-in and OFF by default** (`auto` resolves to off). It
337+
moves code generation off the critical path, which is why it only helps where
338+
a cascade is genuinely owed: `cold` 86.69s → 35.73s and `edit-body` 80.87s →
339+
29.83s, while on the two rows mcpp already skips the cascade it buys nothing.
340+
A scheduling change that is wrong is wrong *silently*, so it does not become
341+
the default on the strength of one machine.
346342

347343
📊 **[Methodology, pinned versions, and the full data →
348344
`bench/README.md`](bench/README.md)** · [简体中文](bench/README.zh-CN.md)

README.zh-CN.md

Lines changed: 31 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -302,37 +302,37 @@ import mcpplibs.cmdline;
302302

303303
## 性能对比
304304

305-
**同一个编译器二进制**、三个构建引擎构建 **mcpp 自己** —— 137 个模块接口单元、
306-
57k 行,每一个都 `import std;`。中位墙钟时间,越低越好。
307-
308-
| 场景 | 改了什么 | **mcpp** | cmake | xmake |
309-
|---|---|---|---|---|
310-
| `cold` | 什么都还没构建 | **79.54s** · 1.2x | 92.33s · 1.0x | 90.30s · 1.0x |
311-
| `noop` | 什么都没改 | **0.16s** · 1.8x | 0.28s · 1.0x | 0.38s · 0.7x |
312-
| `touch-hub` | 给被大量 import 的接口改 mtime | **0.40s** · 208x | 83.39s · 1.0x | 82.08s · 1.0x |
313-
| `edit-body` | 函数体内部一处真实修改 | **76.24s** · 1.1x | 85.64s · 1.0x | 84.61s · 1.0x |
314-
| `edit-comment` | 给被大量 import 的接口加一条注释 | **0.38s** · 218x | 82.96s · 1.0x | 82.73s · 1.0x |
315-
316-
* **冷构建三家都在 15% 以内** —— 依赖图是一条 26 层深的模块接口链,没有可供调度
317-
的余地,任何引擎都缩短不了一条链。
318-
* **`touch-hub` 才是一天里真正花掉的时间。** cmake 和 xmake 按时间戳判断,下游
319-
全量重建;mcpp 把编译器刚产出的 BMI 和上一份比对,接口没变就不级联 ——
320-
**0.40s 对 83s**
321-
* **`edit-body` 是对照组。** 那里接口确实变了,任何引擎都不该快,也确实都不快。
322-
323-
📊 **[测量方法、钉住的版本与完整数据 → `bench/README.zh-CN.md`](bench/README.zh-CN.md)**
324-
· [English](bench/README.md)
325-
326-
<sub>mcpp 为**默认配置**。Linux x86_64 · i9-13900K · gcc 16.1.0 · n=1 ·
327-
钉住的工作负载 `a749e9f`。opt-in 的 `[build] bmi_schedule = "on"` 能把 `cold`
328-
降到 36.4s、`edit-body` 降到 30.5s,但它**只在级联确实欠着的时候有用**:上表
329-
`touch-hub` / `edit-comment` 两行 mcpp 本来就跳过了级联,开了它反而略慢
330-
(0.44s 对 0.40s / 0.38s)。该键仍为 opt-in。
331-
它此前有一个增量正确性缺陷,已于 2026-08-14 修复;修复也推翻了那一列原先的
332-
`touch-hub 0.22s` / `edit-comment 0.18s` —— 那两个数量的是**没跑完的构建**
333-
经过与修正后的数据见 `bench/README.zh-CN.md` §8b。
334-
套件还测量了第二个独立工程(xlings)的两种代码风格;那份对比、已声明的不对称、
335-
以及「什么时候一个格子**不能**拿来比较」的规则,都在 `bench/README.md`。</sub>
305+
**四个构建引擎**编译 **mcpp 自己** —— 137 个模块接口单元、57k 行、每一个都
306+
`import std;` —— 并且**给它们同一个编译器二进制**。每格是 **3 轮的中位数**,以及
307+
相对 cmake 的倍率。所有列出自**同一次跑**,数据在
308+
[`bench/results/standard-20260814-linux-x86_64/gcc-mcpp-2026.8.11.3.json`](bench/results/standard-20260814-linux-x86_64/gcc-mcpp-2026.8.11.3.json)
309+
310+
| 场景 | 改了什么 | `mcpp@2026.8.13.1` | `mcpp@2026.8.13.1+schedule=on` | `mcpp@2026.8.11.3` | `cmake` | `xmake` |
311+
|---|---|---|---|---|---|---|
312+
| `cold` | 还没编过 | 86.69s · 1.1x | **35.73s · 2.6x** | 86.75s · 1.1x | 91.74s · 1.0x | 90.54s · 1.0x |
313+
| `noop` | 什么都没改 | **0.16s · 2.0x** | 0.18s · 1.8x | 0.24s · 1.3x | 0.32s · 1.0x | 0.38s · 0.8x |
314+
| `touch-hub` | 碰一下被大量导入的接口的 mtime,内容不变 | **0.42s · 197.7x** | 0.42s · 197.2x | 81.72s · 1.0x | 83.21s · 1.0x | 82.48s · 1.0x |
315+
| `edit-body` | 真的改了一个函数体 | 80.87s · 1.1x | **29.83s · 2.9x** | 81.19s · 1.1x | 85.30s · 1.0x | 84.33s · 1.0x |
316+
| `edit-comment` | 在被大量导入的接口里加一行注释 | **0.40s · 207.0x** | **0.40s · 207.0x** | 79.11s · 1.1x | 83.21s · 1.0x | 82.15s · 1.0x |
317+
318+
<sub>Linux x86_64 · i9-13900K · gcc 16.1.0 · n=3 · 钉住的工作负载 `a749e9f` ·
319+
cmake 4.4.2 / xmake 3.1.0。`-` 表示未测,本表没有。所有大于 1s 的中位数,
320+
min/max 都在 ±4% 以内。</sub>
321+
322+
* **`touch-hub``edit-comment` 是一天时间的去处 —— 而且这是新的。**
323+
cmake 和 xmake 按时间戳判断,把下游全部重编;mcpp 拿编译器刚产出的 BMI 与上一份
324+
比较,接口没变就**不级联**。已发布的 **2026.8.11.3 那一列说明它以前并没有生效**:
325+
81.72s,和 cmake 一个量级;这里是 0.42s。这是**默认行为**,不需要设任何键。
326+
* **`edit-body` 是对照组。** 那里接口是真的变了,级联是**欠着的** —— mcpp 是 1.1x
327+
而不是 200x。在这一行更快的引擎,是漏掉了它该做的活。`bmi_schedule=on` 也不跳过它,
328+
只是把同样欠着的活做快了 2.9 倍。
329+
* **`bmi_schedule` 是 opt-in,默认关闭**(`auto` 解析为 off)。它把代码生成移出关键
330+
路径,所以只在级联**确实欠着**时有用:`cold` 86.69s → 35.73s、`edit-body`
331+
80.87s → 29.83s;而在 mcpp 本来就跳过级联的那两行,它买不到任何东西。
332+
调度改错是**无声**的,所以不靠一台机器的数据就变成默认值
333+
334+
📊 **[方法、钉住的版本、完整数据 → `bench/README.zh-CN.md`](bench/README.zh-CN.md)** ·
335+
[English](bench/README.md)
336336

337337
## 平台支持
338338

bench/tools/report.py

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -184,11 +184,27 @@ def render(cells, baseline):
184184

185185

186186
def engine_order(engines):
187-
"""mcpp arms first (opt-in, then default, then the released reference)."""
187+
"""mcpp arms first, newest version first, each default before its opt-in arm.
188+
189+
Reading order matters more than it looks: the build under test and its
190+
`+schedule=on` arm answer one question ("what does the key buy?") and the
191+
released reference answers a different one ("did this get faster?"). Sorting
192+
the raw strings interleaved them — reference, opt-in, default — so neither
193+
pair sat together and every comparison was two columns apart.
194+
"""
195+
def version_key(e):
196+
head = e.split("+", 1)[0]
197+
ver = head.split("@", 1)[1] if "@" in head else ""
198+
parts = []
199+
for piece in ver.split("."):
200+
parts.append(int(piece) if piece.isdigit() else 0)
201+
return parts + [0] * (4 - len(parts))
202+
188203
def key(e):
189204
if not e.startswith("mcpp@"):
190-
return (2, e)
191-
return (0 if "+" in e else 1, e)
205+
return (1, [], 0, e) # other engines after every mcpp arm
206+
# negated version → newest first; base arm before its own option arm
207+
return (0, [-p for p in version_key(e)], 1 if "+" in e else 0, e)
192208
return sorted(engines, key=key)
193209

194210

0 commit comments

Comments
 (0)