Signature
docker/qemu/run-aarch64-boot-test-strict.sh, health battery at be412ee9 (aarch64 boot_tests
profile of aarch64-breenix-kernel.json, -M virt,gic-version=3 -cpu cortex-a72 -m 512 -smp 4,
Apple Silicon Mac, QEMU TCG). 40 boots, 35 PASS. 2 of the 5 reds are this shape:
[FAIL] Boot 1: Exec smoke did not complete (663 lines)
[FAIL] Boot 16: Exec smoke did not complete (735 lines)
score_serial emits that string when [EXEC_SMOKE:TARGET_OK] is absent. In both boots the serial
carries 0 EXEC_SMOKE lines of any kind — 0 of the 2 boots reached [EXEC_SMOKE:LAUNCH]. That
separates it from #599, whose wedge is after [EXEC_SMOKE:LAUNCH].
Rate and concurrency
|
boot 1 |
boot 16 |
| gate-internal concurrency at launch |
0 |
1 |
| serial lines |
663 |
759 |
| guest uptime at last heartbeat |
7447 ms |
9537 ms |
EXEC_SMOKE lines |
0 |
0 |
| crash / panic / abort markers |
0 |
0 |
[BOOT_TESTS:PASS] |
2 |
2 |
SCHED_STRAND_ORACLE reports |
2 |
2 |
| last stage reached |
init service sequence, just past block_eintr_oracle child exit |
the TTY/pty oracle, mid sys_posix_openpt |
Rate on this battery: 2 of 40 (5%). A third specimen of the same string was captured in the
preceding health-818 corpus, so 3 occurrences are on record across the two batteries. Neither of the
2 boots here launched at a gate-internal concurrency of 2 or more, so the gate's own parallelism does
not account for them.
Serials (scratch, not in-repo):
.../scratchpad/health-819-serials/strict-boot01-Exec-smoke-did-not-complete-663-lines-serial-tmp-breenix_aar.txt.../scratchpad/health-819-serials/strict-boot16-Exec-smoke-did-not-complete-735-lines-serial-tmp-breenix_aar.txt.../scratchpad/health-818-serials/strict3-boot4-exec-smoke-incomplete.txt
Last lines
Boot 1 ends on a complete line terminated by \r\n — the SIGTERM landed between two writes:
[TTBR0_ASID_CENSUS:untagged=0:tagged=4222:kernel=6167:cleared=10342]
F123456789SC[heartbeat] tid=1207 uptime_ms=6441 kbd_nonzero=0
[syscall] exit(0) pid=93 name=block_eintr_oracle_child_93
[TTBR0_ASID_CENSUS:untagged=0:tagged=6532:kernel=8831:cleared=15286]
[heartbeat] tid=1207 uptime_ms=7447 kbd_nonzero=0
Boot 16 ends mid-line, with no trailing newline — the SIGTERM landed inside a write:
[TTY_ORACLE:ctty:verdict=PASS:dev_tty_aliased_slave=1:survived_alias_close=1:eof_after_last_slave_close=1]
[pty] sys_posix_openpt(flags=0x902)
[pty] Created master fd=4 (pty 3)
[pty] Unlocked PTY 3 (fd 4)
F123456789SC
Heartbeat cadence held at ~1002-1006 ms right up to the final line in both boots, so the guest was
progressing at the instant it was cut off. It was late, not stuck.
Where the deadline sits relative to the work
The gate gives each boot timeout 20 (run-aarch64-boot-test-strict.sh:467) and polls
score_serial 12 times at sleep 1.5 (:513-522) for an 18 s ceiling, then SIGTERMs QEMU
(:525-526) and re-scores the file.
Measured from this battery's own preserved serials, a boot that reaches exec smoke emits
[EXEC_SMOKE:TARGET_OK] at ~12.85 s of guest uptime (boot 17, between the uptime_ms=12851 and
uptime_ms=13857 heartbeats), and is killed at ~17.7-18.6 s of guest uptime. So on a healthy boot
guest uptime tracks the host's 18 s window roughly 1:1, and the marker lands with ~5 s of headroom —
about 28%.
Boots 1 and 16 reached only 7.4 s and 9.5 s of guest uptime inside the same host window, i.e. their
guest clock advanced at roughly 41% and 53% of host wall-clock rate. At that rate the 12.85 s marker
would need 24-31 s of host time, well past both the 18 s poll ceiling and the 20 s timeout.
Two sub-shapes, both consistent with the guest getting a fraction of the CPU it needs
The SCHED_STRAND_ORACLE sampler sleeps a 50 ms guest-clock deadline per sample and reports at 5000 ms intervals of guest time, so a guest keeping pace with its own clock accrues ~100 samples per report.
- Boots 1 and 16 (this battery). Samples per report 9 -> 102 and 7 -> 100, i.e. +93 both — the
same per-guest-second work rate as the healthy boots (+93 to +99, boots 17/24/37 and both health-818
full boots). Their deficit landed entirely on the clock: the guest did a normal amount of work per
guest-second, and the guest clock itself fell behind host wall clock.
worst_cpu_scheduler_silence_ms topped out at 2930 and 2175, against 1664-2368 on the healthy
boots.
- health-818
strict3-boot4. The mirror image: guest uptime reached 18178 ms — pacing the host
window 1:1 — while samples per report ran 8 -> 40 -> 92 -> 153 (+32, +52, +61), a third to two
thirds of nominal, and worst_cpu_scheduler_silence_ms climbed 410 -> 3413 -> 6530 -> 8589 on CPU 0
with stranded=0 and nonprogress=0 throughout. Here the deficit landed on the work.
Both amount to the guest completing roughly a third to a half of a boot's worth of work in the gate's
18 s window. Which of the two clocks absorbs the deficit plausibly depends on whether the vCPU
threads are descheduled while the timer is armed (#657's lost-tick clock permanently drops the
stalled interval) or while running.
Why this is not attributed
The battery reproduced the signature at 5%, so it is a live gate red, and I am not charging it to
host overload on the evidence available. What would settle it is host-side, and the gate records 0 of the 3 facts below — see the companion tooling issue. Specifically:
- No per-boot host wall-clock start or end stamp.
report_failure writes a UTC stamp into the
preserved filename (:439) and the run as a whole has START_TIME/END_TIME (:551, :562), so
a boot's guest-uptime-to-host-wall-clock ratio — the discriminator computed above from a fixed
18 s assumption — cannot be measured directly.
- No record of concurrent
qemu-system-aarch64 processes or host load average. The battery's
operator observed 4-6 concurrent aarch64 QEMU processes from unrelated worktrees during this run,
which is exactly the fact that would settle it, and exactly the fact the script does not write down.
- No marker for which bound ended the boot — poll-loop exhaustion, the
timeout 20, or the crash-marker break land on one and the same kill and the same content-only re-score, so a
budget-exhausted boot and a wedged boot produce an identical verdict string.
Until those exist, this reads as a starved guest against a thin deadline, but that reading rests on
the two in-guest ratios above and cannot be made a gate-side ruling.
Two things worth deciding either way
- The 18 s poll ceiling leaves ~28% headroom over a marker that lands at ~12.85 s of guest uptime.
That margin is thin for a TCG guest on a shared laptop, whether or not the underlying cause here
is host load.
-cpu cortex-a72 on an Apple Silicon host forces TCG. A gate whose verdict depends on the host
finishing 4 emulated vCPUs' worth of work inside a fixed wall-clock window will keep producing
this red under any concurrent load, and will keep producing it with a string that names exec smoke
rather than the budget.
Related: #599, #657, #825, #786, #789.
Signature
docker/qemu/run-aarch64-boot-test-strict.sh, health battery atbe412ee9(aarch64boot_testsprofile of
aarch64-breenix-kernel.json,-M virt,gic-version=3 -cpu cortex-a72 -m 512 -smp 4,Apple Silicon Mac, QEMU TCG). 40 boots, 35 PASS. 2 of the 5 reds are this shape:
score_serialemits that string when[EXEC_SMOKE:TARGET_OK]is absent. In both boots the serialcarries 0
EXEC_SMOKElines of any kind — 0 of the 2 boots reached[EXEC_SMOKE:LAUNCH]. Thatseparates it from #599, whose wedge is after
[EXEC_SMOKE:LAUNCH].Rate and concurrency
EXEC_SMOKElines[BOOT_TESTS:PASS]SCHED_STRAND_ORACLEreportsblock_eintr_oraclechild exitsys_posix_openptRate on this battery: 2 of 40 (5%). A third specimen of the same string was captured in the
preceding health-818 corpus, so 3 occurrences are on record across the two batteries. Neither of the
2 boots here launched at a gate-internal concurrency of 2 or more, so the gate's own parallelism does
not account for them.
Serials (scratch, not in-repo):
.../scratchpad/health-819-serials/strict-boot01-Exec-smoke-did-not-complete-663-lines-serial-tmp-breenix_aar.txt.../scratchpad/health-819-serials/strict-boot16-Exec-smoke-did-not-complete-735-lines-serial-tmp-breenix_aar.txt.../scratchpad/health-818-serials/strict3-boot4-exec-smoke-incomplete.txtLast lines
Boot 1 ends on a complete line terminated by
\r\n— the SIGTERM landed between two writes:Boot 16 ends mid-line, with no trailing newline — the SIGTERM landed inside a write:
Heartbeat cadence held at ~1002-1006 ms right up to the final line in both boots, so the guest was
progressing at the instant it was cut off. It was late, not stuck.
Where the deadline sits relative to the work
The gate gives each boot
timeout 20(run-aarch64-boot-test-strict.sh:467) and pollsscore_serial12 times atsleep 1.5(:513-522) for an 18 s ceiling, then SIGTERMs QEMU(
:525-526) and re-scores the file.Measured from this battery's own preserved serials, a boot that reaches exec smoke emits
[EXEC_SMOKE:TARGET_OK]at ~12.85 s of guest uptime (boot 17, between theuptime_ms=12851anduptime_ms=13857heartbeats), and is killed at ~17.7-18.6 s of guest uptime. So on a healthy bootguest uptime tracks the host's 18 s window roughly 1:1, and the marker lands with ~5 s of headroom —
about 28%.
Boots 1 and 16 reached only 7.4 s and 9.5 s of guest uptime inside the same host window, i.e. their
guest clock advanced at roughly 41% and 53% of host wall-clock rate. At that rate the 12.85 s marker
would need 24-31 s of host time, well past both the 18 s poll ceiling and the 20 s
timeout.Two sub-shapes, both consistent with the guest getting a fraction of the CPU it needs
The
SCHED_STRAND_ORACLEsampler sleeps a 50 ms guest-clock deadline per sample and reports at 5000 ms intervals of guest time, so a guest keeping pace with its own clock accrues ~100 samples per report.same per-guest-second work rate as the healthy boots (+93 to +99, boots 17/24/37 and both health-818
full boots). Their deficit landed entirely on the clock: the guest did a normal amount of work per
guest-second, and the guest clock itself fell behind host wall clock.
worst_cpu_scheduler_silence_mstopped out at 2930 and 2175, against 1664-2368 on the healthyboots.
strict3-boot4. The mirror image: guest uptime reached 18178 ms — pacing the hostwindow 1:1 — while samples per report ran 8 -> 40 -> 92 -> 153 (+32, +52, +61), a third to two
thirds of nominal, and
worst_cpu_scheduler_silence_msclimbed 410 -> 3413 -> 6530 -> 8589 on CPU 0with
stranded=0andnonprogress=0throughout. Here the deficit landed on the work.Both amount to the guest completing roughly a third to a half of a boot's worth of work in the gate's
18 s window. Which of the two clocks absorbs the deficit plausibly depends on whether the vCPU
threads are descheduled while the timer is armed (#657's lost-tick clock permanently drops the
stalled interval) or while running.
Why this is not attributed
The battery reproduced the signature at 5%, so it is a live gate red, and I am not charging it to
host overload on the evidence available. What would settle it is host-side, and the gate records 0 of the 3 facts below — see the companion tooling issue. Specifically:
report_failurewrites a UTC stamp into thepreserved filename (
:439) and the run as a whole hasSTART_TIME/END_TIME(:551,:562), soa boot's guest-uptime-to-host-wall-clock ratio — the discriminator computed above from a fixed
18 s assumption — cannot be measured directly.
qemu-system-aarch64processes or host load average. The battery'soperator observed 4-6 concurrent aarch64 QEMU processes from unrelated worktrees during this run,
which is exactly the fact that would settle it, and exactly the fact the script does not write down.
timeout 20, or the crash-marker break land on one and the samekilland the same content-only re-score, so abudget-exhausted boot and a wedged boot produce an identical verdict string.
Until those exist, this reads as a starved guest against a thin deadline, but that reading rests on
the two in-guest ratios above and cannot be made a gate-side ruling.
Two things worth deciding either way
That margin is thin for a TCG guest on a shared laptop, whether or not the underlying cause here
is host load.
-cpu cortex-a72on an Apple Silicon host forces TCG. A gate whose verdict depends on the hostfinishing 4 emulated vCPUs' worth of work inside a fixed wall-clock window will keep producing
this red under any concurrent load, and will keep producing it with a string that names exec smoke
rather than the budget.
Related: #599, #657, #825, #786, #789.