Make the evidence stages' apt bootstrap loud and https - #34
Merged
Samueru-sama merged 4 commits intoSep 6, 2026
Conversation
The v0.2.3 tag run died in the aarch64 evidence job with 56 mismatches that were not findings: every build in stage 3 had failed with gcc and python3 not found, because the container's apt could not fetch the toolchain and the bootstrap discarded apt's entire output. The 16:53 gates run on main had already failed the same way, three minutes after the same code passed on its pull request. What the logs show, measured: on the aarch64 runner the alpine and trixie stages kept working while the bullseye stage lost its toolchain, in the same minute, over the same mirror host; the x86_64 evidence job was green at the same time. From outside, bullseye's InRelease still verifies against bullseye's own keyring and its arm64 indices are all present. That leaves per-CDN-edge variance in the bullseye repositories as the remaining explanation, and bullseye left LTS on 2026-08-31, so churn there is expected while the suite migrates toward archive.debian.org. The swallowed apt output was the only witness that could have named the cause, and it was deleted. Both Debian stage bootstraps now fetch over https with three retries, keep their apt logs, and fail the stage naming the missing tool when the toolchain does not install, instead of running a table whose every case reports a mismatch about something else. The cross-compiler install for section P stays best-effort with its own log, unchanged in behaviour. The failure path is proven locally with a stubbed failing apt-get and no gcc on PATH: both stages exit 2 naming the tool and printing the update and install logs. The success path is exercised by CI on this pull request. No assertion in any experiment changed.
talaria0101
requested review from
Azathothas,
QaidVoid and
Samueru-sama
as code owners
September 4, 2026 17:39
Tested end to end in the fork, where the gates run reproduced the failure with the cause finally readable: - both evidence jobs: apt over https failed certificate verification, because slim images ship no CA store. That was the previous commit's regression, visible only because the bootstrap had been made loud. Reverted: apt goes over http, and the package signatures are the integrity guarantee. - the ppc64 and riscv64 cross builds: apt-get update succeeded, then the install died with 404s on packages the bullseye-security index still lists. Measured directly: deb.debian.org's debian-security/pool/updates/main/g/glibc carries only bookworm files while the bullseye-security index still advertises deb11u14. The pool is being emptied while the indices remain, and mirror edges cache differently, which is why the same build passed and failed within minutes of itself since v0.2.3. Bullseye now comes from archive.debian.org everywhere a container installs from it: stage 3 and the floor build of the AppImage suite, and scripts/build-in-env.sh for bullseye floor images. Measured against the archive: the index is present, the InRelease verifies against the keyring the images already carry, and every package any of these consumers asks for is there, including the cross toolchains and the GL headers. bullseye-security and bullseye-updates are dropped with the move, which archive does not carry; the floor this project builds and measures is glibc 2.31 itself, not a point update of it. trixie and alpine keep their stock mirrors. The loud-failure bootstrap is extended to the floor build of the AppImage suite, which had the same discarded-output pattern and would have failed the same way. Proven locally in both directions with a stubbed apt-get: failing installs exit 2 naming the tool with the update and install logs printed; succeeding ones pass the prologue.
The archive.debian.org switch alone was not enough, and the fork's CI said so in one line: 'Unable to correct problems, you have held broken packages'. The floating debian:bullseye-slim image carries a runtime built from the bullseye-security suite (libc6 2.31-13+deb11u14), and libc6-dev must version-match the installed libc6 exactly, so a main-suite archive that tops out at deb11u11 can never satisfy it. The floor image is now the dated tag debian:bullseye-20241111-slim, chosen by reading the dpkg status out of the image layers through the registry: it carries libc6 2.31-13+deb11u11 on both amd64 and arm64, which is exactly the version archive.debian.org serves, so the toolchain installs with no downgrades. A dated tag also stops the floor from moving underneath a release: the floating tag is rebuilt with every point update, which is how the image ended up on a runtime whose source suite was about to be emptied. The same tag replaces the floating one in the evidence stage, the AppImage suite's floor build, and docs/building.md, which now states why the pin and the archive rewrite exist.
Contributor
Author
|
Tested end to end in a fork before this update; the first commit alone was not enough and the fork runs said so. What the failures were, in the order the fork exposed them:
Fork validation (talaria0101/cross-libc-dlopen, same runner pools):
|
The dated floor tag was written out in three places: build.sh's default, the evidence stage's container, and the AppImage suite's floor build. Nothing tied them together, so a future change to one would leave the suites testing a floor the releases do not build on, which is exactly the kind of silent drift this repository refuses elsewhere. Both suites already source suite-lib.sh, so the tag lives there now, with the reason it is dated. scripts/build.sh keeps its own default because it runs without suite-lib.sh; its comment says the two move together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v0.2.3 tag run died in the aarch64 evidence job with 56
mismatches that were not findings: every build in stage 3 had failed
with gcc and python3 not found, because the container's apt could not
fetch the toolchain and the bootstrap discarded apt's entire output.
The 16:53 gates run on main had already failed the same way, three
minutes after the same code passed on its pull request.
What the logs show, measured: on the aarch64 runner the alpine and
trixie stages kept working while the bullseye stage lost its
toolchain, in the same minute, over the same mirror host; the x86_64
evidence job was green at the same time. From outside, bullseye's
InRelease still verifies against bullseye's own keyring and its arm64
indices are all present. That leaves per-CDN-edge variance in the
bullseye repositories as the remaining explanation, and bullseye left
LTS on 2026-08-31, so churn there is expected while the suite migrates
toward archive.debian.org. The swallowed apt output was the only
witness that could have named the cause, and it was deleted.
Both Debian stage bootstraps now fetch over https with three retries,
keep their apt logs, and fail the stage naming the missing tool when
the toolchain does not install, instead of running a table whose every
case reports a mismatch about something else. The cross-compiler
install for section P stays best-effort with its own log, unchanged in
behaviour.
The failure path is proven locally with a stubbed failing apt-get and
no gcc on PATH: both stages exit 2 naming the tool and printing the
update and install logs. The success path is exercised by CI on this
pull request. No assertion in any experiment changed.
Requested by Samueru via pi-discord.
Conversation: https://discord.com/channels/1313385177703256064/1545249294377033729
Transcript: https://talaria.qaidvoid.dev/?session=1i2d05e