Skip to content

Make the evidence stages' apt bootstrap loud and https - #34

Merged
Samueru-sama merged 4 commits into
pkgforge-dev:mainfrom
talaria0101:fix/evidence-apt-loud
Sep 6, 2026
Merged

Make the evidence stages' apt bootstrap loud and https#34
Samueru-sama merged 4 commits into
pkgforge-dev:mainfrom
talaria0101:fix/evidence-apt-loud

Conversation

@talaria0101

Copy link
Copy Markdown
Contributor

The v0.2.3 tag run died in the aarch64 evidence job with 56
mismatches that were not findings: every build in stage 3 had failed
with gcc and python3 not found, because the container's apt could not
fetch the toolchain and the bootstrap discarded apt's entire output.
The 16:53 gates run on main had already failed the same way, three
minutes after the same code passed on its pull request.

What the logs show, measured: on the aarch64 runner the alpine and
trixie stages kept working while the bullseye stage lost its
toolchain, in the same minute, over the same mirror host; the x86_64
evidence job was green at the same time. From outside, bullseye's
InRelease still verifies against bullseye's own keyring and its arm64
indices are all present. That leaves per-CDN-edge variance in the
bullseye repositories as the remaining explanation, and bullseye left
LTS on 2026-08-31, so churn there is expected while the suite migrates
toward archive.debian.org. The swallowed apt output was the only
witness that could have named the cause, and it was deleted.

Both Debian stage bootstraps now fetch over https with three retries,
keep their apt logs, and fail the stage naming the missing tool when
the toolchain does not install, instead of running a table whose every
case reports a mismatch about something else. The cross-compiler
install for section P stays best-effort with its own log, unchanged in
behaviour.

The failure path is proven locally with a stubbed failing apt-get and
no gcc on PATH: both stages exit 2 naming the tool and printing the
update and install logs. The success path is exercised by CI on this
pull request. No assertion in any experiment changed.


Requested by Samueru via pi-discord.
Conversation: https://discord.com/channels/1313385177703256064/1545249294377033729
Transcript: https://talaria.qaidvoid.dev/?session=1i2d05e

The v0.2.3 tag run died in the aarch64 evidence job with 56
mismatches that were not findings: every build in stage 3 had failed
with gcc and python3 not found, because the container's apt could not
fetch the toolchain and the bootstrap discarded apt's entire output.
The 16:53 gates run on main had already failed the same way, three
minutes after the same code passed on its pull request.

What the logs show, measured: on the aarch64 runner the alpine and
trixie stages kept working while the bullseye stage lost its
toolchain, in the same minute, over the same mirror host; the x86_64
evidence job was green at the same time. From outside, bullseye's
InRelease still verifies against bullseye's own keyring and its arm64
indices are all present. That leaves per-CDN-edge variance in the
bullseye repositories as the remaining explanation, and bullseye left
LTS on 2026-08-31, so churn there is expected while the suite migrates
toward archive.debian.org. The swallowed apt output was the only
witness that could have named the cause, and it was deleted.

Both Debian stage bootstraps now fetch over https with three retries,
keep their apt logs, and fail the stage naming the missing tool when
the toolchain does not install, instead of running a table whose every
case reports a mismatch about something else. The cross-compiler
install for section P stays best-effort with its own log, unchanged in
behaviour.

The failure path is proven locally with a stubbed failing apt-get and
no gcc on PATH: both stages exit 2 naming the tool and printing the
update and install logs. The success path is exercised by CI on this
pull request. No assertion in any experiment changed.
Tested end to end in the fork, where the gates run reproduced the
failure with the cause finally readable:

- both evidence jobs: apt over https failed certificate verification,
  because slim images ship no CA store. That was the previous commit's
  regression, visible only because the bootstrap had been made loud.
  Reverted: apt goes over http, and the package signatures are the
  integrity guarantee.
- the ppc64 and riscv64 cross builds: apt-get update succeeded, then
  the install died with 404s on packages the bullseye-security index
  still lists. Measured directly: deb.debian.org's
  debian-security/pool/updates/main/g/glibc carries only bookworm
  files while the bullseye-security index still advertises deb11u14.
  The pool is being emptied while the indices remain, and mirror edges
  cache differently, which is why the same build passed and failed
  within minutes of itself since v0.2.3.

Bullseye now comes from archive.debian.org everywhere a container
installs from it: stage 3 and the floor build of the AppImage suite,
and scripts/build-in-env.sh for bullseye floor images. Measured
against the archive: the index is present, the InRelease verifies
against the keyring the images already carry, and every package any
of these consumers asks for is there, including the cross toolchains
and the GL headers. bullseye-security and bullseye-updates are
dropped with the move, which archive does not carry; the floor this
project builds and measures is glibc 2.31 itself, not a point update
of it. trixie and alpine keep their stock mirrors.

The loud-failure bootstrap is extended to the floor build of the
AppImage suite, which had the same discarded-output pattern and would
have failed the same way. Proven locally in both directions with a
stubbed apt-get: failing installs exit 2 naming the tool with the
update and install logs printed; succeeding ones pass the prologue.
The archive.debian.org switch alone was not enough, and the fork's CI
said so in one line: 'Unable to correct problems, you have held broken
packages'. The floating debian:bullseye-slim image carries a runtime
built from the bullseye-security suite (libc6 2.31-13+deb11u14), and
libc6-dev must version-match the installed libc6 exactly, so a
main-suite archive that tops out at deb11u11 can never satisfy it.

The floor image is now the dated tag debian:bullseye-20241111-slim,
chosen by reading the dpkg status out of the image layers through the
registry: it carries libc6 2.31-13+deb11u11 on both amd64 and arm64,
which is exactly the version archive.debian.org serves, so the
toolchain installs with no downgrades. A dated tag also stops the
floor from moving underneath a release: the floating tag is rebuilt
with every point update, which is how the image ended up on a runtime
whose source suite was about to be emptied.

The same tag replaces the floating one in the evidence stage, the
AppImage suite's floor build, and docs/building.md, which now states
why the pin and the archive rewrite exist.
@talaria0101

Copy link
Copy Markdown
Contributor Author

Tested end to end in a fork before this update; the first commit alone was not enough and the fork runs said so.

What the failures were, in the order the fork exposed them:

  1. The bootstrap in stages 2/3 and the AppImage suite's floor build discarded all apt output, so a container whose apt failed ran the whole table with gcc missing and reported 56 mismatches naming the cases instead of the cause. The logs are kept now, and a missing tool fails the stage naming itself.
  2. With the cause readable, the real defect showed: deb.debian.org has emptied the bullseye-security POOL while its indices still list the emptied versions. Measured directly: debian-security/pool/updates/main/g/glibc carries only bookworm files while the bullseye-security index still lists deb11u14. Installs 404 mid-download, per mirror edge, which is why the same build passed and failed within minutes since v0.2.3. Bullseye now installs from archive.debian.org, whose whole bullseye main suite is present and verifies against the keys the images already carry (gpgv checked).
  3. The archive alone still failed with held broken packages: the floating bullseye-slim runtime is deb11u14 from the emptied suite, and libc6-dev must version-match exactly, while the archive tops out at deb11u11. The floor image is pinned to debian:bullseye-20241111-slim, chosen by reading dpkg status out of the image layers through the registry: it carries libc6 2.31-13+deb11u11 on both amd64 and arm64, exactly what the archive serves.

Fork validation (talaria0101/cross-libc-dlopen, same runner pools):

  • gates: 12/12 jobs, evidence 63/63 on x86_64 and 60/60 on aarch64, zero mismatches.
  • a tag ran the full release workflow: 16/16 jobs including publish, 66 assets across all six architectures and both variants.
  • downloaded assets verified: riscv64-gl-fwd.so is ELF RISC-V, ppc64-runtime-select is big-endian PowerPC64, ppc64le-egl-fwd.so little-endian PowerPC64, loongarch64-cross-libc-dlopen.so LoongArch, sha256 matches the release API, and the generated body carries the per-arch floor table (loongarch64 2.36, the rest 2.31).

The dated floor tag was written out in three places: build.sh's
default, the evidence stage's container, and the AppImage suite's
floor build. Nothing tied them together, so a future change to one
would leave the suites testing a floor the releases do not build on,
which is exactly the kind of silent drift this repository refuses
elsewhere. Both suites already source suite-lib.sh, so the tag lives
there now, with the reason it is dated. scripts/build.sh keeps its own
default because it runs without suite-lib.sh; its comment says the two
move together.
@Samueru-sama
Samueru-sama merged commit 34482c7 into pkgforge-dev:main Sep 6, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants