diff --git a/docs/building.md b/docs/building.md index e172789..a636935 100644 --- a/docs/building.md +++ b/docs/building.md @@ -43,7 +43,7 @@ have it. The artefact loads perfectly on the machine that built it and then fails at `dlopen` time in somebody else's application, with a message about a symbol version rather than about a build. -The same source built on `debian:bullseye-slim` needs at most `GLIBC_2.16`. +The same source built on `debian:bullseye-20241111-slim` needs at most `GLIBC_2.16`. ⭐ **glibc's backward compatibility is real, and it runs one way.** Both directions were measured, by preloading the built object onto `/bin/true`: @@ -64,7 +64,7 @@ that breaks, which is why the default is a container and the floor is 2.31. The floor is a **property of the build environment**, which is why the default is a container: it is the only portable way to pin one. The shipped builds use -`debian:bullseye-slim` (glibc 2.31), and the artefacts come out needing at most +`debian:bullseye-20241111-slim` (glibc 2.31), and the artefacts come out needing at most `GLIBC_2.16`. `scripts/build.sh --engine native` exists for a maintainer already on the floor @@ -179,12 +179,22 @@ Which floor image a target builds in is a property of the target: | target | floor image | cross compiler | notes | |---|---|---|---| -| aarch64, riscv64, ppc64, ppc64le | `debian:bullseye-slim`, floor glibc 2.31 | gcc-10 | measured against the Debian archive indices: all four cross packages exist on bullseye | +| aarch64, riscv64, ppc64, ppc64le | `debian:bullseye-20241111-slim`, floor glibc 2.31 | gcc-10 | measured against the Debian archive indices: all four cross packages exist on bullseye | | loongarch64 | `debian:trixie-slim`, floor glibc 2.36 | gcc-14 | the port postdates gcc-10, so bullseye has no cross compiler for it. The floor is 2.36 because that is the first glibc release that runs loongarch64 at all, so no older bundle exists to break | `scripts/build.sh` selects the trixie image for loongarch64 by itself; a `--floor-image` you pass wins over that. +The bullseye image is a DATED tag, and its apt sources are rewritten to +`archive.debian.org` by the build scripts. Bullseye left LTS on 2026-08-31, +and the live mirror's bullseye-security pool was emptied while its indices +stayed behind, so an install against the live mirror asks for versions whose +files no longer exist (measured: the glibc pool directory carries only +bookworm files while the bullseye-security index still lists `deb11u14`). The +archive serves the whole of bullseye main, signed by the keys the image +already trusts, and the dated tag carries the exact `libc6` the archive +offers, read out of the image layers before the pin was chosen. + ⛔ **Do not reach for `podman run --platform linux/` to get there.** Pulling a tag for another platform **replaces the cached image for that tag**, and the next x86-64 job using that image dies with `Exec format error`. That cost a run. diff --git a/experiments/20-build-newglibc.sh b/experiments/20-build-newglibc.sh index 488121f..8e9c07b 100644 --- a/experiments/20-build-newglibc.sh +++ b/experiments/20-build-newglibc.sh @@ -4,8 +4,23 @@ # libthr.so : needs a RE-HOMED symbol (pthread_create@GLIBC_2.34, moved into libc at 2.34) # newglibc/ : the newer runtime, for the "load a second libc" experiment set -eu -apt-get update -qq >/dev/null 2>&1 -apt-get install -y -qq gcc binutils patchelf >/dev/null 2>&1 +# ⚠ Same bootstrap rule as stage 3: the apt output is kept, and a toolchain +# that did not install fails the stage naming itself, because a silent +# failure here hands stage 3 no libraries and the evidence table then reports +# mismatches that name the cases instead of the cause. The stock http sources +# stay: this image ships no CA store, so https fails certificate verification +# (measured), and apt's package signatures are the integrity guarantee anyway. +apt-get update -qq -o Acquire::Retries=3 >/work/.apt2-update.log 2>&1 || true +apt-get install -y -qq -o Acquire::Retries=3 gcc binutils patchelf \ + >/work/.apt2-install.log 2>&1 || true +for _tool in gcc readelf; do + command -v "$_tool" >/dev/null 2>&1 || { + echo "STAGE 2 CANNOT RUN: $_tool did not install; the apt output follows" >&2 + sed 's/^/ update| /' /work/.apt2-update.log >&2 + sed 's/^/ install| /' /work/.apt2-install.log >&2 + exit 2 + } +done cd /work # ⚠ Three things in this stage carry the architecture: the loader name, the diff --git a/experiments/30-run-tests.sh b/experiments/30-run-tests.sh index 22535fa..878b399 100644 --- a/experiments/30-run-tests.sh +++ b/experiments/30-run-tests.sh @@ -3,7 +3,44 @@ # bundles an older glibc". Every experiment states its PREDICTION; the harness reports # MATCH / MISMATCH against it. A MISMATCH is a finding, not a failure of the harness. set -u -apt-get update -qq >/dev/null 2>&1 +# ⚠ THE BOOTSTRAP IS NOT THE TEST, AND ITS FAILURE IS NOT A MISMATCH. All of +# this used to be `>/dev/null 2>&1`, so a container whose apt could not fetch +# the toolchain ran the whole table with gcc missing and reported 56 +# MISMATCHes naming the cases instead of the cause. That happened: on the +# 2026-09-04 v0.2.3 tag run, the aarch64 runner's view of the bullseye +# repositories stopped working mid-day, and the discarded apt output was the +# only witness. The logs below made the cause readable on the next failure: +# deb.debian.org's bullseye-security POOL is being emptied while its indices +# still advertise the emptied versions, so `apt-get install` dies with 404s +# mid-download on some mirror edges and not others, because the edges cache +# differently. Measured: the pool directory for glibc carries only bookworm +# files while the bullseye-security index still lists deb11u14. +# +# So this stage takes its bullseye from archive.debian.org, which is the +# permanent home of an EOL suite and serves the whole of it (measured: the +# index, every package this stage asks for, and an InRelease that verifies +# against the keyring this image already carries). bullseye-security and +# bullseye-updates are dropped: the archive has neither, and the floor this +# stage measures is glibc 2.31 itself, not a point update of it. http, not +# https, because this image ships no CA store and apt's package signatures +# are the integrity guarantee. Check-Valid-Until is disabled because an +# archived suite is never re-dated; it is a no-op today and protective the +# day the archived Release file grows a Valid-Until. +printf '%s\n' 'deb http://archive.debian.org/debian bullseye main' \ + > /etc/apt/sources.list +rm -f /etc/apt/sources.list.d/*.sources 2>/dev/null || true +apt-get update -qq -o Acquire::Check-Valid-Until=false -o Acquire::Retries=3 \ + >/work/.apt-update.log 2>&1 || true +apt-get install -y -qq -o Acquire::Retries=3 gcc binutils python3 \ + >/work/.apt-install.log 2>&1 || true +for _tool in gcc python3 readelf; do + command -v "$_tool" >/dev/null 2>&1 || { + echo "STAGE 3 CANNOT RUN: $_tool did not install; the apt output follows" >&2 + sed 's/^/ update| /' /work/.apt-update.log >&2 + sed 's/^/ install| /' /work/.apt-install.log >&2 + exit 2 + } +done # The aarch64 cross toolchain and qemu-user are for section P, which RUNS the # aarch64 trampolines rather than only assembling them. They are installed # best-effort: if the host has no network for them the section SKIPS by name @@ -12,10 +49,10 @@ apt-get update -qq >/dev/null 2>&1 # ⚠ Not on an aarch64 host, where section P builds with the native gcc and # runs on the CPU. Installing an emulator for the architecture you are # standing on is how E76 came to run under qemu on real aarch64 silicon. -apt-get install -y -qq gcc binutils python3 >/dev/null 2>&1 if [ "$(uname -m)" != aarch64 ]; then - apt-get install -y -qq --no-install-recommends \ - gcc-aarch64-linux-gnu libc6-dev-arm64-cross qemu-user-static >/dev/null 2>&1 + apt-get install -y -qq --no-install-recommends -o Acquire::Retries=3 \ + gcc-aarch64-linux-gnu libc6-dev-arm64-cross qemu-user-static \ + >/work/.apt-cross.log 2>&1 || true fi cd /work diff --git a/experiments/42-build-floor.sh b/experiments/42-build-floor.sh index fd48587..83b51dd 100644 --- a/experiments/42-build-floor.sh +++ b/experiments/42-build-floor.sh @@ -2,9 +2,28 @@ # Build the preload and the probes on the OLDEST supported glibc, so they only # ever need symbols the AppImage's bundled runtime is guaranteed to have. set -eu -apt-get update -qq >/dev/null 2>&1 -apt-get install -y -qq --no-install-recommends gcc libc6-dev make python3 binutils \ - libgl1-mesa-dev libegl1-mesa-dev libx11-dev >/dev/null 2>&1 +# Same bootstrap rule as stages 2 and 3: bullseye comes from +# archive.debian.org because deb.debian.org's bullseye-security pool is being +# emptied while its indices still list the emptied versions (measured +# 2026-09-06), and the apt output is kept so a failed toolchain fails the +# stage naming itself instead of failing every later build with a missing +# file. http, not https, because the slim image ships no CA store. +printf '%s\n' 'deb http://archive.debian.org/debian bullseye main' \ + > /etc/apt/sources.list +rm -f /etc/apt/sources.list.d/*.sources 2>/dev/null || true +apt-get update -qq -o Acquire::Check-Valid-Until=false -o Acquire::Retries=3 \ + >/w/.apt4-update.log 2>&1 || true +apt-get install -y -qq -o Acquire::Retries=3 --no-install-recommends \ + gcc libc6-dev make python3 binutils libgl1-mesa-dev libegl1-mesa-dev \ + libx11-dev >/w/.apt4-install.log 2>&1 || true +for _tool in gcc make python3 readelf; do + command -v "$_tool" >/dev/null 2>&1 || { + echo "STAGE 42 CANNOT RUN: $_tool did not install; the apt output follows" >&2 + sed 's/^/ update| /' /w/.apt4-update.log >&2 + sed 's/^/ install| /' /w/.apt4-install.log >&2 + exit 2 + } +done mkdir -p /build/src && cd /build/src cp /repo/src/*.c /repo/src/*.h /repo/src/Makefile . mkdir -p /build/inventories /build/tools diff --git a/scripts/build-in-env.sh b/scripts/build-in-env.sh index dbeca37..5afcbcb 100644 --- a/scripts/build-in-env.sh +++ b/scripts/build-in-env.sh @@ -45,13 +45,30 @@ if [ "${CLD_INSTALL_DEPS:-0}" = 1 ]; then # ignored, so a package list that never downloaded surfaced one line later # as an install failure, which names the wrong step. export DEBIAN_FRONTEND=noninteractive - if ! apt-get update -qq >/tmp/cld-apt-update.log 2>&1; then + # A bullseye floor image installs from archive.debian.org. Measured on + # 2026-09-06: deb.debian.org's bullseye-security pool is being emptied + # while its indices still list the emptied versions, so an install dies + # with 404s mid-download on mirror edges that no longer carry the files + # and works on edges that still do, which is why the same build passed and + # failed within minutes of itself. The archive is the permanent home of an + # EOL suite and carries the whole of it, signed by the keys the image + # already trusts. bullseye-security and bullseye-updates are dropped with + # it: the floor this build asserts is glibc 2.31, not a point update of + # it. trixie and newer keep the stock mirrors untouched. + archive_flags='' + if grep -qs '^VERSION_CODENAME=bullseye$' /etc/os-release; then + printf '%s\n' 'deb http://archive.debian.org/debian bullseye main' \ + > /etc/apt/sources.list + rm -f /etc/apt/sources.list.d/*.sources 2>/dev/null || true + archive_flags='-o Acquire::Check-Valid-Until=false' + fi + if ! apt-get update -qq $archive_flags >/tmp/cld-apt-update.log 2>&1; then printf 'build-in-env: apt-get update failed. Its output:\n' >&2 sed 's/^/ | /' /tmp/cld-apt-update.log >&2 die "no package list, so the prerequisites cannot be installed" fi # shellcheck disable=SC2086 - if ! apt-get install -y -qq --no-install-recommends $pkgs >/tmp/cld-apt-install.log 2>&1; then + if ! apt-get install -y -qq -o Acquire::Retries=3 --no-install-recommends $pkgs >/tmp/cld-apt-install.log 2>&1; then printf 'build-in-env: apt-get install failed. Its output:\n' >&2 sed 's/^/ | /' /tmp/cld-apt-install.log >&2 die "could not install: $pkgs" diff --git a/scripts/build.sh b/scripts/build.sh index f521b5c..a076de2 100644 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -32,7 +32,16 @@ ROOT=$(dirname -- "$HERE") ENGINE=auto ARCH= -FLOOR_IMAGE=debian:bullseye-slim +# ⚠ A DATED tag, never the floating one. The floor is a property of the build +# environment, so the image is pinned to the one it was verified on: this tag +# carries libc6 2.31-13+deb11u11 on every architecture (read out of the image +# layers), which is exactly the version archive.debian.org serves, so the +# cross toolchain installs with no version gymnastics. The floating +# debian:bullseye-slim moves with point updates from the bullseye-security +# suite, whose pool left deb.debian.org while its indices stayed (measured +# 2026-09-06), which broke installs whose candidate versions no longer had +# files behind them. +FLOOR_IMAGE=debian:bullseye-20241111-slim FLOOR_IMAGE_GIVEN=0 FLOOR_GLIBC=2.31 OUT= diff --git a/scripts/run-appimage.sh b/scripts/run-appimage.sh index a53d0b9..4b5c4b2 100644 --- a/scripts/run-appimage.sh +++ b/scripts/run-appimage.sh @@ -6,7 +6,7 @@ # users actually complain about, on real software, across four host classes and # a second AppImage: # -# debian:bullseye-slim builds the artefacts on the glibc 2.31 FLOOR +# debian:bullseye-20241111-slim builds the artefacts on the glibc 2.31 FLOOR # alpine:3.22 musl host, the case the complaint is about # debian:trixie-slim glibc 2.41, OLDER than the bundled 2.44, so nothing # NEEDS rewriting. The regression case. @@ -88,7 +88,7 @@ if [ ! -d "$WORK/AppDir" ]; then die "extraction failed" fi -in_container debian:bullseye-slim 42-build-floor.sh || die "floor build failed" +in_container "$CLD_FLOOR_IMAGE" 42-build-floor.sh || die "floor build failed" in_container alpine:3.22 45-build-musl-guest.sh || die "musl guest build failed" fail=0 diff --git a/scripts/run-evidence.sh b/scripts/run-evidence.sh index 5fbfc82..befcd06 100644 --- a/scripts/run-evidence.sh +++ b/scripts/run-evidence.sh @@ -4,7 +4,7 @@ # alpine:3.22 builds a faithful musl-linked probe library # debian:trixie-slim builds libraries needing NEWER glibc symbols and # stages that newer runtime -# debian:bullseye-slim glibc 2.31, the "older bundled glibc" under test +# debian:bullseye-20241111-slim glibc 2.31, the "older bundled glibc" under test # # Every experiment declares a prediction; the harness reports MATCH/MISMATCH. # Exit 0 means every prediction held. This is the ~4 minute pre-commit gate. @@ -63,7 +63,7 @@ stage alpine:3.22 10-build-musl.sh sh || die "stage 1 failed" stage debian:trixie-slim 20-build-newglibc.sh sh || die "stage 2 failed" rc=0 -stage debian:bullseye-slim 30-run-tests.sh bash || rc=$? +stage "$CLD_FLOOR_IMAGE" 30-run-tests.sh bash || rc=$? say "" if [ "$rc" = 0 ]; then diff --git a/scripts/suite-lib.sh b/scripts/suite-lib.sh index d08df45..5c2cbde 100644 --- a/scripts/suite-lib.sh +++ b/scripts/suite-lib.sh @@ -181,6 +181,16 @@ fetch_verified() { # fetch_verified