Skip to content

ci(e2e): cut fixed setup and teardown costs of the k3s conformance rows - #5608

Merged
elezar merged 3 commits into
mainfrom
ci/5531-k3s-conformance-fixed-costs/matthewgrossman
Nov 18, 2026
Merged

elezar merged 3 commits into
mainfrom
ci/5531-k3s-conformance-fixed-costs/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Summary

Cut the fixed setup and teardown costs of the k3s conformance rows without touching the tests: cache the k3s setup layers, and kill the throwaway test VM instead of powering it off.

How the k3s disk cache is made safe:

  • The k3s setup layer is now saved without k3s's first-boot credentials, so each run's cluster generates its own.
  • Install layers, which hold the deployed gateway and the build under test, move to disks/install and are never cached; CI caches only the setup layers.

Related Issue

Closes #5531

Changes

  • Disk cache for ubuntu-k3s. The skip came from test(tmachine): add K3s conformance scenario #3848, following ci(qualification): add Ubuntu 24.04 k3s scenario #3455: cached disks would retain cluster credentials, so k3s needed run-scoped state. The setup layer does hold k3s's first-boot state (cluster CA keys, server token, node password, admin kubeconfig, datastore), because Wait for K3s runs before the layer is saved. k3s.yaml now stops k3s with k3s-killall.sh, deletes that state, keeps the containerd image store, and fails the layer build if any credential file remains, so each run's cluster generates its own credentials on first start. Measured on an arm64 VM: about 1 s added to the layer build and about 2 s to a first start; the node and kube-system pods came up healthy with new credentials on three boots. The install layer, which holds the deployed gateway and the build under test, is still never cached: tmachine keeps it under disks/install, and e2e-suites.yml caches only disks/*.qcow2. With no install layers in the cache, the key no longer includes the installer, so rows of one environment share an entry. The key also covers ansible_hash.rs, which feeds the setup hash. integration-runner.yml is unchanged: it no longer runs k3s by default and uses one shared cache key, which is a separate issue.
  • Teardown. Session::down kills QEMU instead of sending system_powerdown and waiting up to 120 s; the clean shutdown took 3–35 s. This is safe because nothing the test VM writes outlives it:
    • The test VM runs on a throwaway copy-on-write overlay of the install layer, deleted right after teardown, so a killed guest can't leave a disk that is reused.
    • Results and diagnostics are copied out before down is called.
    • Layer builds, whose disks are cached and reused, still power off cleanly, and a layer that fails to power off is not cached.
    • Killing was already the fallback: after the 120 s power-off timeout, and whenever a session is dropped on failure or panic.

Follow-ups

Testing

  • cargo clippy -D warnings and cargo nextest run for tests/tmachine and tests/runner
  • Cold and cached Branch E2E runs on the head with the credential wipe (the wipe was tested on an arm64 VM only)
  • actionlint and zizmor on e2e-suites.yml (no new findings), nix build .#tmachine-config, nix fmt, mise run markdown:lint
  • Branch E2E (earlier head, before the timeout and PostgreSQL changes were dropped) 43915914207 passed. Its first attempt saved the k3s disk cache, and re-runs of the two k3s conformance jobs restored it. k3s booted from the restored setup layer, and the install and all tests passed.

Timings

Measured on an earlier head (with the PostgreSQL pre-pull, before the credential wipe). Baseline: runs 43312288101, 43257220972 and 43237169662 (ranges across the three). "Cold" is this PR's first attempt, which built the setup layer. "Cached" is the re-run that restored it (about 6 s to restore 506 MB).

Row Phase Before Cold Cached
k3s Setup layer 47–50 s 47 s restored
k3s Provisioning (setup + install) 126–138 s 114 s 93 s
k3s Teardown 3–35 s 1 s 1 s
k3s Job 10.7–11.3 min 9.8 min 9.6 min
k3s-ha Setup layer 45–50 s 49 s restored
k3s-ha Provisioning (setup + install) 174–182 s 158 s 118 s
k3s-ha Teardown 23–32 s 2 s 1 s
k3s-ha Job 11.9–13.3 min 10.5 min 10.8 min

The test phase varies by about ±50 s between runs, which accounts for most of the spread in job time. The new k3s cache entry is 506 MB, against about 1 GB for the old entries that included install layers.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Cache the tmachine setup layers for ubuntu-k3s too, and leave install
layers out of the disk cache: they hold the build under test and, on k3s,
the gateway's credentials. Install layers now live under disks/install so
the cache can take only the setup layers, keyed per environment.

Kill QEMU at session teardown instead of powering the guest off, since the
session overlay is thrown away. Layer builds still power off cleanly.

Pull the HA PostgreSQL fixture image in the k3s setup layer, and run the
prebuilt test program with a 180-second per-test limit instead of the
900-second default.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 16, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test d52deca

…ixed-cost change

Leave per-test timeouts to the runner work that owns them, and stop
pulling the PostgreSQL fixture image in the k3s setup layer, which saved
no measurable time and tied the tmachine cache key to e2e/.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 9c481ad

The k3s setup layer boots k3s once, so it holds the cluster CA, token, node
password and admin kubeconfig. Stop k3s and delete that state before the layer
is saved, so each run's cluster generates its own credentials, and fail the
layer build if any credential file remains. Also key the disk cache on
ansible_hash.rs, which feeds the setup layer hash.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test d8d1a06

@elezar elezar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked on the head (d8d1a06):

  • The k3s setup layer builds with the credential wipe and the leftover check passes in CI (cold runs) and locally on arm64.
  • A restored layer booted, installed and passed the conformance k3s job in CI (re-run, cache hit).
  • Two boots from one setup layer generated different CA, server token and node password (local, arm64).

Not covered: k3s-ha from a restored layer, and the credential comparison on x86_64.

Non-blocking: the setup layers could move to disks/setup to match disks/install, which I'll do in #5628. A note that existing local setup layers are orphaned and rebuilt once would help, and a comment in the playbook that the wipe list is hand-maintained.

@elezar
elezar added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit ab14373 Nov 18, 2026
469 of 473 checks passed
@elezar
elezar deleted the ci/5531-k3s-conformance-fixed-costs/matthewgrossman branch November 18, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(e2e): cut fixed setup and teardown costs of the k3s conformance rows

2 participants