Skip to content

ci(e2e): key the tmachine disk cache on the setup layer hash - #5628

Draft
matthewgrossman wants to merge 7 commits into
mainfrom
ci/5627-tmachine-layer-key/matthewgrossman
Draft

matthewgrossman wants to merge 7 commits into
mainfrom
ci/5627-tmachine-layer-key/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Summary

CI now asks tmachine for the fingerprint of an environment's prepared disk and uses it as the cache key. Before, CI guessed it from a hand-written file list. When tmachine's inputs change, the cache key now changes with them, with no workflow edit.

Depends on #5608 (this PR is based on its branch, which limits the cache to ~/.cache/tmachine/disks/*.qcow2). Retarget to main after #5608 merges.

Related Issue

Closes #5627

Background

What tmachine caches. The E2E tests run inside VMs. Before tests can run, tmachine prepares each VM (installs Docker or Podman, test tools, Keycloak and so on), which takes several minutes. It saves the prepared VM as a disk file so it doesn't repeat that work.

How tmachine names the disk. It combines everything that went into the preparation (the OS base image, the Ansible scripts, the environment's playbooks and extra input files such as the Keycloak realm file) into one fingerprint, and names the file disks/<fingerprint>.qcow2. When any of those inputs change, the fingerprint changes, and tmachine prepares a fresh disk.

How CI keeps the disks between runs. CI runners start empty, so the workflow saves the disk files with GitHub's cache at the end of a run and restores them at the start of the next. GitHub files each cache entry under a key. If the key matches, it restores the files. After a matching key, it does not save again, even if the files changed.

The problem. The workflow built its key from its own hand-written list of files, which was supposed to match tmachine's inputs but wasn't checked. When someone added an input to tmachine but not to the list:

  1. The key still matched, so CI restored the old disk.
  2. tmachine's fingerprint had changed, so it ignored the old disk and prepared a new one (slow).
  3. Because the key matched, GitHub didn't save the new disk.
  4. Every later run repeated step 2, with no error.

The fix. tmachine gets a layer-key command that prints the fingerprint without starting a VM. The workflow runs it first and uses the output as the cache key. The hand-written list is gone. setup() and layer-key call the same function, so the key and the disk name can't disagree.

Changes

  • setup_hash in tests/tmachine/src/qemu/setup.rs is now the public setup_layer_key. setup() still uses it to name the disk.
  • New tmachine layer-key <environment> command prints that fingerprint.
  • e2e-suites.yml: a new step builds tmachine and runs layer-key, and the cache key is tmachine-<os>-<arch>-<fingerprint>. The hashFiles(...) list and its comment about keeping it in sync are deleted.
  • integration-runner.yml: this workflow runs on every Branch E2E run as podman-e2e-integration (it boots fedora-podman-rootless). It had the same problem in a worse form: one fixed key, tmachine-<os>-<arch>, that never changed, so it kept restoring a stale cache, and it cached all of ~/.cache/tmachine, including install layers that hold the commit's build. It also skipped ubuntu-k3s without a stated reason. It now uses the same layer-key step, the disks/*.qcow2 path and the same key. The k3s skip is removed: with install layers out of the cache and k3s credentials wiped from the setup layer (ci(e2e): cut fixed setup and teardown costs of the k3s conformance rows #5608), k3s setup disks are as safe to cache here as in e2e-suites.yml. Both workflows produce the same key for the same environment, so they can share a cache entry.
  • TESTING.md: the instruction to add setup inputs to the cache key by hand is replaced, since inputs now reach the key automatically.

Cost and other details

  • Base images. The fingerprint reads the base image, so the new step downloads the base images before the cache step. That isn't new work: openshell-test-runner depends on the same tmachine config, and the prepared disk is built on top of the base image, so the run step already downloads them. On Cachix, the Ubuntu image and what it depends on are about 623 MB. The file actually read for the fingerprint is a small overlay (28 MB for Ubuntu, 48 MB for Fedora), so hashing takes well under a second.
  • Building tmachine. Before this PR, e2e-suites.yml didn't build tmachine itself. The new step adds nix build .#tmachine. Its tools (qemu, ansible, git, sshpass) are the same ones the test runner already downloads. The only new item is the tmachine program. After a change to tests/tmachine/src or its lockfile, no copy is on Cachix yet, so each CI job compiles it. That took about 30 s locally on macOS. With CACHIX_AUTH_TOKEN set, later runs download it.
  • Environments with nothing to prepare. setup() uses the base image directly when an environment has no setup playbooks, so there's no disk to cache. layer-key still prints a fingerprint for these, and GitHub's cache would log a "nothing to save" warning. No current environment is like this.
  • Which disks are covered. Only the prepared setup disk. Install layers, which hold the commit's build, live under disks/install and are not cached. The environment name is no longer part of the key. Each environment's fingerprint already differs, and two environments with the same fingerprint would use the same disk anyway.

Testing

  • cargo fmt, cargo clippy --all-targets -- -D warnings and cargo test in tests/tmachine pass. cargo check in tests/runner, which depends on the crate, passes.
  • No new unit tests. The change exposes an existing function and adds a command that prints its result.
  • Ran nix run .#tmachine -- layer-key locally on aarch64-darwin for ubuntu-docker-rootful, ubuntu-k3s and fedora-podman-rootless-keycloak. Each printed a 64-character fingerprint in about 1.3 s once built. An unknown environment fails with environment "nope" is not defined.
  • actionlint on e2e-suites.yml and integration-runner.yml is clean. zizmor reports the same existing findings as the base branch for both.
  • I couldn't boot the amd64 rows on this machine. The cold-cache and warm-cache E2E runs still need to happen in CI. They have not run yet.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Cache the tmachine setup layers for ubuntu-k3s too, and leave install
layers out of the disk cache: they hold the build under test and, on k3s,
the gateway's credentials. Install layers now live under disks/install so
the cache can take only the setup layers, keyed per environment.

Kill QEMU at session teardown instead of powering the guest off, since the
session overlay is thrown away. Layer builds still power off cleanly.

Pull the HA PostgreSQL fixture image in the k3s setup layer, and run the
prebuilt test program with a 180-second per-test limit instead of the
900-second default.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 16, 2026
…ixed-cost change

Leave per-test timeouts to the runner work that owns them, and stop
pulling the PostgreSQL fixture image in the k3s setup layer, which saved
no measurable time and tied the tmachine cache key to e2e/.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The k3s setup layer boots k3s once, so it holds the cluster CA, token, node
password and admin kubeconfig. Stop k3s and delete that state before the layer
is saved, so each run's cluster generates its own credentials, and fail the
layer build if any credential file remains. Also key the disk cache on
ansible_hash.rs, which feeds the setup layer hash.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
elezar and others added 4 commits November 18, 2026 13:49
Install layers already live in disks/install. Give setup layers a
sibling directory so the CI cache path names the setup layers directly
and a wider glob cannot pull install layers into it.

Existing local setup layers in disks/ are no longer found and are
rebuilt once; they can be deleted.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add `tmachine layer-key <environment>`, which prints the hash tmachine
names the environment's setup layer after, or `none` when the
environment has no setup playbooks, without starting a VM. setup() now
derives its layer name from the same function.

Key the e2e-suites disk cache on that value instead of a hand-written
hashFiles list, so a new setup input changes the key without a workflow
edit. Skip the cache for environments without a setup layer.

Closes #5627

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The integration runner cached all of ~/.cache/tmachine, install layers
included, under one fixed key per OS and arch, and skipped ubuntu-k3s.
Cache only the setup layers and key them on `tmachine layer-key`, as
e2e-suites.yml does.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Print the setup-layer hash for every environment instead of `none` for
environments without setup playbooks, and drop the workflow checks for
it. Expose the existing hash function directly, and drop the tests and
the Ansible hashing split that only they needed.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@elezar
elezar force-pushed the ci/5627-tmachine-layer-key/matthewgrossman branch from e553a96 to b3fb5ab Compare November 18, 2026 12:54
Base automatically changed from ci/5531-k3s-conformance-fixed-costs/matthewgrossman to main November 18, 2026 12:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants