fix(ci): recover K3s forwarding and gate test boot readiness

[matthewgrossman]

Summary

A temporary K3s API refusal can exhaust systemd's default start limit and leave the conformance gateway forwarder stopped after the API and gateway recover. Disable that limit, check functional readiness after every test VM boot, and fetch K3s diagnostics before a failing VM exits.

This fixes a reproduced recovery defect. The original merge-queue failure did not retain the journals needed to prove its exact trigger.

Related Issue

Refs #3954 and #2860; neither broader tracker should be closed by this change. This is a localized CI harness bug fix under the issue exemption in the instructions supplied for this work; it introduces no product feature or public API change.

Changes

• Disable the loopback forwarder's start limit and retry every five seconds.
• Add optional installer preparation playbooks, executed after automated test VM boots, including cached installations. Interactive shell suites skip preparation so gateway boot failures remain accessible for debugging. Existing installers default to no preparation.
• Check K3s API/node readiness and the gateway StatefulSet, then require openshell status --output json as tmachine to connect to the registered loopback gateway. This functional check covers forwarding with 24 retries at five-second intervals, replacing separate TCP-port and forwarder-active waits. It does not restart/reset the unit or change scenario timeouts.
• On installation, preparation, or conformance failure, collect bounded unit state/journals, listener state, Pod identity/container status, Service selectors, endpoint readiness, events, and current/previous gateway logs. Omit Secrets, kubeconfigs, environment dumps, and full Pod specifications; redact common credential fields.
• Fetch diagnostics before failure teardown and upload them with if: always(). Preserve the existing system/user gateway journal queries for other runtimes.
• Document diagnostics in CI.md and the CI monitoring skill.

Testing

• Simplification checks for 508035eefd57a1d13a6e126a34597db3ba735b2e: mise run pre-commit including the commit hook, both tmachine configuration tests, tmachine formatting/Clippy (existing ARM firmware lint allowed), Ansible syntax checks for installer/readiness, and diff checks passed. A process fixture with the compiled tmachine binary and fake QEMU/Ansible commands verified that an interactive shell opens despite failed readiness, automated readiness failure prevents conformance, and healthy automated preparation precedes conformance. This fixture validates harness behavior, not real VM/K3s recovery.

• Current-head CI qualification blocked for 508035eefd57a1d13a6e126a34597db3ba735b2e: Branch E2E Checks failed on attempt 1. The original K3s job passed gateway preparation but failed file_transfer::path_safety during sandbox creation: create sandbox failed: sandbox not found (7 tests passed, 1 failed, 0 skipped). The test's own gateway preflight was connected. Two independent fresh K3s repeats passed on attempt 1: repeat 2 and repeat 3. Both used artifacts from 36938749555 and this exact source SHA, with one conformance execution per job. Current campaign: 2 passed / 3 executions, with an unresolved provisioning failure; no clean qualification or absence-of-flakes claim. All first-failure workflow logs and the tmachine diagnostics artifact were preserved before any rerun. Diagnostics show a healthy forwarder; the bounded current gateway log begins after the failing request, so its precise cause is unconfirmed. No test/job reruns were dispatched. Monitoring is paused pending a scope decision for further provisioning investigation. Earlier successful campaigns below apply only to their recorded heads.

• Rebased without conflicts onto main 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2, including Python PTY fix test(python): synchronize interactive exec TTY readiness #4076. Previous tested head: 2e0202c72299f54b0fc1b4922a61757c53008d86. git range-diff confirms the fix is unchanged; both tmachine configuration tests and diff checks passed after rebase; DCO sign-off preserved.

• Previous-head qualification after rebase 2e0202c72299f54b0fc1b4922a61757c53008d86: Branch E2E Checks passed on attempt 1 with 65 successful jobs, including Core E2E and GPU E2E. Optional Kubernetes HA and credential-driver suites were not enabled and are not counted as passing. Three independent fresh ubuntu-k3s QEMU/KVM conformance executions passed on attempt 1: original, repeat 2, repeat 3. All used candidate artifacts from run 36933214181 for this exact source SHA. Logs show one conformance execution per job with no test/job reruns or flaky-test markers. The readiness gates performed 3, 4, and 3 expected boot polls. No flakes were observed in this sample; three successful runs do not guarantee absence of future flakes.

• Previous-head branch CI qualification for 5763091c48a3050a48748a383a0030f20ba8e487: Branch E2E Checks passed on attempt 1 with 65 successful jobs, including Core E2E and GPU E2E. Optional Kubernetes HA and credential-driver suites were not enabled and are not counted as passing.

• Previous-head qualification: three independent fresh ubuntu-k3s QEMU/KVM conformance executions passed on attempt 1: original job, repeat 2, and repeat 3. All used matching candidate artifacts and the unchanged PR commit.

• Reviewed CI logs for flaky-test/retry markers and verified all workflow attempts were 1. The K3s gate performed expected startup polling for the API, forwarder, and CLI connection; conformance ran once per job with no test reruns. No flakes were observed in this sample; three successful runs do not guarantee absence of future flakes.

• mise run pre-commit, including the installed commit hook, passed.

• cargo test --manifest-path tests/tmachine/Cargo.toml: two configuration compatibility tests passed.

• tmachine Clippy passed with only its existing ARM option_env_unwrap warning allowed; no firmware code changed.

• Ansible syntax checks for installer, preparation, and conformance playbooks; Bash syntax, ShellCheck, Nix syntax, and diff checks passed.

• Live regression on isolated Ubuntu 24.04/systemd 255 and pinned K3s v1.36.3+k3s1: a 12-second API refusal left the original unit permanently failed. The actual new readiness role rejected it and fetched diagnostics. The exact proposed unit recovered automatically (StartLimitIntervalUSec=0, RestartUSec=5s, three retries) and the readiness role passed.

• All eight prepared conformance scenarios passed in 242 seconds with the proposed unit and readiness check, using the original failing candidate's CLI, images, chart, and test archive (1ad4e428a67c2a0869fd2043159acacea5d4683e, artifact 11173596960). This exercises the change against that candidate rather than claiming newly built main artifacts.

Full mise run ci was attempted and stopped on an unrelated Snap packaging documentation assertion. The release-range Git fixtures also failed because they inherited the local pre-commit hook; all nine pass with hooks disabled only for the disposable fixture process. Nix evaluation could not download its pinned flake input because of the local certificate trust configuration. These checks are not reported as green.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• CI documentation and affected contributor skill updated; production gateway deployment and configuration are unchanged

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.