Skip to content

ci: an apt-get stall in engine-tests burns the 30-minute job budget and fails the required-check gate #974

Description

@JarryShaw

apt-get in the engine-tests install step has no timeout and no retries, so a
single mirror stall consumes the whole 30-minute job budget and surfaces as
cancelled — which correctly fails Required checks passed and blocks the PR.

Hit twice within forty minutes, on #971 and #972, both docs-only:

run 36887906077  Engines Python 3.13 (PCAP_CT)  cancelled  15:57:45 .. 16:28:00  (30m15s)
run 36888646396  Engines Python 3.10 (PCAP_CT)  cancelled  16:01:29 .. 16:31:47  (30m18s)

The log pins it precisely. apt-get update fetched InRelease metadata, then stalled:

15:58:06  Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
16:27:58  ##[error]The operation was canceled.

29m52s of silence, then the timeout-minutes: 30 at unit-tests.yml:327 killed the
job. Not version-specific (3.13 and 3.10) and not leg-specific: on the same run the
other five PCAP_CT legs finished in 39–81 seconds.

Nothing in either diff can affect an engine's system-package install, so this is
purely infrastructural — but it is expensive, because one stalled leg of six turns a
green PR red and the 30-minute burn delays everything behind it.

Fix is on the apt invocations at unit-tests.yml:403-417: -o Acquire::Retries=3 -o Acquire::http::Timeout=30 so a stalled mirror fails fast and retries, plus a
step-level timeout-minutes well under the job's 30 so a hang cannot eat the budget.

Worth noting Required checks passed behaved exactly right here — refusing to treat
cancelled as a pass is the property #966 and #970 added tests for.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssues reporting a defect (set by the bug report template; a default, not an assessment)ciPull requests that change CI or workflow configuration (ci: subject prefix)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions