Skip to content

test(testkit): fail stalled steps before the runner stops the test - #5455

Open
elezar wants to merge 4 commits into
mainfrom
test/5333-surface-test-stalls/elezar
Open

elezar wants to merge 4 commits into
mainfrom
test/5333-surface-test-stalls/elezar

Conversation

@elezar

@elezar elezar commented Nov 10, 2026

Copy link
Copy Markdown
Member

Stacked on #5450. Review that PR first; this diff is against its branch.

Summary

A hung CLI call in a conformance lane showed only nextest's generic TIMEOUT, with no command name or output, and the gateway log tail began after the stalled test.

  • Time budget. The runner exports the per-test limit as OPENSHELL_TEST_TIMEOUT_SECS, from the same constant as the nextest profile and the program's --timeout. testkit caps its CLI, sandbox exec, sandbox wait and poll timeouts to the time left minus 5 s, so a stuck step fails with its own error. Cleanup keeps its own timeouts. Without the variable, nothing is capped.
  • CLI output. A command that times out reports its command name and the output it printed so far. A command that takes over 10 s, exits non-zero, or is still running (every 30 s) is reported on stderr. Fast successful commands print nothing.
  • Gateway logs. The k3s diagnostics collect gateway logs from the run's start time (--since-time, written by the runner in the guest), with a 30 minute fallback, keeping the first 64 KiB and last 512 KiB.

Related Issue

Part of #5333. Not covered here, so the issue stays open:

Testing

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test pass for tests/testkit and tests/runner.
  • New unit tests cover the budget cap and CLI timeout reporting.
  • collect-k3s.sh was only syntax-checked (bash -n), and no e2e lane was run.

…rows

Program rows relied on the program's 900 s default, so one hung test could use much of the 45-minute run timeout. Derive the per-test timeout from the same constants as the nextest profile (120 s) and pass it with a 10 s grace to the program.

Part of #5416.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
A hung CLI call only showed nextest's generic timeout, with no command name or output, and the gateway log tail started after the stalled test.

- The runner exports the per-test limit as OPENSHELL_TEST_TIMEOUT_SECS. testkit caps its CLI, exec, wait and poll timeouts to the time left, so a stuck step fails with its own error. Cleanup keeps its own timeouts.
- A CLI command that times out reports its output so far. A command that is slow, exits non-zero, or is still running is reported while the test runs.
- The k3s diagnostics collect gateway logs from the run's start time, with a 30 minute fallback, instead of the last 300 lines.

Part of #5333.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

@Karthik-Chowdary Karthik-Chowdary left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At 23edc35, the new per-test cap can expire while poll_until is still inside an observation: it caps the duration but only checks it after observe().await returns.

There is an existing caller affected by that mismatch: tests/support/keycloak/src/lib.rs::wait_ready runs curl --max-time 10 inside the observation. Its comment explicitly relies on the old between-attempt deadline checks, but a remaining test budget can now be shorter than that fixed 10-second request.

I reproduced the polling path with that current curl invocation against a loopback HTTP listener that accepted the discovery request and withheld its response. With OPENSHELL_TEST_TIMEOUT_SECS=6 (a 1-second step budget after the margin), the helper was still awaiting curl at 2 seconds rather than returning its own contextual timeout. The ready-result control passed. A local correction that bounds the observation by the remaining deadline and clamps the retry sleep returned the Keycloak realm at ... timeout at about 1 second; the pending-result control retained its last observation.

Could the capped deadline cover the in-flight observation as well as the retry interval, preserving the last completed observation in the error? Otherwise a late readiness attempt can still reach the runner's generic TIMEOUT before the helper reports which wait stalled. This is a polling/diagnostic check, not a Keycloak or full e2e qualification. No change to authentication behavior is being suggested.

@purp purp added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:dx Making OpenShell easy to contribute to for humans and agents. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 15, 2026
Base automatically changed from test/5416-program-row-timeout/elezar to main November 16, 2026 23:33
…t-stalls/elezar

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

# Conflicts:
#	tests/runner/src/bundle.rs
#	tests/testkit/testkit/src/cli.rs
#	tests/testkit/testkit/src/context.rs

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#5450 has merged, so this PR now targets main. I merged main into this branch as 900ba57 (a normal merge commit, no force push). The conflict resolutions:

  • tests/runner/src/bundle.rs: pub const TEST_TIMEOUT_SECS = SLOW_PERIOD_SECS * SLOW_TERMINATE_AFTER and pub const TEST_GRACE_SECS = 10 live here, and the generated nextest profile uses them. The test checks the full slow-timeout line and the 120 s value.
  • tests/runner/src/execute.rs: passes --timeout/--grace to the program from those constants, next to OPENSHELL_TEST_TIMEOUT_SECS, and drops its local grace constant. The program's 120 s limit that #5450 first carried now lands here, together with the deadline that makes it safe.
  • tests/testkit/testkit/src/cli.rs: keeps this PR's spawn-based handling.
  • tests/testkit/testkit/src/context.rs: crate::budget::release() followed by #5431's self.handles.cleanup.run(CLEANUP_TIMEOUT).

fmt, clippy and the unit tests pass for tests/runner and tests/testkit, and the conformance suite compiles. Nothing from either side was dropped.

Review notes on the change itself, most important first:

1. The stuck step's error is printed only after cleanup (medium-high). In run.rs run_in, the body's error is only shown by the final panic!, after cx.cleanup(). Cleanup lifts the cap and can wait up to CLEANUP_TIMEOUT (2 min) per sandbox delete. If a step times out at about 115 s and the delete takes more than about 15 s, the runner kills the test first and the output is the generic TIMEOUT again, which is what this PR sets out to fix. Suggest eprintln! the error (or panic message) before cx.cleanup() and keep the final panic. Optionally give cleanup a deadline that fits inside the grace period.

2. Some waits aren't capped (medium).

  • Sandbox::delete() waits on wait_deleted(.., DELETE_TIMEOUT, ..), 2 min, without budget::cap.
  • Sandbox::wait_absent(), added on main by #5431, is also uncapped. lifecycle.rs passes it 60 s.
  • Single gRPC calls have no timeout. In poll_until, the timeout is only checked after observe returns, so a hung call there never times out.

Suggest budget::cap on the first two, and a remaining-time tokio::time::timeout around observe (or a small budget helper for gRPC calls). Until then, the README's "the wait helpers give up" isn't quite true.

3. CLI timeout handling (low-medium). In cli.rs, stdout.finish() and stderr.finish() are awaited one after the other. With a grandchild holding a pipe, that can take 4 s of the 5 s margin. child.kill() only signals the direct child, so anything it started keeps running. Suggest tokio::join! for the captures, and process_group(0) plus a group kill as in program/process.rs.

Smaller items:

  • The budget clock starts at budget::start(), not at process spawn, so the real margin is a little under 5 s. Worth a note in the comment.
  • FLOOR gives every capped step at least 1 s, even after the deadline. Fine for tests that stop at the first error.
  • If date fails, a stale /var/tmp/openshell-run-started from an earlier run on a reused VM sets the wrong log window. Suggest rm -f before writing it. The 64 KiB head is mostly startup logs and may not be worth keeping.
  • tests/suites/conformance/src/lifecycle.rs:36-39 still calls the exec timeout "the runner's two-minute kill".
  • Timed-out CLI errors now include up to 2 KiB of stdout and stderr. A CLI that prints a credential would leak it into test output.

Items 1 and 2 seem worth fixing before merge. The rest can be follow-ups.

poll_until now bounds the observation in flight as well as the retry
sleep, and keeps the last completed observation in its timeout error.
run_in prints the body's error before cleanup so a slow cleanup cannot
hide it behind the runner's generic timeout. Sandbox::delete and
wait_absent respect the test budget, and the policy advisor retry loops
use poll_until. The runner removes a stale run-started marker before
writing a new one.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar elezar added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 3f6f04e. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. test:e2e Requires end-to-end coverage topic:dx Making OpenShell easy to contribute to for humans and agents. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants