Skip to content

test: shorten pplx-space-api's slowest tests without changing what they prove - #1425

Closed
zibi-pplx wants to merge 1 commit into
zibi/test-shared-costsfrom
zibi/api-slow-tests
Closed

zibi-pplx wants to merge 1 commit into
zibi/test-shared-costsfrom
zibi/api-slow-tests

Conversation

@zibi-pplx

@zibi-pplx zibi-pplx commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

About 70 of the pplx-space-api tests listed in ci/test-duration-budget.toml get faster. Each one still proves what it proved before, and production keeps today's values. The slow tests fell into four kinds.

The test Kubernetes substrate always waited out its budgets. Its reflectors are seeded once and never fed, so in tests:

  • every pod-cache miss waited the full 250 ms reflector budget;
  • every delete waited the 1 s blocking-delete timeout;
  • every follower of a joined restore saw its owner's change only on the next 2 s restore poll.

The reflector wait and the retry-deferral grace move into SubstrateBudgets, and the restore poll ceiling becomes a field of the restore waiters. The test substrate uses 20 ms, 20 ms and 100 ms. with_seeded_caches, the public test constructor that cluster-controller tests use, keeps production values.

Production bounds the tests waited out. These become injectable; production passes the existing constants:

Bound Production Tests
Sandbox list page byte budget (SandboxController.list_page_max_bytes) 8 MiB KiB-sized pages; the same 1/7/8 page shapes
Quota lock timeout (PublicOperationAdmission.quota_lock_timeout) 2 s 400 ms
Builder connect retry (DirectSubstrate.builder_connect_retry_policy) the default TCP policy 200 ms

Real waits in the tests themselves.

  • The native pod failover test, which proves a caller waits out the full 10 s connect budget, runs on tokio's paused clock. Its 9 s assertion still holds, in virtual time.
  • Fixed 1 s sleeps that showed a request blocked on a lock now poll pg_stat_activity for the waiter. The poll helper moved from the quota tests to test_support.rs.
  • Lock and access timeouts that a test waits out are shorter. Each still sits above the backoffs and sub-cases it has to separate.
  • The update-timeout test's lock observer kept running until its deadline after the update had finished. It now stops with the update.

Sequential sweeps. These loops become rstest cases:

  • state rendering, as 8 snapshot shards. The union of their rows equals the old snapshot, and each shard still generates the full lifecycle list.
  • hidden operations, e2b connect, sessions, owner scope, admin list and id forms;
  • the Kubernetes store behaviours, now 100 tests, each with its own store;
  • lifecycle characterization, native create policy, and create-ending-early.

The file-write tests also build each multi-MiB body once instead of re-encoding it.

Test Before After (slowest case)
routes::e2b::state_render_tests::each_surface_renders_… 15.8 s 0.38 s (8 shards)
native_build::a_failed_system_build_frees_its_key_… 10.6 s 0.38 s
hidden_operations_answer_as_nonexistent_… 10.6 s 0.41 s
updates::postgres_tests::unchanged_timeout_does_not_wait_… 9.8 s 0.32 s
characterization_tests::kubernetes::completed_actions_keep_… 6.8 s 0.67 s
the_write_body_limit_admits_the_largest_documented_payload 5.7 s 0.49 s
owner_scope_tests::scoped_sandbox_page_bounds_the_bytes_it_returns 5.4 s 0.37 s
restoring_access::a_joined_restore_that_its_owner_abandons_… 4.6 s 0.69 s

These are local times on a loaded machine.

Stack

  1. #1407, #1408, #1414, #1417, #1422: the test duration budget and the changes so far.
  2. This PR: pplx-space-api's slow tests.
  3. The same work for the other crates follows on top of this PR.

Validation

  • cargo clippy -p pplx-space-api --all-targets --all-features -- -D warnings and cargo fmt are clean.
  • Locally, 2,498 of the crate's 2,502 tests pass. The other four are native_pod_failover tests: they bind 127.0.0.2, which macOS does not allow, and fail the same way without this change.
  • The paused-clock failover test passed in 0.18 s locally on a routable stand-in address; CI on Linux runs it as written.
  • This PR does not yet remove exceptions. Once CI shows which listed tests now run well under the limit, I will remove theirs. Tests that became rstest cases have new names, so their old entries go too.

Severity and risk

Severity: does not apply; this is a CI speedup.

Risk:

  • Production. The only changes are values moving from constants into fields and parameters. Every production construction path sets them to the same constants.
  • Flakiness. Shorter test waits leave less slack under load. Each shortened wait still exceeds the backoffs and sub-cases its test separates, and the split ineligible-row cases got a longer lock-observation deadline (500 ms) because they now start at once.
  • To roll back, revert this PR.

AI assistance: Claude Code drafted the change, its tests and this description, using parallel agents whose patches it combined.

🤖 Generated with Claude Code

…ey prove

About 70 listed api tests spent their time waiting out production
timeouts, sweeping cases in sequence or moving megabytes. Each change
keeps the outcome the test distinguishes; production keeps today's
values.

- Test Kubernetes substrate: its reflectors are seeded once and never
  fed, so every pod-cache miss waited the full 250 ms reflector budget,
  every delete the 1 s blocking-delete wait, and every joined restore
  the 2 s restore poll. The reflector wait and the retry-deferral grace
  move into SubstrateBudgets, the restore poll ceiling into the restore
  waiters, and the test substrate uses 20 ms, 20 ms and 100 ms.
  with_seeded_caches keeps production values.
- Injectable bounds: the sandbox list page byte budget, the quota lock
  timeout and the builder connect retry policy. Production passes the
  existing constants. Tests use KiB pages, a 400 ms quota bound and a
  200 ms builder retry.
- Real waits: the failover test runs on tokio's paused clock; fixed
  sleeps that showed a request blocked on a lock now poll for the lock
  waiter; lock and access timeouts that a test waits out are shorter.
- Case sweeps become rstest cases: state rendering (8 snapshot shards
  whose union is the old snapshot), hidden operations, e2b connect,
  sessions, owner scope, admin list and id forms, k8s store behaviours
  (100 tests), lifecycle characterization and others.
- File body tests build each multi-MiB body once.

Locally the slowest of the changed tests went from 2-16 s to under
1.6 s; all 2,498 api tests pass except the four failover tests that
bind 127.0.0.2, which macOS does not allow.
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@zibi-pplx
zibi-pplx added this pull request to stack #1415 November 4, 2026 12:03
@zibi-pplx
zibi-pplx removed this pull request from stack #1415 November 4, 2026 12:05
@zibi-pplx
zibi-pplx deleted the branch zibi/test-shared-costs November 4, 2026 12:08
@zibi-pplx zibi-pplx closed this Nov 4, 2026
@zibi-pplx
zibi-pplx deleted the zibi/api-slow-tests branch November 4, 2026 12:08
@zibi-pplx

Copy link
Copy Markdown
Collaborator Author

Combined into #1426 with the rest of the test burn-down, so the stack is #1407, #1408 and #1426.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant