[zibi-pplx]

Summary

The typed Kind smoke now runs on eight shards instead of six. The ECW and native passes are partitioned together, so each shard carries one even load.

Why. After the earlier Kind CI work, the slowest typed shard (about 10.8 min) is the largest part of a Kind PR's roughly 20-minute required check. Each shard runs its ECW scenarios and then its native ones, but each pass was partitioned on its own, so a shard's two loads did not add up evenly. By the duration hints, eight shards partitioned per surface would range from 7.3 to 9.9 minutes.

What changes.

• select_shard takes an optional inventories: the eligible scenarios of every API surface a sharded --all run covers (JOINT_SURFACES, ECW then native, as scripts/tests/smoke-kind-typed.sh runs them).

  • It partitions all their (surface, scenario) pairs at once and returns the current surface's share.
  • Every pass computes the same partition from the same registry and filters, so each pair still runs on exactly one shard.
  • Category runs keep the per-surface partition.

• By the hints, eight shards then carry 7.7 minutes each. That is about the leader-election brownout's own length, so a ninth shard would not lower the slowest.

• The Kind matrix runs shards 1-8, and the job name, SMOKE_SHARD and the Kind README follow.

• A pass whose share of the joint partition is empty, while the shard's other pass has work, says so and exits 0 without a bootstrap. A long scenario can fill a shard by itself: at nine shards, the brownout's shard has no native scenario. A shard with nothing on either surface still fails.

• Cargo.lock bumps pulldown-cmark, which prost-build pulls in, from 0.13.5 to 0.13.6 for RUSTSEC-2026-0433. The advisory was published today and fails the Rust dependency check (cargo deny) on every PR until the lockfile moves. Only that entry and its checksum change; the checksum matches the crates.io index. supply-chain/config.toml carries the existing safe-to-deploy exemption forward from 0.13.5 to 0.13.6, as earlier advisory bumps did (build: update zstd dependencies for published advisories #972, build: update Tokio and ctutils for security advisories #901), with a note that this is not an independent audit.

Value. The slowest typed shard drops from a median of 10.65 to about 8.4 minutes, so the required check is about 2.2 minutes shorter on every Kind PR.

Stack

1. kind-ci: run the typed Kind smoke on eight shards, balanced across both API surfaces #1267 (this PR): eight shards, balanced across both API surfaces.
2. #1268: install the observability profile's images and receivers in parallel.

Validation

• space_smoke/tests/test_sharding.py:
  • the existing exactly-once test now runs both surfaces' passes at four, eight and nine shards. Every (surface, scenario) pair runs exactly once, and a pass with an empty share exits 0 without a bootstrap. The nine-shard case fails without the empty-share exit;
  • by the hints, the joint partition's heaviest shard is strictly lighter than the per-surface one's: 462 s against 592 s. It fails if select_shard ignores inventories.
• All 13 sharding tests pass. 622 of the 623 space_smoke tests pass on macOS; the ECW port-forward test fails there on main too (wc -c padding).
• cargo metadata --locked resolves with the bumped lockfile; the dependency check runs in CI.
• Kind CI: every shard passed in all eight-shard runs so far. CI for this head is running.

The slowest typed shard and summed shard job time, against six-shard PR runs whose shards did not wait for the producer:

Runs	Slowest typed shard	Shard runner-minutes
ten six-shard PR runs	10.4-11.7 min, median 10.65	median 106.4
this PR's previous head (run)	9.1 min	132.7
#1268's previous head (run)	8.3 min	124.7
five eight-shard attempts in all	8.1-10.9 min, median 8.4	124.6-133.1

That is about 2.2 minutes saved for about 20% more runner time. In the 10.9-minute shard, local.reconcile_queue_gc_prunes_only_gone_nodes took 300 s against its 132 s hint. It waits for a GC tick of up to 300 s, so it varies from about 26 to 300 s. With eight tightly balanced shards, such a scenario sets the tail when it runs long; see Not covered.

Severity and risk

Severity: does not apply; this is a CI speedup.

Risk:

• Runners. A Kind run needs nine concurrent 16-core runners (the producer and eight shards), and a main push a tenth for kind_rust_state. The runner's limit is 150. Each shard repeats the cluster setup, so Kind runner time rises by about 20% (Validation). In main runs since six shards merged, shards waited up to 1.9 min for a runner; measure main's shard queue after this merges, as the Kind README asks.
• Scenario order. A shard now runs a different set of scenarios, so a hidden ordering dependency could surface.
• Rollback. Revert this PR.

Not covered

• local.reconcile_queue_gc_prunes_only_gone_nodes waits for a GC tick of up to 300 s. Hinting it at its slow end, or shortening the tick through a test hook as the brownout's budget was, would keep it from setting the tail.

AI assistance: Claude Code drafted the change, its tests and this description.

🤖 Generated with Claude Code

[ac-pplx]

Each Kind CI shard runs its ECW scenarios and then its native ones, as two processes, and until now each pass was split across the shards on its own. A shard could therefore draw a heavy share of both, and the slowest shard sets the required check's wall time. This PR splits the (surface, scenario) pairs of both passes together and goes from six shards to eight.

shard 1 at eight shards, by the duration hints
               each pass split alone       split together (this PR)
ECW pass       brownout 457 s + others     brownout 457 s only
native pass    its own eighth of native    2 scenarios, 5 s
total          592 s (9.9 min)             462 s (7.7 min)

At this head (run) every (surface, scenario) pair ran exactly once (124 of 124), and the slowest typed shard took 9.1 min. Across ten recent six-shard PR runs whose shards did not wait for the producer, the median was 10.65 min.

Verdict: Approved; 2 threads to resolve before merge.

Before merge (inline threads)

1. A shard whose share of one surface is empty now fails
2. Neither new test fails if the joint split is removed

Follow-ups

• Main pushes now ask for ten concurrent 16-core runners: the producer, eight shards and kind_rust_state. In the 13 green main runs since six shards merged, shards waited up to 1.9 min for a runner. Measure main's shard queue after merge, as the README asks (infra/space/kind/README.md:276-281).

Nits

• The Validation table and runner-minute figures come from earlier heads stacked on #1266, which has since closed and is not in this stack. At this head, the slowest typed shard took 9.1 min and the shard jobs summed to 132.7 runner-minutes. At #1268's head (run) they were 8.3 min and 124.7. Five eight-shard attempts took 8.1-10.9 min (median 8.4) and 124.6-133.1 runner-minutes. Ten six-shard PR runs that did not wait for the producer took 10.4-11.7 min (median 10.65) and 106.4 runner-minutes at the median. That is about 2.2 minutes saved at about +20% runner time, rather than 2.7 minutes at +15%.
• infra/space/kind/README.md:292 still says ID/index tie-breaking; equal hints now order by surface, then ID.

[zibi-pplx]

Thanks for the review. Both inline threads are fixed in f3625fc. On the rest:

• Main runner queue: agreed. I will measure main's shard queue across the first main runs after this merges, as infra/space/kind/README.md asks, and report it here.
• Validation numbers: the body now cites this stack's own runs and your six-shard comparison: median 10.65 → 8.4 min for the slowest typed shard, about 2.2 minutes saved for about 20% more runner time, rather than 2.7 minutes at +15%.
• README tie-breaking: it now says equal hints order by surface, then ID, and equal loads go to the lower index.