Split the legacy test stage into four buckets

[Groenbech96]

Stacked on #9736. This branch contains #9736 plus the four-bucket change, so the diff below includes both until #9736 merges — the reviewable delta is the last commit (perf(tests): split the legacy stage into four buckets). It has to be stacked because the bucket lists reference the Tests-ERM-* / Tests-SCM-* apps that #9736 creates.

It targets main rather than copilot/split-tests-scm for a practical reason: .github/workflows/PullRequestHandler.yaml only triggers on pull_request when the base is main, releases/* or features/*, so a PR stacked on a copilot/* branch gets no build at all. Once #9736 merges, this diff collapses to just the four-bucket change.

Why

Buckets are separate jobs on separate runners with separate containers, so they run fully in parallel — run 30354787707 used 44 distinct runners, with Bucket1 and Bucket2 starting in the same minute. Adding a bucket therefore buys real, contention-free capacity, which raising the tenant count inside one container does not.

That distinction matters because the legacy stage has hit its contention ceiling. Measured in 30354787707, both buckets are already tenant-bound, and #9736 showed that adding parallelism inside a container now costs almost as much in per-test slowdown as it gains: splitting Tests-ERM raised Bucket1's concurrency and made every other app in that bucket 10–19% slower, inflating bucket work +27% for a net −5%.

The constraint that shapes the split

Only LegacyTestsBucket1 containers get additionalDemoDataTypes: ["Standard","Evaluation"]. An app cannot move between the two classes without changing the dataset it tests against, so the classes cannot be freely rebalanced.

Keeping every app in its current class, using measured per-app times from run 30354787707 (W1):

Option	resulting buckets	critical path
today	388.4 / 313.1	388.4 min
split Bucket2 in two	388.4 / 156.5 / 156.5	388.4 min — no gain at all
split Bucket1 in two	194.4 / 194.0 / 313.1	313.1 min (−19%)
split both (this PR)	194.4 / 194.0 / 156.5 / 156.5	194.4 min (−50%)

Two non-obvious results:

1. The intuitive "just add a third bucket" is worthless unless it splits Bucket1. Bucket1 carries 388.4 min against Bucket2's 313.1, so splitting Bucket2 leaves the critical path untouched.
2. Four beats three by a wide margin. The unconstrained ideal 3-way split is 233.8 min, but the demo-data boundary makes it unreachable — three buckets can only reach 313.1, while four overshoots the ideal at 194.4.

What

• Bucket3 is a sibling of Bucket1 and inherits its additionalDemoDataTypes; Bucket4 is a sibling of Bucket2. No app changes demo-data class, so no app changes the data it runs against.
• Assignment is LPT within each class using measured per-app times, and each list is ordered longest-first for the FIFO dispatch loop.
• No code changes were needed. Get-AppNamesForBucket already resolves "LegacyTests-Bucket$N" generically, and the Uncategorized path discovers every LegacyTests-* key.

Bucket	apps	work	largest app	floor @ 6 tenants
1 (demo data)	12	194.4 min	Tests-Workflow 51.7	51.7
2	9	156.5 min	Tests-SCM 50.9	50.9
3 (demo data)	13	194.0 min	AlCosting 45.6	45.6
4	15	156.5 min	Tests-SCM-Warehouse 46.0	46.0

Cost

Legacy jobs go from 44 → 88 (22 countries × 4). Roughly +87 runner-hours per build against the 129.2 measured today, because ~97 min of container setup is paid per job. This is the trade to weigh: it is the only lever that adds real capacity, and the only one that costs materially more compute.

Tests

Adds six Pester tests over the bucket configuration. These failure modes are silent, which is why they are worth guarding:

• an app listed in two buckets runs its tests twice;
• an app listed in no bucket does not error — it falls through to the Uncategorized build mode with a different testType and companyName.

Both new guards were negative-tested by injecting a duplicate app and a non-contiguous bucket number, and both failed as intended. Get-AppNamesForBucket previously had no test coverage at all.

build/scripts Pester suite: 124/124 pass.

Follow-ups deliberately left out

Kept to a single variable so the next run measures the bucket change cleanly:

• Re-tune numberOfTenantsForTesting downward. With four buckets each bucket becomes app-bound rather than tenant-bound — work/6 is 26–32 min against a largest app of 45.6–51.7 — so tenants 5 and 6 are provably idle. The model puts 4–5 tenants at the same floor with materially less contention.
• Split Tests-Workflow (51.7 min) and AlCosting (45.6 min). In the current 2-bucket layout these have 30 and 38 min of slack and splitting them is worth nothing. Once buckets are this small they become the binding constraint, and splitting them pays for the first time. AlCosting is a single codeunit, so it needs a source-level split.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[github-actions]

Could not find a linked ADO work item. Please link one by using the pattern 'AB#' followed by the relevant work item number. You may use the 'Fixes' keyword to automatically resolve the work item when the pull request is merged. E.g. 'Fixes AB#1234'