[Open Data] Add instance size flexibility dataset from Azure Catalogs API

[RolandKrummenacher]

Summary

Adds a new instance size flexibility (ISF) open data dataset, sourced from the authenticated Azure Reservations Catalogs API, to replace the two deprecated static CSVs hosted on ccmstorageprod.blob.core.windows.net (AutofitComboMeterData.csv and isfratioblob.csv / aka.ms/isf). Microsoft is retiring those files — no updates after 9 May 2026, removed 30 Aug 2026.

Addresses #2090. This implements the open-data-generation approach discussed on the issue: the authentication complexity lives only in the generator, so downstream tools consume a public CSV and need no Azure credentials. Consumer repoints are tracked separately in #2200.

What's included (foundation only)

• src/scripts/Update-InstanceSizeFlexibility.ps1 — extracts ISF groups/ratios from Microsoft.Capacity/catalogs. Handles both REST (ReservationsAutofit*) and PS-cmdlet (InstanceSizeFlexibility*) property names, pagination with in-place 429/5xx retry, placeholder-SKU filtering, and a cache-merge that preserves SKUs the API drops. Defaults to the four ISF-bearing reserved resource types (VirtualMachines, BlockBlob, RedisCache, DedicatedHost) unioned across a broad region set.
• src/open-data/InstanceSizeFlexibility.csv — populated with 2,436 SKUs / 557 groups of live data (3-column schema: InstanceSizeFlexibilityGroup,ArmSkuName,Ratio).
• .github/workflows/opendata-instance-size-flexibility.yml — weekly refresh; opens a PR on change.
• README + docs download tile.

Coverage vs the deprecated files

Dataset	Groups	SKUs
New InstanceSizeFlexibility.csv	557	2,436
isfratioblob.csv (Optimization Engine)	434	2,524
AutofitComboMeterData.csv (Power BI, frozen since 2023)	132	721

Raw Microsoft ratios are kept as-is for drop-in parity with the deprecated files (normalization is available via -Normalize).

Schema (3 columns, by design)

The file uses the canonical Catalogs API ISF schema — InstanceSizeFlexibilityGroup, ArmSkuName, Ratio — which is a 1:1 match for the Optimization Engine's isfratioblob.csv. The Power BI AutofitComboMeterData.csv had additional columns (ResourceLocation, meter IDs, composite key, normalized SKU), but those are Cost Management connector artifacts not exposed by the Catalogs API, so the Power BI repoint (#2200) will join on ArmSkuName instead.

Authentication

The Catalogs API is authenticated (Microsoft.Capacity/catalogs/read). The workflow reuses the existing OIDC service principal from the aoe-prod environment (the same identity the AOE deployment workflows use), following the repo's azure/login@hf_447_release pattern — no new secrets to provision. The principal's subscription needs the Microsoft.Capacity resource provider registered (done). A one-time workflow_dispatch after merge confirms the principal can read the catalog.

Test plan

• Generator validated end-to-end against the live Catalogs API (response shape, property names, armSkuName fallback all confirmed)
• Discovery confirmed only 4 reserved resource types expose ISF ratios
• Ratios confirmed region-stable (merge reported Modified: 0 across all regions)
• Offline tests: multi-type/region union, dedup, cache-merge/preserve, 429-retry, placeholder filtering; script parses clean
• Reviewer spot-checks a few SKUs against the Azure Pricing Calculator
• workflow_dispatch after merge to confirm CI auth path

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR introduces a new open-data dataset for instance size flexibility (ISF) ratios by generating InstanceSizeFlexibility.csv from the authenticated Azure Reservations Catalogs API, replacing reliance on deprecated static CSV blobs.

Changes:

• Added a PowerShell generator script to fetch, normalize (optional), and cache-merge ISF ratios from Microsoft.Capacity/catalogs.
• Added the generated InstanceSizeFlexibility.csv open-data artifact and documented it in open-data docs/README.
• Added a scheduled GitHub Actions workflow to refresh the dataset weekly and open a PR when it changes.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/scripts/Update-InstanceSizeFlexibility.ps1	New generator that pages the Catalogs API and emits the ISF CSV (with cache-merge + optional normalization).
src/open-data/InstanceSizeFlexibility.csv	New published open-data dataset containing ISF group/SKU/ratio rows.
.github/workflows/opendata-instance-size-flexibility.yml	Weekly automation to refresh the dataset and open an update PR.
src/open-data/README.md	Documents the new ISF dataset, its source, and manual update instructions.
docs/open-data.md	Adds a download tile linking to the CSV in the latest release assets.

[RolandKrummenacher]

Addressed the Copilot review (commit pushed):

1. 429/5xx retry semantics (✅ fixed) — moved transient HTTP status handling inside the inner retry loop so a throttled/5xx page is retried in place without advancing $page or resetting the retry cap. Verified with a mock that returns 429 then 200: the page is retried once and the cap holds.
2. arm_sku_name_placeholder in the CSV (✅ fixed) — the live Catalogs API returned a placeholder name for one unreleased DedicatedHost SKU. The generator now skips any *placeholder* SKU name, and I removed the one row that had reached the published CSV (it was the sole member of its group, so a meaningless single-SKU ISF group went with it). CSV is now 2,436 SKUs.
3. Hidden spacer anchors in docs/open-data.md (ℹ️ no change) — these use style="visibility:hidden", which already removes them from the tab order and the accessibility tree, so they aren't surfaced to AT. They're also the established layout pattern used by every tile in this file, so changing only the new tile would be inconsistent. Happy to do a site-wide pass separately if desired.

On the column schema (raised in review discussion): the dataset intentionally uses the canonical three-column Catalogs API schema (InstanceSizeFlexibilityGroup, ArmSkuName, Ratio).

• This is a 1:1 match for the deprecated isfratioblob.csv (Optimization Engine) — that file was also exactly these 3 columns.
• The deprecated AutofitComboMeterData.csv (Power BI) had 9 columns, but the 6 extras (ResourceLocation, ComboMeter/HardwareMeter/SoftwareMeter meter IDs, CompositeKey, NormalizedSKU) are Cost Management connector artifacts — they are not ISF data and the Catalogs API does not expose them, so they can't be reproduced here regardless of column count.
• Consequence for the deferred Power BI repoint: the model currently joins via x_CommitmentDiscountKey (derived from those connector columns), so the repoint will need to join on ArmSkuName instead. I've documented this in src/open-data/README.md.

[RolandKrummenacher]

Update on the auth setup — no new secrets/SP needed.

The earlier note said maintainers would need to create an OIDC service principal and AZURE_* secrets. On closer inspection, the repo already has that identity: the aoe-prod environment holds AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID (the same OIDC principal the AOE deployment workflows use), and it has no protection rules, so a scheduled job can use it.

I've updated the workflow to reuse that identity rather than create a parallel one:

• environment: aoe-prod
• the repo's established azure/login@hf_447_release + enable-AzPSSession: true pattern, with an explicit Az.Accounts install

So there's nothing to provision. The only residual is a one-time confirmation that the aoe-prod principal can read the Catalogs API on its subscription — i.e. it has Microsoft.Capacity/catalogs/read (a deployment principal with Contributor/Owner already does) and the subscription has the Microsoft.Capacity resource provider registered (done). Easiest way to confirm is a manual workflow_dispatch once this is merged.

If you'd prefer the open data workflow not to borrow the AOE-named environment, a maintainer can create a dedicated opendata environment with the same federated credential + secrets and I'll repoint to it — purely cosmetic.

[flanakin]

🤖 [AI][Claude Code] PR Review

Summary: Clean, well-structured addition that closely mirrors the established Update-CommitmentDiscountEligibility sibling pattern (cache-merge, paging, retry, weekly PR workflow). The earlier Copilot items (in-place 429/5xx retry, placeholder-SKU filtering) are already addressed in-tree, and the generator/CSV logic looks correct. Two non-blocking findings below.

⚠️ Should fix (1)

1. No changelog entry. The sibling Commitment discount eligibility dataset shipped with an ### [Open data] changelog section; this is a comparably substantial external-facing dataset (2,436 SKUs, new download tile, weekly workflow) with no entry in the current docs-mslearn/toolkit/changelog.md. Add an ### [Open data] entry under the next/unreleased version.

💡 Suggestions (1)

1. -Normalize is typed [bool] (requires -Normalize $true); [switch] would be more idiomatic PowerShell. Optional — the [bool] form does allow scripted toggling.

[github-actions]

⚠️ Open Data CI task failed: https://github.com/microsoft/finops-toolkit/actions/runs/28363541029

[helderpinto]

The earlier note said maintainers would need to create an OIDC service principal and AZURE_* secrets. On closer inspection, the repo already has that identity: the aoe-prod environment holds AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID (the same OIDC principal the AOE deployment workflows use), and it has no protection rules, so a scheduled job can use it.

The aoe-prod principal belongs to a personal Entra ID tenant associated to a MSDN benefits subscription. Given that the ISF workflow is not going to create resources, I don't anticipate issues, but if the subscription type can be a problem, we should switch to a more permanent subscription type, such as PAYG/enterprise.

So there's nothing to provision. The only residual is a one-time confirmation that the aoe-prod principal can read the Catalogs API on its subscription — i.e. it has Microsoft.Capacity/catalogs/read (a deployment principal with Contributor/Owner already does) and the subscription has the Microsoft.Capacity resource provider registered (done). Easiest way to confirm is a manual workflow_dispatch once this is merged.

The aoe-prod principal can read the Catalogs API on its subscription and the subscription has the Microsoft.Capacity resource provider registered.

[RolandKrummenacher]

Two follow-ups:

Open Data CI failure was transient — now green. The failure wasn't in the data or generator: the Build Open Data step regenerates the Get-OpenData* helper functions and auto-commits them, and its git push was rejected (fetch first) because concurrent commits had moved the branch head mid-run. Re-ran the job with no concurrent push and both steps pass (Build Open Data ✓, Test Open Data ✓). The two earlier Open Data CI runs on this branch had also passed, confirming it was a one-off race. No generated function is committed (consistent with the sibling Commitment discount eligibility dataset — git commit -a skips the untracked generated files).

Thanks @helderpinto for confirming the aoe-prod principal can read the Catalogs API and that the subscription has Microsoft.Capacity registered — that clears the only residual, so the post-merge workflow_dispatch is now just a nice-to-have sanity check rather than a gate. Noted on the MSDN-benefits subscription: since this workflow is read-only (no resource creation, one lightweight catalog read per week), the subscription type shouldn't matter, but easy to point at a PAYG/enterprise sub later if you'd prefer.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

docs/open-data.md:41

• These hidden spacer elements are anchors with empty href values. Even though they’re hidden, empty links can show up in HTML/link validation and create confusing focus/interaction semantics. Use a non-interactive element (e.g., ) for layout spacing instead.

        <a class="btn mb-4 mb-md-0 mr-4" href="" style="visibility:hidden; width:100px">&nbsp;</a>
        <a class="btn mb-4 mb-md-0 mr-4" href="" style="visibility:hidden; width:100px">&nbsp;</a>

docs-mslearn/toolkit/open-data.md:7

• For docs-mslearn pages, ms.date should be updated to today when the file changes to avoid merge conflicts with the automated date updater.

ms.date: 06/29/2026

docs-mslearn/toolkit/changelog.md:6

• For docs-mslearn pages, ms.date should be updated to today when the file changes to avoid merge conflicts with the automated date updater.

ms.date: 06/29/2026

[MSBrett]

Request changes

This cannot be approved while the privileged workflow uses mutable third-party action refs and while its published data is unsafe for the documented join.

Requirement	Evidence	Status
Pin privileged third-party actions	Workflow lines 24 and 35 use actions/checkout@v4 and azure/login@hf_447_release, both named refs; the latter resolves to branch commit 264ef4588d3ca547fea5296679ee9f8a4766d4df. The job has contents: write, pull-requests: write, and id-token: write.	Blocked — comment posted
Preserve a safe consumer key	Import-Csv analysis finds 45 duplicate ArmSkuName values; archive_grs_data_stored_1 pb has 26 rows in distinct groups. The README recommends joining on this non-unique field.	Blocked — comment posted
Make weekly output converge on the source	Cache rows are copied into $merged before observed rows overwrite them; no path removes an unseen record. Confirmed by executing the merge logic against a synthetic retired-SKU fixture, not just reading it.	Blocked — comment posted
Test changed behavior	No test references Update-InstanceSizeFlexibility; existing Data selection and lint scope do not exercise src/scripts. Direct Invoke-ScriptAnalyzer run against the script also surfaces a PSUseSingularNouns violation that would fail immediately if lint scope were widened as requested.	Blocked — comment posted
Validate documentation	./src/scripts/Test-PowerShell.ps1 -Docs: 1,510 passed, 0 failed.	Passed
Validate syntax and diff hygiene	PowerShell parser clean; git diff --check clean.	Passed

I reviewed the existing discussion and verified the prior retry, placeholder-filter, changelog, and -Normalize feedback is reflected in the current diff. The four requirements above remain unresolved.

Update: every claim above has since been independently re-verified against the current PR HEAD by direct execution (mocked pagination/retry test, live merge-logic replay, CSV reproduction, PSScriptAnalyzer run, and a live authenticated call to the Azure Reservations Catalogs API to validate ISF group shape). All four blocking findings hold. One earlier internal concern about single-SKU flexibility groups was raised and then retracted after the live API confirmed Microsoft's own catalog legitimately returns singleton groups for isolated/high-memory/specialty SKUs — that was not posted here and required no correction.

[RolandKrummenacher]

@MSBrett Thanks for the thorough review — all four blocking items are addressed in ee1dc3b and f7f434c.

1. Pinned privileged actions — actions/checkout and azure/login are now pinned to full commit SHAs (11d5960a… = v4.4.0, 264ef458… = current hf_447_release), with the named refs retained as comments.

2. Safe join key — Root-cause analysis showed all 45 duplicated ArmSkuName values came exclusively from BlockBlob: its catalog entries carry meter-style names rather than real ARM SKU names (the armSkuName fallback), every one of its 984 rows has Ratio = 1 (no usable flexibility information), and its groups are region-specific. BlockBlob is now excluded from the default scope (it can still be generated explicitly via -ReservedResourceType), which makes ArmSkuName globally unique — a safe standalone join key. This is enforced twice: the generator throws if the API ever returns the same SKU in multiple groups, and a unit test asserts uniqueness on the published CSV. The README documents the guarantee.

3. Convergence on the source — The cache-merge is removed entirely. Each run publishes exactly what the API returns for a fully successful sweep; any scope that still fails after retries aborts the run without touching the output file, so the workflow fails and the last published dataset stays in place. No path retains an unseen record anymore. The regenerated CSV (1,352 SKUs) also drops 100 legacy-series rows (G/GS, Dv2/Dv3, Ev3, F/FS, LSv2, …) — verified as no longer returned by the Catalogs API in any queried region, i.e. convergence working as intended. Reviewers of the weekly PR see removals in the diff before anything merges.

4. Test coverage — Added Update-InstanceSizeFlexibility.Tests.ps1 with 15 Pester unit tests mocking the Catalogs API: nextLink pagination, bare-array responses, 429/5xx same-page retry, fatal non-retryable 4xx, placeholder filtering, armSkuName fallback, regional union dedup, retired-record removal, duplicate-SKU guard, default-scope composition (BlockBlob excluded), -Normalize, and published-CSV schema/uniqueness. Wired into Test-PowerShell.ps1 -Data. All 15 pass locally, as do the 1,510 docs tests.

#needs-review

[RolandKrummenacher]

Follow-up hardening (6fa45eb): the workflow no longer uses the shared aoe-prod deployment principal.

It now authenticates as a dedicated finops-toolkit-opendata-reader app registration in the FinOps Toolkit tenant:

• Least privilege: the principal holds only the Reader role (the Catalogs API needs Microsoft.Capacity/catalogs/read), instead of the deployment principal's write-capable role. The catalog content is subscription-independent, so the subscription only serves as an API routing scope.
• Scoped trust: the OIDC federated credential trusts exactly repo:microsoft/finops-toolkit:ref:refs/heads/dev — no client secret exists anywhere.
• No GitHub configuration needed: client/tenant/subscription IDs are identifiers, not secrets, and live inline in the workflow (same pattern as the tenant/subscription IDs already documented in the repo's contributor READMEs). The environment: reference is gone.

@helderpinto this also resolves the continuity concern you raised about the personal MSDN-benefits subscription — the identity now lives in the shared FinOps Toolkit tenant. The trade-off of branch-scoped (vs. environment-scoped) trust: any workflow on dev could request this token, but since it's Reader-only, the blast radius is minimal — and aoe-prod had no protection rules either, with a far more privileged principal behind it.

Validation after merge: a one-time manual workflow_dispatch run confirms the OIDC exchange end to end.