Feature: Run FinOps hubs on your own hardware - #2187
Conversation
Local install of FinOps Hubs: runs the toolkit's ADX KQL against a local
Kusto Emulator (Kustainer) with the real two-database topology
(Ingestion + Hub), so dashboards / Power BI / the query catalog run
unchanged against a local endpoint.
T-5000.1 — load-ftk-kql.ps1 creates the Ingestion + Hub databases and loads
them by combining the toolkit's own finops-hub/.build.config fabric-setup
bundles (real v1_0 included; database('Ingestion'). cross-DB references
preserved; zero object renames — strict name parity). No hand-coded load
order, no strip hack, no v1_0 stubs.
T-5000.2 — ingest.ps1 -> Ingestion; ftk.ps1 -> Hub; run-parity-checks.ps1 is
DB-aware (raw/final checks on Ingestion, Hub-view checks 9/10 on Hub) and
adds a name-parity check. Verified end-to-end on R&D data: Costs and Prices
ingested at exact parity (BilledCost preserved to $0.00; Prices raw==final
12,735,587), Hub cross-DB queries resolve, parity suite 10 pass / 0 fail /
2 manual.
Prototype docs/notes are intentionally not migrated here yet — they are
authored cleanly (sanitized, two-DB-accurate) in the docs task (T-5000.7).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Red-team caught that `make kql` defaulted to Ingestion, so Hub view-function queries (e.g. Costs_v1_2) failed with 400 unless DB=Hub was passed. The DB var was shared with the parity recipe, which needs Ingestion as its base DB. Split into KQL_DB (?= Hub, used by `make kql`) and PARITY_DB (?= Ingestion, passed to run-parity-checks.ps1). ingest-status now passes KQL_DB=Ingestion explicitly since it inspects Ingest_Manifest in Ingestion. Header/help/topology comments updated. ingest unaffected (ingest.ps1 defaults to Ingestion). Verified: `make kql QUERY='Costs_v1_2 | count'` -> 1350561 (Hub) with no override; KQL_DB=Ingestion override works; make parity still 10/0/2 + name parity PASS; make ingest-status reads Ingest_Manifest on Ingestion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…00.4)
Document the data-in path: Cost Management export configuration via the
toolkit's New-/Get-/Start-FinOpsCostExport cmdlets (Azure-only; ftklocal
consumes the produced FOCUS parquet), and the local staging contract that
ingest.ps1 consumes — export/{scope}/{type}/{period}/{run-uuid}/ with
manifest.json, the ms--focus-cost / ms--pricesheet dataset tokens, latest-run
selection, SHA-256 idempotency, and supersede semantics (correct/replace data
by staging a NEW run-uuid; same run-uuid + changed file double-ingests).
Maps the msexports/ (no hub) and ingestion/ (hub deployed) source modes;
auto-download bridge deferred (T-5000.9). Contract cross-checked against
ingest.ps1.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on (T-5000.10) Add a sanitized settings.example.json (HubInstance schema: scopes + retention) and scripts/settings.ps1 (init/show/get) for local-hub fidelity; gitignore the real settings.json. Wire load-ftk-kql.ps1 to derive the raw soft-delete window ($$rawRetentionInDays$$) from settings.json retention.raw.days with a 3650 fallback and a guard (0/negative/non-numeric -> fallback) plus a warning when a positive value is surprisingly short (<90 days) for local analysis. Document the config-container decision: Mode A (ingestion container) needs no config; Mode B (msexports) would need the msexports->ingestion conversion + the in-repo Exports/schemas (deferred to the bridge, T-5000.9). The ADX/KQL layer consumes no config schemas; load-ftk-kql.ps1 reads settings.json only for retention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Localize the FinOps hub ADX dashboard for the local Kusto emulator: point the single data source at http://localhost:8082 / database Hub (135 tiles, 87 queries unchanged from upstream). Add notes/dashboard.md documenting the proven ftk.ps1 CLI path, the ADX web UI as best-effort with the HTTPS->HTTP mixed-content caveat, and an ngrok HTTPS-tunnel workaround. Power BI dropped (local Mac, no AAD endpoint). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ocal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e-ingest guard (T-5000.3) Add proactive per-extent chunked backfill for large final tables (default threshold 2,000,000 rows OR >4 source extents, both env-configurable via BACKFILL_CHUNK_ROW_THRESHOLD / BACKFILL_CHUNK_EXTENT_THRESHOLD), so the Prices transform never attempts a single-pass backfill that OOMs; Costs stays single-pass with the reactive chunked fallback intact. Add a fail-fast guard: re-staging a file under the same (scope,type,period,run-uuid) with a changed checksum now halts with a non-zero exit and a clear message instead of silently appending duplicate rows. Harden Invoke-KustoPost strict-mode catch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sers/ literal) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…resolution (T-5000.8) Add the finops-hub-local agent skill (SKILL.md thin router, README, local-vs-remote reference) that drives the local Kusto-emulator FinOps hub via the ftk.ps1 CLI and the finops-hub-local Makefile, reusing the published query catalog (no new KQL). Also fix ftk.ps1 catalog discovery for the in-repo layout: Get-FtkRepo assumed the old prototype's sibling-folder layout (<parent>/finops-toolkit) and failed to find src/queries/catalog when ftklocal lives inside the repo, breaking 'ftk list' and 'ftk run'. Resolve the toolkit root in-repo (three levels up from the script dir) with a git-toplevel fallback and the legacy sibling layout as last resort. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Under Set-StrictMode + $ErrorActionPreference='Stop', calling '& git' when git is absent throws CommandNotFoundException and skips the legacy sibling-layout fallback. Guard the git-toplevel probe with Get-Command so a missing git degrades gracefully. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…alog fix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…+ harden Prices value parity (T-5000.3) Per architect-duo ruling: drop the extents>4 proactive-chunk criterion (OOM is row-driven; >4 extents mis-fires on every multi-part export and would chunk Costs, which single-passes safely at 15 extents). Gate proactive chunking on rawCount > 2,000,000 alone; keep the reactive single-pass->chunked fallback. Extent count is now informational logging only. Remove the BACKFILL_CHUNK_EXTENT_THRESHOLD knob; note that adding a new RawToFinalPolicy table needs a fresh single-pass OOM measurement. Fix the Makefile threshold comment (3,000,000 -> 2,000,000). Harden parity CHECK 13 (Prices value parity): emit delta_pct so the runner enforces a 0.1% relative tolerance, with a sentinel that fails on zero rows or row-count mismatch so the check cannot pass vacuously. Verified: real data 9,804,381 rows / sums equal at $58.39B -> PASS; x1.01 corruption -> FAIL; zero rows -> FAIL. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…uble-ingest guard, Prices value parity) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ss-db strip in ftk.ps1 (T-5000.7)
Add user-facing docs under src/templates/finops-hub-local/: README.md (quickstart,
two-database Ingestion+Hub architecture, ingest with the fail-fast double-ingest guard
and automatic chunked backfill, querying via ftk.ps1 + the agent skill, dashboard
best-effort, measured limits table, troubleshooting), notes/ftk-kql-adaptations.md (the
load-time vs ftk.ps1 on-read adapter layers), notes/parity-gaps.md (the 13-check parity
suite, grounded in a live run).
Also remove the now-stale database('Ingestion'). strip from the ftk.ps1 on-read adapter:
on the two-database topology cross-database references resolve correctly, so stripping
them would break any catalog query that legitimately references an Ingestion table.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…trip removal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add notes/performance.md with measured memory characterization on MEM_LIMIT=16g (amd64-on-Rosetta), instrumented via cgroup v2: cold loaded floor ~1.12 GiB (7%), hot post-ingest working set 13.99 GiB (87.4%), Prices chunked-backfill peak 15.03 GiB (94%), and the single-pass Prices transform ceiling where Kusto self-terminates at 15.12 GiB (~94.5%). Includes the row-count to memory curve, the ingest wall-clock (raw 7.0 min, full 14.0 min), and an operating-states table that answers when the engine approaches the limit (only during ingest+transform of a 28M-row dataset). Reconcile the README limits table with these measured numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…red) Replace the speculative 'mixed-content blocked / not proven' caveat with measured findings: the Kusto emulator returns CORS headers for https://dataexplorer.azure.com (OPTIONS 204 + POST 200 with Access-Control-Allow-Origin), real dashboard tile queries return renderable data over that path, and dashboard.json (schema v60, 135 tiles / 87 queries / 12 pages) has zero dangling references so it imports cleanly. localhost is exempt from mixed-content blocking; the real browser gate is the Edge 143+/Chrome 142+ Local Network Access permission prompt (click Allow), with browser-flag and ngrok fallbacks documented for locked-down environments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… + cross-platform bring-up Add per-OS install guides (notes/install-windows.md primary, notes/install-mac.md secondary) and make the README quickstart cross-platform: 'docker compose up -d --wait' (honors the container healthcheck) plus the self-contained 'pwsh scripts/*.ps1' commands replace the Unix-only 'make' front door, which remains a macOS/Linux convenience. The scripts already self-resolve HOST_PORT from .env, so they run standalone on Windows. Windows guide covers Docker Desktop + WSL 2, PowerShell 7 (not Windows PowerShell 5.1), WSL 2 memory tuning, forward-slash EXPORT_DIR, and Windows-specific troubleshooting; notes that linux/amd64 runs natively on Windows x64 (no emulation) so it outperforms the Apple-Silicon numbers in performance.md. Supporting correctness: add .gitattributes (LF for .ps1/.sh/.kql/.json/Makefile) to prevent CRLF corruption on Windows checkout, and '#Requires -Version 7.0' to the entry scripts so PowerShell 5.1 fails with a clear message instead of cryptic parser errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run repo Prettier (.prettierrc) over the finops-hub-local markdown so the docs conform to the toolkit coding standard (docs-wiki/Coding-guidelines.md): aligned table columns and underscore emphasis. Formatting only — no content, measured numbers, or links changed. 'prettier --check' now passes for all finops-hub-local markdown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Matches the repo-wide markdown convention (all .md stored LF) and prevents CRLF drift for Windows contributors under core.autocrlf=true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add notes/build-by-hand.md — the manual runbook for standing up a local FinOps hub: six steps, about a dozen raw commands (start emulator, create the two databases, load the schema, ingest Parquet, build the final tables, query). Each schema bundle is a single .execute database script command, so loading is one POST per database; the doc shows the exact curl/KQL for every step, verified against the emulator (118/118 statements load in one call). A closing table maps each manual step to the helper that automates it, so the scripts are no longer a black box. Link it from the README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The engine runs a whole bundle natively via '.execute database script with (ContinueOnErrors=true)', returning a per-statement result table. So wrap each built bundle and POST it once instead of splitting it into ~157 commands and posting each in a loop. Removes the command splitter and per-command loop (~48 lines); per-statement diagnostics are preserved by parsing the result table. Verified: 118/118 Ingestion and 39/39 Hub statements load in one call each, schema reload is non-destructive (data intact at the known total), and the dry-run path is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Hubs (manual) deployment failed. View logs. |
|
Hubs (manual) deployment failed. View logs. |
|
Hubs (manual) deployment failed. View logs. |
|
Hubs (manual) deployment failed. View logs. |
|
Closing/reopening to trigger PR resource cleanup (ftk-pr-cleanup.yml) and clear an orphaned Azure deployment lock blocking the Hubs (manual) check. No code changes. |
|
Test environments cleaned up. |
|
Reopening — resource group cleanup completed, orphaned deployment lock cleared. Re-running Hubs (manual) now. |
|
Hubs (manual) deployed to |
RolandKrummenacher
left a comment
There was a problem hiding this comment.
Overview
The feature described in the PR body is good work — the build-generated open-data artifact is the right call over hand-maintained externaldata schemas, the retry-on-empty handling is well reasoned, and the UAT evidence against a real emulator with real FOCUS data is genuinely more than most PRs bring. I'd happily take that.
The problem is that it's about a quarter of the diff.
Undisclosed scope — ~3,750 of 4,972 added lines
| Path | Lines | What it is |
|---|---|---|
.github/extensions/ftk-local-dashboard/ |
~3,750 | A GitHub Copilot CLI canvas extension — Node HTTP server, 1,879-line app.js, 815-line app.css, kusto.mjs client, PRODUCT.md |
PRODUCT.md (repo root) |
42 | "Product context" for the dashboard |
.gitignore |
3 | .impeccable/ — "impeccable design skill cache" |
None of it appears in the PR description, the changelog, TOC.yml, or the docs. There's no test coverage and nothing in the build or release pipeline references it.
@flanakin's June review asked for the agent skill to be removed from this PR as an anti-pattern. The skill went, but a Copilot CLI extension and a design-skill cache entry arrived in its place — same class of artifact, different form. I don't think that's what was being asked for.
To be fair to the code: I read through extension.mjs and kusto.mjs and the extension isn't dangerous. It binds 127.0.0.1 rather than 0.0.0.0, serves static assets from an allowlist map so there's no path traversal, and has no exec/spawn/eval. My objection is placement and disclosure, not safety.
Please pull all three out into their own PR so they can be evaluated on their merits. The remaining ~1,200 lines are a clean, well-tested feature that I'd expect to move quickly.
Issues in the intended scope
Four things I'd like fixed on the PowerShell — details inline. Briefly: -WhatIf still performs network and disk I/O; -replace is used with a user-supplied replacement string that can carry regex substitution metacharacters; the default timeout of 0 means wait forever; and the command isn't re-runnable.
Verified as fine
So these don't get relitigated: .set-or-replace in IngestionSetup_OpenDataExternal.kql makes the five-attempt retry loop safely idempotent (no double-append). $Destination = [System.IO.Path]::GetTempPath() matches the existing Save-FinOpsHubTemplate convention. New-Directory exists as a private function. The integration tests self-skip cleanly on missing env vars, so they don't break CI. The Toolkit.Tests.ps1 asset check correctly gates finops-hub-local-opendata.kql at 15.0. And [int64](...).Tables[0].Rows[0][0] binds the way you want it to — the cast applies after member access.
…Initialize-FinOpsHubLocal bugs Per RolandKrummenacher's review (2026-07-31): - Removed .github/extensions/ftk-local-dashboard/, root PRODUCT.md, and the .impeccable/ .gitignore entry: undisclosed, out-of-scope content (~3,752 lines) unrelated to the 'run FinOps hubs locally' feature, not referenced in the PR description, changelog, or docs. Reverts .gitignore to be byte-identical with origin/dev. - Initialize-FinOpsHubLocal: gated the release-asset download (and New-Directory) behind its own ShouldProcess check so -WhatIf performs zero network/disk I/O, matching every other operation in the command. - Initialize-FinOpsHubLocal: switched database creation from '.create database' to '.create-merge database' so re-running the command after a partial failure does not error when the databases already exist. Applied the same change to the manual walkthrough in run-hubs-locally.md. - Initialize-FinOpsHubLocal: replaced '-replace' with '.Replace()' for the RawRetentionInDays and OpenDataPath placeholder substitutions, since -replace treats sequences like \ or \$& in the replacement value as regex backreferences instead of literal text. - Initialize-FinOpsHubLocal: changed the TimeoutSec default from 0 (wait indefinitely) to 30 seconds so an unresponsive emulator fails with a clear timeout instead of hanging forever. - Documented the /kustodata storage-path assumption in the command's comment-based help. - Updated Initialize-FinOpsHubLocal.Tests.ps1: added assertions that -WhatIf makes zero New-Directory/Invoke-WebRequest calls, a regression test for the -replace/.Replace() metacharacter bug, a test for the new finite TimeoutSec default, and updated the create-merge command-text assertions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Hubs (manual) deployment failed. View logs. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
# Conflicts: # docs-mslearn/toolkit/changelog.md
|
All feedback from @RolandKrummenacher's review has been addressed in commit b4d5eb2 (and a subsequent merge from dev + CI fix):
|
|
Hubs (manual) deployed to |
🛠️ Description
Adds and UAT-hardens a self-hosted path for FinOps hubs: a documented way to run the hub analytics layer on your own hardware using the Kusto emulator. Customers, consultants, contributors, and agents can exercise the same hub KQL, schema, transforms, update policies, and open-data enrichment without deploying the full Azure-hosted stack.
This PR is no longer docs-only. Based on review feedback (see below), it now also ships the supporting automation so the guide stays correct and easy to follow:
finops-hub-local-opendata.kqlbuild artifact — the open-data load script is generated by the toolkit build from the same source a deployed hub uses (.build.configcombineKql), so the guide downloads and runs it instead of hand-codingexternaldataschemas that could silently drift.Initialize-FinOpsHubLocalPowerShell command — sets up a local hub against a running emulator in one call: creates theIngestionandHubdatabases, downloads the released setup scripts, applies raw retention, and loads open data with verify-and-retry. It does not install or manage Docker, and it does not ingest cost data.Prices_final_v1_2.Why this changed from the original docs-only PR
@flanakin requested removing the agent skill (anti-pattern) and asked whether the fragile inline blocks could be build-generated and driven by a PowerShell command. This PR does exactly that: the skill is removed, the open-data load is build-generated, and the deterministic steps are a tested command — no Docker management in the module.
Verification
Verified end-to-end against a real Kusto emulator (Docker
kustainer-linux) with real release artifacts served over HTTP and real FOCUS data (Cost Management export, billing account 8611537, 2025-04). No mocks.src/powershell/Tests/Integration/Initialize-FinOpsHubLocal.Tests.ps1: 7/7 passed.Costs_raw117,559 →Costs_final_v1_2117,559;Prices_raw/Prices_final_v1_2853,636; prices present before the costs phase;Costs_rawretention90.00:00:00; cost enrichment 117,559/117,559.externaldataread after setup can silently return 0 rows; the command and the guide now load, verify, and retry.Boundaries
The Kusto emulator is not for production: no security, no managed ingestion, limited by host capacity. The local hub is an analysis and validation tier, not a replacement for an Azure Data Explorer or Microsoft Fabric hub.
No linked issue.
📷 Screenshots
Not applicable.
📋 Checklist
🔬 How did you test this change?
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
📑 Did you update
docs/changelog.md?📖 Did you update documentation?