[lorenzorubi-db]

Changes

Ports a standalone ADF cost profiler into flowx as a new flowx profile --source adf phase and a flowx-profile skill. It answers "what does this ADF estate cost today?": a live Azure scan of recent pipeline/activity runs, priced at Azure list rates, compared with what Azure actually billed (Cost Management).

What's in it

• New phase: flowx profile --source adf, standalone. It talks to Azure directly, doesn't read metadata/inventory.json, and doesn't need discover to have run. --source airflow is rejected cleanly ("profiling is not supported for the airflow source"). It's wired via a new optional profile_module slot on flowx.sources.Source.
• Cost calcs: The cost math is ported unchanged (1-minute billing floor, DIU-based Copy, pipeline/external/Data Flow rates, trigger orchestration, rounding). Rates come from the public Azure Retail Prices API for the factory's region, falling back to list rates.
• Output: <output_dir>/metadata/tco/cost_comparison.{csv,md}, overwritten each run (no timestamped folders), with factory-level estimate vs actual and pipeline-level estimates. It doesn't collide with discover's metadata/profile_report.csv.
• Optional dependency: a new profile extra (azure-identity, requests). It's imported lazily, so other phases and Airflow users never need azure. A missing extra gives an install hint (exit 2), not a traceback. Plugin venv: pip install 'azure-identity>=1.17'.
• Credentials: one DefaultAzureCredential code path. Locally: az login. Genie Code / serverless / CI: a service principal from a Databricks secret scope via env vars. Missing roles degrade with a warning naming the role to add (Reader, Data Factory Contributor, Cost Management Reader). A failed sign-in prints setup guidance and exits 2.
• Robustness for long scans: the token refreshes near expiry, with one retry on 401. A run with unusable activity data drops only itself, and uncosted runs are reported. Network errors give a one-line hint ("run locally if egress is blocked"). An empty scan explains why (no subscriptions → wrong-tenant hint; no factories → scope/Reader hint).

Tests

• manually tested

• added unit tests

• added integration tests

• ruff + mypy clean. Full unit suite: 1379 passing (~100 new profiler tests, all from hand-written fixtures, no live Azure needed). They cover the cost math, the pricing parser, discovery and pagination through an injectable replay transport, report output, runner errors, and phase routing.

• Preliminary live validation (read-only, field-eng subscription, one factory with 45 runs over 45 days): the phase runs end to end (sign-in, discovery, run queries, live regional pricing, Cost Management actuals, report) and exits 0. The live run surfaced two bugs inherited from the original script, both fixed here with tests built from the live response shapes:

  • Actuals were always $0. The script only counted rows with an empty meter subcategory as compute, but live Cost Management tags ADF compute Azure Data Factory v2. The match is exact, so infrastructure meters such as Azure Data Factory v2 - Managed Airflow and Private Link stay excluded.
  • Managed-VNet IRs were priced at the Azure IR rate (~4000× too low). Runs report the IR as name (Region), so the lookup missed. Separately, every Azure IR has type Managed; a managed VNet is now detected from the IR's managedVirtualNetwork property.

Limitations

Managed-VNet keep-warm time (TTL) isn't modelled, and on managed-VNet factories it's most of the bill. A managed-VNet IR keeps its external compute warm for a time-to-live (60 min on the test factory) after each activity, and Azure bills that time. On the live factory, Azure billed 44.6 h (≈ $26.80 after discount) for 45 short notebook runs, while the estimate counts 7.25 h of activity time ($7.34 at list price).

modifications to README and documentation on the profiler: TBD

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[ghanse]

@lorenzorubi-db thanks for adding this. I've added some changes in #152 to streamline the dependency installation if the user runs /setup or uses our notebook to install the MCP app. Can you take a look?

[lorenzorubi-db]

@ghanse thanks for your previous review, I've reviewed and commented #152

Quite a few changes in this PR done after your review:

1. Validated against real Azure billing.

I've built two small test factories (Azure IR + managed VNet, ARM template in tests/integration/azure/profiler/factory/), ran them for a weekend, and compared the estimate with Cost Management meter by meter. That surfaced a few fixes, each its own commit with tests:

• Actuals attributed by exact factory name. The substring match put all of x-adf-vnet's billing on x-adf.
• --days defaults to 45, ADF's run-history limit. With 90, actuals covered twice the window the estimate could see.
• Mapping Data Flow meters count in actuals. They have their own meter subcategories and were missing.
• Azure IR runs are priced from the run's billingReference (Azure's own billed quantities and bucket), with the duration estimate as the fallback. Data flow went from +37% to ~2% of billed vCore-hours. DatabricksJob is now an external activity.

2. Golden-reference test (tests/integration/test_profiler_golden.py, follows Add integration tests #154's layout).

It replays one live capture offline and checks the port against the original script's per-pipeline costs. Every place we deliberately differ is listed with its reason, and the test fails if a difference disappears. It runs in make integration (similarly to #154 but for profiler).

3. Deliberately left for a follow-up PR:

The biggest pending gap is adding the Managed-VNet start-up/keep-warm (TTL) time (about 2× costs under on our managed-VNet factory).
Others we could add:

• Trigger orchestration charged per pipeline run instead of per trigger firing (inherited).
• Control-flow activities (ForEach, Wait, …) priced as execution (inherited, cents on Azure IR, more on managed VNet).
• Exact-name matching for Retail Prices meters (Azure IR pipeline activities currently get the external rate, inherited).
• An async MCP profile, since long scans won't fit in one MCP call (as in my comment in Profiler setup opt-in + MCP profile command (stacks on #143) #152)

Thanks!