[ghanse]

Stacks on #143 (feat/adf-profiler). Merge that first; this PR's base is feat/adf-profiler.

What changed

Setup asks about profiling, so neither path installs the Azure deps unless the user says yes

• skills/flowx-setup/SKILL.md: new Step 2 asks "Do you want to profile your ADF resources?" and both paths act on the answer.
• scripts/bootstrap.sh: new --with-profile flag. With it, bootstrap also runs pip install -e '<plugin_root>[profile]'. That installs from the local checkout, so the installed version always matches the source tree. Without the flag nothing changes. Unknown flags exit 2 with a usage line before any work starts.
• MCP app: the with_profile widget in app/deploy_app.py (default no) and WITH_PROFILE=1 for app/deploy.sh append the profile extra to the app's requirements.txt. Both read the extra straight from pyproject.toml, so there is no second requirements list to keep in sync (this matches the intent of test_profiler_agents_doc).

MCP profile command (src/flowx/mcp/server.py)

• _cmd_profile is a thin wrapper over python -m flowx.adapter profile through runner.run_adapter. It takes source (required), output_dir, days, subscription_id, resource_group, and factory_name.
• On success it returns report_md (metadata/tco/cost_comparison.md). It skips that read on failure, so a stale report from an earlier run is never returned.
• If the extra is missing, the adapter's existing clean error comes back unchanged: ok: false, returncode: 2, and the install hint in process.stderr, with no traceback.
• Azure credentials are not accepted as parameters. They come from the server's AZURE_* env vars, so a secret can't show up in the echoed process.command.
• The command is listed in the server instructions, the tool docstring, and the app/README.md command table.
• Importing flowx.mcp / flowx.mcp.server still doesn't import azure.

Docs

• skills/flowx-profile/SKILL.md: explains MCP usage, replaces the "not exposed as an MCP command yet" note, and adds: ADF Datasets are intentionally not scanned (definitions only, no runtime meter).

How I verified

• PYTHONPATH=src uv run pytest tests/unit -v: 1383 test cases collected, 1381 passed, 3 skipped, 2 failed.
  • The 2 failures (test_airflow_provider_sync.py) also fail on the base commit. They are local-environment only: my global commit.gpgsign=true makes the fixture's git commit fail with "No secret key". They are not related to this change.
  • The 3 skips are the MCP test files (test_mcp_profile.py, test_mcp_migrate.py, test_mcp_source_routing.py), which skip when the mcp extra isn't installed. That is also true in CI.
• I also ran the MCP tests with mcp available (pinned to uv.lock versions): 26 passed, including the 6 new tests in test_mcp_profile.py.
• I ran a real, unmocked flowx(command="profile", …) call through FastMCP without azure installed. It returned returncode: 2 and the install hint.
• uv run ruff format --check src/ tests/, uv run ruff check src/ tests/, and uv run mypy src/flowx/ all pass.
• New tests:
  • test_mcp_profile.py: argv dispatch, report readback, missing extra (exit 2 + hint, no stale report), source required, no azure on server import.
  • test_setup_profile_opt_in.py: bootstrap rejects unknown flags, bootstrap installs -e "${PLUGIN_ROOT}[profile]", app requirements leave out azure by default and add the pyproject extra when asked.
  • Extended test_profiler_lazy_imports.py to import flowx.mcp and flowx.mcp.runner.

Notes

• inputs discovery doesn't cover profile: the adapter's inputs subcommand only accepts discover/convert/package, and widening it was outside this change.
• I didn't update CHANGELOG.md: it only has the 0.1.0 entry, and flowx profile ADF TCO profiler phase + flowx-profile skill #143 didn't add one either.

This pull request and its description were written by Isaac.

[lorenzorubi-db]

Looks great: setup opt-in + MCP profile is the right shape, and the tests are great. One thing is needed before merge (rebase + profile → azure, below); the inline comments are small and optional.

1. Rebase onto the new feat/adf-profiler. I rebased #143 onto main, which now includes #154 (integration tests) and #157 (extract phase). The old base commit eb3a6f9 is gone; the branch is now eb654cc (the profiler) plus three commits: the exact factory-name match for actuals, a golden-reference integration test, and --days defaults to 45. To replay just your commit:

git fetch origin
git rebase --onto origin/feat/adf-profiler eb3a6f9 feat/adf-profiler-setup-mcp

There's one conflict, in app/deploy_app.py. main reworded app_description to "flowx MCP server — Lakeflow jobs migration tools"; keep that line and your with_profile lines. Everything else applies cleanly.

2. Rename the profile extra to azure. #157 added an azure extra with exactly the same packages as profile (azure-identity>=1.17, requests>=2.31), so #143 dropped profile and the profiler now uses [azure]. As-is, after the rebase:

• app/deploy_helpers.py (app_requirements) and app/deploy.sh read optional-dependencies["profile"], so both raise a KeyError.
• scripts/bootstrap.sh runs pip install -e "${PLUGIN_ROOT}[profile]". That fails silently: pip only warns that the extra doesn't exist and installs flowx without the Azure packages, and profile then exits 2 with the install hint.

Places to update (all [profile] / ["profile"] / "the profile extra"):

• app/deploy_helpers.py, app/deploy.sh, app/README.md
• scripts/bootstrap.sh
• skills/flowx-setup/SKILL.md, skills/flowx-profile/SKILL.md
• src/flowx/mcp/server.py (server instructions + tool docstring)
• tests/unit/test_setup_profile_opt_in.py

Design question:

extract needs the same azure extra, and its skill already tells users to run pip install -e '<plugin_dir>[azure]' by hand. Should setup ask a single question covering both, e.g. "Do you want the live Azure features (pull ADF definitions with extract, estimate current spend with profile)?", with --with-azure / with_azure / WITH_AZURE=1 instead of the profile-only names? Then the extract skill could point to setup instead of its manual install line. I'd lean yes, but I'm happy either way, and fine as a follow-up if you'd rather not widen this PR.

[lorenzorubi-db]

tomllib needs Python ≥ 3.11, a guard upfront gives a clear message

Suggested change

	if [ "${WITH_PROFILE:-0}" = "1" ]; then
	echo "==> Adding the profile command's Azure deps (WITH_PROFILE=1)"
	python3 -c 'import sys, tomllib; print("\n".join(tomllib.load(open(sys.argv[1], "rb"))["project"]["optional-dependencies"]["profile"]))' \
	"$REPO_ROOT/pyproject.toml" >> "$STAGE_DIR/requirements.txt"
	fi
	if [ "${WITH_PROFILE:-0}" = "1" ]; then
	if ! python3 -c 'import tomllib' 2>/dev/null; then
	echo "WITH_PROFILE=1 needs python3 >= 3.11 (it reads pyproject.toml with tomllib); found $(python3 --version 2>&1)" >&2
	exit 1
	fi
	echo "==> Adding the profile command's Azure deps (WITH_PROFILE=1)"
	python3 -c 'import sys, tomllib; print("\n".join(tomllib.load(open(sys.argv[1], "rb"))["project"]["optional-dependencies"]["profile"]))' \
	"$REPO_ROOT/pyproject.toml" >> "$STAGE_DIR/requirements.txt"
	fi

[lorenzorubi-db]

Long scans may not fit in a single MCP call. The scan makes one REST call per pipeline run (~0.3 s each), so a subscription-wide scan of a busy factory can run for hours. The MCP client's request limit is probably shorter than FLOWX_MCP_TIMEOUT (30 min), so raising that doesn't help, and catching TimeoutExpired here would only improve the error message for a call the client has already abandoned.

Not for this PR, but I think MCP profile should eventually be async: start the scan, return a run id, and poll for the report. Until then, could the docstring and the skill say that MCP profile is for scoped scans (resource_group / factory_name, fewer days), and that subscription-wide scans should run from a notebook?

[lorenzorubi-db]

app/app.yaml has no env: section, so "as app env vars, e.g. from a secret" may leave the user stuck:

Suggested change

	install hint. The app also needs Azure service principal credentials (`AZURE_CLIENT_ID`,
	`AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`) as app env vars, e.g. from a secret.
	install hint. The app also needs an Azure service principal. Add the client secret to the app as a
	**Secret** resource (Apps → your app → Edit → App resources; resource key e.g. `azure-sp-secret`,
	permission *Can read*), then add to `app/app.yaml` before deploying:
	env:
	- name: AZURE_TENANT_ID
	value: "<tenant id>"
	- name: AZURE_CLIENT_ID
	value: "<client id>"
	- name: AZURE_CLIENT_SECRET
	valueFrom: azure-sp-secret
	The service principal needs Reader and Data Factory Contributor on the subscription or resource group
	it scans, and Cost Management Reader on the subscription (see `skills/flowx-profile/sources/adf.md`).

[lorenzorubi-db]

In the app, output_dir is inside the container, so only report_md reaches the user; cost_comparison.csv is left out. Since it should be small, returning it too is the simplest:

Suggested change

	report_md = runner.read_text(out / "metadata" / "tco" / "cost_comparison.md") if result.ok else None
	return _phase_result(result, out, report_md=report_md)
	# Only read the reports on success, so a failed run never returns stale ones from an earlier run.
	report_dir = out / "metadata" / "tco"
	report_md = runner.read_text(report_dir / "cost_comparison.md") if result.ok else None
	report_csv = runner.read_text(report_dir / "cost_comparison.csv") if result.ok else None
	return _phase_result(result, out, report_md=report_md, report_csv=report_csv)

[lorenzorubi-db]

and add report_csv next to report_md here

[lorenzorubi-db]

ADF keeps only 45 days of pipeline run history, so days above 45 adds nothing, and 90 as the documented default is misleading.

Suggested change

	- "profile": source(req: "adf"), output_dir, days(default 90), subscription_id, resource_group,
	- "profile": source(req: "adf"), output_dir, days(default 45, ADF's run-history limit), subscription_id,
	resource_group,