Skip to content

fix(airflow): scope default args to DAG context - #178

Open
rudra337-lgtm wants to merge 2 commits into
databricks-solutions:mainfrom
rudra337-lgtm:fix/150-dag-default-args-scope
Open

rudra337-lgtm wants to merge 2 commits into
databricks-solutions:mainfrom
rudra337-lgtm:fix/150-dag-default-args-scope

Conversation

@rudra337-lgtm

@rudra337-lgtm rudra337-lgtm commented Oct 18, 2026 •

Copy link
Copy Markdown

Fixes #150.

Commit 1 — scope default args to DAG context

Airflow operators created after a with DAG(...) block were still captured by flowx and inherited that DAG's default_args, even though Airflow does not make unwired operators outside the context members of the DAG.

The visitor now requires either an active DAG context or an explicit dag= reference before registering a classic operator call. Added a regression test covering the context boundary.

Commit 2 — follow statically invoked wiring-only helpers

Issue #150 repro 2: a plain helper whose body only wires already-captured tasks (def wire(): start >> t followed by wire()) was skipped by both the capture pass and the independent source audit, so the DAG was verified while the dependency edge was missing.

Both passes now follow the same rule — module-level definition, no decorators, no parameters, wiring-only statements (shift chains, chain/cross_downstream, set_upstream/set_downstream), and at least one static invocation — so Airflow's one-shot execution is reproduced statically and reconciliation compares identical edge sets on both sides. Statements inside a followed helper that cannot be resolved fail closed (unclaimed_dag_statement) instead of verifying with a missing edge. An uncalled helper stays uncaptured, matching Airflow (its wiring never runs).

Verification

  • uv run pytest tests/unit/test_airflow_reconciliation.py -q — 32 passed, 0 failed (29 before this PR's work; +1 default-args regression test, +3 wiring-helper tests).
  • uv run ruff check clean on the touched files.
  • uv run pytest tests/unit -q — 1386 passed, 2 skipped, 121 failed; all 121 failures reproduce identically on the pre-change branch (1de4f21) out of the box on this Windows machine (missing tzdata in the vendored Python 3.14 runtime, path-separator comparisons, provider JSON canonicalization, deployer summary reads) — no failures introduced, none of the related failures touched.
  • Issue repro scripts: repro 1 (t no longer in the DAG, no default_args leak), repro 2 (edge start -> t now captured, verified), the with DAG helper variant, and the uncalled-helper Airflow-parity case all behave as expected.

Remaining (issue part 3)

The module-wide "account for every operator call" check and the operator-wired-outside-context shape (currently fail-closed via unclaimed_dag_statement) are larger follow-ups; both were left out of this PR to keep the change reviewable.

A plain helper whose body only wires captured tasks (start >> t, chain,
cross_downstream, set_upstream/set_downstream) was skipped by both the
capture pass and the independent audit, so the DAG verified while the
dependency was missing (databricks-solutions#150 repro 2). Both passes now follow the same
rule: a module-level, parameterless, undecorated helper with wiring-only
statements that is statically invoked, so Airflow's one-shot execution
is reproduced statically and reconciliation compares identical edge
sets. Unfollowed statements still fail closed through unclaimed
statement checks.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airflow loader: DAG membership and edge gaps beyond #147

1 participant