feat: add DQDL support via EvaluateDataQuality (#205)

[AMC-hawk]

Issue #, if available: Closes #205

Description of changes:

Deequ already supports data freshness as a DQDL rule (awslabs/deequ#661, in every release since 2.0.14, including the 2.0.21 jar PyDeequ pins). PyDeequ doesn't expose DQDL, so Python users can't use it. This PR adds a thin wrapper around com.amazon.deequ.dqdl.EvaluateDataQuality.

from pydeequ.dqdl import EvaluateDataQuality

ruleset = 'Rules=[IsComplete "id", DataFreshness "updated_at" <= 24 hours]'

EvaluateDataQuality.process(spark, df, ruleset)      # one row per rule
EvaluateDataQuality.processRows(spark, df, ruleset)  # originalData / ruleOutcomes / rowLevelOutcomes

Changes

• pydeequ/dqdl.py: EvaluateDataQuality.process and processRows, matching the Scala API. Supports additionalDataSources={"alias": df} for rules like RowCountMatch, pandas=True output, and pandas input (same as VerificationSuite.onData).
• tests/test_dqdl.py: output schema, passing and failing DataFreshness rules (including the compliance metric), days/minutes units, additionalDataSources, pandas output, row-level outcomes, and input validation.
• README.md and docs/source/pydeequ.rst: usage section and API docs.

No changes to existing APIs or the Deequ dependency.

Notes

• DataFreshness doesn't have row-level support in Deequ, so processRows lists it under DataQualityRulesSkip. The README says this.
• Spark 3.5 logs ANTLR version mismatch warnings (4.7 vs 4.9.3) when the DQDL parser loads. Parsing works; the warnings come from the upstream dqdl dependency.

Testing

pytest tests/test_dqdl.py tests/test_verification.py: 15 passed locally (Spark 3.5.9, Python 3.11, Java 17, Deequ 2.0.21-spark-3.5). black and flake8 pass on the new files.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow

[github-actions]

NIT: Prefer the existing to_scala_map helper from scala_utils for consistency rather than inlining PythonUtils.toScalaMap.

dqdl.py:94 return data._jdf, rulesetDefinition, spark_session._jvm.PythonUtils.toScalaMap(sources) vs scala_utils.py:92 which defines to_scala_map as exactly spark_session._jvm.PythonUtils.toScalaMap(d).

Refutation trail (why this survived the Critic's disprove pass)

Hypothesis (Investigator): Calling spark_session._jvm.PythonUtils.toScalaMap(sources) directly instead of the existing to_scala_map helper is an inconsistency / possible bug for empty maps.

Disprove attempt (Critic): Read scala_utils.py:83-92; to_scala_map is literally spark_session._jvm.PythonUtils.toScalaMap(d). The diff inlines the same call. For the None case, additionalDataSources or {} yields {}, so toScalaMap({}) produces an empty Scala map — the same construct used for empty tags elsewhere. Behaviorally equivalent; only a style inconsistency (bypasses the helper).

The Critic's default verdict is OVERTURNED. UPHELD findings are those it tried — and failed — to refute.

[AMC-hawk]

@sudsali Can you please review it. Thanks