Skip to content

fix: Correctly evaluate pl.len() with lazy validation in plan#371

Merged
Oliver Borchert (borchero) merged 2 commits into
mainfrom
fix-length
Jul 22, 2026
Merged

fix: Correctly evaluate pl.len() with lazy validation in plan#371
Oliver Borchert (borchero) merged 2 commits into
mainfrom
fix-length

Conversation

@borchero

Copy link
Copy Markdown
Member

Motivation

Moritz Potthoff (@MoritzPotthoffQC) discovered a bug where pl.len() was reported incorrectly.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an incorrect pl.len() result when using pl.scan_parquet(...).pipe(schema.validate, eager=False) by adjusting the Polars plugin fast-path for the “no rule failures” case, and adds coverage to prevent regressions.

Changes:

  • Add a regression test asserting pl.len() remains correct after lazy validation on a scanned parquet.
  • Update the Rust all_rules_required plugin to return an all-true boolean bitmap (length = input) when there are no validation failures, to support a filter fast-path.
  • Extend the row-wise validation benchmark to parameterize eager vs lazy validation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests/schema/test_validate.py Adds a regression test for lazy validation over scan_parquet ensuring pl.len() stays correct.
tests/benches/test_schema.py Updates benchmark to cover both eager and lazy validation modes.
src/polars_plugin/mod.rs Changes all_rules_required success-path output to a full-length all-true bitmap series to avoid incorrect lazy plan behavior.

Comment thread tests/benches/test_schema.py
@borchero
Oliver Borchert (borchero) merged commit ced2b96 into main Jul 22, 2026
31 checks passed
@borchero
Oliver Borchert (borchero) deleted the fix-length branch July 22, 2026 14:18
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (0cbc66a) to head (83cae0a).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #371   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           56        56           
  Lines         3507      3507           
=========================================
  Hits          3507      3507           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants