Repository navigation
Native engines silently drop pandas extension dtypes (Int64 -> int64, string -> object) with nothing in backend_differences #465
Description
Severity: S3 (silent dtype change, values preserved, undisclosed) — found by the production-readiness campaign
On the fully native path (strategy="conservative", fix_dtypes=False), native ingestion does not carry pandas extension dtypes back, and records nothing in backend_differences:
| input dtype | pandas engine | polars | duckdb |
|---|---|---|---|
Int64 (no nulls) |
Int64 |
int64 |
int64 |
Float64 |
Float64 |
float64 |
float64 |
string |
string |
object |
object |
boolean (with nulls) |
boolean |
object |
boolean |
The values are equal in every row above — this is a dtype-parity gap, not data loss. The data-loss case (Int64/UInt64 with nulls rounding through float64) is #444 and is being fixed separately; please keep the two apart.
Fix: either restore the input extension dtype on egress, or record a backend_difference for each column whose dtype the native path changed.
Activity
WilliamK112 commentedon Oct 13, 2026
/assign
I reproduced the extension-dtype changes in both native engines. I am taking the audit option requested here: record a per-column backend difference after pandas output conversion when an input extension dtype changes. This preserves native results and requested conversions, keeps the existing wide-nullable-integer precision fix intact, and avoids materializing native handles solely for dtype checks.
github-actions commentedon Oct 13, 2026
Hey @WilliamK112! 🎉 You have claimed this issue.
Quick Onboarding Checklist
pytest tests/<test_file>.py --no-cov(bypasses CI 93% global gate during local development).ruff check .(Need to unassign yourself? Simply reply with
/unassign.)