Repository navigation
_knn_fill repeats the raw-corrwith pattern fixed in #464 and crashes on py3.9 #470
Description
Severity: S2 (crash on a supported input, py3.9 lane) — sibling of #464
#464 fixed the raw-corrwith pattern in engine/model_select.py. The same pattern survives at src/freshdata/engine/missing.py:637 (_knn_fill), which is reachable when len(df) < 30.
Root cause is the one #464 turned up: on pandas 1.5, Series.corr receives the raw values, so a nullable Int64 column holding pd.NA arrives as an object array, and numpy 1.26's average() builds its scale factor as a bare Python float, so scl.shape raises:
AttributeError: 'float' object has no attribute 'shape'
fd.clean's gates make this hard to reach through the public API, but a direct call to _knn_fill crashes on py3.9 for any nullable-integer partner column with a missing value.
Fix: the same as #464 — cast to float64 before correlating (what pandas 2.x does internally) and treat ±inf as missing.
Reported by the collaborator working #464; filing here because engine/missing.py is outside that PR's scope.
Activity
test(missing): pin _knn_fill against nullable-integer partner columns (…
WilliamK112 commentedon Oct 17, 2026
This is fixed on current
mainby #487 (48011ca)._knn_fillnow casts both sides tofloat64before eachcorrwith, replaces±infwithNaN, andtests/test_knn_nullable_correlations.pycovers the supported nullable-integer cases, including the Python 3.9/pandas 1.5 failure shape. This issue can be closed.