fix(conform): never write NaN coordinates to the output GeoJSON - #2
Merged
Merged
Conversation
ESRI services report a null point geometry as the string "NaN" rather than omitting the geometry. Bradford County PA (source 1078) has 67 such records, and they reached the addresses output as "coordinates": [NaN, NaN] — invalid GeoJSON per RFC 7946, which made the whole 39,638-feature file unparseable by Go's encoding/json. Two guards were meant to catch this and neither did: - openaddr/cache.py walked traverse(geom), which yields the geometry dict itself rather than its coordinates, so the isnan() test never ran on a coordinate. It also only recognized float NaN, not the string ESRI sends. - openaddr/conform.py compared the WKT against the literal "POINT (nan nan)", which was Shapely 1.x's spelling. Shapely 2 / GEOS 3.13 writes "POINT (NaN NaN)", so the comparison stopped matching. Skip the feature at download time, matching how every other geometry-less ESRI feature is already handled, and fall back to a null geometry further down the pipeline for the non-ESRI paths. json.dumps now runs with allow_nan=False so this class of bug fails loudly here instead of silently downstream. test_single_ny_orange asserted a golden row hash derived from the cache file's md5 fingerprint; dropping the bogus row changes that fingerprint, so the hash moves and the test now also asserts the feature count and the absence of NaN. Claude-Session: https://claude.ai/code/session_01JWLFBWRfhWhLUqFFqbPCGm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1 -- review that one first; this PR's base is
sync/upstream-10.2.0.Summary
ESRI services return records with null geometry as the JSON string
"NaN"-- querying the live Bradford County service for the affected records returns"geometry": {"x": "NaN", "y": "NaN"}.pyesridump's
convert_esri_pointguards withif x_coord and y_coord:-- a non-empty string is truthy -- so it emitscoordinates: ["NaN","NaN"],shape()coerces those tonan, andjson.dumps(defaultallow_nan=True) writes[NaN, NaN]into the output GeoJSON.NaNis invalid GeoJSON per RFC 7946, so every strict parser downstream rejects the file.Real impact: 67 of 39,638 features in Bradford County, PA (source 1078) broke the entire ChopChop merge stage on every run for weeks.
The codebase already has two guards for exactly this, and both have been dead:
openaddr/cache.py:443(added 2016,e8148048) --any((isinstance(g, float) and math.isnan(g)) for g in traverse(geom)).traverse()only recurses intolists, butgeomis adict, so it yields the dict itself and theisinstance(g, float)test is never true. This has never fired once. It also only ever knew about floatnan, not the string.openaddr/conform.py:900("fix-esri-nan",65a5fffa) --if source_geom == "POINT (nan nan)". That was Shapely 1.x's spelling; Shapely 2.1.2 / GEOS 3.13.1 in the current image writesPOINT (NaN NaN), so the literal silently stopped matching.Changes
openaddr/cache.py:50-56-- newcoordinate_is_usable():math.isfinite(float(value)), returningFalseonTypeError/ValueError.openaddr/cache.py:450-- walkgeom.get('coordinates')instead ofgeom, so the guard actually runs. Catches floatnan, the string"NaN", andNone.openaddr/conform.py:49-51--NONFINITE_COORDINATE_PATTERN = re.compile(r'\b(nan|inf(inity)?)\b', re.IGNORECASE).openaddr/conform.py:901,1258-- replace the two brittle literal comparisons with that pattern. Now catchesPOINT (NaN NaN),POINT (nan nan)(OGR's spelling),POINT Z (nan nan nan), polygons, and infinities.openaddr/conform.py:1335--json.dumps(out_row, allow_nan=False)as a loud backstop, so this class of bug fails at the source instead of silently downstream.A regex rather than a shapely re-parse because
row_extract_and_reprojectruns per row on multi-million-row sources, where a WKT parse per row is a real cost. WKT contains only geometry keywords and numbers, so there are no false positives.Skip, not
geometry: null, for the ESRI path --cache.pyalready skips every other geometry-less ESRI feature (raise TypeError("No geometry parsed"),cache.py:446). These records are the same class and only survived because the filter was broken, so skipping is the behavior already in force and the least surprising. Non-ESRI paths keep the codebase's other convention (geometry: null), which is what theconform.pyguard always intended.Tests
Three regression tests, each confirmed to fail without the fix:
openaddr/tests/cache.py:415test_skip_esri_features_with_nan_geometry-- new fixturesus-pa-bradford-{metadata,count-only,0}.jsonmodeled on the real Bradford response, including the"x": "NaN"string. Without the fix:AssertionError: 3 != 2.openaddr/tests/conform.py:81test_row_convert_to_out_nonfinite_geometry-- without the fix:AssertionError: {'type': 'Point', 'coordinates': (nan, nan)} is not None.openaddr/tests/conform.py:96test_transform_to_out_geojson_skips_nonfinite_geometry-- reproduces production exactly; without the fix,'NaN' unexpectedly found inthe output.Upstream's own
test_single_ny_orange("Test complete process_one.process on data NaN values in ESRI response") passed on the unfixed baseline -- its fixture has a NaN feature, but it only asserted onrows[0]and never checked that the NaN row was gone. Hardened here withassertNotIn('NaN', contents)andlen(rows) == 8.One golden value moves:
rows[0]['properties']['hash']8a72112f6b1404d8->a0127261b0619522.row_calculate_hashseeds SHA-1 withcache_fingerprint, which is the md5 of the cache file, so dropping the bogus row shifts every row hash in that source. Benign -- all other assertions on that row (number, city, street, postcode, coordinates) pass unchanged.Full suite via the CI path (
docker build+docker run machine): 128 tests, OK.Note for ChopChop
Feature count for source 1078 drops 39,638 -> 39,571.