Skip to content

fix(conform): never write NaN coordinates to the output GeoJSON - #2

Merged
MiniCodeMonkey merged 1 commit into
sync/upstream-10.2.0from
fix/skip-nan-geometry
Aug 28, 2026
Merged

fix(conform): never write NaN coordinates to the output GeoJSON#2
MiniCodeMonkey merged 1 commit into
sync/upstream-10.2.0from
fix/skip-nan-geometry

Conversation

@MiniCodeMonkey

Copy link
Copy Markdown
Member

Stacked on #1 -- review that one first; this PR's base is sync/upstream-10.2.0.

Summary

ESRI services return records with null geometry as the JSON string "NaN" -- querying the live Bradford County service for the affected records returns "geometry": {"x": "NaN", "y": "NaN"}.

pyesridump's convert_esri_point guards with if x_coord and y_coord: -- a non-empty string is truthy -- so it emits coordinates: ["NaN","NaN"], shape() coerces those to nan, and json.dumps (default allow_nan=True) writes [NaN, NaN] into the output GeoJSON. NaN is invalid GeoJSON per RFC 7946, so every strict parser downstream rejects the file.

Real impact: 67 of 39,638 features in Bradford County, PA (source 1078) broke the entire ChopChop merge stage on every run for weeks.

The codebase already has two guards for exactly this, and both have been dead:

  • openaddr/cache.py:443 (added 2016, e8148048) -- any((isinstance(g, float) and math.isnan(g)) for g in traverse(geom)). traverse() only recurses into lists, but geom is a dict, so it yields the dict itself and the isinstance(g, float) test is never true. This has never fired once. It also only ever knew about float nan, not the string.
  • openaddr/conform.py:900 ("fix-esri-nan", 65a5fffa) -- if source_geom == "POINT (nan nan)". That was Shapely 1.x's spelling; Shapely 2.1.2 / GEOS 3.13.1 in the current image writes POINT (NaN NaN), so the literal silently stopped matching.

Changes

  • openaddr/cache.py:50-56 -- new coordinate_is_usable(): math.isfinite(float(value)), returning False on TypeError/ValueError.
  • openaddr/cache.py:450 -- walk geom.get('coordinates') instead of geom, so the guard actually runs. Catches float nan, the string "NaN", and None.
  • openaddr/conform.py:49-51 -- NONFINITE_COORDINATE_PATTERN = re.compile(r'\b(nan|inf(inity)?)\b', re.IGNORECASE).
  • openaddr/conform.py:901,1258 -- replace the two brittle literal comparisons with that pattern. Now catches POINT (NaN NaN), POINT (nan nan) (OGR's spelling), POINT Z (nan nan nan), polygons, and infinities.
  • openaddr/conform.py:1335 -- json.dumps(out_row, allow_nan=False) as a loud backstop, so this class of bug fails at the source instead of silently downstream.

A regex rather than a shapely re-parse because row_extract_and_reproject runs per row on multi-million-row sources, where a WKT parse per row is a real cost. WKT contains only geometry keywords and numbers, so there are no false positives.

Skip, not geometry: null, for the ESRI path -- cache.py already skips every other geometry-less ESRI feature (raise TypeError("No geometry parsed"), cache.py:446). These records are the same class and only survived because the filter was broken, so skipping is the behavior already in force and the least surprising. Non-ESRI paths keep the codebase's other convention (geometry: null), which is what the conform.py guard always intended.

Tests

Three regression tests, each confirmed to fail without the fix:

  • openaddr/tests/cache.py:415 test_skip_esri_features_with_nan_geometry -- new fixtures us-pa-bradford-{metadata,count-only,0}.json modeled on the real Bradford response, including the "x": "NaN" string. Without the fix: AssertionError: 3 != 2.
  • openaddr/tests/conform.py:81 test_row_convert_to_out_nonfinite_geometry -- without the fix: AssertionError: {'type': 'Point', 'coordinates': (nan, nan)} is not None.
  • openaddr/tests/conform.py:96 test_transform_to_out_geojson_skips_nonfinite_geometry -- reproduces production exactly; without the fix, 'NaN' unexpectedly found in the output.

Upstream's own test_single_ny_orange ("Test complete process_one.process on data NaN values in ESRI response") passed on the unfixed baseline -- its fixture has a NaN feature, but it only asserted on rows[0] and never checked that the NaN row was gone. Hardened here with assertNotIn('NaN', contents) and len(rows) == 8.

One golden value moves: rows[0]['properties']['hash'] 8a72112f6b1404d8 -> a0127261b0619522. row_calculate_hash seeds SHA-1 with cache_fingerprint, which is the md5 of the cache file, so dropping the bogus row shifts every row hash in that source. Benign -- all other assertions on that row (number, city, street, postcode, coordinates) pass unchanged.

Full suite via the CI path (docker build + docker run machine): 128 tests, OK.

Note for ChopChop

Feature count for source 1078 drops 39,638 -> 39,571.

ESRI services report a null point geometry as the string "NaN" rather than
omitting the geometry. Bradford County PA (source 1078) has 67 such records,
and they reached the addresses output as "coordinates": [NaN, NaN] — invalid
GeoJSON per RFC 7946, which made the whole 39,638-feature file unparseable by
Go's encoding/json.

Two guards were meant to catch this and neither did:

- openaddr/cache.py walked traverse(geom), which yields the geometry dict
  itself rather than its coordinates, so the isnan() test never ran on a
  coordinate. It also only recognized float NaN, not the string ESRI sends.
- openaddr/conform.py compared the WKT against the literal "POINT (nan nan)",
  which was Shapely 1.x's spelling. Shapely 2 / GEOS 3.13 writes
  "POINT (NaN NaN)", so the comparison stopped matching.

Skip the feature at download time, matching how every other geometry-less ESRI
feature is already handled, and fall back to a null geometry further down the
pipeline for the non-ESRI paths. json.dumps now runs with allow_nan=False so
this class of bug fails loudly here instead of silently downstream.

test_single_ny_orange asserted a golden row hash derived from the cache file's
md5 fingerprint; dropping the bogus row changes that fingerprint, so the hash
moves and the test now also asserts the feature count and the absence of NaN.

Claude-Session: https://claude.ai/code/session_01JWLFBWRfhWhLUqFFqbPCGm
@MiniCodeMonkey
MiniCodeMonkey merged commit 9962aa5 into sync/upstream-10.2.0 Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant