Skip to content

Index postcode lookups; don't geocode blank coordinates to 0,0 - #508

Merged
ev-sc merged 5 commits into
mainfrom
flood-postcode-index
Sep 23, 2026
Merged

ev-sc merged 5 commits into
mainfrom
flood-postcode-index

Conversation

@ev-sc

@ev-sc ev-sc commented Sep 23, 2026

Copy link
Copy Markdown
Member

Index postcode lookups; drop a duplicate index; don't geocode blank coordinates to 0,0

  • New index: adds data_record_postcode_idx on (data_source_id, lower(json->>'postcode')). This matches exactly what the REST API's EXACT filter generates, so no code change is needed. The API's postcode query drops from 1,642 ms to 0.022 ms, measured locally on a 1.75M-row data source. Estimated size on production: ~60 MB now (about 44% of rows have a postcode key), ~200 MB after the Flooded People import.
  • Duplicate index dropped: removes idx_data_record_source_id_covering, a hand-made index identical to the migration-managed data_record_source_id_covering_id (btree (data_source_id) INCLUDE (id)). It had 7 scans against 3,123 for its twin since April and backs no constraint. It's the only exact duplicate in the production database. This frees ~350 MB, more than the new index adds. IF EXISTS, since no other environment has it.
  • Geocoding: records with blank latitude/longitude are now left ungeocoded instead of being placed at 0,0 (Number("") is 0). Adds a unit test.
  • README: adds a troubleshooting note for the Turbopack Google Fonts dev error. A stale .next/dev cache makes every route return 500; rm -rf .next/dev fixes it.

Deploying the migrations

  • Postcode index: migrations run inside a transaction, so it's built without CONCURRENTLY. That blocks writes to data_record, but not reads, while it builds: 3.3 s on a 7 GB, 1.95M-row table locally, and production's table is smaller (4.1 GB heap). To avoid even that, build it by hand before deploying, and the migration becomes a no-op:

    CREATE INDEX CONCURRENTLY IF NOT EXISTS data_record_postcode_idx
    ON data_record (data_source_id, (lower(json->>'postcode')));
    

    If a concurrent build fails, Postgres leaves an invalid index behind under that name. The migration detects that, drops it and rebuilds it.

  • Duplicate drop: DROP INDEX is instant, but it needs a brief exclusive lock on data_record, so it waits for queries already running on the table. The migration tool's 10 s lock_timeout limits that wait.

🤖 Generated with Claude Code

ev-sc and others added 4 commits September 23, 2026 00:08
FilterType.EXACT on the postcode column was a sequential scan over every record
in the data source. The expression index matches the filter's
lower(json->>'postcode') exactly, so the planner uses it with no code change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Number("") is 0, so a Coordinates-geocoded record with empty latitude and
longitude cells passed the isNaN check and was placed at 0,0 in the Gulf of
Guinea. Treat blank values as missing, so the record is left ungeocoded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turbopack can cache a Google Fonts response that uses dynamic-subset
/l/font?kit=...&skey=... URLs, which its font loader can't parse. In dev
the resulting compile error makes every route, API included, return 500.
Clearing .next/dev fixes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Document the manual CONCURRENTLY build for deploys that can't take the
brief write block, and drop an INVALID index left by a failed concurrent
build instead of letting IF NOT EXISTS accept it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Sep 23, 2026

Copy link
Copy Markdown

MAP-1584

Production has idx_data_record_source_id_covering, a hand-made index
identical to the migration-managed data_record_source_id_covering_id
(btree (data_source_id) INCLUDE (id), 350 MB each). It is the only exact
duplicate in the production database. Dropping it frees ~350 MB and one
index's write cost. IF EXISTS, as no other environment has it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ev-sc
ev-sc merged commit c428664 into main Sep 23, 2026
1 check passed
@ev-sc
ev-sc deleted the flood-postcode-index branch September 23, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants