Index postcode lookups; don't geocode blank coordinates to 0,0 - #508
Merged
Merged
Conversation
FilterType.EXACT on the postcode column was a sequential scan over every record in the data source. The expression index matches the filter's lower(json->>'postcode') exactly, so the planner uses it with no code change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Number("") is 0, so a Coordinates-geocoded record with empty latitude and
longitude cells passed the isNaN check and was placed at 0,0 in the Gulf of
Guinea. Treat blank values as missing, so the record is left ungeocoded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turbopack can cache a Google Fonts response that uses dynamic-subset /l/font?kit=...&skey=... URLs, which its font loader can't parse. In dev the resulting compile error makes every route, API included, return 500. Clearing .next/dev fixes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Document the manual CONCURRENTLY build for deploys that can't take the brief write block, and drop an INVALID index left by a failed concurrent build instead of letting IF NOT EXISTS accept it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production has idx_data_record_source_id_covering, a hand-made index identical to the migration-managed data_record_source_id_covering_id (btree (data_source_id) INCLUDE (id), 350 MB each). It is the only exact duplicate in the production database. Dropping it frees ~350 MB and one index's write cost. IF EXISTS, as no other environment has it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joaquimds
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Index postcode lookups; drop a duplicate index; don't geocode blank coordinates to 0,0
data_record_postcode_idxon(data_source_id, lower(json->>'postcode')). This matches exactly what the REST API'sEXACTfilter generates, so no code change is needed. The API's postcode query drops from 1,642 ms to 0.022 ms, measured locally on a 1.75M-row data source. Estimated size on production: ~60 MB now (about 44% of rows have apostcodekey), ~200 MB after the Flooded People import.idx_data_record_source_id_covering, a hand-made index identical to the migration-manageddata_record_source_id_covering_id(btree (data_source_id) INCLUDE (id)). It had 7 scans against 3,123 for its twin since April and backs no constraint. It's the only exact duplicate in the production database. This frees ~350 MB, more than the new index adds.IF EXISTS, since no other environment has it.Number("")is 0). Adds a unit test..next/devcache makes every route return 500;rm -rf .next/devfixes it.Deploying the migrations
Postcode index: migrations run inside a transaction, so it's built without
CONCURRENTLY. That blocks writes todata_record, but not reads, while it builds: 3.3 s on a 7 GB, 1.95M-row table locally, and production's table is smaller (4.1 GB heap). To avoid even that, build it by hand before deploying, and the migration becomes a no-op:If a concurrent build fails, Postgres leaves an invalid index behind under that name. The migration detects that, drops it and rebuilds it.
Duplicate drop:
DROP INDEXis instant, but it needs a brief exclusive lock ondata_record, so it waits for queries already running on the table. The migration tool's 10 slock_timeoutlimits that wait.🤖 Generated with Claude Code