Add a way to retry imports that previously failed - #58
Merged
Conversation
Nothing re-runs a spatial import once it has failed, so a record broken by a defect stays broken after the defect is fixed. Every host wanting to sweep those up has to write the same three things by hand: the jsonb predicate for the failure status, the iteration, and the choice to queue rather than run inline — and getting the last one wrong leaves a record flagged as failed even when the retry succeeded, because `#update_features!` called directly bypasses the `SpatialProcessingJob` callbacks that maintain the status cache. `::with_failed_feature_updates` and `::retry_failed_feature_updates!` put all three in the gem that owns the status cache. Options pass through to the queued job, so a caller can lower the priority to keep a large sweep behind live uploads. This belongs here rather than in a host concern because `referrals` and `sites` carry `spatial_processing_status_cache` without including Connect Engine's `SpatialFeaturesDefaults`; anything defined there would miss them. Refs culturecode/stolo_connect#1962 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HLgkg1oN6dkKfLzhKyDUFt
Member
Author
|
Opened #59 for the design question this raised: That's why |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #57, prompted by review of the host-side sweep in stolo_connect#1964: "Is there no mechanism in spatial_features or connect_engine that does this for us? This seems like a lot to reimplement to do something that should probably just be a single method call."
There wasn't, and the reviewer is right that there should be.
FeatureImport::ClassMethods#update_features!covers bulk re-import, but nothing selects the records whose last attempt failed, and nothing queues the retry. So every host that wants to sweep them up writes the same three things by hand:The third is the one that bites. Calling
#update_features!directly bypasses theSpatialProcessingJobcallbacks that maintainspatial_processing_status_cache, so a record that imports successfully stays flagged as failed — and is picked up again by the next sweep. That was a real bug in the host-side task before this.::with_failed_feature_updatesand::retry_failed_feature_updates!put all three in the gem that owns the status cache. Options pass through to the queued job, so a caller can lower the priority to keep a large sweep behind live uploads:Why here rather than Connect Engine
referralsandsitesboth carryspatial_processing_status_cachewithout including the engine'sSpatialFeaturesDefaultsconcern, so anything defined there would silently miss them. The gem is wherehas_spatial_featuresand the status cache live, so every spatial model gets it regardless of which concern the host mixed in.Verification
bundle exec rspec— 266 examples, 0 failures (19 pending, pre-existing). Three new examples cover the failure-only selection, that successes and never-imported records are left alone, and that options reach the queued job.Deploying
3.11.1 — additive, no behaviour change to existing calls.
connect_engine's~> 3.11constraint already admits it, so only the locks move.Refs https://github.com/culturecode/stolo_connect/issues/1962