Skip to content

Add a way to retry imports that previously failed - #58

Merged
njakobsen merged 1 commit into
masterfrom
retry-failed-imports
Aug 3, 2026
Merged

Add a way to retry imports that previously failed#58
njakobsen merged 1 commit into
masterfrom
retry-failed-imports

Conversation

@njakobsen

Copy link
Copy Markdown
Member

Follow-up to #57, prompted by review of the host-side sweep in stolo_connect#1964: "Is there no mechanism in spatial_features or connect_engine that does this for us? This seems like a lot to reimplement to do something that should probably just be a single method call."

There wasn't, and the reviewer is right that there should be. FeatureImport::ClassMethods#update_features! covers bulk re-import, but nothing selects the records whose last attempt failed, and nothing queues the retry. So every host that wants to sweep them up writes the same three things by hand:

  • the jsonb predicate for the failure status
  • the iteration
  • the choice to queue rather than run inline

The third is the one that bites. Calling #update_features! directly bypasses the SpatialProcessingJob callbacks that maintain spatial_processing_status_cache, so a record that imports successfully stays flagged as failed — and is picked up again by the next sweep. That was a real bug in the host-side task before this.

::with_failed_feature_updates and ::retry_failed_feature_updates! put all three in the gem that owns the status cache. Options pass through to the queued job, so a caller can lower the priority to keep a large sweep behind live uploads:

ReferralSubmission.retry_failed_feature_updates!(priority: 10)

Why here rather than Connect Engine

referrals and sites both carry spatial_processing_status_cache without including the engine's SpatialFeaturesDefaults concern, so anything defined there would silently miss them. The gem is where has_spatial_features and the status cache live, so every spatial model gets it regardless of which concern the host mixed in.

Verification

bundle exec rspec — 266 examples, 0 failures (19 pending, pre-existing). Three new examples cover the failure-only selection, that successes and never-imported records are left alone, and that options reach the queued job.

Deploying

3.11.1 — additive, no behaviour change to existing calls. connect_engine's ~> 3.11 constraint already admits it, so only the locks move.

Refs https://github.com/culturecode/stolo_connect/issues/1962

Nothing re-runs a spatial import once it has failed, so a record broken by a defect stays broken after the defect is fixed. Every host wanting to sweep those up has to write the same three things by hand: the jsonb predicate for the failure status, the iteration, and the choice to queue rather than run inline — and getting the last one wrong leaves a record flagged as failed even when the retry succeeded, because `#update_features!` called directly bypasses the `SpatialProcessingJob` callbacks that maintain the status cache.

`::with_failed_feature_updates` and `::retry_failed_feature_updates!` put all three in the gem that owns the status cache. Options pass through to the queued job, so a caller can lower the priority to keep a large sweep behind live uploads.

This belongs here rather than in a host concern because `referrals` and `sites` carry `spatial_processing_status_cache` without including Connect Engine's `SpatialFeaturesDefaults`; anything defined there would miss them.

Refs culturecode/stolo_connect#1962

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HLgkg1oN6dkKfLzhKyDUFt
@njakobsen

Copy link
Copy Markdown
Member Author

Opened #59 for the design question this raised: spatial_processing_status_cache is defined as queued-job state but read everywhere as last-import outcome, so an inline import that succeeds leaves a record flagged as failed permanently.

That's why ::retry_failed_feature_updates! here queues rather than importing inline — it's the only path that maintains the status honestly. It also means the sweep will over-select on hosts that import inline (stolo_connect's nightly geometry task does), picking up records that already recovered. They self-correct on the first run, but it's worth knowing before #59 is decided.

@njakobsen
njakobsen merged commit ff19fdd into master Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant