Destroy orphan BlockedExecution rows when the job class no longer resolves - #783
Conversation
|
@rosa any chance getting a look at this? |
|
Hey @julik, sorry for the delay! I've been on-call for a couple of weeks, and working on a lot of other stuff, but I'm on vacation this week and catching up with PRs. I have a few in the queue before this one, so it'll take a few days more. |
|
Hey @julik! Sorry again for the delay. I'm looking into this one now. I wonder: would it make sense, instead of just discarding these jobs, to mark them as failed instead, so you have the chance of doing something with them? Or discarding if you don't need them, that's always an option. |
|
I will check - I think anything but having things "occupying a slot" will work |
|
Done - hope this works ;-) |
…olves If an ActiveJob class with limits_concurrency is renamed or removed between deploys, any BlockedExecution rows referencing the old class name would cause the dispatcher's concurrency-maintenance tick to raise DelegationError forever: release -> acquire_concurrency_lock -> Semaphore.wait -> job.concurrency_limit, which delegates to a nil job_class. Guard the release path (symmetric with Job#acquire_concurrency_lock) and short-circuit set_expires_at with the default concurrency period when the class is unresolvable.
…g it Per review feedback: instead of silently discarding a blocked job whose class no longer resolves, mark it as failed so it shows up in Mission Control where it can be retried (once the class is restored) or discarded. Either way the blocked row is destroyed so the dispatcher stops re-picking it.
ccfac3b to
2b31294
Compare
Both a worker picking up a job whose class is gone and the concurrency maintenance releasing a blocked one now fail the job with SolidQueue::Job::ClassMissingError. Before, the first case surfaced as whatever NameError Active Job's deserialization raised, and the second one, just added, had its own BlockedExecution::JobClassMissingError, so the same user mistake read differently in failed jobs depending on where it was caught. The error subclasses NameError, which is what resolving the class raises, so anything matching on NameError still matches. Also surface the new failed: key in the release_blocked log line, document the behaviour in the README's concurrency controls section, and add the magic comment to the new test file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41
2b31294 to
c4426a3
Compare
|
Thanks @julik! I've pushed a couple of small tweaks to your solution, but basically this is it. Thank you! |
Fixes #784.
Summary
Fixes a
DelegationErrorraised on everySolidQueue::Dispatcher::ConcurrencyMaintenancetick when aBlockedExecutionrow references aJobwhoseclass_nameno longersafe_constantizes — e.g. because the concurrency-limited ActiveJob class was renamed or removed between deploys.Job#acquire_concurrency_lockalready gates onconcurrency_limited?(which checksjob_class.present?), butBlockedExecution#acquire_concurrency_lock— invoked from the dispatcher-driven release path — went straight toSemaphore.wait(job), which callsjob.concurrency_limit, which is delegated to aniljob_class:Because the exception is raised inside
BlockedExecution#release's transaction, the row is never destroyed or promoted, and the dispatcher re-picks the same orphans forever — flooding error reporters and starving any legitimately blocked jobs that share the concurrency key.Changes
BlockedExecution#release: ifjob.job_classisnil, destroy the orphan row and emitorphaned: truein therelease_blockedinstrumentation payload. Otherwise fall through to the existing acquire/promote/destroy path.BlockedExecution#set_expires_at: fall back toSolidQueue.default_concurrency_control_periodwhen the class does not resolve (guards the same latent bug at create time).Job#job_classpromoted from private to public soBlockedExecutioncan consult it.Test plan
bin/rails test test/models/solid_queue/blocked_execution_test.rb— new test passes.bin/rails test test/models/— full model suite green (97 runs, 585 assertions, 0 failures).