Skip to content

fix(process): recover from broken process pool - #95

Open
olivermeyer wants to merge 1 commit into
TkTech:mainfrom
olivermeyer:fix/process-pool-recovery
Open

fix(process): recover from broken process pool#95
olivermeyer wants to merge 1 commit into
TkTech:mainfrom
olivermeyer:fix/process-pool-recovery

Conversation

@olivermeyer

@olivermeyer olivermeyer commented Jul 21, 2026

Copy link
Copy Markdown

Fixes #94.

Why?
When a subprocess dies abruptly (via os._exit(), a signal, OOM, or an exception that can't be pickled back to the parent), concurrent.futures marks the pool as broken and raises BrokenProcessPool on every subsequent submit() call. This exception escaped ProcessExecutor.push() into the worker's job-pull loop, killing the entire worker — triggering crash-restart loops under a supervisor and unbounded max_attempts inflation via the Recovery plugin.

How?
Pool construction is extracted into a _create_pool() helper. In push(), the pool.submit() call is wrapped in a try/except BrokenProcessPool: on failure the broken pool is replaced with a fresh one (old.shutdown(wait=False)), and the submit is retried once. If the retry also raises, the job is routed through on_job_completed(exc=...) inline so it fails/retries per its own max_attempts without BrokenProcessPool ever reaching the worker loop. A defensive pids_for_job.pop() is added to _on_job_completed to clean up entries left by processes that died before their finally block ran.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@olivermeyer
olivermeyer marked this pull request as ready for review July 21, 2026 09:14
@olivermeyer

Copy link
Copy Markdown
Author

Hi @TkTech any chance you take a look at this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ProcessExecutor: abrupt subprocess death breaks the pool permanently and kills the whole worker

1 participant