Skip to content

Prevent task ack on Windows build dir restart - #5586

Open
Xeicker wants to merge 1 commit into
masterfrom
fix-windows-unrecoverable-restart
Open

Xeicker wants to merge 1 commit into
masterfrom
fix-windows-unrecoverable-restart

Conversation

@Xeicker

@Xeicker Xeicker commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

When clearing a build directory fails on Windows (shell.remove_directory returns False), _handle_unrecoverable_error_on_windows() triggers utils.restart_machine() (shutdown /f /r /t 0) to recover from lingering file locks or disk issues, but previously returned None immediately while Windows initiated shutdown asynchronously.

Returning normally caused two races during the few seconds before the OS terminated run_bot.py:

  1. False Pub/Sub ack() of the failed task: _unpack_build / _unpack_builds / _unpack_custom_build returned False, allowing commands.process_command(task) to return without raising and PubSubTask.lease() to acknowledge the failed task message.
  2. Shutdown lease race: task_loop() looped around and pulled/leased a new task right before the OS killed the bot process.

Changes

  • build_manager._handle_unrecoverable_error_on_windows(): Sleep for 60s after utils.restart_machine() while OS shutdown terminates the process, and call sys.exit(1) as a safety net if shutdown fails so PubSubTask.lease() unwinds via finally: without acknowledging the failed task or leasing another one.
  • utils.restart_machine(): Log an error if the OS shutdown command returns a non-zero exit code.
  • build_manager_test.py: Add RegularBuildUnpackTest.test_unpack_build_windows_remove_directory_fails_restarts_and_exits covering the Windows directory-cleanup failure path.

@Xeicker
Xeicker requested a review from a team as a code owner November 9, 2026 17:35
@Xeicker
Xeicker requested a review from IvanBM18 November 9, 2026 17:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant