Repository navigation
fix(widget): relaunch when the UI thread stops answering - #168
Merged
Merged
Conversation
A UI thread parked in a call that never returns keeps every window handle valid and every parent alive, so the surface watchdog sees nothing wrong while the widget stops drawing and stops polling. It was seen staying frozen for well over an hour, through an explorer.exe restart, with nothing in the log to show for it. A dedicated thread now asks the UI thread to acknowledge an empty message every couple of seconds and relaunches the widget once it has stayed silent for 30 seconds and at least 10 probes in a row, so a system sleep that begins while a probe is failing cannot trigger it on wake. It never takes the state mutex, since a UI thread wedged while holding that lock is the case it has to survive. It does not rely on IsHungAppWindow either: that stayed false while the UI thread of a taskbar-embedded widget was suspended, and only turned true once a frozen widget had been orphaned. The probe stands down while a debugger is attached. A replacement that freezes again within ten minutes waits twice as long before the next attempt, up to eight minutes, instead of being replaced every half minute for as long as the cause lasts. The frozen process ends with TerminateProcess rather than ExitProcess, as Microsoft recommends when the state of a process's threads is unknown: DLL detach code can deadlock on a lock the wedged thread held. Two watchdogs can now reach the relaunch, so the first one to commit blocks the other from spawning a second replacement. A relaunch also carries --diagnose and --diagnose-append over whenever diagnostics are on, including when they were switched on from the dashboard, so the replacement keeps the log entry explaining why it was started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IsWindowand the parent window, sees nothing wrong. I ran into this on my machine: the widget stayed frozen for well over an hour, through an explorer.exe restart, with nothing in the log, until it was restarted by hand. ASendMessageTimeoutto its window timed out, and the usage cache had not been written since the last successful poll.WM_NULLto the main window every 2 seconds with a 1 second timeout. Once the window has stayed silent for 30 seconds and at least 10 probes in a row, it relaunches the widget through the existing relaunch path. The probe count keeps a system sleep that begins while a probe is failing from triggering a relaunch on wake. The thread never takes the state mutex, since a UI thread stuck while holding that lock is a case it has to survive.IsHungAppWindow: that stayed false while the UI thread of a taskbar-embedded widget was suspended, and only turned true once a frozen widget had been orphaned.CCUM_HANG_RELAUNCHESenvironment variable, like the existing relaunch flags.TerminateProcessrather thanExitProcess, following theExitProcessdocumentation: DLL detach code can deadlock on a lock held by a thread in an unknown state. The shell-restart path keepsexit(0).Win32_System_Diagnostics_Debugfeature forIsDebuggerPresent;Cargo.lockis unchanged.--diagnoseand--diagnose-appendover whenever diagnostics are on, including when they were switched on from the dashboard, so the replacement keeps the log line explaining why it was started.Testing
window::ui_hang_watchdog_tests: a message-only window on a thread that pumps messages and one on a thread that blocks without pumping, which the probe tells apart; a destroyed window left to the surface watchdog; the doubling of the delay and its cap; the streak resetting after ten minutes. I checked that the first test fails when the probe is short-circuited.SuspendThread:watchdog: UI thread stopped answering for 30s -> relaunchingkept in the replacement's log;--diagnose, diagnostics switched on throughWM_APP_ENABLE_DIAGNOSTICS, then frozen: the replacement started with--diagnose --diagnose-appendand kept the log.cargo fmt --all -- --check: clean.cargo build --locked --release: OK.cargo clippy --all-targets: no new warnings.cargo test --locked: 534 passed.poller::retry_after::tests::oversized_headers_store_bounded_cooldowns_and_requests_resume_after_expirycomputesInstant::now() - Duration::from_secs(86_400). It panicked with "overflow when subtracting duration from instant" onmainwhile my machine had been up for 19 hours, and passed once uptime went past 24 hours.Checklist