Skip to content

fix(widget): relaunch when the UI thread stops answering - #168

Merged
CodeZeno merged 1 commit into
CodeZeno:mainfrom
amorel:pr/ui-hang-watchdog
Oct 19, 2026
Merged

CodeZeno merged 1 commit into
CodeZeno:mainfrom
amorel:pr/ui-hang-watchdog

Conversation

@amorel

@amorel amorel commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The widget can stop responding while its process keeps running. Its UI thread gets stuck in a call that never returns, so it no longer draws and no longer polls. Every window handle stays valid and every parent stays alive, so the existing watchdog, which checks IsWindow and the parent window, sees nothing wrong. I ran into this on my machine: the widget stayed frozen for well over an hour, through an explorer.exe restart, with nothing in the log, until it was restarted by hand. A SendMessageTimeout to its window timed out, and the usage cache had not been written since the last successful poll.
  • I could not identify which call blocked, so this does not fix the cause. It makes the widget recover, whatever the cause.
  • A dedicated thread sends WM_NULL to the main window every 2 seconds with a 1 second timeout. Once the window has stayed silent for 30 seconds and at least 10 probes in a row, it relaunches the widget through the existing relaunch path. The probe count keeps a system sleep that begins while a probe is failing from triggering a relaunch on wake. The thread never takes the state mutex, since a UI thread stuck while holding that lock is a case it has to survive.
  • It does not rely on IsHungAppWindow: that stayed false while the UI thread of a taskbar-embedded widget was suspended, and only turned true once a frozen widget had been orphaned.
  • If the replacement freezes again within ten minutes, the next attempt waits twice as long (30 s, then 1, 2, 4 and at most 8 minutes), so a cause that persists, such as a hung explorer.exe, does not lead to a relaunch every half minute. The streak travels in a CCUM_HANG_RELAUNCHES environment variable, like the existing relaunch flags.
  • The frozen process ends with TerminateProcess rather than ExitProcess, following the ExitProcess documentation: DLL detach code can deadlock on a lock held by a thread in an unknown state. The shell-restart path keeps exit(0).
  • The probe stands down while a debugger is attached, so stepping through the UI thread does not end the session. This adds the Win32_System_Diagnostics_Debug feature for IsDebuggerPresent; Cargo.lock is unchanged.
  • Two watchdogs can now reach the relaunch, so the first one to commit to it blocks the other from spawning a second replacement.
  • A relaunch carries --diagnose and --diagnose-append over whenever diagnostics are on, including when they were switched on from the dashboard, so the replacement keeps the log line explaining why it was started.

Testing

  • New tests in window::ui_hang_watchdog_tests: a message-only window on a thread that pumps messages and one on a thread that blocks without pumping, which the probe tells apart; a destroyed window left to the surface watchdog; the doubling of the delay and its cap; the streak resetting after ten minutes. I checked that the first test fails when the probe is short-circuited.
  • Manual, on a release build including this commit, by suspending the UI thread of the running widget with SuspendThread:
    • single freeze: relaunched after 32 to 34 seconds, one instance left, watchdog: UI thread stopped answering for 30s -> relaunching kept in the replacement's log;
    • second freeze right after that relaunch: relaunched after a little over 60 seconds, as expected from the doubling;
    • widget started without --diagnose, diagnostics switched on through WM_APP_ENABLE_DIAGNOSTICS, then frozen: the replacement started with --diagnose --diagnose-append and kept the log.
  • cargo fmt --all -- --check: clean. cargo build --locked --release: OK. cargo clippy --all-targets: no new warnings. cargo test --locked: 534 passed.
  • Not covered: the debugger guard.
  • Unrelated heads-up: poller::retry_after::tests::oversized_headers_store_bounded_cooldowns_and_requests_resume_after_expiry computes Instant::now() - Duration::from_secs(86_400). It panicked with "overflow when subtracting duration from instant" on main while my machine had been up for 19 hours, and passed once uptime went past 24 hours.

Checklist

  • I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md.
  • The change is focused and contains no credentials or sensitive data.
  • I have added or updated relevant documentation and tests, where applicable.
  • I have left changelog and version changes to maintainers unless requested.

A UI thread parked in a call that never returns keeps every window handle
valid and every parent alive, so the surface watchdog sees nothing wrong
while the widget stops drawing and stops polling. It was seen staying
frozen for well over an hour, through an explorer.exe restart, with
nothing in the log to show for it.

A dedicated thread now asks the UI thread to acknowledge an empty message
every couple of seconds and relaunches the widget once it has stayed
silent for 30 seconds and at least 10 probes in a row, so a system sleep
that begins while a probe is failing cannot trigger it on wake. It never
takes the state mutex, since a UI thread wedged while holding that lock is
the case it has to survive. It does not rely on IsHungAppWindow either:
that stayed false while the UI thread of a taskbar-embedded widget was
suspended, and only turned true once a frozen widget had been orphaned.
The probe stands down while a debugger is attached.

A replacement that freezes again within ten minutes waits twice as long
before the next attempt, up to eight minutes, instead of being replaced
every half minute for as long as the cause lasts. The frozen process ends
with TerminateProcess rather than ExitProcess, as Microsoft recommends
when the state of a process's threads is unknown: DLL detach code can
deadlock on a lock the wedged thread held.

Two watchdogs can now reach the relaunch, so the first one to commit
blocks the other from spawning a second replacement. A relaunch also
carries --diagnose and --diagnose-append over whenever diagnostics are on,
including when they were switched on from the dashboard, so the
replacement keeps the log entry explaining why it was started.
CodeZeno added a commit that referenced this pull request Oct 19, 2026
Integrate PR #168 UI hang recovery, avoid log mutex waits, and reset hang timing after sleep.
@CodeZeno
CodeZeno merged commit a582e74 into CodeZeno:main Oct 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants