Skip to content

docs: add a debugging recipe for localizing a deferred GPU KernelException#18

Draft
haakon-e wants to merge 1 commit into
mainfrom
he/gpu-exception-localization
Draft

docs: add a debugging recipe for localizing a deferred GPU KernelException#18
haakon-e wants to merge 1 commit into
mainfrom
he/gpu-exception-localization

Conversation

@haakon-e

@haakon-e haakon-e commented Jul 9, 2026

Copy link
Copy Markdown
Member

What

Adds a recipe to docs/dev-guides/workflow/debugging.md for localizing a device-side throw (a DomainError from an out-of-domain sqrt/log/x^y, a bounds error, a SingularException) that surfaces on the host as a generic CUDA.KernelException with no kernel name and no device line. A row in the "silently-wrong values" lookup table points to the new section.

Why

The exception is deferred: CUDA detects it at the next synchronization, so the host stacktrace names that synchronization point, not the throw, and the named line often contains no transcendental operation at all. Compiling with device line info (-g2) is not an option here because ptxas rejects the non-ASCII identifiers common in CliMA kernels. The section documents the practical alternative: force a synchronous throw with CUDA_LAUNCH_BLOCKING=1, read the reported line correctly, bisect with ClimaComms.@sync barriers paired with @info markers, restart from a crash-proximate checkpoint to keep iterations short, and confirm the candidate site with a floor or clamp.

Scope

Documentation only; no source changes. The page lives under the shared docs/dev-guides/, so the recipe is kept repo-agnostic. No docs build is required (CI covers it).

…ption

Document CUDA_LAUNCH_BLOCKING plus sync-barrier bisection for a device-side
throw that surfaces as a CUDA.KernelException with no kernel or line.
@haakon-e
haakon-e force-pushed the he/gpu-exception-localization branch from b248242 to 0800787 Compare July 16, 2026 00:31
@haakon-e
haakon-e requested a review from dennisYatunin July 16, 2026 00:31
@haakon-e

Copy link
Copy Markdown
Member Author

@dennisYatunin could you have a quick look here if these suggestions seem sufficiently generic and useful in your view? As always, things can be improved and expanded upon in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant