diff --git a/.claude/skills/browser-javascript-profiling/SKILL.md b/.claude/skills/browser-javascript-profiling/SKILL.md new file mode 100644 index 0000000..72dd39c --- /dev/null +++ b/.claude/skills/browser-javascript-profiling/SKILL.md @@ -0,0 +1,243 @@ +# Browser JavaScript Performance Profiling with Chrome DevTools + +Use this skill when the subject is JavaScript running in a browser page: page-load cost, interaction latency, rendering jank, long tasks, layout/paint work, or objects retained by a page. The primary reference is the [Chrome DevTools Performance features reference](https://developer.chrome.com/docs/devtools/performance/reference). + +This is **browser-page profiling**, not Node.js process profiling. Node.js flags include `node --cpu-prof`, `node --heap-prof`, `--trace-gc`, V8 heap APIs, and server-side tools. Do not use those Node flags as a substitute for recording a page: they profile a Node process and its isolate. A Node `.cpuprofile` or `.heapprofile` can be opened in DevTools, but it is not a browser Performance trace and does not include the page's renderer, network, layout, paint, or user interaction. Use the Performance and Memory panels below for a browser page. + +## Choose the panel for the question + +| Question | First workflow | Main evidence | +| --- | --- | --- | +| Why is an interaction or animation janky? | Performance panel, runtime recording | Main-thread tasks, frames, interactions, scripting, style/layout, paint, and user-timing markers | +| Why is initial load slow? | Performance panel, **Record and reload** | Network request timing, parse/compile, script execution, rendering milestones, and screenshots | +| Which objects remain alive after teardown? | Memory panel, heap snapshots | Snapshot comparison, retained size, dominators, and retainer paths to GC roots | +| What allocates repeatedly during a scenario? | Memory panel, **Allocation instrumentation on timeline** | Allocation intervals, allocation stacks, and objects that survive later collections | +| Is the issue caused by a slow service or transport? | Performance panel plus Network track/conditions | Request queueing, connection, response, and page work; corroborate with server and field data | + +## Establish a repeatable capture + +1. **State the hypothesis and observable.** Record whether the target is load time, interaction latency, frame drops, CPU self-time, retained heap, allocation rate, or a particular user-timing measure. Define the exact URL, viewport, device emulation, input sequence, data set, and success criterion. +2. **Use the right build.** Profile the same production-like bundle, feature flags, and source-map revision that users receive. Development builds add assertions, logging, unminified code, and different scheduling behavior. Keep the browser/OS/Chrome version, viewport, zoom, display scale, and extension set constant. +3. **Control state.** Decide whether the test is cold-cache or warm-cache and use that state consistently. Use a test account and deterministic fixtures where possible. Close unrelated tabs and background work, but do not silently change the workload between runs. +4. **Prepare capture settings.** Open **Performance** and inspect **Capture settings**. Choose CPU and Network throttling deliberately; write the exact presets down. Enable screenshots only when visual correlation is needed. Keep JavaScript samples enabled when function-level attribution matters; disable them only to reduce trace size when broad browser/renderer activity is sufficient. +5. **Warm up, then capture only the scenario.** Let one run settle if the page has startup compilation or cache effects. Clear the old recording, start at a known state, perform the same actions, stop promptly, and repeat several times. A long trace containing unrelated idle time is harder to interpret and consumes more memory. +6. **Compare like with like.** Keep an unprofiled run and profiled runs under the same conditions. Profiling, screenshots, throttling, DevTools itself, and forced garbage collection can change scheduling and memory behavior. Treat a single trace as a lead, not a regression result. + +For a load investigation, use **Record and reload** so the navigation is included in the trace. For a runtime investigation, use **Record**, then perform only the interaction under test. Save a trace only after confirming that its selected interval and settings represent the intended scenario. + +### Agent-executable capture with the Browser tool + +The DevTools panel instructions above are useful for a person, but an agent cannot open DevTools inside the Browser tool's page tab. Do not stop after telling the user to record a trace manually. When the page can be driven by the Browser tool, use a Chrome DevTools Protocol (CDP) session from Puppeteer's `page.createCDPSession()` and capture the scenario directly. + +1. Open the target with the Browser tool and reach the deterministic starting state. +2. In one `browser.run` call, create a CDP session, start `Tracing`, perform the interaction, stop tracing, wait for `Tracing.tracingComplete`, and drain its `IO` stream. +3. Keep the trace in memory unless the user explicitly needs an artifact. Saved traces can contain sensitive data. +4. Report the browser version, URL/build, viewport, cache state, throttling, exact action, and whether `dataLossOccurred` was false. + +Use this tested `browser.run` body as the starting point. Replace only the marked scenario and the analysis needed for the hypothesis: + +```js +const client = await page.createCDPSession(); +let tracingStarted = false; + +try { + const browserVersion = await client.send('Browser.getVersion'); + + await client.send('Tracing.start', { + transferMode: 'ReturnAsStream', + streamFormat: 'json', + traceConfig: { + recordMode: 'recordAsMuchAsPossible', + enableSampling: true, + includedCategories: [ + 'devtools.timeline', + 'v8.execute', + 'blink.user_timing', + 'disabled-by-default-devtools.timeline', + ], + }, + }); + tracingStarted = true; + + // Scenario: replace this selector/action and keep the capture bounded. + await page.evaluate(() => document.querySelector('#work').click()); + await new Promise(resolve => setTimeout(resolve, 100)); + + // Subscribe before Tracing.end so the completion event cannot be missed. + const completed = new Promise(resolve => { + client.once('Tracing.tracingComplete', resolve); + }); + await client.send('Tracing.end'); + tracingStarted = false; + + const completion = await completed; + if (completion.dataLossOccurred) { + throw new Error('Trace buffer lost data; repeat with a shorter scenario'); + } + if (!completion.stream) { + throw new Error('Tracing completed without a result stream'); + } + + let traceJson = ''; + for (;;) { + const chunk = await client.send('IO.read', { + handle: completion.stream, + }); + traceJson += chunk.data; + if (chunk.eof) break; + } + await client.send('IO.close', {handle: completion.stream}); + + const {traceEvents} = JSON.parse(traceJson); + const completeEvents = traceEvents.filter( + event => event.ph === 'X' && event.dur, + ); + const totalMs = name => + completeEvents + .filter(event => event.name === name) + .reduce((sum, event) => sum + event.dur / 1000, 0); + + const evidence = { + browser: browserVersion.product, + traceEventCount: traceEvents.length, + eventDispatchMs: totalMs('EventDispatch'), + layoutMs: totalMs('Layout'), + styleMs: totalMs('UpdateLayoutTree'), + gcMs: totalMs('MinorGC') + totalMs('MajorGC'), + }; + display(evidence); + return evidence; +} finally { + // A failed action must not leave browser-global tracing active. + if (tracingStarted) { + try { + await client.send('Tracing.end'); + } catch {} + } + await client.detach(); +} +``` + +The totals above are a smoke-test summary, not a complete diagnosis. Trace events are nested, so never add `EventDispatch`, JavaScript, style, and layout totals into one “total CPU” number. For a real investigation, restrict analysis to the operation's User Timing interval, identify the renderer main thread, rank complete events by duration/self-time, and inspect the event arguments and stacks that support the hypothesis. Use `performance.getEntriesByName()` only to corroborate a marker duration; it cannot replace the trace because it has no layout, paint, network, task, or call-stack attribution. + +For **load profiling**, start tracing before `tab.goto(...)` or `page.goto(...)` in the same `browser.run` call, then wait for the agreed readiness condition before ending the trace. For **runtime profiling**, navigate and warm up first, then start tracing immediately before the action. Set cache and CPU/network emulation through CDP before capture when required, and repeat with identical settings. + +If `Tracing.start` reports that tracing is already active, an earlier run failed without cleanup. Close and kill the Browser tool tab/process, reopen it, and repeat with the `try`/`finally` pattern above. If `tab.click()` times out on an otherwise present element during a synthetic fixture, use the normal observed element handle or `page.evaluate()` only when programmatic dispatch is an acceptable representation of the scenario; trusted pointer/input timing requires the real click path. + +### Throttling and environment notes + +- CPU throttling is relative to the machine running Chrome. A `4x slowdown` is not a real replica of a particular phone's CPU architecture, thermal state, scheduler, or GPU. Calibrate a custom preset when appropriate, and report the preset with the result. +- Network throttling approximates latency, throughput, and request behavior; it does not reproduce a carrier radio, packet loss, server queue, CDN, DNS, or all service-worker/cache behavior. Record cache state and explain whether the server was local, staging, or remote. +- Use throttling to expose bottlenecks and compare alternatives, not to claim a universal user experience. Validate important conclusions with field data such as real-user Web Vitals, CrUX, or application telemetry. +- Keep browser device emulation, viewport, DPR, orientation, and input method fixed. A desktop renderer under emulation is still not the same hardware as a mobile device. + +## Add user-timing markers + +Use the browser User Timing API to label the business operation in the trace. Marks and measures appear in the **Timings** track and let the operation be correlated with Main-thread work, screenshots, and network activity. + +```js +performance.mark('search-start'); +await runSearch(); +performance.mark('search-end'); +performance.measure('search', 'search-start', 'search-end'); +``` + +For an event-driven flow, put the end mark in the callback or promise continuation that represents completion, not immediately after scheduling work. Use stable names plus an operation id when several instances overlap. Clear obsolete entries in a long-lived test page when appropriate (`performance.clearMarks()` / `performance.clearMeasures()`), and avoid logging sensitive input in marker names. A measure is elapsed page time; it does not by itself say how much time was CPU, network, rendering, or waiting. Inspect the corresponding trace interval. + +## Performance panel: recording and analysis + +### Record the smallest useful trace + +1. Open DevTools on the target page and select **Performance**. +2. Set Capture settings, CPU/Network conditions, and the screenshot/JavaScript-sample options before starting. +3. Choose **Record and reload** for load work, or **Record** for a runtime interaction. +4. Reproduce the scenario once it reaches the agreed starting state, then stop immediately. +5. In the overview, drag-select the interesting interval. Use the **Main**, **Network**, **Frames**, **Timings**, and interaction tracks to connect cause and symptom. Do not infer a page problem from an unselected unrelated part of the recording. +6. Select an event to inspect its Summary, source location, initiator, and related details. Use the Call tree or Bottom-up views to rank inclusive work and self-time, then verify the largest entries in the trace itself. + +Screenshots are useful for answering “which frame was late?” and connecting a layout/paint event to a visual change. They add capture overhead and size, so leave them off for CPU-only investigations unless the visual evidence is necessary. + +### Read the overview and flame chart + +The timeline is horizontal: width is elapsed time, and nested bars are work occurring within a parent event. A wide parent can contain several children; do not add parent and child durations as if they were independent. **Self-time** is work attributed to the selected function/event itself; **inclusive time** includes descendants. Sort or inspect both when deciding whether to optimize a leaf function or the caller that caused many descendants. + +Start with the overview's CPU activity and frame cadence, then zoom to the slow interaction or frame. On the Main track, look for: + +- **Long tasks and blocked frames.** A task over roughly 50 ms is a long task and can delay input, rendering, and other work. At 60 Hz a frame has about 16.7 ms, and at 120 Hz about 8.3 ms, before accounting for browser and compositor work. These are useful budgets, not guarantees that every task must fit exactly. Identify the task's top-level cause and its self-time rather than blaming the widest nested bar. +- **JavaScript and event handlers.** Repeated handlers, large synchronous loops, parsing/compilation, JSON work, and promise/microtask chains can monopolize the main thread. Correlate function names with the input event and User Timing mark; do not assume a function is slow merely because it appears often in a sampling view. +- **Style and layout.** Recalculate Style and Layout events indicate rendering work. Repeated DOM writes followed by layout reads can force synchronous layout (“layout thrashing”); inspect the stack and invalidated elements. A large layout may be legitimate for a large DOM, so compare the affected subtree and frequency, not only one duration. +- **Paint, raster, and compositing.** Paint and layer/compositor work can explain a visually late frame even when JavaScript is short. Use screenshots and the event details to distinguish expensive visual updates from main-thread scripting. The trace is not a complete GPU profiler; use a GPU-specific tool when GPU execution is the unresolved question. +- **Garbage collection.** GC events are pauses and evidence of allocation/collection activity, not automatically a leak. Frequent collections during an interaction suggest allocation churn or pressure; a large pause suggests collection cost. Compare heap after the operation has had time to settle. A baseline that rises after repeated equivalent cycles, especially after collection, is stronger leak evidence than one large temporary peak. +- **Network and scheduling.** A request's queueing, connection, wait, download, and subsequent parse/execute/render work are different costs. Follow the request to its initiator and then to the Main-thread work it unlocks. A slow request may be expected under the selected condition rather than a JavaScript defect. + +The flame chart is a time-attribution view, not a proof of causality. Use event initiators, stack traces, screenshots, marks, and repeated captures to establish the order of cause and effect. Sampling may omit very short calls; a trace with JavaScript samples disabled cannot answer function-level CPU questions. + +### Useful comparisons + +- Compare the same selected interval before and after a change, under identical settings, input, cache state, and warm-up. +- Compare a cold-cache load and a warm-cache load separately; do not average them into one unexplained number. +- Compare unthrottled and throttled recordings to learn whether the bottleneck is CPU-sensitive or network-sensitive, but keep the two claims separate. +- For a suspected interaction regression, use the interaction/INP details and the trace's event-to-paint sequence, not only total page-load time. + +## Memory panel: heap snapshots and allocation timelines + +Open **Memory** and choose the capture type that matches the question. Memory captures can pause the page and require substantial memory, so use a bounded test page or test account and avoid taking them casually in production. + +### Heap snapshots + +Use **Heap snapshot** for point-in-time ownership and retention: + +1. Take a baseline snapshot after the page reaches a known idle state. +2. Perform one bounded create/use/destroy cycle, or repeat the suspected operation a fixed number of times. +3. Return to the intended idle state and take a second snapshot. A third snapshot after another equivalent cycle is often more informative. +4. If the question is “what remains live?”, use DevTools' garbage-collection control between captures where available and record that intervention. Forced GC changes the workload; do not use it as evidence of normal user timing. +5. Use the comparison view to inspect new or growing constructors, then follow **retainers** toward GC roots. Sort by shallow size to find large objects and by retained size to find objects keeping subgraphs alive. + +A shallow size is the object's own storage; retained size estimates what would become collectible if the object were released. A retained path through a global, event listener, timer, closure, cache, or framework registry is a lead to the ownership bug. Detached DOM nodes are suspicious only when they persist unexpectedly across equivalent teardown cycles. Caches and pools can be intentional; confirm the lifecycle and an expected upper bound before calling growth a leak. + +A heap snapshot describes the JavaScript heap and related browser-visible objects at a point in time. It does not account for all native allocations, GPU memory, browser-process memory, network buffers, or another tab. Use process/OS telemetry and the appropriate native tool for those questions. + +### Allocation instrumentation on timeline + +Use **Allocation instrumentation on timeline** when the important question is *when* allocations occur and which call stacks produce them: + +1. Start the allocation recording at a known idle point. +2. Perform the same interaction or repeated cycle. +3. Stop recording promptly and select the interval containing the spike. +4. Inspect allocation stacks and the objects that survive later collections. Correlate a surviving group with a heap snapshot if ownership is unclear. + +A rising allocation rate can be harmless if objects are short-lived and collected efficiently. Conversely, a modest rate can leak if each cycle retains a reference. Distinguish temporary allocated bytes, live bytes, and retained bytes; they answer different questions. Allocation instrumentation has more overhead than a normal page run, so use it to explain a hypothesis and verify with a lighter repeat. + +## Source maps and readable attribution + +Enable JavaScript and CSS source maps in DevTools **Settings > Preferences > Sources** when the deployed assets have matching maps. Confirm that the map corresponds to the exact deployed bundle and release; a stale map can point at the wrong source line or hide the real chunk. With maps available, inspect original modules and source lines in the trace and heap allocation stacks. Without maps, minified bundle names and generated offsets are still evidence, but attribution must be checked against the build output. + +Source maps can expose original source, paths, comments, and internal names to anyone who can fetch them. Keep private maps behind the approved access policy, use a sanitized staging deployment for shared investigation, and do not attach a map to a profile or issue tracker without checking its contents. Maps do not make a profile deterministic: inlining, code splitting, async boundaries, and sampling still affect what is visible. + +## Artifacts, privacy, and production safety + +Treat saved Performance traces, heap snapshots, and allocation profiles as sensitive diagnostics. Depending on the page and capture settings they may contain: + +- URLs, query strings, origins, script paths, request/initiator metadata, and user-timing names; +- source locations and source-map content or identifiers; +- DOM text, object property strings, form data, cached application state, tokens, or other values reachable from the heap; +- console arguments, framework state, and names of users or records used in the scenario. + +Before saving or sharing, use synthetic data and a test account, remove or redact sensitive inputs where possible, restrict artifact permissions, and follow retention/deletion policy. Do not commit traces or heap snapshots to the repository. Avoid uploading them to an unapproved third-party viewer. If the artifact cannot be safely sanitized, keep it in the authorized local evidence store and share only derived timings or a screenshot with sensitive text removed. + +Do not profile a production session containing real user data merely because it is convenient. Prefer a representative staging build and a reproducible fixture. If production capture is explicitly approved, minimize scope and duration, disable unnecessary screenshots, protect the artifact, and document who may access it. Memory snapshots and forced GC can pause a tab and change behavior; they are particularly unsuitable for an unplanned live user session. + +## Limits of lab captures + +A DevTools trace is a controlled observation of one renderer, browser version, machine, workload, and set of emulated conditions. It is not a population-wide measurement. DevTools overhead, CPU/network emulation, warm caches, extensions, power/thermal state, browser scheduling, service workers, cross-origin frames, and server variance can all change the result. A lab trace is excellent for locating a causal chain and validating a fix under stated conditions; it cannot by itself establish field INP/LCP/CLS, battery impact, crash rates, or behavior on every device. Pair the trace with repeatable local measurements and real-user telemetry before making a production-wide claim. + +## Completion checklist + +- [ ] The question, URL/build, data fixture, viewport/device settings, cache state, Chrome version, and throttling presets are recorded. +- [ ] The capture type matches the question: runtime, load/reload, heap snapshot, or allocation timeline. +- [ ] The trace is bounded to the scenario and repeated under identical conditions. +- [ ] Main-thread self-time, long tasks, layout/paint work, GC, network dependencies, and user-timing marks were interpreted together. +- [ ] Memory conclusions are based on post-cycle retention and retainer paths, not a single heap peak. +- [ ] Source-map revision and access policy were checked before sharing attribution. +- [ ] Saved artifacts contain no unapproved secrets or personal data and are outside version control. +- [ ] The conclusion states the lab limitations and, where user impact matters, is checked against field data. diff --git a/.claude/skills/game-profiling/SKILL.md b/.claude/skills/game-profiling/SKILL.md new file mode 100644 index 0000000..d13817e --- /dev/null +++ b/.claude/skills/game-profiling/SKILL.md @@ -0,0 +1,118 @@ +--- +name: game-profiling +description: Profile Unity and Godot frame hitches, low frame rates, allocations, retained memory, rendering cost, and project-wide performance risks with reproducible captures. +--- + +# Game Development Performance Profiling Skill + +Use this skill when investigating frame hitches, low frame rate, memory growth, allocation spikes, rendering cost, or project-wide performance risks in Unity or Godot. Treat a profiler capture as evidence: preserve the workload and environment, separate measured facts from hypotheses, and recommend one change at a time so that a before/after capture can verify the result. + +## Start with a reproducible question + +Before opening a tool, ask for (or record): + +- engine and package versions, target platform/device, graphics API, build configuration, and commit/build identifier; +- target frame rate and frame budget (`1000 / target FPS` ms: 16.67 ms at 60 FPS, 33.33 ms at 30 FPS); +- exact scene, player input, camera, quality settings, resolution, VSync/frame cap, and a short step-by-step reproduction; +- whether the symptom is a sustained low rate, an intermittent hitch, a memory/GC increase, or a visual/rendering issue. + +Prefer a Development/Profiling player on the target hardware over the Editor. Keep the same quality, resolution, refresh rate, renderer, scripting backend, and asset data as the reported scenario. Warm up loading and shaders, then record a short steady-state window and repeat the scenario several times. Clarify whether a reported hitch duration is the total frame time, time over budget, or an added stall. For sustained workloads, compare like-for-like frame-time distributions. For event-triggered hitches, define a fixed event window and report the trigger frame or maximum frame per trial, the individual values, median, and sample count; report a tail percentile only when enough repeated events make it meaningful. + +If a project-specific MCP server is available, use it to gather engine context before manual inspection: use the Unity MCP if available for Unity project/editor state, assets, scenes, and profiler setup; use the Godot MCP if available for Godot project/editor state, scenes, scripts, and profiler setup. Treat MCP output as context for choosing and configuring the profiler, not as a replacement for the runtime captures below. + +## Choose the tool + +| Question | First tool | What it can establish | Important boundary | +| --- | --- | --- | --- | +| Which thread, system, script, physics step, or wait makes a frame slow? | [Unity Profiler](https://docs.unity3d.com/6000.1/Documentation/Manual/Profiler.html) | Runtime CPU/GPU frame timeline, hierarchy, markers, worker threads, and allocation samples | A marker's cost and a tool's instrumentation overhead are not the same as production cost; Editor data is not a device measurement | +| Where does memory go, what remains reachable, or what grows after repeated loads? | [Memory Profiler](https://docs.unity3d.com/Packages/com.unity.memoryprofiler@latest) | Point-in-time Unity memory snapshots and comparisons across managed/native and other supported categories | A snapshot is not a per-frame allocation trace; capture can pause or materially perturb the player, and support/details vary by platform/package version | +| Which draw calls and rendering events build a frame, and why are batches/pass count high? | [Frame Debugger](https://docs.unity3d.com/6000.1/Documentation/Manual/FrameDebugger.html) | Ordered rendering events, state changes, draw-call/batching decisions, and the visible result at an event | It explains *what* was submitted, not reliable GPU milliseconds, bandwidth, or shader occupancy; use a platform GPU tool when those are the question | +| Which project settings, assets, or serialized/static patterns are risky before runtime? | [Project Auditor](https://docs.unity3d.com/Packages/com.unity.project-auditor@1.0/manual/index.html) | Static rules, findings, severities, and project-wide configuration/asset/API checks | It cannot prove a runtime hotspot, actual device frame time, or that every finding matters to this game | +| Is a Godot script, process/physics step, frame metric, or renderer counter abnormal? | [Godot Profiler](https://docs.godotengine.org/en/stable/tutorials/scripting/debug/the_profiler.html) | Script function timings plus the Debugger Profiler and available frame/process/physics/rendering/memory monitors | Monitor names and availability vary by Godot version/renderer; script profiling does not account for all engine or GPU work | + +## Unity Profiler: CPU/frame timing and allocations + +Use the Unity Profiler first for a runtime symptom. + +1. Attach to or launch the same player/device and select the relevant modules (CPU Usage, GPU Usage where supported, Rendering, Memory, and related modules). Record the scenario after warm-up; do not profile an idle menu and generalize to gameplay. +2. In the CPU Usage timeline, find the long frame(s), then compare Main Thread, Render Thread, Jobs/worker threads, and waits. Switch between Timeline and Hierarchy: use Timeline to understand overlap/order and Hierarchy to aggregate self time and total time by marker. Follow a marker to its caller and the work it invokes rather than optimizing a broad parent label. +3. Compare the frame time with the target budget. A busy main thread over budget is a CPU bottleneck; a render/GPU path over budget points toward rendering or GPU work. `WaitForTargetFPS`, `Present`, or a similar wait may be a frame cap or back-pressure, not a slow game system. CPU and GPU can overlap, so do not add their times as if they were serial without evidence. +4. Look for repeated `GC.Alloc` in the exact workload and correlate it with managed-GC pauses. Distinguish a transient allocation from retained memory: the Profiler allocation sample says when managed allocation occurs, while Memory Profiler snapshots show what remains reachable. Confirm a suspected fix by running the same scenario and checking both allocation rate and hitch distribution. +5. Use markers, custom profiling samples, and a narrow time range to make a hypothesis testable. Deep Profile can expose call detail for a small reproduction, but it changes timings and can create substantial overhead; turn it off for the baseline and final validation. + +The Profiler may show Editor overhead, synchronization waits, profiler transport cost, or a different graphics path from a player. On-device captures and a non-profiled confirmation build are required before treating a small regression as real. + +## Unity Memory Profiler: snapshots and allocation investigations + +Use the [Memory Profiler](https://docs.unity3d.com/Packages/com.unity.memoryprofiler@latest) when memory grows, a load/unload cycle leaks, an out-of-memory failure is suspected, or asset duplication is expensive. + +1. Establish a repeatable lifecycle: capture after a clean boot, after the first load, after the target scene/action, and after the same unload/reload cycle. Keep the player state and cache/warm-up policy consistent. Name captures with build, device, scene, and lifecycle step. +2. Capture a baseline and a suspect snapshot, then compare them. Inspect managed objects, native Unity objects, textures/meshes/asset data, and other categories exposed by the package. For a candidate object, follow references and ownership to determine whether it is retained intentionally, held by a static/event/cache reference, duplicated by bundles, or simply still in a legitimate cache. +3. Pair snapshots with Unity Profiler `GC.Alloc` and Memory module evidence. A larger snapshot does not automatically mean a leak, and a low managed heap does not rule out native or graphics memory pressure. Verify that a repeated cycle returns to a stable range rather than relying on one before/after difference. +4. After changing load/unload, pooling, asset import, or references, repeat the same lifecycle and compare snapshots again. Keep the largest captures outside source control unless the project explicitly manages them; they can contain sensitive scene names, paths, and large binary data. + +Snapshot capture can pause the player, consume substantial memory/storage, and alter timing. Do not use a snapshot's pause duration as the game's hitch time. Check package/platform support and compare captures made under the same Unity and Memory Profiler versions. + +## Unity Frame Debugger: rendering passes and draw-call decisions + +Use the [Frame Debugger](https://docs.unity3d.com/6000.1/Documentation/Manual/FrameDebugger.html) after runtime evidence points at rendering, excessive draw calls, overdraw, state changes, or an unexpected pass. + +1. Reproduce one deterministic view (camera, resolution, quality, lights, and frame index). Pause on the relevant frame and open the Frame Debugger event list. Step through the ordered events from clears and shadow/depth work through opaque/transparent objects, post-processing, UI, and final presentation as applicable. +2. For a suspicious event, inspect the source renderer/material/shader, pass, render target, keywords, state changes, and batching/instancing decision. Identify why a batch split occurs (for example, material/state/mesh/light/shadow differences) instead of assuming that reducing object count alone will help. +3. Record the event/draw/pass pattern and the visual change at the event. Test a targeted change (material sharing, batching/instancing eligibility, culling, resolution, shader/pass selection, or an unnecessary effect), then capture the same view again. A smaller event list is useful evidence, but verify frame time on the target GPU. +4. Use a GPU timing/counter tool appropriate to the platform when the question is GPU milliseconds, bandwidth, shader occupancy, tiler behavior, or thermal throttling. The Frame Debugger is a submission/event inspection tool, not a substitute for hardware GPU timing. + +Pausing, attaching, and displaying events can change resource lifetime and timing. Editor rendering, Game view scaling, and debug visualization can differ from a player. Keep the camera and frame selection fixed; never compare unrelated frames just because they contain the same object. + +## Unity Project Auditor: project/static analysis + +Use [Project Auditor](https://docs.unity3d.com/Packages/com.unity.project-auditor@1.0/manual/index.html) for a project-wide preflight, regression check, or when runtime profiling suggests a broad configuration/asset issue. + +1. Run the Auditor against the same project revision and package configuration used for the build. Start with all relevant analyzers/categories, then filter by severity, platform, category, and location to make the report actionable. +2. Triage each finding: understand the rule and affected asset/setting/API, verify that it applies to the target platform and shipping configuration, and record a minimal remediation. High severity is a prioritization signal, not proof of an observed frame cost. +3. Fix one class of issue at a time, rerun the audit, and keep the report with the commit/build metadata. If a finding is expected, document the reason and ensure it does not hide a newly introduced finding. Validate performance-related fixes with a Unity Profiler or device test; validate memory/rendering fixes with the corresponding runtime tool. + +Auditor is static analysis: it can find risky import settings, serialized data, project configuration, and API/usage patterns without running gameplay, but it cannot see runtime frequency, actual frame overlap, dynamic content, or hardware-specific GPU behavior. Rules and package output can change with Unity/package versions. + +## Godot Profiler: script timing and frame monitors + +Use the [Godot Profiler](https://docs.godotengine.org/en/stable/tutorials/scripting/debug/the_profiler.html) for a Godot runtime symptom. + +1. Run the target scene or build with the same renderer, resolution, VSync/frame cap, quality, and device as the report. In the Debugger, record a short, repeatable scenario with the Profiler, then stop recording before inspecting it. Use the function list/call tree to compare self time (work in that function) with total time (including called functions), and separate script process work from physics/process work where the version exposes it. +2. Check the Profiler's frame monitors during the same window. Correlate frame time/FPS with available `Process`, `Physics Process`, rendering counters such as draw calls, objects, and primitives, pipeline-compilation counters where available, and memory/video-memory counters. Preserve the exact labels because monitor sets differ by Godot release and renderer. Record each monitor's observed update cadence: values sampled every frame may repeat a slower-updating value and are not independent samples, so do not derive frame-time percentiles from duplicated monitor readings. Use Profiler frame samples or direct frame timestamps for per-frame distributions. Rendering counters show that submitted work changed; they are not GPU timings, bandwidth, occupancy, or proof of a GPU bottleneck. +3. For a rendering symptom, make a controlled A/B capture in one warmed build: keep the camera, scene, resolution/render scale, quality, VSync/frame cap, renderer/API, and effect state deterministic, then compare fixed windows with only the suspect effect disabled versus enabled. If script/process evidence does not explain the frame-time change, record the target OS, GPU, renderer, and graphics API before selecting a compatible platform GPU tool. Preserve the existing renderer/API rather than switching APIs for capture. +4. Use that GPU tool to collect supported GPU timestamps or hardware counters when the question is GPU milliseconds, fill/fragment cost, bandwidth, or occupancy. A frame debugger's pass/event list and draw count remain submission evidence, not GPU-cost evidence. Repeat intermittent hitches; do not conclude from first-use shader or pipeline compilation, and do not micro-optimize scripts merely because total frame time is high. +5. Confirm a suspected change with the game's own frame metrics in a non-profiled run. The Godot script Profiler alone cannot identify every render pass or account for all engine and GPU work. + +Profiler recording and remote inspection add overhead and can perturb short frames. The Editor and a running export may differ. Treat `await`/idle, draw, physics, and synchronization time according to the version's labels, and preserve the exact Godot version, renderer, monitor names, and capture settings with the result. + +## Capture hygiene and reproducibility + +- Prefer a player/export on target hardware. If only the Editor is available, label it explicitly and do not present it as device evidence. +- Keep VSync, frame caps, refresh rate, resolution, dynamic resolution, quality, power mode, thermal state, and background apps consistent. Record when a cap is intentionally enabled. +- Warm up shaders, caches, scene streaming, and asset loading. Exclude startup/loading frames when investigating steady-state gameplay, but capture them separately for load-time problems. +- Record a short named interval around the symptom, repeat at least three times when practical, and retain the raw capture/snapshot plus metadata (engine/package versions, commit, build type, device/OS, renderer/API, scene, action, and settings). +- Give automated captures both a deterministic frame/event limit and a wall-clock timeout. Fail loudly and label the capture incomplete if either limit is exceeded; never let a benchmark or unattended player run indefinitely. +- Disable unrelated Editor windows, gizmos, logging, breakpoints, and visual debug overlays. Avoid Deep Profile or extra monitors for the baseline; use them only in a targeted diagnostic capture. +- Never infer a leak from one larger snapshot, a GPU bottleneck from draw-call count alone, or a CPU bottleneck from a single frame. Compare like-for-like captures and validate the proposed fix in a non-profiled run. +- Keep captures reproducible and safe to share: scrub sensitive paths/project names where required, avoid committing large binary snapshots by default, and do not alter production settings solely to make a profiler capture look better. + +## Practical triage for Claude + +1. Ask whether the report is CPU/frame timing, allocation/memory, rendering submission/passes, static project risk, or Godot frame-monitor behavior; select the narrowest first tool. +2. Request the target build/device and a capture around the exact reproduction. If no capture exists, give the user a deterministic capture recipe rather than guessing from code or object counts. +3. Translate observations into a bounded hypothesis: e.g. “Main Thread exceeds 16.67 ms during the spawn step,” “managed allocations recur every frame,” “a transparent effect adds repeated passes,” “native texture memory remains after reload,” or “the Auditor flags platform-inappropriate import settings.” +4. Recommend one change, then repeat the same capture and report the measured delta, regressions, and remaining budget. Keep static findings separate from runtime proof and keep Frame Debugger event counts separate from GPU timing. + +## Repository links + +- [Unity Profiler](https://docs.unity3d.com/6000.1/Documentation/Manual/Profiler.html) +- [Memory Profiler](https://docs.unity3d.com/Packages/com.unity.memoryprofiler@latest) +- [Frame Debugger](https://docs.unity3d.com/6000.1/Documentation/Manual/FrameDebugger.html) +- [Project Auditor](https://docs.unity3d.com/Packages/com.unity.project-auditor@1.0/manual/index.html) +- [Getting started with Unity Project Auditor video](https://www.youtube.com/watch?v=8QQG0J624LY) +- [Ultimate Guide to Profiling Unity Games](https://unity.com/resources/ultimate-guide-to-profiling-unity-games) +- [Unity Profiler Walkthrough & Tutorial](https://www.youtube.com/watch?v=xjsqv8nj0cw) +- [Memory Profiler Walkthrough & Tutorial](https://www.youtube.com/watch?v=Uuzd39AjFWQ) +- [Godot Profiler](https://docs.godotengine.org/en/stable/tutorials/scripting/debug/the_profiler.html) +- [Locate Performance Issues Using Godot's Profiler](https://www.youtube.com/watch?v=eafdNo1kMA8) diff --git a/.claude/skills/gpu-profiling/SKILL.md b/.claude/skills/gpu-profiling/SKILL.md new file mode 100644 index 0000000..721e704 --- /dev/null +++ b/.claude/skills/gpu-profiling/SKILL.md @@ -0,0 +1,78 @@ +# GPU Profiling Tools Skill + +Use this skill when the question involves GPU frame time, GPU/CPU overlap, graphics API behavior, kernel throughput, GPU counters, mobile graphics, heterogeneous CPU/GPU execution, or power/thermal effects. Start by identifying the target device, graphics/compute API, and the question being asked; do not select a profiler solely because it is installed. + +This skill covers the seven tools listed in this repository's GPU profiling section: + +- [NVIDIA Visual Profiler](https://developer.nvidia.com/nvidia-visual-profiler) +- [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems) +- [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute) +- [Android GPU Inspector (AGI)](https://developer.android.com/agi) +- [Metal debugger](https://developer.apple.com/documentation/xcode/metal-debugger) +- [AMD uProf](https://www.amd.com/en/developer/uprof.html) +- [Intel VTune Profiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html) + +The commands below are examples for an installed tool; adapt executable paths, options, target API, and result names to the installed version. Claude must not claim to have run any command unless it actually did so. + +## Choose the measurement first + +| Question | Start with | Why | Do not infer from it alone | +|---|---|---|---| +| Is a frame slow, stuttering, or serialized? | A system/timeline profile | Shows CPU threads, graphics/compute queues, API calls, synchronization, transfers, and idle gaps on one time axis | A busy GPU does not prove that a particular shader is the cause | +| Which CUDA/HIP/OpenCL/SYCL kernel is slow? | Kernel or GPU-hotspot analysis | Ranks kernels and exposes throughput, stalls, memory, occupancy, and source correlation | A kernel percentage is not the same as frame contribution; account for invocation count and overlap | +| Which draw, pass, resource, or shader caused a rendering result? | A graphics API frame capture | Replays a bounded frame and exposes command order, attachments, resources, pipeline state, and shader data | Capture timing is representative of the shipping workload; capture can perturb it substantially | +| Is the Android game CPU-, GPU-, memory-, battery-, or driver-bound? | AGI system profile, then AGI frame profile | System trace establishes cross-device context; frame trace narrows to Vulkan/OpenGL ES work | A counter from one Android GPU is not necessarily comparable to a counter with the same name on another GPU | +| Is the workload limited by AMD or Intel CPU/GPU hardware? | AMD uProf or Intel VTune, respectively | Their sampling, hardware-event, accelerator, and power analyses match the vendor's PMU and driver | Vendor metrics and derived percentages do not transfer directly across architectures | +| Is the issue specifically Metal command behavior? | Xcode Metal debugger | It can capture/replay Metal workloads and inspect dependencies, counters, resources, and shaders | A GPU trace is not a substitute for measuring an uninstrumented run | + +### Timeline/system profiling versus kernel counters + +Keep these analyses separate: + +- **Timeline/system profiling** answers *when* work happened and *what was waiting*. Look for CPU submission gaps, queue bubbles, synchronization, copies, page faults, thermal throttling, context switches, and frame-to-frame variance. It is the first pass for game stutter and end-to-end latency. +- **Kernel counters** answer *why a selected GPU program is inefficient*. Look for arithmetic or memory throughput, occupancy, cache behavior, wave/warp stalls, instruction mix, and source-line correlation. They are a second pass after the timeline proves that the kernel matters. +- **Graphics API captures** answer *which commands and state produced a frame*. They are bounded replay/debug captures, not low-overhead production measurements. + +A GPU can show high utilization while the frame is still slow because of poor scheduling, a long queue, copies, or synchronization. A GPU can show low utilization because the CPU is late, the workload is too small, the queue is blocked, or the application is waiting on presentation. Correlate CPU submission time, GPU execution time, queue wait time, and frame percentile data before naming a bottleneck. + +## A repeatable workflow + +1. **Define the acceptance metric.** Record the unprofiled baseline: frame-time percentiles (not only average FPS), CPU and GPU frame duration, kernel duration or throughput, and power/temperature if relevant. +2. **Describe the target.** Record GPU model and memory, driver/firmware, OS and device build, API (CUDA, HIP, Vulkan, OpenGL ES, Metal, DirectX, SYCL/OpenCL), display mode, resolution, refresh/vsync, compiler flags, and profiler version. +3. **Make the workload deterministic.** Use a fixed scene/input, seed, asset set, camera, resolution, workload size, and run duration. Warm up shader compilation, caches, asset streaming, and the runtime before capture. Use the same power/performance mode and background-load policy. +4. **Capture a narrow interval.** Prefer a representative 2–20 second timeline or one to a few representative frames. Use API markers (for example, NVTX or application labels) and explicit frame boundaries where the tool supports them. Avoid capturing the whole game session by default. +5. **Start broad, then narrow.** Use a timeline/system view to locate the bad frame, queue, or phase. Then profile only the relevant kernel, range, pass, or counter set. Do not start with an exhaustive counter set on every kernel. +6. **Repeat the baseline and capture.** Run at least three comparable unprofiled and profiled trials when the effect is noisy. Treat a profiler-induced change as overhead until it survives a lighter collection and an unprofiled measurement. +7. **Compare like with like.** Keep tool/driver/device, metric set, clock mode, input, and warm-up identical. A derived metric, percentage of peak, or occupancy number must be interpreted against the exact architecture and collection mode. +8. **Record evidence.** Preserve the report/trace, command line or GUI settings, application commit, symbols/shader binaries, tool version, and a short note describing the selected interval and expected result. + +## Additional resources + +Reference the supporting file that matches the target hardware, API, and question so Claude knows when to load detailed workflows: + +- [nvidia.md](nvidia.md): NVIDIA Visual Profiler legacy CUDA workflow, Nsight Systems timelines, and Nsight Compute kernel analysis. +- [android.md](android.md): Android GPU Inspector system and frame profiling for Android Vulkan/OpenGL ES workloads. +- [metal.md](metal.md): Xcode Metal debugger capture/replay and `gpudebug` inspection for Metal workloads. +- [amd.md](amd.md): AMD uProf CPU, threading, power, and supported AMD accelerator profiling. +- [intel.md](intel.md): Intel VTune CPU hotspots, threading, microarchitecture, GPU offload, and GPU compute/media analysis. + +## Graphics API capture guidance + +A graphics API capture is a different artifact from a system timeline or kernel-counter report: + +- **Metal debugger:** use Xcode GPU Frame Capture or `MTLCaptureManager` to capture Metal command buffers, then inspect/replay the `.gputrace` in Xcode or `gpudebug`. +- **AGI:** use Frame Profiler and choose **Vulkan** or **OpenGL on ANGLE** exactly as the app uses it. Use System Profiler alongside it to verify that the selected frame is representative on the Android device. +- **Nsight Systems:** trace supported NVIDIA graphics APIs to correlate API submission and GPU workload with CPU/system activity. It is not a complete per-draw resource/pipeline debugger; move a confirmed graphics bottleneck to the platform's dedicated graphics debugger when draw-state inspection is required. +- **NVIDIA Nsight Compute, AMD uProf, and Intel VTune Profiler:** use them for CUDA/OptiX kernels, AMD supported GPU/CPU activity, or Intel GPU/CPU/offload analyses respectively. They do not replace a graphics frame capture when the question is attachment contents, draw ordering, resource binding, or shader debugging. +- **NVIDIA Visual Profiler:** treat its CUDA API/kernel timeline as a legacy compute trace only; it is not a modern cross-API graphics capture tool. + +For every frame capture, record API, device, OS/driver, app build, capture scope/count, debug-layer or validation-layer state, and whether replay occurred on the original device. Never compare a captured frame's wall time directly to a production frame without an uninstrumented check. + +## Overhead, permissions, and safety + +- Sampling, tracing, hardware counters, replay, shader instrumentation, validation layers, and frame capture have different overhead. State which one is enabled; do not call all profilers “low overhead.” +- Counters may be privileged or unavailable under a hypervisor/container. Ask for the least privilege that enables the chosen analysis; record when root/administrator/debugfs/driver permissions were used. +- Never leave Android global GPU-debug settings, Metal capture enablement, debug layers, or elevated profiling services enabled in a release/test device after the experiment. Clean up settings and terminate sessions. +- Traces can contain source paths, symbol names, shader source/binaries, textures, frame contents, process arguments, device identifiers, and application data. Store them securely and redact before sharing. +- A profiler result is an observation under a collection configuration. Distinguish “the queue was idle while the CPU waited” from “this function caused the wait”; establish causality by changing the suspected work and checking the resulting unprofiled metric. +- When reporting a result, include the exact command or GUI settings, target and API, tool/driver versions, permissions, warm-up policy, capture interval, metric set, number of trials, and the baseline/profiling overhead. diff --git a/.claude/skills/gpu-profiling/amd.md b/.claude/skills/gpu-profiling/amd.md new file mode 100644 index 0000000..769ac13 --- /dev/null +++ b/.claude/skills/gpu-profiling/amd.md @@ -0,0 +1,31 @@ +# AMD GPU and CPU/GPU profiling + +## AMD uProf + +**Use when:** the target is an AMD Zen-based CPU or AMD Instinct MI accelerator and the question spans CPU sampling, hardware events/IBS, thread and OS timelines, power/thermal behavior, or supported MI GPU/HIP/HSA activity. uProf is useful for heterogeneous application behavior and for connecting CPU-side submission or waiting to AMD hardware activity. + +**Target constraints:** current AMD documentation describes uProf for 64-bit x86 applications on Windows, Linux, and FreeBSD, with metrics for AMD Zen processors and AMD Instinct MI accelerators. The feature matrix is important: GUI and CLI coverage differs by OS, and GPU profiling/tracing is documented for Linux rather than Windows/FreeBSD. Check the installed release's system requirements and supported processor/accelerator before promising a metric. Hardware-event and IBS collection may require elevated privileges, kernel support, compatible firmware, or bare-metal access; virtualization can hide or multiplex counters. + +**CLI workflow (version-stable shape, version-specific configurations):** + +```bash +# Discover the configurations supported by this installed uProf build. +AMDuProfCLI info --list collect-configs + +# A CPU sampling collection; config names and options are release-specific. +AMDuProfCLI collect --config overview -o uprof-overview ./app [args] +AMDuProfCLI report -i uprof-overview/ +``` + +For a supported MI GPU workload, select the GPU configuration shown by `info --list collect-configs` and run the same `collect` form. Do not assume a `gpu` configuration or a counter set exists on every uProf release, OS, or accelerator. Use `AMDuProfCLI --help` and the [AMD uProf user guide](https://docs.amd.com/go/en-US/57368-uProf-user-guide) for exact options. + +**GUI workflow:** on supported Windows or Linux installations, select CPU **Hotspots**, **Threading**, **Microarchitecture**, **System**, **Power**, or the supported GPU analysis, configure a short interval and sampling/counter set, launch or attach to the target, then inspect hotspots, thread states, system timeline, source correlation, GPU dispatches, and power/thermal trends. Remote profiling can collect on a remote Linux system and translate/view on a Windows host when supported. + +**Interpretation:** + +- Sampling hotspots show where CPU time is observed; they are statistical and need enough samples before ranking close functions. +- Threading/OS traces reveal runnable, blocked, synchronization, I/O, and scheduling time; use them to explain why a GPU queue is starved or why CPU work misses a frame deadline. +- IBS/PMC metrics are AMD-architecture-specific. Explain cache/TLB/branch or memory findings with the event definition and collection mode, not with a generic counter name. +- MI GPU profiling/tracing can expose HIP/HSA API and GPU activity, kernels, dispatch, and hardware statistics. Validate that the selected process and accelerator are actually included before interpreting an empty GPU view. + +**Overhead and limits:** sampling is usually lower overhead than tracing, while OS/runtime tracing, call-stack stitching, IBS, and GPU counters increase cost and result size. Counter availability, call-stack quality, and GPU analysis depend on kernel/driver privileges and application symbols. Run a matching unprofiled baseline, keep sessions on local storage when possible, and do not treat a remote or virtualized session as equivalent to bare metal. diff --git a/.claude/skills/gpu-profiling/android.md b/.claude/skills/gpu-profiling/android.md new file mode 100644 index 0000000..d028294 --- /dev/null +++ b/.claude/skills/gpu-profiling/android.md @@ -0,0 +1,50 @@ +# Android GPU profiling + +## Android GPU Inspector (AGI) + +**Use when:** the target is a supported Android device and the problem involves mobile game frame pacing, Vulkan/OpenGL ES commands, CPU/GPU scheduling, memory, battery, thermal behavior, or vendor GPU counters. AGI's system profiler is a Perfetto-like system-wide view; its frame profiler captures one application frame for in-depth Vulkan or OpenGL-on-ANGLE analysis. + +**Target constraints:** the AGI quickstart requires a supported device running Android 11 or later, USB/ADB access, and a debuggable application (`android:debuggable="true"`). Supported GPU families and exact driver/device combinations are version-dependent; the AGI landing page calls out Qualcomm Adreno, Arm Mali, and Imagination PowerVR for system profiling. If using native Vulkan, enable validation layers as required by the installed AGI release and fix validation errors before interpreting performance data. The desktop host and device must meet the current AGI quickstart requirements. + +The current AGI quickstart notes that Android Performance Analyzer is the newer recommended choice for system profiling. Use AGI when its frame profiler, supported counters, existing traces, or graphics workflow is the requirement; do not present AGI system profiling as the only current Android system-tracing option. + +**GUI system-profile workflow:** + +1. Connect the device by USB, enable ADB/Install via USB, launch AGI, and select the `adb` executable. +2. Choose **Capture a new trace** and select **System profile**. Select only relevant CPU, GPU, memory, battery, and GPU counter options; use a short manual duration. +3. Start the trace, reproduce the same scene/input, stop after the chosen interval, and open the trace. +4. Correlate app/render threads, scheduling, GPU queue activity, memory pressure, thermal or battery events, and frame-time outliers. + +**GUI frame-profile workflow:** + +1. Choose **Capture a new trace** and select **Vulkan** or **OpenGL on ANGLE** to match the app's actual graphics path. +2. Select manual start and a short capture, choose the app/process (including a separate graphics process if the app uses one), start it, and press **Start** at a known frame. +3. Open the trace and inspect the frame's API calls, render passes, dependencies, GPU work, and counters. Use the system profile to check whether the captured frame is representative. + +AGI is primarily a GUI workflow; do not invent a command-line interface for frame capture. `adb` is used for device setup and diagnostics, not as a replacement for AGI's capture UI. If Vulkan validation layers must be forced for a debuggable app, the documented setup is analogous to: + +```bash +app_package= +abi=arm64v8a +adb shell settings put global enable_gpu_debug_layers 1 +adb shell settings put global gpu_debug_app "$app_package" +adb shell settings put global gpu_debug_layer_app "com.google.android.gapid.$abi" +adb shell settings put global gpu_debug_layers VK_LAYER_KHRONOS_validation +``` + +After profiling, remove the temporary global settings and restore the device's prior state: + +```bash +adb shell settings delete global enable_gpu_debug_layers +adb shell settings delete global gpu_debug_app +adb shell settings delete global gpu_debug_layers +adb shell settings delete global gpu_debug_layer_app +``` + +**Interpretation:** + +- System profile: use scheduling and queue timelines to separate CPU starvation, GPU saturation, memory bandwidth/pressure, synchronization, and thermal throttling. +- Frame profile: identify expensive passes and API submissions, then inspect dependencies and counters for the selected frame. API calls are evidence of recorded work, not automatically the root cause. +- GPU counter names and availability are vendor/driver-specific. Compare a change on the same device and driver; never compare an Adreno counter directly with a Mali or PowerVR counter. + +**Overhead and limits:** AGI instrumentation, Vulkan validation, frame capture, and counters can change CPU/GPU timing, memory usage, driver behavior, and thermal state. Validation is especially unsuitable for final frame-time claims. Captures may include app resources, shader data, and device identifiers. Keep traces short, rerun without instrumentation, and use a production-like non-debug build for final measurements while retaining a debuggable build for capture. diff --git a/.claude/skills/gpu-profiling/intel.md b/.claude/skills/gpu-profiling/intel.md new file mode 100644 index 0000000..88703c6 --- /dev/null +++ b/.claude/skills/gpu-profiling/intel.md @@ -0,0 +1,36 @@ +# Intel GPU and CPU/GPU profiling + +## Intel VTune Profiler + +**Use when:** the target uses an Intel CPU, Intel GPU, or supported Intel accelerator path and the question is CPU hotspots/microarchitecture, threading, memory, GPU offload, GPU compute/media hotspots, power, or CPU/GPU interaction. VTune can analyze SYCL, OpenCL, OpenMP offload, DirectX, and other supported application paths; select the analysis type that matches the actual API and hardware. + +**Target constraints:** VTune analyses and drivers are OS-, processor-, GPU-, and version-dependent. Current documentation covers Windows/Linux hosts and targets for the main workflows, with additional target support (including FreeBSD) for selected analyses. GPU hardware metrics may require administrator/root privileges, a sampling driver, i915/GPU setup, or an allowed debugfs/perf configuration. A missing driver or permission can produce an incomplete analysis rather than a zero-cost result; read collection diagnostics. + +**CLI examples:** + +```bash +# CPU user-mode sampling; use hardware sampling only when the platform permits it. +vtune -collect hotspots -result-dir vtune-cpu \ + -knob sampling-mode=sw -- ./app [args] + +# Intel GPU compute/media hotspots and GPU/CPU correlation. +vtune -collect gpu-hotspots -result-dir vtune-gpu \ + -knob enable-gpu-runtimes=true -- ./intel-gpu-app [args] + +# Produce a text summary; open the same result in vtune-gui for exploration. +vtune -report summary -result-dir vtune-gpu +vtune-gui vtune-gpu +``` + +Use `vtune -help collect gpu-hotspots` and `vtune -help collect hotspots` for current knobs. Characterization presets can require multiple runs and elevated privileges; source/basic-block or memory-latency modes can have much higher overhead than sampling. For reproducible automation, set a unique `-result-dir`, preserve the command and knobs, and use the same binary/symbols and device power state. + +**GUI workflow:** launch VTune GUI, choose the local or remote target, select **Hotspots**, **Microarchitecture Exploration**, **GPU Offload**, **GPU Compute/Media Hotspots**, **Threading**, or **System Overview**, configure the sampling/counter/characterization mode, collect a short interval, and inspect Summary, Timeline, Bottom-up/Top-down, GPU, and source views. Use the GUI to discover an analysis configuration, then export or reproduce it with the CLI when it will run in CI or on a remote host. + +**Interpretation:** + +- CPU hotspots are sampled execution observations, not a complete trace of every call. Use call stacks and source/assembly correlation to distinguish inclusive from exclusive cost. +- Microarchitecture views relate pipeline, cache, branch, memory, and bandwidth metrics to Intel CPU execution; verify that the metric set applies to the exact CPU generation. +- GPU offload analysis first answers whether host-to-device transfers, synchronization, or a kernel dominate. GPU Compute/Media Hotspots then explores Intel GPU utilization, EU occupancy/stalls, memory bandwidth, engine loading, and kernel source modes. +- GPU metrics such as EU active/stalled, occupancy, memory bandwidth, and GPU cycles describe Intel GPU hardware; they are not interchangeable with NVIDIA SM/warp or AMD CU/wave metrics. + +**Overhead and limits:** stack collection, hardware event sampling, GPU characterization, source/basic-block latency, and multiple-run presets alter runtime or require replay. Hardware event multiplexing and missing privileges can reduce precision. Use low-overhead hotspots/system analysis to locate the phase, then one focused GPU analysis. Confirm improvements with an unprofiled run and preserve the result database, symbols, tool version, driver, and analysis knobs. diff --git a/.claude/skills/gpu-profiling/metal.md b/.claude/skills/gpu-profiling/metal.md new file mode 100644 index 0000000..4ad2e2b --- /dev/null +++ b/.claude/skills/gpu-profiling/metal.md @@ -0,0 +1,33 @@ +# Metal GPU profiling + +## Metal debugger + +**Use when:** the target is an Apple Metal workload and the question is command order, render/compute pass dependencies, resources, attachments, pipeline state, shader behavior, GPU timeline, or Apple GPU counters. The Metal debugger is part of Xcode and supports GPU frame capture/replay; it is the graphics API capture workflow for Metal. + +**Xcode GUI capture:** + +1. In the Run scheme, open **Edit Scheme > Run > Options** and set **GPU Frame Capture** to **Metal** (or **Automatically** when appropriate). +2. Run the app, click the **Metal Capture** button in the debug bar, select a frame/command queue/device/custom capture scope and count, and click **Capture**. +3. Inspect the replayed GPU trace: command buffers, encoders/passes, dependencies, attachments, resources, pipeline state, shader source, timeline, and counters. Export with **File > Export** when a `.gputrace` artifact is needed. + +To capture a bounded workload from code, use `MTLCaptureManager` and `MTLCaptureDescriptor` around the desired command buffers. The capture destination can be a GPU trace (`.gputrace`) when supported. On macOS 14 and later, `MTL_CAPTURE_ENABLED=1` enables programmatic capture support; it has a small but measurable CPU effect, so keep it out of release builds. + +**CLI trace inspection:** current Xcode versions provide the interactive `gpudebug` command for GPU traces: + +```bash +gpudebug -t /path/to/Scene.gputrace +# At the gpudebug prompt, navigate and inspect a node: +# go commands/cb1/re0 +# info pipeline +# fetch color0 +``` + +The exact trace node paths depend on the capture; use `man gpudebug` and the tool's `list`/`go` commands rather than hard-coding paths. CLI replay is useful for scripted inspection and resource extraction, while Xcode is better for visual exploration and shader debugging. + +**Interpretation:** + +- Use the timeline to find long encoders, gaps, and pass dependencies; then inspect the command that owns the cost. +- Use counter statistics and per-line shader profiling only for the captured pass/command and the exact Apple GPU; counters are not portable across Apple GPU generations or to non-Apple GPUs. +- A frame capture can reveal incorrect attachments or state even when the performance symptom is elsewhere. Pair the capture with uninstrumented frame-time data. + +**Overhead and limits:** frame capture and replay can add CPU work, memory, disk, synchronization, and shader instrumentation effects. The debugger may replay on a different device than the one that produced the trace unless configured otherwise. Capture only a small reproducible scope, record device/OS/Xcode versions, and do not use a capture-enabled build as the performance baseline. diff --git a/.claude/skills/gpu-profiling/nvidia.md b/.claude/skills/gpu-profiling/nvidia.md new file mode 100644 index 0000000..fa87f3c --- /dev/null +++ b/.claude/skills/gpu-profiling/nvidia.md @@ -0,0 +1,93 @@ +# NVIDIA GPU profiling + +## NVIDIA Visual Profiler (legacy CUDA workflow) + +**Use when:** an existing CUDA project still depends on `nvprof`/`nvvp`, or an old `.nvprof` session must be inspected. The tool provides a GUI view of CUDA API calls, kernel launches, memory transfers, a CPU/GPU timeline, and configurable hardware metrics. + +**Important lifecycle constraint:** NVIDIA's current page says Visual Profiler was discontinued as of CUDA Toolkit 13.0. Prefer Nsight Systems for system/timeline analysis and Nsight Compute for kernel analysis on new work. Do not recommend installing a legacy profiler merely to investigate a new workload. CUDA 11 and later do not support developing or running applications on macOS; NVIDIA's macOS-host Visual Profiler was available only through CUDA 12.4 and macOS host packages were dropped in CUDA 12.5. Targets are CUDA-capable NVIDIA platforms supported by the corresponding legacy toolkit (commonly Linux or Windows; ARM support depends on that toolkit). + +**Minimal legacy collection (CLI, only if the installed toolkit provides it):** + +```bash +nvprof --export-profile=legacy.nvprof ./cuda-app [args] +nvvp legacy.nvprof +``` + +If option spelling differs in an older toolkit, use `nvprof --help`; do not silently substitute an Nsight report format. Open the resulting profile in the Visual Profiler GUI and inspect the unified CPU/GPU timeline before running metric experiments. + +**Interpretation:** + +- Start with host API launch and synchronization time, device kernel intervals, and host-to-device/device-to-host copies. +- A large gap between a host launch and device execution suggests submission, dependency, or synchronization issues; a long device interval makes the kernel a candidate for Nsight Compute. +- Use occupancy and memory/SM counters as architecture-specific evidence. Compare the same kernel and launch configuration across sessions, not raw counter values across GPU generations. +- Metric collection may replay kernels or serialize activity. A metric result can describe a replayed kernel and may not represent concurrent production behavior. + +**Overhead and limits:** tracing, instrumentation, metric experiments, and replay can materially change launch order, overlap, clocks, and wall time. Visual Profiler is not a modern Vulkan/DirectX/Metal frame debugger. Keep it for legacy CUDA evidence and migrate the question to Nsight Systems or Nsight Compute when possible. + +## NVIDIA Nsight Systems + +**Use when:** the question is end-to-end scheduling and overlap on an NVIDIA system: game frame stutter, CPU submission, CUDA/graphics queue occupancy, copies, synchronization, library calls, network/OS activity, or GPU metrics over time. It traces CUDA and can trace Vulkan, OpenGL, DirectX 11/12, DXR, and OptiX API activity; this is an API/system timeline, not a full draw-state debugger. + +**CLI collection:** + +```bash +# CUDA and application markers; keep the interval short and name the report. +nsys profile --trace=cuda,nvtx -o nsys-cuda-run ./cuda-app [args] + +# Choose the graphics API used by the target; do not enable unrelated APIs. +nsys profile --trace=vulkan,nvtx -o nsys-vulkan-run ./game [args] +# For another supported target, replace vulkan with opengl, dx11, or dx12. + +# Open the report in the GUI and optionally generate summary statistics. +nsys-ui nsys-cuda-run.nsys-rep +nsys stats nsys-cuda-run.nsys-rep +``` + +`nsys profile` starts and collects in one command. For an application with reliable markers, bound collection to a marker rather than tracing startup: + +```bash +nsys profile --trace=cuda,nvtx --capture-range=nvtx \ + --nvtx-capture=profile-range@* -o nsys-range ./app [args] +``` + +Check `nsys profile --help` for version-specific option names. GPU metric sampling and system-wide sampling can require root/administrator privileges or special driver permissions; request the smallest permission and metric set needed. + +**GUI workflow:** launch `nsys-ui`, configure the local or remote target, select only the needed CPU/GPU/API traces, start the app after the target is configured (especially for Vulkan), and inspect the unified timeline. Zoom to a bad frame or marker, correlate CPU thread submission with the graphics/compute queue, then inspect API calls, memory operations, synchronization, and GPU metrics. Use the CLI for repeatable runs and the GUI for interactive correlation. + +**Interpretation:** + +- A long CPU interval with no GPU work often means CPU submission, synchronization, or an application-side bottleneck. +- A queue gap after a launch can indicate dependencies, a late producer, or a host wait; verify the corresponding API and thread rather than blaming the next kernel. +- A continuously busy queue with high frame time means the workload is GPU-limited at that phase; hand the dominant kernel to Nsight Compute or use a graphics frame debugger for draw/pipeline state. +- GPU metrics are sampled over time and are not a replacement for per-kernel counters. Sampling frequency, metric set, and available hardware vary by GPU. + +**Overhead and limits:** Nsight Systems is designed for low-overhead system analysis, but tracing every API, backtrace, OS event, or high-frequency GPU metric increases overhead and report size. Disable CPU sampling or reduce sampling frequency for a GPU-only question. It does not provide the detailed resource/shader/pipeline replay of a graphics frame debugger, and it cannot turn a timeline into proof of a particular shader's cause. Keep CLI and GUI versions aligned; a newer report may not open correctly in an older GUI. + +## NVIDIA Nsight Compute + +**Use when:** the NVIDIA CUDA or OptiX timeline identifies a kernel worth optimizing and the question is kernel throughput, memory behavior, occupancy, warp stalls, instruction mix, source/SASS correlation, or roofline-style limits. It is not a Vulkan/OpenGL/DirectX frame debugger and should not be the first tool for unexplained game stutter. + +**CLI collection and GUI review:** + +```bash +# The basic set is a practical first pass; use a report name for comparison. +ncu -o ncu-basic ./cuda-app [args] + +# Limit collection to one named kernel when the application has many launches. +ncu --kernel-name regex:myKernel --launch-count 1 -o ncu-kernel ./cuda-app [args] + +# Open the report in the GUI (the report extension is produced by ncu). +ncu-ui ncu-basic.ncu-rep +``` + +Use `ncu --list-sets`, `ncu --list-sections`, and `ncu --query-metrics` before selecting architecture-specific sections or metrics. NVTX range filtering can restrict collection to a phase, but the exact filter syntax is version-specific; check the installed CLI help. For an interactive GUI run, select the application, kernel/range filters, and the smallest section set that answers the current question. + +**Interpretation:** + +- Begin with GPU throughput and the roofline/memory workload view. Distinguish arithmetic saturation from memory bandwidth, cache, latency, or launch-bound behavior. +- Inspect occupancy together with register/shared-memory usage and launch dimensions; low occupancy is a clue, not automatically a defect. +- Use warp-state/stall metrics to ask what the warp is waiting on (memory, dependency, barrier, instruction issue, or another cause); then correlate to source/PTX/SASS. +- Check total frame contribution: a fast kernel called thousands of times may dominate a slower kernel called once. +- Compare one kernel, input, launch configuration, metric set, and GPU architecture before attributing an improvement. + +**Overhead and limits:** Nsight Compute commonly replays a kernel multiple times to collect mutually exclusive hardware counters. It can serialize concurrent work, perturb caches and clocks, expose nondeterminism, or fail to make progress for mandatory concurrent kernels. Limit launches, use deterministic input, avoid profiling a whole training/game session, and treat replayed timings as diagnostic rather than shipping latency. Many counters are unavailable or renamed on different compute capabilities; use the tool's metric query and report warnings. diff --git a/.claude/skills/javascript-profiling/SKILL.md b/.claude/skills/javascript-profiling/SKILL.md new file mode 100644 index 0000000..6c300bf --- /dev/null +++ b/.claude/skills/javascript-profiling/SKILL.md @@ -0,0 +1,266 @@ +--- +name: javascript-profiling +description: Profile Node.js CPU usage, allocation churn, retained heap, garbage collection, event-loop latency, and scoped operations with built-in Node and V8 tooling. +--- + +# JavaScript and Node.js performance profiling + +Use this skill for Node.js workloads. For browser JavaScript, rendering, network waterfalls, or browser main-thread work, use `browser-javascript-profiling` instead. + +Prefer built-in Node.js facilities before native profiler packages. A useful profile requires a representative workload that finishes successfully; validate that first. + +## Choose the profiler by question + +| Question | First tool | Artifact | Important limitation | +|---|---|---|---| +| Which JavaScript/native call paths consume CPU? | `node --cpu-prof` | `.cpuprofile` | Sampling can miss short work; CPU attribution is not wall-clock latency | +| Which call paths allocate the most bytes? | `node --heap-prof` | `.heapprofile` | Samples allocations, not the objects still retaining memory | +| Which objects retain memory at one point in time? | `v8.writeHeapSnapshot()` | `.heapsnapshot` | Stops the main thread and can require roughly twice the current heap | +| Is garbage collection frequent or expensive? | `node --trace-gc` | Diagnostic text | Adds logging overhead and needs representative allocation pressure | +| Is the event loop delayed? | `perf_hooks.monitorEventLoopDelay()` | Histogram values | Reports delay, not the responsible call path | +| How long does one operation take? | `performance.mark()` / `performance.measure()` | Performance entries | Instrumentation measures only the marked scope | + +Do not treat RSS, heap allocation, retained heap, and total allocated bytes as interchangeable. External memory such as `Buffer` backing stores can increase RSS without appearing as ordinary V8 heap growth. + +## Reproducible workflow + +1. Define the observable: CPU time, end-to-end latency, allocation rate, retained objects, GC pauses, event-loop delay, or peak RSS. +2. Run the workload without profiling. Confirm correct output and record the Node version, command, input, concurrency, warm-up state, and environment. +3. Make the workload long enough to produce useful samples. Repeat or loop short operations rather than profiling process startup alone. +4. Capture the narrowest relevant profile. Keep generated artifacts outside source directories or in ignored output directories. +5. Check that the artifact is non-empty and contains samples before interpreting it. A valid but sample-free `.heapprofile` is not evidence that the workload allocates nothing. +6. Inspect self time and total/inclusive time separately. A hot parent can merely contain the expensive descendant. +7. Change one variable, then rerun the same unprofiled workload and the same profiler. Use repeated measurements; one run is not a regression result. + +Profilers perturb execution. Compare runs only when Node version, input, process flags, workload duration, and profiler settings match. + +## CPU profiling + +Capture a V8 CPU sample profile: + +```bash +mkdir -p profiles +node \ + --cpu-prof \ + --cpu-prof-dir=profiles \ + --cpu-prof-name=app.cpuprofile \ + your-app.js +``` + +For an npm script, invoke the actual Node entry point when practical so the profile covers the target rather than the package manager. Preserve all application arguments after the entry script. + +Load `profiles/app.cpuprofile` in Chrome DevTools using **Performance > Load profile**. Confirm that it has samples and that the intended workload appears in the call tree. Use bottom-up view for expensive leaves and call-tree view for their calling context. + +CPU samples answer where on-CPU execution occurred. Waiting on sockets, timers, filesystem operations, worker results, or scheduling may dominate wall time without becoming a hot JavaScript frame. Pair the CPU profile with operation timing or event-loop diagnostics when latency is the real symptom. + +### Deno and Bun CPU profiles + +Deno and Bun also expose V8 CPU-profile output through `--cpu-prof`. Use these only for the process started by that runtime; they do not replace browser Performance traces for a page. + +```bash +mkdir -p profiles + +deno run --cpu-prof --cpu-prof-dir=profiles --cpu-prof-name=deno.cpuprofile your_script.ts +deno run --cpu-prof --cpu-prof-md --cpu-prof-dir=profiles server.js + +bun --cpu-prof --cpu-prof-dir ./profiles --cpu-prof-name bun.cpuprofile script.js +bun --cpu-prof-md script.js +``` + +Load `.cpuprofile` output in Chrome DevTools or another compatible viewer and confirm the intended workload appears in the sample tree. The Markdown profile options (`--cpu-prof-md`) are summaries for review, not substitutes for the raw CPU profile when detailed stack inspection is needed. Do not assume Node-only heap, GC, or inspector examples in this skill apply unchanged to Deno or Bun. + +### Scoped CPU profiling with the built-in inspector + +Use `node:inspector` instead of adding `v8-profiler-next`. This captures only the operation between `Profiler.start` and `Profiler.stop`: + +```javascript +const fs = require('node:fs/promises'); +const inspector = require('node:inspector'); + +const session = new inspector.Session(); +const post = (method, params = {}) => new Promise((resolve, reject) => { + session.post(method, params, (error, result) => { + if (error) reject(error); + else resolve(result); + }); +}); + +async function captureProfile() { + session.connect(); + try { + await post('Profiler.enable'); + await post('Profiler.start'); + + await runRepresentativeWorkload(); + + const { profile } = await post('Profiler.stop'); + await fs.writeFile('operation.cpuprofile', JSON.stringify(profile)); + } finally { + session.disconnect(); + } +} + +captureProfile().catch((error) => { + console.error(error); + process.exitCode = 1; +}); +``` + +Do not perform unrelated logging, setup, or teardown inside the measured scope unless it is part of the behavior being investigated. + +## Allocation sampling + +Capture sampled allocation stacks: + +```bash +mkdir -p profiles +node \ + --heap-prof \ + --heap-prof-dir=profiles \ + --heap-prof-name=app.heapprofile \ + your-app.js +``` + +Load the result in Chrome DevTools under **Memory**. This is allocation sampling: it identifies allocation-heavy stacks, including objects later collected. It does not by itself prove a leak. + +Short workloads can produce a structurally valid profile with no samples at the default sampling interval. For a bounded diagnostic run, lower the interval to collect more samples: + +```bash +node \ + --heap-prof \ + --heap-prof-interval=1024 \ + --heap-prof-dir=profiles \ + --heap-prof-name=app-1k.heapprofile \ + your-app.js +``` + +A smaller interval increases overhead and artifact size. Record the interval and never compare profiles captured with different intervals as if their sample counts were equivalent. + +## Retained-heap snapshots + +Use a heap snapshot when the question is which objects remain reachable and what retains them: + +```javascript +const v8 = require('node:v8'); + +const snapshotPath = v8.writeHeapSnapshot( + `heap-${process.pid}-${Date.now()}.heapsnapshot`, +); +console.log(`Heap snapshot written to ${snapshotPath}`); +``` + +`v8.writeHeapSnapshot()` returns only after the file is complete. Load it in Chrome DevTools under **Memory**. For leak analysis, compare snapshots around the same application state after allowing expected temporary objects to be collected; inspect growing retained objects and their retaining paths. + +Heap snapshots stop the main thread and may require about twice the current heap, which can terminate a process under memory pressure. They can contain secrets and user data. Do not capture production heaps without authorization, capacity headroom, secure storage, and a retention/deletion plan. + +Node also supports signal-triggered snapshots through `--heapsnapshot-signal=` on supported platforms. There is no general `node --heapsnapshot` flag. + +## Garbage-collection tracing + +Run a bounded representative workload: + +```bash +node --trace-gc your-app.js +``` + +Interpret each event by collection type, before/after heap size, pause duration, and reason. Repeated scavenges can be normal for allocation-heavy work. Frequent major collections, long pauses, or little memory reclaimed are stronger signals of heap pressure or retention. Do not “fix” GC pressure by increasing `--max-old-space-size` unless the application genuinely needs a larger live heap; a larger heap can defer rather than remove the cause. + +For a controlled experiment that explicitly calls `global.gc()`, launch with `--expose-gc` and fail if the flag is missing: + +```javascript +if (typeof global.gc !== 'function') { + throw new Error('Run this diagnostic with node --expose-gc'); +} +global.gc(); +``` + +Manual GC changes application behavior. Use it to establish controlled snapshot boundaries, not as a production memory-management strategy. + +## Scoped timing and event-loop delay + +Measure a named operation with `node:perf_hooks`: + +```javascript +const { performance } = require('node:perf_hooks'); + +performance.mark('operation:start'); +await runRepresentativeWorkload(); +performance.mark('operation:end'); +const measurement = performance.measure( + 'operation', + 'operation:start', + 'operation:end', +); +console.log(`${measurement.name}: ${measurement.duration.toFixed(3)} ms`); +performance.clearMarks(); +performance.clearMeasures(); +``` + +`performance.measure()` returns the completed entry, so this form cannot lose a queued `PerformanceObserver` callback by disconnecting too early. + +For event-loop delay: + +```javascript +const { monitorEventLoopDelay } = require('node:perf_hooks'); +const { setImmediate: nextTurn } = require('node:timers/promises'); + +const delay = monitorEventLoopDelay({ resolution: 20 }); +delay.enable(); +await nextTurn(); + +await runRepresentativeWorkload(); + +await nextTurn(); +delay.disable(); +console.log({ + meanMs: delay.mean / 1e6, + p99Ms: delay.percentile(99) / 1e6, + maxMs: delay.max / 1e6, +}); +``` + +Histogram values are nanoseconds; divide by `1e6` for milliseconds. Let the observation window span event-loop turns on both sides of the work. A very short window can have no samples and report `NaN` for the mean; increase the workload or observation time before interpreting it. + +## Webpack and Jest + +Avoid version-specific internal paths such as `webpack-cli/bin.js` and `jest-cli/bin/jest.js`. Use the package entry points installed by the project: + +```bash +# Webpack production build CPU profile +node \ + --cpu-prof \ + --cpu-prof-dir=profiles \ + --cpu-prof-name=webpack.cpuprofile \ + ./node_modules/webpack/bin/webpack.js \ + --mode production + +# Jest CPU profile; serial execution makes attribution and comparison clearer +node \ + --cpu-prof \ + --cpu-prof-dir=profiles \ + --cpu-prof-name=jest.cpuprofile \ + ./node_modules/jest/bin/jest.js \ + --runInBand + +# Jest per-test heap reporting +node \ + --expose-gc \ + ./node_modules/jest/bin/jest.js \ + --runInBand \ + --logHeapUsage +``` + +Keep the project’s existing arguments, config, environment, and test selection. Profile a focused build or test target before a full suite. `--runInBand` changes concurrency, so compare it only with another serial run. + +## Reporting and cleanup + +Report: + +- the exact command and Node version; +- workload input, duration, concurrency, and warm-up state; +- profiler type and settings, including heap sampling interval; +- the hottest or largest call paths with self and total values; +- whether the result concerns allocated, retained, or process memory; +- repeated unprofiled before/after measurements for any claimed optimization. + +Delete generated `.cpuprofile`, `.heapprofile`, `.heapsnapshot`, and GC trace artifacts when the investigation ends unless the repository intentionally retains them. Never commit heap snapshots containing application or user data. diff --git a/.claude/skills/jvm-profiling/SKILL.md b/.claude/skills/jvm-profiling/SKILL.md new file mode 100644 index 0000000..709acb4 --- /dev/null +++ b/.claude/skills/jvm-profiling/SKILL.md @@ -0,0 +1,96 @@ +# JVM Profiling with Java Flight Recorder + +Use this skill when a Java application's CPU time, latency, allocation rate, garbage collection, heap growth, or thread contention needs evidence rather than guesses. It is deliberately independent of a build system: replace the example application command and output paths with the command used to launch the service. + +## Choose the tool + +- **Java Flight Recorder (JFR)** is the low-overhead event recorder built into modern JDKs. Use it for a bounded production investigation, startup or intermittent latency, GC/safepoint behavior, allocation rates, locks, and correlated JVM/OS events. +- **JDK Mission Control (JMC)** is the GUI for opening and analyzing `.jfr` recordings. It is especially useful when the problem happened in the past or on another host. +- A useful default sequence is: establish a baseline, collect a short JFR recording, inspect it in JDK Mission Control, and take a heap dump only when the evidence points to a live heap/object-graph question. Do not treat a profiler's percentage as a diagnosis by itself. + +Official repository resources: + +- [Java Flight Recorder and JDK Mission Control](https://www.oracle.com/java/technologies/jdk-mission-control.html) — the repository's JFR/JMC link. +- The `jcmd` syntax and options are versioned with the JDK; use the documentation for the JDK actually running the target JVM (for example, the [JDK 25 `jcmd` reference](https://docs.oracle.com/en/java/javase/25/docs/specs/man/jcmd.html)). + +## Before collecting data + +1. Record the JDK vendor/version, JVM flags, host/container limits, PID, deployment version, and the workload window. Keep these with every profile. +2. Define one question and a short capture interval (for example, “why do requests pause for 90 seconds?”), and capture a comparable healthy interval if possible. +3. Use a JDK tool from the same release family as the target (`jcmd -l`, `java -version`). Local attach normally requires the same OS user or an explicitly permitted service account. Containers, PID namespaces, and hardened JVMs can hide or deny attach. +4. Create a private output directory with enough space and predictable ownership. Use absolute paths for recordings and dumps; avoid writing diagnostic files into an application's working directory or a world-readable temporary directory. + +## JFR and JMC workflows + +JFR recordings are read in JMC. Keep the recording's JDK/JMC versions compatible, copy the `.jfr` file out of the host through an access-controlled channel, and open it in JMC's automated analysis plus event pages. `default.jfc` is intended for low-overhead, potentially continuous recordings; `profile.jfc` enables more detail and is better for a short investigation with more overhead. + +### Attach to an already-running JVM with `jcmd` + +Find the process and check whether a recording already exists: + +```bash +jcmd -l +jcmd VM.command_line +jcmd JFR.check +``` + +Start a bounded recording using the JDK's low-overhead configuration: + +Use a concrete, unique filename for each capture. JFR's `filename` option does **not** expand the `%p` and `%t` placeholders supported by unified JVM logging; passing those placeholders can make JVM startup fail. Generate the timestamp/PID in the shell, service manager, or deployment configuration before invoking `java` or `jcmd`. The filenames below are examples of already-expanded capture IDs. + +```bash +jcmd JFR.start name=latency settings=default duration=90s filename=/secure/profiles/latency-20260710T120000Z-12345.jfr +jcmd JFR.check name=latency +``` + +`duration` lets the JVM close the recording and write the file without a second stop command. When the incident is event-driven and the end time is unknown, omit `duration`, then dump or stop it explicitly: + +```bash +jcmd JFR.start name=incident settings=default +# Wait for the known failure or latency spike. +jcmd JFR.dump name=incident filename=/secure/profiles/incident-20260710T120000Z-12345.jfr +# Use JFR.stop instead when you want to finish the recording and release it. +jcmd JFR.stop name=incident +``` + +`JFR.dump` writes a snapshot while the recording continues; `JFR.stop` ends it. Do not use an unbounded recording without a disk/retention plan. For a short, higher-detail capture, use `settings=profile` and a small duration only after checking that the extra overhead is acceptable: + +```bash +jcmd JFR.start name=short-profile settings=profile duration=30s filename=/secure/profiles/profile-20260710T120000Z-12345.jfr +``` + +If attach fails, do not repeatedly retry against a live production process. Check PID namespace and permissions, use the target JDK's `jcmd`, and consult the service/container's diagnostic policy. A restart may be required for startup-only flags or for a JVM that disallows attach. + +### Start with a recording from JVM startup + +Add a startup option to the normal Java launch command when startup, class loading, or an early failure matters: + +```bash +java -XX:StartFlightRecording=name=startup,settings=default,duration=2m,filename=/secure/profiles/startup-20260710T120000Z.jfr +``` + +For a continuously available, bounded rolling window, configure retention and dump on exit. Check the path and disk budget before enabling it: + +```bash +java -XX:StartFlightRecording=name=rolling,settings=default,maxage=1h,maxsize=512m,dumponexit=true,filename=/secure/profiles/rolling-20260710T120000Z.jfr +``` + +A startup option is part of process configuration: deploy it deliberately, verify the file is created, and remove it after the investigation if it is not an intentional baseline. Do not assume that a startup recording captures data from before the JVM was launched; it starts during JVM initialization. + +### Interpret a recording in JMC + +Start with the time range containing the symptom and correlate these views: + +- **CPU/execution:** execution samples show where threads were observed running; compare Java method stacks with process/host CPU. A method with high wall-clock presence may be waiting or blocked, while a high-CPU stack is a stronger candidate for compute cost. Look for repeated application frames, not only framework dispatch. +- **Threads and locks:** inspect thread states, Java monitor blocked events, parks, waits, and thread dumps around the incident. Many blocked threads behind one owner indicate contention; many runnable threads with low host CPU can indicate scheduling or I/O rather than a CPU algorithm. +- **Garbage collection:** correlate young/old collection pauses, pause duration, heap occupancy before/after GC, allocation rate, promotion, and safepoints. Frequent short collections suggest allocation pressure; a rising post-GC live set suggests retention or an undersized workload window. A long pause can be GC, a safepoint operation, or an external stall—use the event type and thread stacks to distinguish them. +- **Memory/allocation:** JFR allocation events (including TLAB and outside-TLAB allocations where enabled) identify allocation sites and rates. They do not replace a full heap graph. Use a heap dump or object-graph tool when the question is “what is retaining these objects?” rather than “where are they being allocated?” +- **Latency:** use event timestamps and duration, not just averages. A tail-latency spike should be aligned with thread blocking, GC/safepoint pauses, I/O, CPU saturation, or a deployment/configuration change. + +## Overhead and production safety + +- JFR's `default` configuration is designed for low overhead, but it is not free; stack depth, high-rate events, custom events, and `profile` settings increase CPU, memory, and disk use. Keep captures short, set `maxage`/`maxsize` for rolling use, and monitor the output filesystem. +- Heap dumps and class histograms can pause the JVM, require significant disk, and expose object contents. Treat dumps as sensitive data: restrict permissions, encrypt in transit/storage, avoid collecting during the busiest interval unless the pause is acceptable, and delete them according to the incident-retention policy. +- JFR recordings contain stack traces, thread names, class/method names, and any values emitted by application/custom events. Restrict access and redact before sharing. Do not enable remote JMX or expose a profiler port without authentication, authorization, network controls, and TLS. +- Never run `jcmd` as an unreviewed “fix.” Profiling explains behavior; it does not repair a deadlock, leak, or overload. Record the exact command, start/stop time, PID, JDK, settings, workload, and observed impact so another engineer can reproduce the analysis. +- Stop when the question is answered, restore the normal launch configuration, verify that temporary recordings/dumps are no longer accumulating, and compare against a baseline or controlled reproduction before changing application code. diff --git a/.claude/skills/python-cpu-profiling/SKILL.md b/.claude/skills/python-cpu-profiling/SKILL.md new file mode 100644 index 0000000..79c0fc8 --- /dev/null +++ b/.claude/skills/python-cpu-profiling/SKILL.md @@ -0,0 +1,97 @@ +# Python CPU and Runtime Profiling + +Use this skill when you need to explain, investigate, or improve Python execution time in this repository. It covers the four profilers that are relevant here: **cProfile**, **pyinstrument**, **py-spy**, and **yappi**. The target program is `python/program.py`; commands below assume they are run from the `python/` directory unless stated otherwise. + +## Scope and selection + +Choose the clock and profiler based on the question, not on the smallest reported number. This program performs an HTTP request, parses HTML with Beautiful Soup, and counts words, so network wait, imports, and parsing can show up differently in each profile. + +| Tool | Method and clock | Use it when you need | Primary output | Cost and important limits | +|---|---|---|---|---| +| `cProfile` | Deterministic function-call instrumentation; the CLI's default timer is elapsed real time (the profiler's built-in real-time clock) | Exact call counts and inclusive/self timing for Python functions | `profile.out`, inspected with `pstats` | More overhead for call-heavy code; it changes the execution it measures and is not a benchmark. It profiles the interpreter process that starts the script, not child interpreters automatically. | +| `pyinstrument` | Statistical stack sampling at an interval; its normal report represents wall-clock/elapsed activity | A readable call tree showing where user-perceived time, including waits, is spent | Terminal tree, or HTML/JSON/etc. report | Usually less intrusive than deterministic tracing, but samples can miss very short functions; a smaller interval costs more CPU and memory. | +| `py-spy` | External OS-level sampling profiler; samples the running process without importing code into it and normally focuses on active/on-CPU threads | A low-overhead view of a running or production-like process, including stacks without changing the target | Live `top`, one-shot `dump`, or SVG flame graph from `record` | Requires OS permission to read another process (especially on macOS and for PID attach on Linux). Sampling is approximate, and a short script may finish before enough samples are collected. | +| `yappi` | Deterministic in-process function/thread profiling; clock is explicitly selectable (`cpu` or `wall`) | CPU-vs-wall comparisons, per-function totals, and thread-aware reports from Python code | Function and thread tables printed by `print_all()` | Instrumentation can be substantial, especially with many calls or threads; child processes need their own profiler setup. | + +### CPU time versus wall-clock time + +- **CPU time** answers “How much processor time did this process consume?” It excludes time sleeping or waiting for the network, although kernel CPU time can still appear separately in the shell's `sys` measurement. Use it for CPU-bound Python work and algorithmic cost. +- **Wall-clock time** answers “How long did a user wait?” It includes scheduling, network, file, lock, and sleep delays. Use it for request latency and end-to-end responsiveness. +- The shell's `time` command is the baseline for the distinction: `real` is elapsed wall time, `user` is user-mode CPU time, and `sys` is kernel-mode CPU time. Do not compare a profiler's function total directly to a shell `real` value without checking that profiler's clock. +- `cProfile`'s repository command uses its default real-time timer. `yappi` is the clearest way in this set to run the same code with either `cpu` or `wall`. `pyinstrument` is normally used here as an elapsed-time sampler. `py-spy`'s default active-thread samples are most useful for CPU hot spots; use a wall-time profiler when blocked time is the question. + +## Setup and safe baseline + +The repository README sets up a Python virtual environment and installs the pinned dependencies in `python/requirements.txt` (`pyinstrument==4.6.2`, `py-spy==0.3.14`, and `yappi==1.6.0` among them). From the repository root, the README's setup commands are: + +```bash +cd python +pyenv install $(cat .python-version) +pyenv local +python -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +``` + +If the environment already exists, the minimum assumption for every command is: + +```bash +cd python +source .venv/bin/activate +``` + +Verify that `python` and the console tools resolve from the intended environment before profiling: + +```bash +python --version +command -v python +command -v pyinstrument +command -v py-spy +``` + +Do not silently use a system interpreter: the target imports `requests` and `bs4`, and the pinned virtual environment is what makes those imports and profiler versions reproducible. The target fetches `https://en.wikipedia.org/wiki/Intuitive_Machines_Nova-C`, writes a temporary file, and prints its result. Run only when network access and that side effect are acceptable. The response, server latency, DNS, proxy settings, import caches, and machine load can all change the profile; compare runs under the same conditions and treat one short run as directional evidence. + +Establish an unprofiled timing before interpreting profiler overhead: + +```bash +time python program.py +``` + +The repository's `python/timing.sh` activates `.venv`, runs each command with `time`, and redirects program output to `/dev/null`. Its complete profiling-relevant commands are: + +```bash +time pyinstrument program.py > /dev/null +time python -m cProfile -o profile.out program.py > /dev/null +time python run_yappi.py > /dev/null +``` + +The README also runs the whole comparison with: + +```bash +./timing.sh +``` + +That script includes other memory/profiling tools and performs multiple network requests; use it only when that full comparison is intended. The redirection is for timing, not report inspection: run the profiler without `> /dev/null` when you need to read its text output. + +## Additional resources + +Reference the supporting file that matches the profiler question so Claude knows when to load detailed workflows: + +- [cprofile.md](cprofile.md): deterministic function-call attribution with `cProfile` and `pstats`. +- [pyinstrument.md](pyinstrument.md): wall-clock stack sampling and readable call trees. +- [py-spy.md](py-spy.md): external sampling, live summaries, dumps, and flame graphs. +- [yappi.md](yappi.md): CPU/wall-clock function and thread profiling from Python code. +- [pyperformance.md](pyperformance.md): benchmark regression comparisons; not a substitute for a profiler when a hot call path is needed. + +## A repeatable investigation workflow + +1. Activate `.venv` from `python/`, verify `python`, and run `time python program.py` once. Save the `real`, `user`, and `sys` values and note network conditions. +2. Decide whether the question is CPU consumption (`user`/`sys`, [yappi](yappi.md) `cpu`, [cProfile](cprofile.md), or [py-spy](py-spy.md) hot stacks) or user-visible latency (`real`, yappi `wall`, [pyinstrument](pyinstrument.md)). +3. Start with py-spy when you need a low-overhead view of a live/production-like process. Use its launched `record` form for this short script and its PID form only for an authorized long-running process. +4. Use pyinstrument for an approachable elapsed-time call tree and to expose blocking/waiting phases. Use `--show-all` when library frames matter. +5. Use cProfile when exact function counts and caller/callee attribution are needed. Inspect `profile.out` with `pstats`; sort by cumulative time, then check self time and call counts. +6. Use yappi when CPU versus wall or per-thread attribution is the deciding question. Keep the clock type beside the report and stop profiling before printing stats. +7. Repeat the same profiler under the same workload when a conclusion matters. A single HTTP response and a short script can produce sparse samples and high run-to-run variance. +8. Compare profiler overhead to the unprofiled shell timing. Do not use any of these profiled commands as a benchmark, and do not add profiler output redirects when the purpose is report interpretation. + +Keep generated artifacts (`profile.out`, `profile.svg`, `pyinstrument.html`) separate from source changes. No profiling command should be described as having run unless its output was actually observed; examples in this skill are instructions only. diff --git a/.claude/skills/python-cpu-profiling/cprofile.md b/.claude/skills/python-cpu-profiling/cprofile.md new file mode 100644 index 0000000..56dcd77 --- /dev/null +++ b/.claude/skills/python-cpu-profiling/cprofile.md @@ -0,0 +1,54 @@ +# cProfile: deterministic call attribution + +## Run the repository command + +The README's exact command writes a binary profile database: + +```bash +python -m cProfile -o profile.out program.py +``` + +Inspect it with the standard-library statistics browser, using the README's exact interaction: + +```bash +python -m pstats profile.out +profile.out% sort cumulative +profile.out% stats 10 +``` + +`sort cumulative` puts the functions with the largest inclusive (`cumtime`) totals first; `stats 10` limits the displayed report to ten rows. The file is not source text, so do not try to read `profile.out` with a text editor. Keep the artifact if a later analysis needs the same run, and remove or ignore it when it is no longer needed. + +## Read the report + +The standard report columns mean: + +- `ncalls`: number of calls; `151/1` means 151 total calls caused by one primitive call through recursion. +- `tottime`: time spent in that function body, excluding its callees (self time). +- First `percall`: `tottime / ncalls`. +- `cumtime`: time in the function and all functions it called (inclusive time). It is the useful sort for finding an expensive path, but parent and child rows overlap and must not be added together. +- Second `percall`: `cumtime / primitive calls`. +- `filename:lineno(function)`: the source location and function name. + +A high `cumtime` in `program.py:main` is a boundary, not automatically the defect: follow its children and compare `tottime`. A function with high `tottime` is a candidate for direct optimization; a function with low self time but high cumulative time is often an orchestrator or an expensive call boundary. Library rows such as `requests` or Beautiful Soup can be legitimate contributors, and the target's HTTP wait may dominate a wall-clock profile. + +## Clock and overhead cautions + +`cProfile` records every Python function-call/return event, which gives exact counts but perturbs call-heavy code. C-level work is not charged the same way as Python call dispatch, so do not use a profiled run as a microbenchmark or infer production latency from it. Compare the unprofiled `time` output with the profiled `time` output, and make decisions from repeated profiles and relative hot spots. + +If a CPU-only call profile is required for a controlled experiment, use a custom timer around the code under investigation rather than changing the repository command: + +```python +import cProfile +import time + +profiler = cProfile.Profile(timer=time.process_time) +profiler.enable() +try: + # call the code under investigation + pass +finally: + profiler.disable() +profiler.print_stats(sort="cumulative") +``` + +This is an example of the standard `cProfile.Profile` API, not a modification requested for `program.py`. A custom timer changes what the numbers mean and should be documented alongside the result. For this repository's unmodified target, use the README command and the shell's `user`/`sys` values when you need the normal baseline. diff --git a/.claude/skills/python-cpu-profiling/py-spy.md b/.claude/skills/python-cpu-profiling/py-spy.md new file mode 100644 index 0000000..973e463 --- /dev/null +++ b/.claude/skills/python-cpu-profiling/py-spy.md @@ -0,0 +1,46 @@ +# py-spy: external sampling and flame graphs + +`py-spy` appears in `python/requirements.txt` as version `0.3.14`, but the repository README and `timing.sh` do not provide a py-spy run. The following are CLI examples for the installed dependency; they do not replace the repository's exact commands above. + +Launch and record the target in one command: + +```bash +py-spy record -o profile.svg -- python program.py +``` + +This writes an interactive SVG flame graph. The `--` separates py-spy's options from the target command. To watch a live summary instead: + +```bash +py-spy top -- python program.py +``` + +To inspect a process that is already running, first obtain the PID from a process you own and then use: + +```bash +py-spy top --pid +py-spy dump --pid +py-spy record -o profile.svg --pid +``` + +`top` is useful for a long-lived process; `dump` is useful for a one-time stack snapshot, such as diagnosing a hang. `program.py` is short and performs one request, so a launched `record` run is generally more useful than trying to attach after it has already exited. + +A flame graph's horizontal width is proportional to sampled time, not to exact invocation count. Read from the bottom (the process/thread root) upward: broad branches are the stacks that occupied the most samples; narrow branches may still be important if they are latency-sensitive or under-sampled. A missing short function is expected with sampling. The default view attempts to omit idle threads, so a missing network wait should not be interpreted as proof that no wall time was spent waiting. Use pyinstrument or yappi with `wall` when blocked time is the question. + +For a process that creates Python child processes, ask py-spy to follow them when supported by the installed version: + +```bash +py-spy record --subprocesses -o profile.svg -- python program.py +py-spy top --subprocesses -- python program.py +``` + +Without `--subprocesses`, a parent profile does not automatically explain work done in a child interpreter. `cProfile`, pyinstrument, and yappi similarly require profiling code in each child or a separately launched profiler; do not assume that an in-process profile covers multiprocessing workers. + +## Permission and safety caveats + +py-spy reads another interpreter's memory from outside the target process, which is why it has low in-process overhead but also why OS security rules matter. On macOS, attaching commonly requires root; on Linux, attaching to an unrelated PID can require root or ptrace permission. If a PID attach fails with an access/permission error, first prefer the launched form (`py-spy ... -- python program.py`) for a process you own. If the OS still requires elevation, rerun only the specific command with `sudo` and use an explicit path so the intended installed binary is selected, for example: + +```bash +sudo "$(command -v py-spy)" record -o profile.svg -- python program.py +``` + +Do not attach to an unrelated process or use `sudo` to bypass a policy without authorization. A root-run target may have different environment, proxy, file permissions, and output ownership, so record that fact with the profile. `--locals` (where supported) can expose application data; avoid it for this target unless the data is safe to disclose. diff --git a/.claude/skills/python-cpu-profiling/pyinstrument.md b/.claude/skills/python-cpu-profiling/pyinstrument.md new file mode 100644 index 0000000..c2623e4 --- /dev/null +++ b/.claude/skills/python-cpu-profiling/pyinstrument.md @@ -0,0 +1,32 @@ +# pyinstrument: wall-clock sampling + +The repository's exact `python/timing.sh` invocation is: + +```bash +time pyinstrument program.py > /dev/null +``` + +For an inspectable terminal report, omit the redirect: + +```bash +pyinstrument program.py +``` + +For a saved interactive report, use the installed CLI renderer explicitly: + +```bash +pyinstrument -r html -o pyinstrument.html program.py +``` + +The report is a sampled call tree. A wide/high-percentage branch means many samples observed that stack, so it is an estimate of elapsed activity rather than an exact call count. Repeated frames may be condensed in the normal view. Use `--timeline` when the order of phases matters, or `--show-all` if the default filtering hides library frames that are relevant to this target: + +```bash +pyinstrument --timeline program.py +pyinstrument --show-all program.py +``` + +Run those modes separately with the repository's pinned pyinstrument 4.6.2. Combining `--timeline --show-all` can produce an effectively empty terminal report even when the header reports many samples. + +The default sampling interval is designed for general profiling. A shorter interval can resolve shorter calls but increases profiler overhead and report size; a longer interval reduces overhead and memory use but loses detail. For a very short `program.py` run, “no samples” or a sparse tree is a sampling limitation, not evidence that the program did no work. Prefer a longer representative workload when available, or compare several runs rather than making a conclusion from one sample. + +Because pyinstrument is an elapsed-time profiler, a `requests.get` wait or file operation can occupy a large branch even when it consumes little CPU. That is exactly the signal to use for user-visible latency. If the question is “which Python code burns CPU while the request is already available?”, pair the report with `time` or use yappi's CPU clock/cProfile rather than treating a wall-time branch as CPU cost. diff --git a/.claude/skills/python-cpu-profiling/pyperformance.md b/.claude/skills/python-cpu-profiling/pyperformance.md new file mode 100644 index 0000000..f68e098 --- /dev/null +++ b/.claude/skills/python-cpu-profiling/pyperformance.md @@ -0,0 +1,11 @@ +# pyperformance: benchmark comparisons + +The repository root links to [pyperformance](https://github.com/psf/pyperformance), a benchmark suite rather than a replacement for cProfile or a flame graph. Use it when the question is whether a repeatable benchmark regressed across Python builds or commits; use the profilers above when you need to explain a hot function or allocation. It is not pinned in `python/requirements.txt`, so install and invoke the version approved by the project, then check its local `--help` for version-specific options: + +```bash +python -m pyperf system tune # optional; use only on an authorized benchmark host +pyperformance run --python="$(command -v python)" -o pyperformance.json +pyperformance compare baseline.json pyperformance.json +``` + +Run a benchmark suite for long enough to reduce noise, keep CPU frequency/power state and workload configuration stable, and compare distributions rather than one elapsed time. Record the Python executable, pyperformance version, benchmark selection, warm-up, calibration, and machine state. A benchmark result can establish a regression; it does not identify the responsible call path. Follow a significant result with cProfile, pyinstrument, py-spy, or yappi under a controlled reproduction. diff --git a/.claude/skills/python-cpu-profiling/yappi.md b/.claude/skills/python-cpu-profiling/yappi.md new file mode 100644 index 0000000..8fe7bf8 --- /dev/null +++ b/.claude/skills/python-cpu-profiling/yappi.md @@ -0,0 +1,61 @@ +# yappi: programmatic CPU/wall and thread profiles + +The repository's runner is `python/run_yappi.py`, and the README's exact command is: + +```bash +python run_yappi.py +``` + +`python/timing.sh` expands its `run run_yappi.py` helper to: + +```bash +time python run_yappi.py > /dev/null +``` + +Run without redirection to inspect the report. The existing runner does the following: + +```python +import yappi + +from program import main + +yappi.set_clock_type("cpu") +yappi.start() +main() +yappi.get_func_stats().print_all() +yappi.get_thread_stats().print_all() +print(f"Memory usage {yappi.get_mem_usage()}") +``` + +The important choices are `set_clock_type("cpu")` and starting before `main()`. A controlled version should stop in a `finally` block before reading stats, so a failure does not leave profiling enabled: + +```python +import yappi + +from program import main + +yappi.set_clock_type("cpu") # processor time; excludes network sleep +# Use "wall" instead when end-to-end elapsed time is the question. +yappi.start() +try: + main() +finally: + yappi.stop() + +yappi.get_func_stats().print_all() +yappi.get_thread_stats().print_all() +``` + +Use `yappi.set_clock_type("wall")` for the same function/thread report with elapsed time. Do not mix CPU-clock and wall-clock runs when comparing totals. If the target uses threads, retain the thread report: it can reveal a worker that the aggregate function table obscures. Yappi's `get_mem_usage()` output in `run_yappi.py` is an auxiliary value printed by the repository runner; it is not a replacement for a memory profiler. + +Interpret the function table shown by the README (`Clock type: CPU`, `Ordered by: totaltime, desc`) as follows: + +- `ncall`: number of function calls. +- `tsub`: time spent in the function body itself (self time). +- `ttot`: total time attributed to that function, including callees. +- `tavg`: average total time per call. +- Thread rows include a thread name/id and total time plus a switch/count field; use them to identify uneven work or scheduling. + +Yappi is deterministic instrumentation, so it can be more disruptive than pyinstrument or py-spy on code with very high call rates. Its exact function totals are valuable for a CPU-vs-wall experiment, but the profiler's own cost must be checked against `time python program.py`. Start and stop around the smallest meaningful region when profiling a larger application; for this repository's complete target, use the runner so imports and `main()` are consistent with the README. + +A child process has a separate interpreter and does not inherit useful yappi samples merely because the parent called `yappi.start()`. Initialize yappi in the child, or use py-spy's `--subprocesses` mode for an external view. diff --git a/.claude/skills/python-dtrace-profiling/SKILL.md b/.claude/skills/python-dtrace-profiling/SKILL.md new file mode 100644 index 0000000..98ece99 --- /dev/null +++ b/.claude/skills/python-dtrace-profiling/SKILL.md @@ -0,0 +1,166 @@ +# Python DTrace Profiling Skill + +Use this skill when you need an event-by-event view of CPython execution on macOS: Python function entry/return events, source-line events, and their ordering in a real process. + +DTrace is **instrumentation tracing**, not a replacement name for every Python profiler. It observes CPython's static probes and prints a timestamped text stream. Use `cProfile`/`yappi` for function call timing, `pyinstrument` for sampled call stacks, `tracemalloc`/`memray`/`guppy3` for Python memory behavior, and `psutil` for process resource snapshots. DTrace is useful when the exact sequence and nesting of interpreter events matters. + +## When to use it + +Choose DTrace when you need to answer questions such as: + +- Which Python functions were entered and returned around a particular operation? +- Which source lines in `program.py` executed, and in what order? +- Did an event happen before or after another event in the running process? +- Are native/interpreter-level boundaries relevant, rather than only an aggregate profile? + +Do not start with DTrace for a broad “what is slow?” question. Begin with the repository's lower-overhead or purpose-built profilers, then use a short DTrace run to explain a specific behavior. DTrace output is raw and often needs filtering or aggregation after the run. + +## Prerequisites: an instrumented Python + +DTrace probes are not enabled in an ordinary prebuilt Python. CPython must be configured and compiled with `--with-dtrace`; this also applies when the interpreter is built in a Docker image. `--with-dtrace` is a build-time option, not a flag that can be added to `python` at run-time. + +From this repository's `python/` directory, a fresh pyenv setup is: + +```bash +cd python +PYTHON_CONFIGURE_OPTS="--with-dtrace" pyenv install +pyenv shell +python -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +``` + +The first command builds the chosen supported CPython version with DTrace support. If that version is already installed by pyenv without the option, installing it again will not convert the existing build; rebuild the version in a controlled pyenv environment (without deleting an environment you still need), then recreate the virtual environment if necessary. Verify the active interpreter before tracing: + +```bash +python -VV +command -v python +``` + +The repository README's regular setup uses the same virtualenv and requirements commands. The DTrace scripts themselves do not require a Python package beyond the instrumented interpreter, but `program.py` needs the dependencies installed by `requirements.txt` (`requests` and `beautifulsoup4`). + +### Confirm that probes exist + +DTrace's Python provider name includes the target process ID. Start the exact interpreter in a quiet interactive shell, list its probes, then return to the shell and exit it: + +```bash +source .venv/bin/activate +python -q & +sudo dtrace -l -P python$! +fg # press Ctrl+D to exit the Python shell +``` + +`$!` is the shell's PID for the background Python process, so `python$!` expands to a provider such as `python22133`. Look for at least `function-entry`, `function-return`, and `line` in the probe list. The provider may also expose audit, import, and garbage-collection probes. The scripts in this repository use the target wildcard (`python$target`) and therefore attach to the process launched by DTrace's `-c` option rather than requiring you to copy a PID. + +Probe availability must be checked on the actual interpreter. CPython has supported DTrace static markers since Python 3.6, but a usable provider still depends on the interpreter build, OS support, and the probes exposed by that build. If the required probes are absent, rebuild CPython with `--with-dtrace` or use another DTrace-enabled CPython build; changing the `.d` file cannot fix a missing provider. + +## Run the repository scripts + +Run these commands from `python/`, with the virtual environment activated, so `program.py` and the checked-in `.d` files resolve as shown in the README: + +```bash +sudo dtrace -s call_stack.d -c 'python program.py' +sudo dtrace -s lines.d -c 'python program.py' +``` + +These are the exact repository workflows: + +- `-s call_stack.d` loads `call_stack.d`. +- `-s lines.d` loads `lines.d`. +- `-c 'python program.py'` makes DTrace launch the target and stop tracing when it exits. The single quotes protect the command from the outer shell; they are not part of the Python command. +- `sudo` is normally required on macOS to enable DTrace and access the process. Expect a password prompt. + +The `-c` command assumes `python` resolves to the DTrace-enabled interpreter in the environment used by DTrace. If `sudo` or pyenv changes `PATH` on a particular machine, keep the same command shape but substitute the absolute path printed by `command -v python`, for example: + +```bash +sudo dtrace -s lines.d -c '/absolute/path/to/python program.py' +``` + +The target's own output (the word counts printed by `program.py`) can appear alongside DTrace output. Redirect DTrace's combined output when a run is large: + +```bash +sudo dtrace -s lines.d -c 'python program.py' > lines.trace +sudo dtrace -s call_stack.d -c 'python program.py' > call-stack.trace +``` + +## What the scripts trace + +Python's DTrace marker arguments used by these scripts are: + +- `arg0`: source filename +- `arg1`: Python function name (for a module body this may be the module-level code name) +- `arg2`: source line number + +Both scripts call `copyinstr()` to read the string arguments and use `basename()` for filename filtering. Their output is deliberately plain text: + +```text +\t::: +``` + +`timestamp` is DTrace's monotonic timestamp (use differences between events for elapsed time; it is not a wall-clock date). The probe name identifies the event: `function-entry`, `function-return`, or `line`. + +### `call_stack.d` + +`call_stack.d` combines function and line events: + +1. A `function-entry` whose `arg0` basename is `program.py` sets the per-thread `self->trace` flag. +2. While that flag is set, every subsequent function entry is printed, including nested calls into libraries. The current per-thread indentation is printed, then incremented. +3. A matching `function-return` decrements indentation before printing, so returns line up with the caller's depth. +4. A `function-return` for `program.py` clears the trace flag. +5. `line` events are printed only when `basename(copyinstr(arg0)) == "program.py"`, with the current indentation. + +Thus the call events show the nested call activity reached from `program.py`, while line events are limited to that basename. The script's predicates are basename filters, not full-path filters: another file also named `program.py` could match. Repeated line events are expected in loops. With multiple threads, `self->trace` and `self->indent` are per-thread, while output lines can still interleave. + +### `lines.d` + +`lines.d` listens only to `python$target:::line` and applies the same `program.py` basename predicate. It prints every matching line event without indentation. It is the smaller script when the question is strictly “which lines executed?”; it is not a duration or line-cost profiler. A loop or parser can produce a very large number of repeated events. + +### Interpreting entry, return, and line events + +- Pair a `function-entry` and `function-return` at the same nesting level to identify an invocation's interval. Subtract their timestamps for an approximate observed interval, remembering that output and other threads affect the trace. +- A `function-entry` is an observation that CPython entered a Python function; it is not itself proof that the function consumed most of the runtime. +- A `function-return` marks the corresponding function exit event. Match events by per-thread nesting and function/file/line rather than by adjacent lines if multiple threads are active. +- A `line` event marks a source-line execution event from CPython. It is useful for control-flow evidence, but repeated events count executions, not exclusive time. +- DTrace's printed source is the filename basename, function name, and line number supplied by the static marker. C-extension work may appear only as surrounding Python events; these scripts do not turn native functions into Python source-line events. + +## Filtering and overhead + +The scripts already filter line events to `program.py` and use `call_stack.d`'s trace flag to avoid printing function events before the target program enters. They still print every matching line, including lines executed many times, and `call_stack.d` also prints nested library function entries and returns. Filtering is therefore not free: the DTrace predicates avoid irrelevant events in the output, but `printf()` for each selected event and the cost of formatting/reading strings can dominate the run. + +The README records one run with “Tracing every line” at **0m35.006s real, 0m7.995s user, and 0m28.952s system**, compared with an unprofiled run around 0m0.344s real. Treat those numbers as an illustration rather than a benchmark—the program performs a network request and timings vary—but expect line tracing to be orders of magnitude slower for event-heavy workloads. Never trace every line casually in production or on a long-running service. Prefer a short workload, a narrow source predicate, and redirected output; only enable `lines.d` when line-level evidence is actually needed. + +## macOS permissions and System Integrity Protection + +On macOS, DTrace may print: + +```text +dtrace: system integrity protection is on, some features will not be available +``` + +SIP can restrict parts of DTrace, especially probes or operations involving protected system processes. In a hosted environment you generally cannot disable SIP. For this repository's own, user-launched Python process, the warning is not automatically fatal: continue if the probe listing contains the needed Python probes and the smoke run emits events. If `dtrace` reports that a probe is unavailable, permission is denied, or no events are emitted, check the build and provider listing before changing SIP. Do not disable SIP merely to obtain a larger trace; doing so changes system security and requires Apple's supported recovery procedure. + +Use `sudo` for the DTrace commands, trace a process you own, and avoid including secrets in either command arguments or traced program output. macOS may still refuse tracing protected, hardened, or otherwise restricted processes even with `sudo`; use a local non-protected interpreter for this repository. A missing `function-entry`, `function-return`, or `line` probe is different from a `sudo` failure: the former is a Python build/version issue, while the latter is an authorization or system-policy issue. + +## DTrace versus the repository's other profilers + +| Tool | Best question | Difference from these DTrace scripts | +| --- | --- | --- | +| `call_stack.d` / `lines.d` | What exact Python events and source lines occurred, and in what order? | macOS DTrace; requires a `--with-dtrace` CPython, `sudo`, and emits high-volume text. | +| `python -m cProfile -o profile.out program.py` + `python -m pstats profile.out` | Which functions accumulated CPU/cumulative time and how many calls? | Deterministic Python profiler with an aggregate report; no DTrace build or root tracing required. | +| `pyinstrument program.py` | Which call stacks are hot over time with sampling overhead? | Statistical sampling rather than an event for every function/line. | +| `python run_yappi.py` | Function timing and call statistics (the repository's Yappi example) | Aggregated profiler output, not a source-line event stream. | +| `python run_tracemalloc.py` | Where Python allocations grew between snapshots? | Allocation tracking; DTrace scripts do not identify memory growth. | +| `memray run program.py`, `python run_guppy3.py`, `fil-profile run program.py` | Native/Python allocation or heap shape | Memory-oriented reports/flamegraphs; not call/line chronology. | +| `python run_psutil.py` | Process CPU and RSS/page-fault snapshots | Coarse resource measurements; no CPython probe stream. | + +For a first pass on a performance regression, use an aggregate or sampling profiler. Use DTrace after you have a bounded hypothesis that requires chronology or exact event counts. + +## Cleanup + +DTrace instrumentation from `-c` is attached only for that command; it does not permanently modify Python or leave probes enabled after the target exits. If you started the probe-listing shell, return it to the foreground with `fg` and press `Ctrl+D` as shown above. If a run was interrupted, confirm that the child is no longer running before starting another trace. Remove any redirected `*.trace` files when they contain sensitive output, and leave the virtual environment with: + +```bash +deactivate +``` + +No DTrace-specific Python package needs to be uninstalled. Keep the instrumented pyenv build if you will trace again; otherwise switch `pyenv local` to the interpreter you normally use rather than expecting a runtime option to disable instrumentation. diff --git a/.claude/skills/python-memory-profiling/SKILL.md b/.claude/skills/python-memory-profiling/SKILL.md new file mode 100644 index 0000000..ceb4a40 --- /dev/null +++ b/.claude/skills/python-memory-profiling/SKILL.md @@ -0,0 +1,295 @@ +# Python memory profiling + +Use this skill when Claude needs to explain, reproduce, or investigate memory growth in this repository's Python example. The target is `python/program.py`, a CPython 3.11.5 program that downloads a Wikipedia page, writes a temporary copy, parses it with BeautifulSoup, and counts words. Its network response and import/cache state can vary, so compare runs made with the same interpreter, dependencies, input/network conditions, and workload. + +## Prepare a reproducible run + +The repository pins Python 3.11.5 in `python/.python-version` and the profiling dependencies in `python/requirements.txt` (`guppy3==3.1.4.post1`, `memray==1.11.0`, `psutil==5.9.8`, and `filprofiler==2023.3.1`). The README setup is: + +```bash +cd python +pyenv install "$(cat .python-version)" +pyenv local +python -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +``` + +Run from `python/`, not the repository root: the runner scripts import `program` as a local module. The program performs a live HTTP request, so a failed request is an input/environment failure rather than a profiler result. Do not treat one run's absolute byte counts as a benchmark; repeat the same command when comparing a change. + +Before changing a profiler runner, inspect the existing scripts. They intentionally print a simple before/after view: + +- `python/run_tracemalloc.py`: starts `tracemalloc`, snapshots before/after `main()`, and prints `snapshot2.compare_to(snapshot1, "lineno")`. +- `python/run_guppy3.py`: prints `hpy().heap()` before and after `main()`. +- `python/run_psutil.py`: prints `psutil.Process().memory_info()` before and after `main()`. + +The runner imports `program` before starting tracemalloc. If import-time allocations matter, start tracing before imports with the interpreter option (the program itself still remains unchanged): + +```bash +cd python +python -X tracemalloc=25 run_tracemalloc.py +# Equivalent environment form: +PYTHONTRACEMALLOC=25 python run_tracemalloc.py +``` + +## First decide what “memory” means + +| Question | Tool and repository command | Artifact/view | Main limitation | +|---|---|---|---| +| Which Python source lines retain or allocate traced Python blocks between two points? | `python run_tracemalloc.py` | Text top-10 diff; optional serialized snapshots | Does not see native allocations that bypass Python's traced allocators; not RSS | +| Which Python/native call stacks account for allocations and peak/live memory? | `memray run program.py`; then `memray flamegraph ...` | Memray binary plus HTML flame graph/table | Higher runtime/storage overhead; native symbol reports are machine-specific | +| Which Python objects and types exist, and how large are they now? | `python run_guppy3.py` | Heapy object census on stdout | No allocation history or source-line attribution; Python objects only | +| How much memory does the OS report for this process? | `python run_psutil.py` | `memory_info()` text (RSS/VMS and platform fields) | No allocation stack or object ownership; RSS includes shared/native/interpreter memory | +| What call stacks account for memory at the high-water mark, including native extensions? | `fil-profile --no-browser run program.py` | Interactive SVG report in `fil-result/` | Offline profiler with substantial overhead; not a production or multiprocessing profiler | + +Do not add numbers from these tools together. A `tracemalloc` size, a Memray allocation, a Heapy object size, and RSS measure different populations and points in time. Use a process-level tool (psutil or Fil) to establish impact, then an allocation/object tool to explain it. + +## `tracemalloc`: Python allocation snapshots and diffs + +Use the standard-library `tracemalloc` when a suspected growth is in Python-managed objects and source-line attribution is useful. The repository's exact baseline command is: + +```bash +cd python +python run_tracemalloc.py +``` + +The existing runner does this around `main()`: + +```python +import tracemalloc +from program import main + +tracemalloc.start() +before = tracemalloc.take_snapshot() +main() +after = tracemalloc.take_snapshot() +for stat in after.compare_to(before, "lineno")[:10]: + print(stat) +``` + +Interpret output as `size=... (+/-...), count=... (+/-...), average=...`. Positive entries are more traced blocks/bytes in the second snapshot; negative entries indicate blocks freed between snapshots. The `"lineno"` key groups by filename and line. For call-chain attribution, start with more frames and group by traceback: + +```python +tracemalloc.start(25) +# ... code under test ... +snapshot = tracemalloc.take_snapshot() +for stat in snapshot.statistics("traceback")[:10]: + print(stat) + for frame in stat.traceback.format(): + print(" ", frame) +``` + +Avoid blaming import overhead or profiler bookkeeping. Start the snapshot immediately before the operation being compared, and optionally collect first if the question is about retained objects rather than transient garbage: + +```python +import gc + +gc.collect() +before = tracemalloc.take_snapshot() +main() +gc.collect() +after = tracemalloc.take_snapshot() +``` + +Filter third-party noise before ranking application lines: + +```python +only_program = snapshot.filter_traces( + (tracemalloc.Filter(True, "*program.py"),) +) +for stat in only_program.statistics("lineno")[:10]: + print(stat) +``` + +For a temporary allocation/high-water check, `get_traced_memory()` returns `(current, peak)` bytes for traced blocks. Reset the peak immediately before the operation if the process has already done setup: + +```python +tracemalloc.reset_peak() +main() +current, peak = tracemalloc.get_traced_memory() +print(f"traced current={current / 1024**2:.2f} MiB peak={peak / 1024**2:.2f} MiB") +``` + +Snapshots can be retained for offline comparison without keeping both large objects in memory: + +```python +before.dump("/tmp/program-before.snapshot") +after.dump("/tmp/program-after.snapshot") +# In a later process: +import tracemalloc +old = tracemalloc.Snapshot.load("/tmp/program-before.snapshot") +new = tracemalloc.Snapshot.load("/tmp/program-after.snapshot") +for stat in new.compare_to(old, "lineno")[:10]: + print(stat) +``` + +`tracemalloc` records Python allocation traces, not the process's resident memory. A large RSS increase can come from a C/C++ extension, a shared-library mapping, allocator arenas retained for reuse, memory-mapped files, or the OS; none necessarily appears in a `tracemalloc` diff. Conversely, a positive traced diff can be reclaimed later and need not explain peak RSS. The tracing tables and traceback metadata also impose runtime and memory overhead, increasing with the frame limit; use the smallest frame depth that answers the question. + +Save text outside the repository while experimenting: + +```bash +python run_tracemalloc.py | tee /tmp/program-tracemalloc.txt +``` + +## Memray: allocation lifetimes and native call stacks + +Use Memray when Python-only attribution is insufficient, especially when an extension or allocator is involved, or when you need a peak/live allocation flame graph. The README's default workflow is: + +```bash +cd python +memray run program.py +memray flamegraph memray-program.py.*.bin +open memray-flamegraph-program.py.*.html # macOS +``` + +Prefer deterministic paths for automation and artifact handoff: + +```bash +memray run --force --output /tmp/program-memray.bin program.py +memray stats /tmp/program-memray.bin +memray summary /tmp/program-memray.bin +memray flamegraph --force --output /tmp/program-memray.html /tmp/program-memray.bin +open /tmp/program-memray.html +``` + +`memray run` starts a new process and records allocation/deallocation events. The default capture name is `memray-