[ankitjoju-google]

Adds a self-contained, cgo-free binary, gpu-bug-report, that ships in the existing device plugin image and runs as a second container in the nvidia-gpu-device-plugin DaemonSet pod. It watches its own Node and, on request (either manually via a Node label or automatically via the Maintenance Handler status annotation handshake), executes the host's nvidia-bug-report.sh and publishes the result metadata on the Node. The existing nvidia-gpu-device-plugin binary is untouched.

Supported Features

1. On-Demand Manual Trigger via Node Label (pkg/gpu/nvidia/diagnostics/controller.go)

   • Setting cloud.google.com/trigger-gpu-report (any value except "false") on a GPU node triggers a bug-report collection on that node.
   • CAS Label Removal & Verified Precondition Comparison: Clears the trigger label via an RFC 6902 JSON Patch guarded by a test op on the observed label value. If a patch precondition fails, the controller fetches the live Node and compares the current TriggerLabel and StatusAnnotation values against the expected test values so only the non-conflicting fields are updated on retry.

2. Automatic Maintenance Handler Handshake (pkg/gpu/nvidia/diagnostics/controller.go)

   • Status Annotation State Machine (cloud.google.com/gpu-report-status):
     • collect → in-progress: Acknowledges an automatic request from the Maintenance Handler via an RFC 6902 test + replace CAS patch.
     • in-progress → completed or failed: <reason>: Publishes the terminal status and cloud.google.com/gpu-report-output atomically when collection finishes (on both manual and automatic runs so status and output annotations never drift).
   • Opt-Out Label (cloud.google.com/disable-gpu-report):
     • Presence-based (active unless value is "false"). When active, automatic collect requests are immediately transitioned to failed: disabled (with a Warning Event) without running the script, while manual triggers remain allowed.
   • Startup Deadlock Recovery (recoverStartup):
     • On startup, if cloud.google.com/gpu-report-status is left at in-progress from a prior container crash or OOM kill, the controller CAS-patches it to failed: interrupted and emits a Warning Event so the Maintenance Handler never waits for a dead run.
   • Workqueue Coalescing & Retry Without Re-Execution:
     • Watch events feed a k8s.io/client-go/util/workqueue (RateLimitingInterface) worker that always reconciles against a fresh GET of the Node. Concurrent manual and automatic requests share a single collection run, and transient Get/Patch errors retry with exponential backoff via pendingResult without re-running nvidia-bug-report.sh.

3. Host-Isolated Execution from a Distroless Container (pkg/gpu/nvidia/diagnostics/collector.go)

   • Executes the host's nvidia-bug-report.sh via syscall.SysProcAttr{Chroot: <host-root>}, probing /home/kubernetes/bin/nvidia/bin, /opt/nvidia/bin, and /usr/bin and automatically deriving PATH and LD_LIBRARY_PATH.
   • Stages execution inside <reports-dir>/.work ($TMPDIR) and atomically renames only the finished report into /home/kubernetes/bin/nvidia-bug-reports/nvidia-bug-report-<UTC-timestamp>-<manual|automatic>.log.gz (0700 permissions), removing .work afterwards. This directory is sibling to—never inside—/home/kubernetes/bin/nvidia, ensuring diagnostic reports are never mounted into user GPU workloads.

4. Result Publishing & Observability (pkg/gpu/nvidia/diagnostics/types.go)

   • Node Annotation: Writes structured JSON metadata to cloud.google.com/gpu-report-output:

     {
       "status": "completed",
       "trigger": "manual",
       "filePath": "/home/kubernetes/bin/nvidia-bug-reports/nvidia-bug-report-20260930T221010Z-manual.log.gz",
       "fileSizeBytes": 558947,
       "complete": true,
       "durationSeconds": 6.739,
       "collectedAt": "2026-09-30T22:10:10Z"
     }

   • Kubernetes Node Events: Emits Normal/GPUReportCompleted or Warning/GPUReportFailed events on the Node (visible in kubectl describe node).

5. Process, Log-Age, Disk & Memory Safety (pkg/gpu/nvidia/diagnostics/collector.go)

   • Journal Log-Age Cap (--max-log-age, default 168h / 7 days): Places a lightweight journalctl wrapper inside <reports-dir>/.work/journalctl at the front of PATH that injects --since="<start - max-log-age>". Because nvidia-bug-report.sh scans journalctl -b twice per boot without a time bound, capping journalctl to the last 7 days uses systemd's timestamp index to keep log scanning fast on long-lived nodes and prevents false failed: timeout runs.
   • Timeout & Process-Group Termination: Enforces a configurable timeout (default 60s) with Setpgid: true and sends SIGKILL to the entire process group (-pgid) so hung nvidia-smi children cannot leak or hold /dev/nvidia* open.
   • D-State Protection: If a kernel-wedged GPU leaves a child in uninterruptible sleep (D state) after SIGKILL, the collector marks itself busy and refuses further runs rather than accumulating stuck processes.
   • Streaming Gzip & Marker Validation: Streams the .log.gz output to verify both a clean gzip EOF and the "End of NVIDIA bug report log file" footer (since nvidia-bug-report.sh can exit 0 even when subcommands fail or truncate). Partial reports on timeout are retained on disk with "complete": false.
   • Disk Rotation & Cleanup: Retains the newest 5 reports (--max-reports=5) and removes the .work staging directory after each run.
   • Memory Reclamation:
     • Redirects TMPDIR to <reports-dir>/.work on disk so nvidia-debugdump temporary archives do not allocate unreclaimable tmpfs (shmem) inside the container's cgroup.
     • Calls fsync + unix.Fadvise(..., unix.FADV_DONTNEED) on the finished report to immediately evict its page cache from the container's memory cgroup.
     • Bounds stderr capture to a 4 KiB ring buffer (tailBuffer) and calls debug.FreeOSMemory() after each run.
   • Dependency Guard (cmd/gpu_bug_report/deps_test.go): Built statically with CGO_ENABLED=0; unit test enforces that the binary never imports google.golang.org/grpc, github.com/NVIDIA/* (NVML), or runtime/cgo.

---

Performance & Resource Benchmarks

Validated end-to-end on GKE (COS) across four GPU machine families (x86_64 and ARM64 baremetal):

Machine Type	GPUs	Script Duration	End-to-End Latency	Report Size (.log.gz / Raw)	Steady-State Memory (current / anon)	Peak Memory (memory.peak / anon)	CPU Used
g4-standard-48	1× RTX PRO 6000 Blackwell	6.7–6.8s	7.5–8.2s	0.56 MB / 1.0 MB	5.8–6.2 MiB / 5.1–5.4 MiB	26.6–27.3 MiB / 21.6 MiB	4.8s
g4-standard-384	8× RTX PRO 6000 Blackwell	6.4–8.0s	~8.5s	5.8–9.6 MB / 9.7–12.9 MB	~4.1–4.3 MiB*	34.0–38.0 MiB / 18.9 MiB	3.1–4.6s
a4-highgpu-8g	8× B200 Blackwell	13.3–13.6s	~14.5s	7.2 MB / 14.6 MB	—	—	—
a4x-maxgpu-4g-metal	4× GB300 Grace Blackwell Ultra (ARM64)	17.6–18.8s	~18.5–19.5s	4.5–5.5 MB	14.1 MiB / 8.3 MiB	127.6 MiB / 13.4 MiB (24.1 MiB kernel)	—

*Note: g4-standard-384 steady-state memory was measured prior to switching from the stdlib HTTP client to client-go (+3.5–4 MiB idle anon overhead).

Key Performance Observations

1. Collection Time Scales Sub-Linearly with GPU Count & Node Uptime:
   • Going from 1× to 8× RTX PRO 6000 GPUs keeps runtime nearly flat (6.7s vs 6.4–8.0s), 8× B200 with NVLink 5 completes in 13.3–13.6s, and 4× GB300 (a4x-maxgpu-4g-metal ARM64 baremetal) completes in 17.6–18.8s—all well within the default 60s timeout.
   • On an 8.75-day-old a4x-maxgpu-4g-metal node (325.6 MB journal), --max-log-age=168h cleanly excluded boot logs older than 7 days via systemd's timestamp index.
2. Impact of nvidia-persistenced on Multi-GPU NVLink Nodes (8× B200):
   • With nvidia-persistenced active (standard on GKE GPU nodes with the device plugin), 8× B200 collection takes 13.3s.
   • When tested without persistence mode (Persistence Mode: Disabled), collection took 97.0s (~7.3× slower) because each sequential nvidia-smi / nvidia-debugdump invocation (~9 separate calls in nvidia-bug-report.sh) had to cold-initialize all 8 GPUs, load the GSP firmware, and retrain NVLink on every command:
     • nvidia-smi -q: 10.39s (cold) → 0.73s (with persistence mode, 14× faster)
     • nvidia-smi nvlink --status: 9.67s → 0.20s (48× faster)
     • nvidia-smi nvlink --errors: 9.68s → 0.09s (100× faster)
     • nvidia-smi topo -m: 11.14s → 1.60s (7× faster)
3. Process Memory & 144-Core Grace ARM64 (64 KB Page Size) Kernel Overhead:
   • Per-process sampling (8ms interval) showed that peak anonymous memory from child processes is flat at ~13–19 MiB across 1-GPU, 8-GPU, and 4× GB300 nodes (only one ~15 MiB NVIDIA CLI tool runs at a time).
   • Redirecting TMPDIR off /tmp (tmpfs) to <reports-dir>/.work: Eliminated 8.6–9.6 MiB of unreclaimable shmem created by nvidia-debugdump temporary zip archives on 8-GPU nodes, with no measurable runtime penalty.
   • fadvise64(POSIX_FADV_DONTNEED) after report completion: Evicts finished report page cache immediately, keeping post-run file cache at 64–140 KiB.
   • Grace ARM64 (a4x-maxgpu-4g-metal) Memory Limit Sizing (128 MiB): On 144-core, 34-NUMA-node Grace ARM64 baremetal nodes running a 64 KB page size kernel (KernelPageSize: 64 kB), unreclaimable cgroup kernel memory (percpu: 7.3 MiB + 64 KB pagetables: 3.7 MiB + kernel_stack: 0.8 MiB + slab: 0.9 MiB) reaches 24.1 MiB during script execution. Setting limits.memory: 128Mi (while keeping requests.memory: 10Mi) provides ample headroom above the ~37.5 MiB anon + kernel working set.
4. Report Completeness vs. Native Host Execution:
   • Comparing the container-generated report against a native host run on 8-GPU nodes confirmed all 137 sections (and 143 sections on 8× B200) match, including per-GPU nvidia-smi -q, ECC/row-remapper states, VBIOS/GSP versions, /proc/driver/nvidia/gpus/*, lspci, kernel/NVRM logs, and a valid multi-GPU nvidia-debugdump zip archive.

---

Deployment Requirements (for GKE Manifest Integration)

• Container: Second container in the device plugin pod running /usr/bin/gpu-bug-report --host-root=/root, privileged: true, readOnlyRootFilesystem: true, GOMAXPROCS=1, with 5m CPU / 10Mi memory request and 128Mi memory limit.
• Mounts: Host / mounted read-only at /root, and host /home/kubernetes/bin mounted read-write at /root/home/kubernetes/bin (reusing the pod's existing root-mount and kubernetes-bin volumes).
• Pod Settings: hostPID: true and hostNetwork: true (already enabled on the device plugin pod) so systemctl status nvidia-persistenced and host hostname resolve from the host namespace; NODE_NAME passed via the downward API.
• Permissions: Uses the existing gpu-device-plugin-sa ServiceAccount (get/watch/patch on nodes, create/patch on events, and own-node ValidatingAdmissionPolicy).

[Jiaqicao257]

cloud.google.com/trigger-gpu-diagnostics can we rename this to be gpu-report to avoid conflict with diagnostic tests?

[ankitjoju-google]

cloud.google.com/trigger-gpu-diagnostics can we rename this to be gpu-report to avoid conflict with diagnostic tests?

Sounds good! Renamed all four labels/annotations and the Node event reasons to variations of gpu-report.

[leiyiz]

Switching finishCollection to context.WithoutCancel(ctx) ensures StatusAnnotation transitions out of "in-progress" (-> "failed: interrupted") on SIGTERM so the maintenance handler does not block. However, p.wantManual is still unconditionally set to wantManual:

• If a DaemonSet rolling update or pod eviction sends SIGTERM while a manual (or coalesced) collection is running, Collector.Run aborts with ReasonInterrupted, and finishCollection removes TriggerLabel from the Node. When the replacement pod starts moments later, the manual trigger is gone and never fulfilled.
• Additionally, unconditionally wrapping finishCtx with a 5s timeout even when ctx.Err() == nil caps total API requests (Patch + Get conflict retries) during normal operation to 5 seconds total.

[ankitjoju-google]

Good catch! Updated in c10849b:

1. pending.wantManual is now wantManual && res.Reason != ReasonInterrupted, so when a run is interrupted by SIGTERM, finishCollection still transitions cloud.google.com/gpu-report-status to "failed: interrupted" while leaving cloud.google.com/trigger-gpu-report on the Node for the replacement pod to fulfill.
2. We now only wrap finishCtx in the 5s context.WithoutCancel(ctx) deadline when finishCtx.Err() != nil, so normal runs aren't capped to 5s.

[ankitjoju-google]

Follow-up: I tested c10849b end-to-end on A4X Max (GB300, arm64) nodes:

• Pod deleted (SIGTERM) 4s into a manual run: the node went to gpu-report-status: failed: interrupted, with output {"status":"failed","reason":"interrupted","trigger":"manual","complete":false}. cloud.google.com/trigger-gpu-report=true was still on the Node after the old pod exited. The replacement pod picked up the label, finished with completed in 20.9s, and removed the label.
• That run also showed a separate bug: the old pod's GPUReportFailed "interrupted" event never reached the API server. The broadcaster writes events on a background goroutine, and Shutdown() only waits until they are queued, so the process exited first. Fixed in d4ebdf5: main now tracks in-flight event sink writes and waits for them (bounded to 5s) before exiting. Unit tests cover the flush case and the hung-API-server timeout.

[leiyiz]

I think we can submit probably. it's your call i don't think they are too big of issues.