From 4ac38479d2f1f366a8e97ea17354e6c44c1792dd Mon Sep 17 00:00:00 2001 From: Tuomas Katila Date: Mon, 31 Aug 2026 15:22:18 +0300 Subject: [PATCH] recovery: add the GPURecoveryPlan API, CRDs and documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce the cluster-scoped GPURecoveryPlan kind: a plan per GPU model, an admin-approval model over the recovery events the operator reports, and the reset/reflash types it can run. This is the API surface only, so it can be reviewed before the implementation lands. There is no controller, no webhook and no kubectl plugin here, and the CRD is not yet reconciled by anything — RECOVERY.md describes the whole feature, including the parts (metrics, the kubectl gpurecovery plugin) that arrive with the implementation. Signed-off-by: Tuomas Katila --- Makefile | 1 + PROJECT | 8 + RECOVERY.md | 230 ++++++++ api/v1alpha1/gpurecoveryplan_types.go | 452 ++++++++++++++ api/v1alpha1/groupversion_info.go | 2 + api/v1alpha1/zz_generated.deepcopy.go | 369 ++++++++++++ .../crds/gpurecoveryplans.yaml | 550 ++++++++++++++++++ .../crd/bases/intel.com_gpurecoveryplans.yaml | 550 ++++++++++++++++++ config/crd/kustomization.yaml | 1 + config/rbac/gpurecoveryplan_admin_role.yaml | 27 + config/rbac/gpurecoveryplan_editor_role.yaml | 33 ++ config/rbac/gpurecoveryplan_viewer_role.yaml | 29 + config/rbac/kustomization.yaml | 3 + .../samples/recoveryplan/gpurecoveryplan.yaml | 88 +++ .../samples/recoveryplan/kustomization.yaml | 4 + 15 files changed, 2347 insertions(+) create mode 100644 RECOVERY.md create mode 100644 api/v1alpha1/gpurecoveryplan_types.go create mode 100644 charts/gpu-base-operator/crds/gpurecoveryplans.yaml create mode 100644 config/crd/bases/intel.com_gpurecoveryplans.yaml create mode 100644 config/rbac/gpurecoveryplan_admin_role.yaml create mode 100644 config/rbac/gpurecoveryplan_editor_role.yaml create mode 100644 config/rbac/gpurecoveryplan_viewer_role.yaml create mode 100644 config/samples/recoveryplan/gpurecoveryplan.yaml create mode 100644 config/samples/recoveryplan/kustomization.yaml diff --git a/Makefile b/Makefile index 5e1ac01..32c46cc 100644 --- a/Makefile +++ b/Makefile @@ -403,6 +403,7 @@ endif crdsync: generate manifests cp config/crd/bases/intel.com_clusterpolicies.yaml charts/gpu-base-operator/crds/clusterpolicies.yaml cp config/crd/bases/intel.com_gpufirmwareupdates.yaml charts/gpu-base-operator/crds/gpufirmwareupdates.yaml + cp config/crd/bases/intel.com_gpurecoveryplans.yaml charts/gpu-base-operator/crds/gpurecoveryplans.yaml .PHONY: check-generated-files check-generated-files: crdsync diff --git a/PROJECT b/PROJECT index db0b0e7..e45c2d8 100644 --- a/PROJECT +++ b/PROJECT @@ -30,4 +30,12 @@ resources: webhooks: validation: true webhookVersion: v1 +- api: + crdVersion: v1 + namespaced: false + controller: true + domain: intel.com + kind: GPURecoveryPlan + path: github.com/intel/gpu-base-operator/api/v1alpha1 + version: v1alpha1 version: "3" diff --git a/RECOVERY.md b/RECOVERY.md new file mode 100644 index 0000000..48ec54c --- /dev/null +++ b/RECOVERY.md @@ -0,0 +1,230 @@ +# GPU Recovery + +In some cases the Intel GPU needs to go through a low-level recovery operations which can be of the following kinds: + +* **Inoperative state** — every operation on the GPU fails, but the device is still visible to the `xe` driver. + A harder, PCIe-level reset is required. +* **Corrupted firmware** — card boots up into FDO (survivability) mode where only the PCIe link is + active and needs to be reflashed with an up to date firmware image. + +Recovering from either case is potentially disruptive: a PCIe reset can disturb other devices on the +same host, or wedge the host itself. The operator therefore never resets a GPU on its own — it +detects the need, reports it, and waits for a cluster admin to approve the operation. + +Recovery is driven by the cluster-scoped `GPURecoveryPlan` CRD. + +## How it works + +``` +GPU fault → Xe KMD → xpumd → GPU DRA driver → ResourceSlice device taint + ↓ + operator detects taint, adds a recovery event + ↓ + admin approves it (spec.approvals) + ↓ + node drain (resets only) → recovery Job runs xpu-smi + ↓ + taint clears → event removed, plan back to idle +``` + +The operator watches `ResourceSlice` device taints published by the GPU DRA driver: + +| Taint key | Meaning | Recovery | +|---|---|---| +| `health-xpumd-gpu.wedged` | GPU wedged at runtime | reset (`spec.defaultResetType`) | +| `health-Survivability` | card booted into survivability/FDO mode | `reflash` | +| `health-xpumd-gpu.survivability` | card fell into survivability at runtime | `reflash` | + +Each affected GPU gets one entry in `status.events` with a deterministic ID +(`evt---`). A device whose condition worsens (reset → reflash) is *escalated in +place*, which regenerates the event ID so an approval for the lighter operation cannot silently +authorise the heavier one. + +Recovery Jobs run `xpu-smi` from the image in `spec.xpuSmi`, privileged, pinned to the affected node, +and are retained after completion for post-mortem diagnostics (they are removed when the event is). + +### Reset types + +| Type | Command | Notes | +|---|---|---| +| `slot` | `xpu-smi config -d --coldreset` | PCIe slot power cycle; requires PCIe hot-plug support | +| `amc` | `xpu-smi amc --gpuReset -d ` | Out-of-band reset through the card's AMC | +| `sbr` | `xpu-smi config -d --reset` | Secondary Bus Reset; per-card backup, via an approval override only | +| `reflash` | `xpu-smi updatefw -d -t FDO -f ` | Flash the known good firmware onto a card in FDO mode | + +These are **not** a severity ladder. Exactly one of `slot` and `amc` works on a given platform — +slot where the PCIe slots do hot-plug, AMC where they do not — and the DRA driver can only say "this +device needs a reset", not which mechanism applies. That is why `spec.defaultResetType` is mandatory +with no default: a reset the platform cannot perform **exits 0**, so a wrong value produces a clean +run to `succeeded` over a GPU that was never touched. + +### Event states + +| State | Meaning | +|---|---| +| `waiting-approval` | Detected, waiting for `spec.approvals`. Also where an event lands when its pre-flight image check failed | +| `missing-firmware` | Reflash needed but `spec.firmware` is absent, empty or volume-sourced | +| `blocked` | Approved, but another recovery is already running on this node. The approval is kept, so it resumes on its own | +| `draining` | Node is being drained before a reset | +| `in-progress` | Recovery Job is running | +| `succeeded` / `failed` | Job finished. A failure inside `spec.maxRetries` is re-queued for approval | + +`status.events[].stateMessage` explains any state that is not self-explanatory (which recovery holds +the node, which image cannot be pulled, what a stalled drain was waiting on). An empty value means +there is nothing to add. + +The plan-level `status.state` is `idle`, `active`, or `error`. `error` means an admin is needed: an +event out of retries, or one blocked on missing firmware. + +## Safety mechanisms + +* **Admin approval** for every recovery, singular or by selector (see below). +* **Node drain before a reset** (`spec.drain`). The whole node is drained, not just GPU pods, because + an SBR or slot reset can wedge the host. A reflash never drains — it writes firmware without + driving the bus. The drain uses the Eviction API, so PodDisruptionBudgets are honoured, and is + bounded by `spec.drain.timeoutSeconds`. The operator's own namespace, DaemonSet pods, static pods + and already-terminated pods are always skipped; `spec.drain.namespacesToSkip` adds to that. +* **Job deadlines** (`spec.timeouts`). Each recovery Job carries an `activeDeadlineSeconds`: + `spec.timeouts.resetSeconds` (default 300) for the resets, `spec.timeouts.reflashSeconds` + (default 600) for a reflash, which also covers copying the firmware out of the known good firmware + image. They are configurable because how long the hardware takes is a property of the platform, and + a deadline that expires early kills the Job mid-operation and fails the event over a card that was + recovering — so raise them where a reset or flash is known to be slow. +* **In-use check.** A reset waits until every `ResourceClaim` reserving the GPU is released. Claims + the drain can never release are excluded (`adminAccess` claims, and claims held only by pods the + drain leaves in place) — waiting for those would deadlock the recovery. +* **One recovery per node at a time.** A second approved event on the same node is held in `blocked` + rather than run concurrently: a PCIe reset during another card's firmware write can leave that card + unrecoverable. +* **Pre-flight image verification.** `spec.xpuSmi.image` and the firmware image are resolved + against their registries before a Job is created, so a mistyped reference does not produce a Job + that reports `in-progress` from `ImagePullBackOff`. Checked once per spec generation; disable with + `spec.skipImageVerification` where nodes hold pull credentials the operator cannot see. +* **Retry budget.** `spec.maxRetries` bounds automatic retries; an exhausted event needs an explicit + per-event re-approval, so a standing group approval cannot loop a dying card forever. +* **Finalizer.** Deleting a plan blocks until every recovery Job is terminal, and releases all drain + taints first. +* **The operator tolerates its own drain taint**, since it is the only thing that removes it. + +## Example plan + +```yaml +apiVersion: intel.com/v1alpha1 +kind: GPURecoveryPlan +metadata: + name: recoveryplan-bmg +spec: + deviceId: "0xe20b" # mandatory; one plan per GPU model + defaultResetType: "slot" # mandatory: "slot" or "amc" + maxRetries: 3 + + drain: + enable: true + timeoutSeconds: 300 + namespacesToSkip: ["kube-system"] + + timeouts: # how long a recovery Job itself may run + resetSeconds: 300 + reflashSeconds: 600 + + xpuSmi: + image: "docker.io/intel/xpu-smi:devel" + pullPolicy: "IfNotPresent" + + firmware: # required for reflash recovery + source: + containerSource: + name: "docker.io/intel/intel-gpu-fw-binaries:devel" + file: "fdo_firmware.bin" +``` + +Heterogeneous clusters use one plan per device ID. A full, commented sample lives in +[`config/samples/recoveryplan/gpurecoveryplan.yaml`](config/samples/recoveryplan/gpurecoveryplan.yaml). + +## Approving recovery + +An approval either names one event or selects a group. Selector approvals are one-shot unless marked +`persistent: true`. `override` changes the recovery type that will actually run, and the original +suggestion is kept in `status.events[].recoveryType.suggestedType` for audit. + +```yaml +spec: + approvals: + # one specific event + - eventId: evt-node05-slot-02-00-0 + + # escalate one card to a Secondary Bus Reset + - eventId: evt-node03-slot-02-00-0 + override: + recoveryType: sbr + + # all current reset events on nodes labelled rack=rack-04-32 + - selector: + recoveryType: slot + nodeSelector: + rack: rack-04-32 + + # standing order: auto-approve future slot resets (CI/test clusters) + - selector: + recoveryType: slot + persistent: true +``` + +The operator fills in a missing `id`, and marks non-persistent approvals `consumed: true` once acted +upon; consumed entries stay as an audit trail and can be removed by hand. + +### kubectl plugin + +`kubectl-gpurecovery` wraps the patching, with tab completion for plan names, event IDs and approval +IDs: + +```sh +make install-kubectl-plugin # builds to bin/ and installs to ~/.local/bin +make setup-completion # optional: shell completion + +kubectl gpurecovery plans +kubectl gpurecovery events +kubectl gpurecovery messages +kubectl gpurecovery approvals +kubectl gpurecovery approve +kubectl gpurecovery confirm [--persistent] +kubectl gpurecovery remove +``` + +Plain `kubectl patch` works too, e.g.: + +```sh +kubectl patch gpurecoveryplan --type=json \ + -p='[{"op":"add","path":"/spec/approvals/-","value":{"eventId":""}}]' +``` + +## Metrics + +Exposed on the operator's own `/metrics` endpoint (there is deliberately no `status.stats` field — +the CR holds current state, the counters hold history): + +| Metric | Labels | +|---|---| +| `gpu_recovery_events_total` | `plan`, `type`, `reason` | +| `gpu_recovery_attempts_total` | `plan`, `type` | +| `gpu_recovery_outcomes_total` | `plan`, `type`, `result` | +| `gpu_recovery_overrides_total` | `plan`, `suggested_type`, `chosen_type` | +| `gpu_recovery_events` (gauge) | `plan`, `node`, `type`, `state` | +| `gpu_recovery_plan_state` (gauge) | `plan`, `state` | + +## Current limitations + +* **Sibling devices are not protected.** Nothing stops a slot reset or SBR on one card from + disturbing another card behind the same PCIe root port. Drain the node and check the topology + before approving. +* **Job targeting is privileged + `nodeName`**, not a DRA claim with device tolerations and CEL + expressions as originally designed. Still an open decision. +* **`spec.subDeviceId` / `spec.subVendorId` are validated but not used for matching** — the DRA + driver does not publish those attributes yet. +* **`firmware.source.volumeSource` is not implemented.** A reflash event on a + volume-only plan stays in `missing-firmware`; use `containerSource`. +* **Reset efficacy on BMG.** On some B580 cards a reset leaves the GPU non-working; end-to-end + validation depends on driver/firmware fixes. +* **The `health-xpumd-gpu.wedged` taint key is not yet confirmed against a shipping DRA driver.** + The survivability key and the `pciId` / `pciAddress` device attributes are. + diff --git a/api/v1alpha1/gpurecoveryplan_types.go b/api/v1alpha1/gpurecoveryplan_types.go new file mode 100644 index 0000000..322c45a --- /dev/null +++ b/api/v1alpha1/gpurecoveryplan_types.go @@ -0,0 +1,452 @@ +/* +Copyright 2026 Intel Corporation. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + core "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// RecoveryType represents the type of GPU recovery operation. +// +kubebuilder:validation:Enum=sbr;slot;amc;reflash +type RecoveryType string + +// RecoveryEventState represents the lifecycle state of a recovery event. +// +kubebuilder:validation:Enum=waiting-approval;missing-firmware;blocked;draining;in-progress;succeeded;failed +type RecoveryEventState string + +// PlanState represents the overall state of the recovery plan. +// +kubebuilder:validation:Enum=idle;error;active +type PlanState string + +const ( + // Secondary Bus Reset + RecoveryTypeSBR RecoveryType = "sbr" + // Power cycle/cold reset through the PCIe slot, also called the PM reset. + RecoveryTypeSlot RecoveryType = "slot" + // Out-of-band reset through the card's AMC (Advanced Management Controller). + RecoveryTypeAMC RecoveryType = "amc" + // Reflash to bring a GPU with corrupted FW into an operational state. + RecoveryTypeReflash RecoveryType = "reflash" + + // RecoveryEventStateWaitingApproval means the event is pending admin approval. This is the first state + // for an event, but an error in event processing can also return it to this state. + RecoveryEventStateWaitingApproval RecoveryEventState = "waiting-approval" + // RecoveryEventStateMissingFirmware means reflash cannot proceed due to missing firmware. + RecoveryEventStateMissingFirmware RecoveryEventState = "missing-firmware" + // RecoveryEventStateBlocked means the event is approved but another recovery is already + // running on the same node, so this one is held back. + RecoveryEventStateBlocked RecoveryEventState = "blocked" + // RecoveryEventStateDraining means the event is approved and the node hosting the GPU is + // being drained before the reset runs. Reset-type recoveries only: a reflash writes + // firmware without resetting the PCIe bus, so it goes straight to in-progress. + RecoveryEventStateDraining RecoveryEventState = "draining" + // RecoveryEventStateInProgress means a recovery Job is currently running. + RecoveryEventStateInProgress RecoveryEventState = "in-progress" + // RecoveryEventStateSucceeded means the recovery Job completed successfully. + RecoveryEventStateSucceeded RecoveryEventState = "succeeded" + // RecoveryEventStateFailed means the recovery Job failed. + RecoveryEventStateFailed RecoveryEventState = "failed" + + PlanStateIdle PlanState = "idle" + PlanStateError PlanState = "error" + PlanStateActive PlanState = "active" +) + +// GPURecoveryPlanSpec defines the desired state of GPURecoveryPlan. +type GPURecoveryPlanSpec struct { + // DeviceID is the mandatory PCI device ID of the target GPU. Format: '0x' followed by 4 hex digits. + // +kubebuilder:validation:Pattern=`^0x[0-9a-fA-F]{4}$` + DeviceID string `json:"deviceId"` + + // SubDeviceID is the optional PCI sub-device ID. Format: '0x' followed by 4 hex digits. + // +kubebuilder:validation:Pattern=`^0x[0-9a-fA-F]{4}$` + // +optional + SubDeviceID string `json:"subDeviceId,omitempty"` + + // SubVendorID is the optional PCI sub-vendor ID. Format: '0x' followed by 4 hex digits. + // +kubebuilder:validation:Pattern=`^0x[0-9a-fA-F]{4}$` + // +optional + SubVendorID string `json:"subVendorId,omitempty"` + + // Approvals contains admin-provided authorisations for specific or grouped recovery events. + // The operator generates an ID for any entry that is missing one. + // +optional + Approvals []RecoveryApproval `json:"approvals,omitempty"` + + // XpuSmi configures the container image providing the xpu-smi tool. + // +kubebuilder:default={pullPolicy: "IfNotPresent"} + // +optional + XpuSmi XpuSmiSpec `json:"xpuSmi,omitempty"` + + // DefaultResetType is the reset the operator runs for every reset-type recovery event it + // creates on this plan. Either "slot" (PCIe slot power cycle, also called the PM reset) or + // "amc" (out-of-band reset through the card's AMC). + // +kubebuilder:validation:Enum=slot;amc + // +kubebuilder:validation:Required + DefaultResetType RecoveryType `json:"defaultResetType"` + + // Firmware holds configuration for reflash-type recovery operations. + // These fields are protected: changes are rejected while any reflash event is active. + // +optional + Firmware *FirmwareSpec `json:"firmware,omitempty"` + + // Tolerations are added to recovery Job pods on top of the blanket toleration the operator + // always sets, for cases where a cluster needs an extra entry. + // +optional + Tolerations []core.Toleration `json:"tolerations,omitempty"` + + // Drain configures the node drain that precedes a reset-type recovery. + // +kubebuilder:default={enable: true, timeoutSeconds: 300} + // +optional + Drain DrainSpec `json:"drain,omitempty"` + + // Timeouts bounds how long the recovery Jobs themselves may run. + // +kubebuilder:default={resetSeconds: 300, reflashSeconds: 600} + // +optional + Timeouts RecoveryTimeoutsSpec `json:"timeouts,omitempty"` + + // SkipImageVerification disables the pre-flight registry check on the images a recovery Job + // needs (spec.xpuSmi.image, and the firmware image for a reflash). + // +optional + SkipImageVerification bool `json:"skipImageVerification,omitempty"` + + // MaxRetries is the maximum number of times a failed recovery event is automatically + // re-queued for approval and retried while its device taint persists. Once this limit + // is reached the event stays in the failed state and requires manual intervention + // (e.g. delete the event entry or increase MaxRetries). Setting 0 disables automatic + // retries entirely. + // +kubebuilder:default=3 + // +kubebuilder:validation:Minimum=0 + MaxRetries int32 `json:"maxRetries"` +} + +// DrainSpec configures the node drain a reset-type recovery performs before it touches the +// hardware. +type DrainSpec struct { + // Enable controls whether the node is drained before a reset. + // +kubebuilder:default=true + // +optional + Enable *bool `json:"enable,omitempty"` + + // TimeoutSeconds bounds how long a reset waits for the node to drain before the event is + // failed. + // +kubebuilder:default=300 + // +kubebuilder:validation:Minimum=1 + // +optional + TimeoutSeconds int32 `json:"timeoutSeconds,omitempty"` + + // NamespacesToSkip lists namespaces the drain leaves alone: their pods are neither evicted + // nor waited for. Use it for cluster infrastructure that is fine to keep running through a + // reset and that a whole-node drain would otherwise have to evict. + // +kubebuilder:validation:MaxItems=64 + // +kubebuilder:validation:items:MaxLength=63 + // +kubebuilder:validation:items:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$` + // +optional + NamespacesToSkip []string `json:"namespacesToSkip,omitempty"` +} + +// Enabled reports whether the pre-reset drain should run. A nil Enable is enabled; see the field. +func (d *DrainSpec) Enabled() bool { + return d.Enable == nil || *d.Enable +} + +// RecoveryTimeoutsSpec bounds the runtime of the recovery Jobs, i.e. the Job's activeDeadlineSeconds. +type RecoveryTimeoutsSpec struct { + // ResetSeconds bounds a reset-type recovery Job (SBR, slot power cycle, AMC reset). + // +kubebuilder:default=300 + // +kubebuilder:validation:Minimum=1 + // +optional + ResetSeconds int32 `json:"resetSeconds,omitempty"` + + // ReflashSeconds bounds a reflash recovery Job. + // +kubebuilder:default=600 + // +kubebuilder:validation:Minimum=1 + // +optional + ReflashSeconds int32 `json:"reflashSeconds,omitempty"` +} + +// XpuSmiSpec describes the xpu-smi container image every recovery Job runs. +type XpuSmiSpec struct { + // Image is the container image providing the xpu-smi tool, used for both hardware resets + // and firmware reflash operations. + // +optional + Image string `json:"image,omitempty"` + + // PullPolicy controls the image pull policy for the xpu-smi container. + // +kubebuilder:validation:Enum=Always;IfNotPresent;Never + // +kubebuilder:default="IfNotPresent" + // +optional + PullPolicy string `json:"pullPolicy,omitempty"` + + // InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + // registry to check that Image resolves. Use it where the registry serves a self-signed or + // private-CA certificate. + // +optional + InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"` +} + +// RecoveryApproval authorises one or more recovery events. +// Exactly one of EventID or Selector must be set. +// An approval is active as long as it exists in the list and Consumed is false. +type RecoveryApproval struct { + // ID is the unique identifier for this approval entry. + // The operator generates an ID when this field is empty. + // +optional + ID string `json:"id,omitempty"` + + // EventID references a specific event by its status.events[].id. + // Mutually exclusive with Selector. + // +optional + EventID string `json:"eventId,omitempty"` + + // Selector applies this approval to all currently matching events. + // Mutually exclusive with EventID. + // +optional + Selector *ApprovalSelector `json:"selector,omitempty"` + + // Override substitutes a different recovery type than the system recommendation. + // When set on an EventID approval, the system records the original suggestion in + // status.events[].recoveryType.suggestedType for audit purposes. + // +optional + Override *RecoveryOverride `json:"override,omitempty"` + + // Persistent keeps this approval alive after matched events are processed so that + // future events matching the Selector are approved automatically. + // Only meaningful when Selector is set. + // +optional + Persistent bool `json:"persistent,omitempty"` + + // Consumed is set to true by the operator after a non-persistent approval has been + // matched and acted upon. Persistent approvals are never consumed. + // +optional + Consumed bool `json:"consumed,omitempty"` + + // Comment is an optional human-readable note about why this approval was granted. + // +optional + Comment string `json:"comment,omitempty"` +} + +// ApprovalSelector filters recovery events by type, node, or node labels. +type ApprovalSelector struct { + // RecoveryType selects events whose suggested recovery type matches. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + // +optional + RecoveryType RecoveryType `json:"recoveryType,omitempty"` + + // NodeName selects events on a specific node. + // +optional + NodeName string `json:"nodeName,omitempty"` + + // NodeSelector selects events on nodes that have all given labels. + // +optional + NodeSelector map[string]string `json:"nodeSelector,omitempty"` +} + +// RecoveryOverride allows the admin to escalate or change the suggested recovery type. +type RecoveryOverride struct { + // RecoveryType is the recovery operation to use instead of the system suggestion. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + RecoveryType RecoveryType `json:"recoveryType"` +} + +// FirmwareSpec holds everything required for a firmware reflash operation. +type FirmwareSpec struct { + // Source specifies where the firmware file can be found. + Source FirmwareSource `json:"source"` + + // File is the filename of the FDO firmware image to flash, relative to the root of the + // source (no path components). + File string `json:"file"` +} + +// FirmwareSource describes where the firmware file is located. +// At least one of ContainerSource or VolumeSource must be set. +type FirmwareSource struct { + // ContainerSource specifies a container image that holds the firmware file. + // +optional + ContainerSource *ContainerFirmwareSource `json:"containerSource,omitempty"` + + // VolumeSource specifies a Kubernetes volume that holds the firmware file. + // + // NOT YET SUPPORTED. The field is validated but not acted upon: a reflash event on a plan + // whose source sets only volumeSource stays in the missing-firmware state, because the + // reflash Job copies firmware from a container image. Use containerSource until this is + // implemented. + // +optional + VolumeSource *VolumeFirmwareSource `json:"volumeSource,omitempty"` +} + +// ContainerFirmwareSource references a container image holding the firmware file. +type ContainerFirmwareSource struct { + // Name is the container image reference (e.g. registry/image:tag). + Name string `json:"name"` + + // InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + // registry to verify this image and the firmware file inside it. Use it where the registry + // serves a self-signed or private-CA certificate. + // +optional + InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"` +} + +// VolumeFirmwareSource references a Kubernetes volume holding the firmware file. +// See FirmwareSource.VolumeSource: not yet supported. +type VolumeFirmwareSource struct { + // Name is the name of the PersistentVolumeClaim. + Name string `json:"name"` +} + +// GPURecoveryPlanStatus defines the observed state of GPURecoveryPlan. +type GPURecoveryPlanStatus struct { + // State is the overall state of this recovery plan. + // +optional + State PlanState `json:"state,omitempty"` + + // Messages contains recent informational or error messages (most recent last, capped at ~50). + // +optional + Messages []string `json:"messages,omitempty"` + + // Events is the list of active and recently completed recovery needs (capped at 1000). + // +optional + Events []RecoveryEvent `json:"events,omitempty"` +} + +// RecoveryEvent represents a single detected GPU recovery need and its lifecycle state. +type RecoveryEvent struct { + // ID is the operator-assigned unique event identifier (e.g. "evt-a3f2"). + ID string `json:"id"` + + // ApprovalID is the ID of the spec.approvals entry that matched and authorised this event. + // +optional + ApprovalID string `json:"approvalId,omitempty"` + + // ApprovalMatchedAt is the timestamp when an approval first matched this event. + // +optional + ApprovalMatchedAt *metav1.Time `json:"approvalMatchedAt,omitempty"` + + // NodeName is the Kubernetes node hosting the affected GPU. + NodeName string `json:"nodeName"` + + // GPUBDF is the PCI Bus:Device.Function address of the affected GPU (e.g. "0000:02:00.0"). + GPUBDF string `json:"gpuBDF"` + + // Reason is the human-readable cause of this event (e.g. "gpu-wedged", "survivability-mode"). + // +optional + Reason string `json:"reason,omitempty"` + + // RecoveryType describes the recovery operation to perform. + RecoveryType RecoveryTypeSpec `json:"recoveryType"` + + // State is the current lifecycle state of this recovery event. + // +kubebuilder:validation:Enum=waiting-approval;missing-firmware;blocked;draining;in-progress;succeeded;failed + State RecoveryEventState `json:"state"` + + // StateMessage explains, in one sentence, why this event is in the state it is in, where the + // state alone does not say: which other recovery is holding the node, which spec field carries + // an image that cannot be pulled, what a timed-out drain was still waiting on, which Job + // failed and on which attempt, why an approved event is unapproved again. + // +optional + StateMessage string `json:"stateMessage,omitempty"` + + // ImageVerifyGeneration is the plan's metadata.generation at the time of the last failed + // pre-flight image verification for this event, and the reason the event is back in + // waiting-approval with its approval still in place. + // The failure itself is reported in StateMessage, which is what an admin reads; this field is + // only the bookkeeping that keeps the check from repeating. Both are cleared once verification + // succeeds. + // +optional + ImageVerifyGeneration int64 `json:"imageVerifyGeneration,omitempty"` + + // DrainStartedAt is when the node drain for this event began. Reset from nil on each + // attempt so a retry gets a full drain timeout rather than inheriting the previous one. + // +optional + DrainStartedAt *metav1.Time `json:"drainStartedAt,omitempty"` + + // PodsBlockingDrain names the pods still keeping the node from being drained, in + // "namespace/name" form, so an admin can see what a stalled drain is waiting on without + // reading operator logs. Capped at a handful of entries. + // +optional + PodsBlockingDrain []string `json:"podsBlockingDrain,omitempty"` + + // ClaimsBlockingReset names the ResourceClaims that still reserve this GPU, in + // "namespace/name" form. A non-empty list means the reset is held back because a workload + // has not released the device yet. + // +optional + ClaimsBlockingReset []string `json:"claimsBlockingReset,omitempty"` + + // JobName is the name of the Kubernetes Job created to execute the recovery, if any. + // +optional + JobName string `json:"jobName,omitempty"` + + // PastJobs is the names of all Jobs created for this event across all attempts. + // Jobs are retained alive until the event is removed (i.e. the device taint clears), + // so their Pods remain available for diagnostics throughout the event lifecycle. + // +optional + PastJobs []string `json:"pastJobs,omitempty"` + + // RetryCount is the number of times this recovery has been retried after failure. + RetryCount int32 `json:"retryCount"` + + // LastUpdated is the timestamp of the most recent state change for this event. + LastUpdated *metav1.Time `json:"lastUpdated"` +} + +// RecoveryTypeSpec describes which recovery operation to perform, including any +// admin override. +type RecoveryTypeSpec struct { + // Type is the recovery operation to execute: a reset (SBR, slot power cycle, or + // AMC reset) or a firmware reflash. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + Type RecoveryType `json:"type"` + + // SuggestedType is the recovery type originally recommended by the system before + // any admin override was applied. Recorded for audit purposes. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + // +optional + SuggestedType RecoveryType `json:"suggestedType,omitempty"` +} + +// IsReflash returns true when this recovery operation is a firmware reflash rather +// than a hardware reset. +func (rt RecoveryTypeSpec) IsReflash() bool { + return rt.Type == RecoveryTypeReflash +} + +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster +// +kubebuilder:printcolumn:name="DeviceID",type=string,JSONPath=`.spec.deviceId` +// +kubebuilder:printcolumn:name="State",type=string,JSONPath=`.status.state` +// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp` + +// GPURecoveryPlan is the Schema for the gpurecoveryplans API. +type GPURecoveryPlan struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec GPURecoveryPlanSpec `json:"spec,omitempty"` + Status GPURecoveryPlanStatus `json:"status,omitempty"` +} + +// +kubebuilder:object:root=true + +// GPURecoveryPlanList contains a list of GPURecoveryPlan. +type GPURecoveryPlanList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []GPURecoveryPlan `json:"items"` +} diff --git a/api/v1alpha1/groupversion_info.go b/api/v1alpha1/groupversion_info.go index 72ff125..0dc581e 100644 --- a/api/v1alpha1/groupversion_info.go +++ b/api/v1alpha1/groupversion_info.go @@ -42,6 +42,8 @@ func addKnownTypes(s *runtime.Scheme) error { &ClusterPolicyList{}, &GPUFirmwareUpdate{}, &GPUFirmwareUpdateList{}, + &GPURecoveryPlan{}, + &GPURecoveryPlanList{}, ) metav1.AddToGroupVersion(s, GroupVersion) diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index f129114..9d0c599 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -25,6 +25,28 @@ import ( "k8s.io/apimachinery/pkg/runtime" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalSelector) DeepCopyInto(out *ApprovalSelector) { + *out = *in + if in.NodeSelector != nil { + in, out := &in.NodeSelector, &out.NodeSelector + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalSelector. +func (in *ApprovalSelector) DeepCopy() *ApprovalSelector { + if in == nil { + return nil + } + out := new(ApprovalSelector) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *BuildArg) DeepCopyInto(out *BuildArg) { *out = *in @@ -221,6 +243,21 @@ func (in *ClusterQueueSpec) DeepCopy() *ClusterQueueSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ContainerFirmwareSource) DeepCopyInto(out *ContainerFirmwareSource) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ContainerFirmwareSource. +func (in *ContainerFirmwareSource) DeepCopy() *ContainerFirmwareSource { + if in == nil { + return nil + } + out := new(ContainerFirmwareSource) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *DevicePluginSpec) DeepCopyInto(out *DevicePluginSpec) { *out = *in @@ -251,6 +288,31 @@ func (in *DevicePluginSpec) DeepCopy() *DevicePluginSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *DrainSpec) DeepCopyInto(out *DrainSpec) { + *out = *in + if in.Enable != nil { + in, out := &in.Enable, &out.Enable + *out = new(bool) + **out = **in + } + if in.NamespacesToSkip != nil { + in, out := &in.NamespacesToSkip, &out.NamespacesToSkip + *out = make([]string, len(*in)) + copy(*out, *in) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DrainSpec. +func (in *DrainSpec) DeepCopy() *DrainSpec { + if in == nil { + return nil + } + out := new(DrainSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *DynamicResourceAllocationSpec) DeepCopyInto(out *DynamicResourceAllocationSpec) { *out = *in @@ -271,6 +333,47 @@ func (in *DynamicResourceAllocationSpec) DeepCopy() *DynamicResourceAllocationSp return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *FirmwareSource) DeepCopyInto(out *FirmwareSource) { + *out = *in + if in.ContainerSource != nil { + in, out := &in.ContainerSource, &out.ContainerSource + *out = new(ContainerFirmwareSource) + **out = **in + } + if in.VolumeSource != nil { + in, out := &in.VolumeSource, &out.VolumeSource + *out = new(VolumeFirmwareSource) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new FirmwareSource. +func (in *FirmwareSource) DeepCopy() *FirmwareSource { + if in == nil { + return nil + } + out := new(FirmwareSource) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *FirmwareSpec) DeepCopyInto(out *FirmwareSpec) { + *out = *in + in.Source.DeepCopyInto(&out.Source) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new FirmwareSpec. +func (in *FirmwareSpec) DeepCopy() *FirmwareSpec { + if in == nil { + return nil + } + out := new(FirmwareSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *GPUFirmwareContent) DeepCopyInto(out *GPUFirmwareContent) { *out = *in @@ -481,6 +584,129 @@ func (in *GPUFirmwareUpdateSubsetStatus) DeepCopy() *GPUFirmwareUpdateSubsetStat return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlan) DeepCopyInto(out *GPURecoveryPlan) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlan. +func (in *GPURecoveryPlan) DeepCopy() *GPURecoveryPlan { + if in == nil { + return nil + } + out := new(GPURecoveryPlan) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *GPURecoveryPlan) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlanList) DeepCopyInto(out *GPURecoveryPlanList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]GPURecoveryPlan, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlanList. +func (in *GPURecoveryPlanList) DeepCopy() *GPURecoveryPlanList { + if in == nil { + return nil + } + out := new(GPURecoveryPlanList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *GPURecoveryPlanList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlanSpec) DeepCopyInto(out *GPURecoveryPlanSpec) { + *out = *in + if in.Approvals != nil { + in, out := &in.Approvals, &out.Approvals + *out = make([]RecoveryApproval, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + out.XpuSmi = in.XpuSmi + if in.Firmware != nil { + in, out := &in.Firmware, &out.Firmware + *out = new(FirmwareSpec) + (*in).DeepCopyInto(*out) + } + if in.Tolerations != nil { + in, out := &in.Tolerations, &out.Tolerations + *out = make([]v1.Toleration, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + in.Drain.DeepCopyInto(&out.Drain) + out.Timeouts = in.Timeouts +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlanSpec. +func (in *GPURecoveryPlanSpec) DeepCopy() *GPURecoveryPlanSpec { + if in == nil { + return nil + } + out := new(GPURecoveryPlanSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlanStatus) DeepCopyInto(out *GPURecoveryPlanStatus) { + *out = *in + if in.Messages != nil { + in, out := &in.Messages, &out.Messages + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.Events != nil { + in, out := &in.Events, &out.Events + *out = make([]RecoveryEvent, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlanStatus. +func (in *GPURecoveryPlanStatus) DeepCopy() *GPURecoveryPlanStatus { + if in == nil { + return nil + } + out := new(GPURecoveryPlanStatus) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *HealthinessSpec) DeepCopyInto(out *HealthinessSpec) { *out = *in @@ -621,6 +847,119 @@ func (in *LocalQueueSpec) DeepCopy() *LocalQueueSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryApproval) DeepCopyInto(out *RecoveryApproval) { + *out = *in + if in.Selector != nil { + in, out := &in.Selector, &out.Selector + *out = new(ApprovalSelector) + (*in).DeepCopyInto(*out) + } + if in.Override != nil { + in, out := &in.Override, &out.Override + *out = new(RecoveryOverride) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryApproval. +func (in *RecoveryApproval) DeepCopy() *RecoveryApproval { + if in == nil { + return nil + } + out := new(RecoveryApproval) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryEvent) DeepCopyInto(out *RecoveryEvent) { + *out = *in + if in.ApprovalMatchedAt != nil { + in, out := &in.ApprovalMatchedAt, &out.ApprovalMatchedAt + *out = (*in).DeepCopy() + } + out.RecoveryType = in.RecoveryType + if in.DrainStartedAt != nil { + in, out := &in.DrainStartedAt, &out.DrainStartedAt + *out = (*in).DeepCopy() + } + if in.PodsBlockingDrain != nil { + in, out := &in.PodsBlockingDrain, &out.PodsBlockingDrain + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.ClaimsBlockingReset != nil { + in, out := &in.ClaimsBlockingReset, &out.ClaimsBlockingReset + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.PastJobs != nil { + in, out := &in.PastJobs, &out.PastJobs + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.LastUpdated != nil { + in, out := &in.LastUpdated, &out.LastUpdated + *out = (*in).DeepCopy() + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryEvent. +func (in *RecoveryEvent) DeepCopy() *RecoveryEvent { + if in == nil { + return nil + } + out := new(RecoveryEvent) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryOverride) DeepCopyInto(out *RecoveryOverride) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryOverride. +func (in *RecoveryOverride) DeepCopy() *RecoveryOverride { + if in == nil { + return nil + } + out := new(RecoveryOverride) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryTimeoutsSpec) DeepCopyInto(out *RecoveryTimeoutsSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryTimeoutsSpec. +func (in *RecoveryTimeoutsSpec) DeepCopy() *RecoveryTimeoutsSpec { + if in == nil { + return nil + } + out := new(RecoveryTimeoutsSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryTypeSpec) DeepCopyInto(out *RecoveryTypeSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryTypeSpec. +func (in *RecoveryTypeSpec) DeepCopy() *RecoveryTypeSpec { + if in == nil { + return nil + } + out := new(RecoveryTypeSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RegistryTLSSpec) DeepCopyInto(out *RegistryTLSSpec) { *out = *in @@ -636,6 +975,21 @@ func (in *RegistryTLSSpec) DeepCopy() *RegistryTLSSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *VolumeFirmwareSource) DeepCopyInto(out *VolumeFirmwareSource) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new VolumeFirmwareSource. +func (in *VolumeFirmwareSource) DeepCopy() *VolumeFirmwareSource { + if in == nil { + return nil + } + out := new(VolumeFirmwareSource) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *XpuManagerSpec) DeepCopyInto(out *XpuManagerSpec) { *out = *in @@ -655,3 +1009,18 @@ func (in *XpuManagerSpec) DeepCopy() *XpuManagerSpec { in.DeepCopyInto(out) return out } + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *XpuSmiSpec) DeepCopyInto(out *XpuSmiSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new XpuSmiSpec. +func (in *XpuSmiSpec) DeepCopy() *XpuSmiSpec { + if in == nil { + return nil + } + out := new(XpuSmiSpec) + in.DeepCopyInto(out) + return out +} diff --git a/charts/gpu-base-operator/crds/gpurecoveryplans.yaml b/charts/gpu-base-operator/crds/gpurecoveryplans.yaml new file mode 100644 index 0000000..d92b3cd --- /dev/null +++ b/charts/gpu-base-operator/crds/gpurecoveryplans.yaml @@ -0,0 +1,550 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.18.0 + name: gpurecoveryplans.intel.com +spec: + group: intel.com + names: + kind: GPURecoveryPlan + listKind: GPURecoveryPlanList + plural: gpurecoveryplans + singular: gpurecoveryplan + scope: Cluster + versions: + - additionalPrinterColumns: + - jsonPath: .spec.deviceId + name: DeviceID + type: string + - jsonPath: .status.state + name: State + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: GPURecoveryPlan is the Schema for the gpurecoveryplans API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: GPURecoveryPlanSpec defines the desired state of GPURecoveryPlan. + properties: + approvals: + description: |- + Approvals contains admin-provided authorisations for specific or grouped recovery events. + The operator generates an ID for any entry that is missing one. + items: + description: |- + RecoveryApproval authorises one or more recovery events. + Exactly one of EventID or Selector must be set. + An approval is active as long as it exists in the list and Consumed is false. + properties: + comment: + description: Comment is an optional human-readable note about + why this approval was granted. + type: string + consumed: + description: |- + Consumed is set to true by the operator after a non-persistent approval has been + matched and acted upon. Persistent approvals are never consumed. + type: boolean + eventId: + description: |- + EventID references a specific event by its status.events[].id. + Mutually exclusive with Selector. + type: string + id: + description: |- + ID is the unique identifier for this approval entry. + The operator generates an ID when this field is empty. + type: string + override: + description: |- + Override substitutes a different recovery type than the system recommendation. + When set on an EventID approval, the system records the original suggestion in + status.events[].recoveryType.suggestedType for audit purposes. + properties: + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType is the recovery operation to use + instead of the system suggestion. + type: string + required: + - recoveryType + type: object + persistent: + description: |- + Persistent keeps this approval alive after matched events are processed so that + future events matching the Selector are approved automatically. + Only meaningful when Selector is set. + type: boolean + selector: + description: |- + Selector applies this approval to all currently matching events. + Mutually exclusive with EventID. + properties: + nodeName: + description: NodeName selects events on a specific node. + type: string + nodeSelector: + additionalProperties: + type: string + description: NodeSelector selects events on nodes that have + all given labels. + type: object + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType selects events whose suggested + recovery type matches. + type: string + type: object + type: object + type: array + defaultResetType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - slot + - amc + description: |- + DefaultResetType is the reset the operator runs for every reset-type recovery event it + creates on this plan. Either "slot" (PCIe slot power cycle, also called the PM reset) or + "amc" (out-of-band reset through the card's AMC). + type: string + deviceId: + description: 'DeviceID is the mandatory PCI device ID of the target + GPU. Format: ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + drain: + default: + enable: true + timeoutSeconds: 300 + description: Drain configures the node drain that precedes a reset-type + recovery. + properties: + enable: + default: true + description: Enable controls whether the node is drained before + a reset. + type: boolean + namespacesToSkip: + description: |- + NamespacesToSkip lists namespaces the drain leaves alone: their pods are neither evicted + nor waited for. Use it for cluster infrastructure that is fine to keep running through a + reset and that a whole-node drain would otherwise have to evict. + items: + maxLength: 63 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + maxItems: 64 + type: array + timeoutSeconds: + default: 300 + description: |- + TimeoutSeconds bounds how long a reset waits for the node to drain before the event is + failed. + format: int32 + minimum: 1 + type: integer + type: object + firmware: + description: |- + Firmware holds configuration for reflash-type recovery operations. + These fields are protected: changes are rejected while any reflash event is active. + properties: + file: + description: |- + File is the filename of the FDO firmware image to flash, relative to the root of the + source (no path components). + type: string + source: + description: Source specifies where the firmware file can be found. + properties: + containerSource: + description: ContainerSource specifies a container image that + holds the firmware file. + properties: + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to verify this image and the firmware file inside it. Use it where the registry + serves a self-signed or private-CA certificate. + type: boolean + name: + description: Name is the container image reference (e.g. + registry/image:tag). + type: string + required: + - name + type: object + volumeSource: + description: |- + VolumeSource specifies a Kubernetes volume that holds the firmware file. + + NOT YET SUPPORTED. The field is validated but not acted upon: a reflash event on a plan + whose source sets only volumeSource stays in the missing-firmware state, because the + reflash Job copies firmware from a container image. Use containerSource until this is + implemented. + properties: + name: + description: Name is the name of the PersistentVolumeClaim. + type: string + required: + - name + type: object + type: object + required: + - file + - source + type: object + maxRetries: + default: 3 + description: |- + MaxRetries is the maximum number of times a failed recovery event is automatically + re-queued for approval and retried while its device taint persists. Once this limit + is reached the event stays in the failed state and requires manual intervention + (e.g. delete the event entry or increase MaxRetries). Setting 0 disables automatic + retries entirely. + format: int32 + minimum: 0 + type: integer + skipImageVerification: + description: |- + SkipImageVerification disables the pre-flight registry check on the images a recovery Job + needs (spec.xpuSmi.image, and the firmware image for a reflash). + type: boolean + subDeviceId: + description: 'SubDeviceID is the optional PCI sub-device ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + subVendorId: + description: 'SubVendorID is the optional PCI sub-vendor ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + timeouts: + default: + reflashSeconds: 600 + resetSeconds: 300 + description: Timeouts bounds how long the recovery Jobs themselves + may run. + properties: + reflashSeconds: + default: 600 + description: ReflashSeconds bounds a reflash recovery Job. + format: int32 + minimum: 1 + type: integer + resetSeconds: + default: 300 + description: ResetSeconds bounds a reset-type recovery Job (SBR, + slot power cycle, AMC reset). + format: int32 + minimum: 1 + type: integer + type: object + tolerations: + description: |- + Tolerations are added to recovery Job pods on top of the blanket toleration the operator + always sets, for cases where a cluster needs an extra entry. + items: + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . + properties: + effect: + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. + type: string + operator: + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. + Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). + type: string + tolerationSeconds: + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. + format: int64 + type: integer + value: + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. + type: string + type: object + type: array + xpuSmi: + default: + pullPolicy: IfNotPresent + description: XpuSmi configures the container image providing the xpu-smi + tool. + properties: + image: + description: |- + Image is the container image providing the xpu-smi tool, used for both hardware resets + and firmware reflash operations. + type: string + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to check that Image resolves. Use it where the registry serves a self-signed or + private-CA certificate. + type: boolean + pullPolicy: + default: IfNotPresent + description: PullPolicy controls the image pull policy for the + xpu-smi container. + enum: + - Always + - IfNotPresent + - Never + type: string + type: object + required: + - defaultResetType + - deviceId + - maxRetries + type: object + status: + description: GPURecoveryPlanStatus defines the observed state of GPURecoveryPlan. + properties: + events: + description: Events is the list of active and recently completed recovery + needs (capped at 1000). + items: + description: RecoveryEvent represents a single detected GPU recovery + need and its lifecycle state. + properties: + approvalId: + description: ApprovalID is the ID of the spec.approvals entry + that matched and authorised this event. + type: string + approvalMatchedAt: + description: ApprovalMatchedAt is the timestamp when an approval + first matched this event. + format: date-time + type: string + claimsBlockingReset: + description: |- + ClaimsBlockingReset names the ResourceClaims that still reserve this GPU, in + "namespace/name" form. A non-empty list means the reset is held back because a workload + has not released the device yet. + items: + type: string + type: array + drainStartedAt: + description: |- + DrainStartedAt is when the node drain for this event began. Reset from nil on each + attempt so a retry gets a full drain timeout rather than inheriting the previous one. + format: date-time + type: string + gpuBDF: + description: GPUBDF is the PCI Bus:Device.Function address of + the affected GPU (e.g. "0000:02:00.0"). + type: string + id: + description: ID is the operator-assigned unique event identifier + (e.g. "evt-a3f2"). + type: string + imageVerifyGeneration: + description: |- + ImageVerifyGeneration is the plan's metadata.generation at the time of the last failed + pre-flight image verification for this event, and the reason the event is back in + waiting-approval with its approval still in place. + The failure itself is reported in StateMessage, which is what an admin reads; this field is + only the bookkeeping that keeps the check from repeating. Both are cleared once verification + succeeds. + format: int64 + type: integer + jobName: + description: JobName is the name of the Kubernetes Job created + to execute the recovery, if any. + type: string + lastUpdated: + description: LastUpdated is the timestamp of the most recent + state change for this event. + format: date-time + type: string + nodeName: + description: NodeName is the Kubernetes node hosting the affected + GPU. + type: string + pastJobs: + description: |- + PastJobs is the names of all Jobs created for this event across all attempts. + Jobs are retained alive until the event is removed (i.e. the device taint clears), + so their Pods remain available for diagnostics throughout the event lifecycle. + items: + type: string + type: array + podsBlockingDrain: + description: |- + PodsBlockingDrain names the pods still keeping the node from being drained, in + "namespace/name" form, so an admin can see what a stalled drain is waiting on without + reading operator logs. Capped at a handful of entries. + items: + type: string + type: array + reason: + description: Reason is the human-readable cause of this event + (e.g. "gpu-wedged", "survivability-mode"). + type: string + recoveryType: + description: RecoveryType describes the recovery operation to + perform. + properties: + suggestedType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + SuggestedType is the recovery type originally recommended by the system before + any admin override was applied. Recorded for audit purposes. + type: string + type: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + Type is the recovery operation to execute: a reset (SBR, slot power cycle, or + AMC reset) or a firmware reflash. + type: string + required: + - type + type: object + retryCount: + description: RetryCount is the number of times this recovery + has been retried after failure. + format: int32 + type: integer + state: + allOf: + - enum: + - waiting-approval + - missing-firmware + - blocked + - draining + - in-progress + - succeeded + - failed + - enum: + - waiting-approval + - missing-firmware + - blocked + - draining + - in-progress + - succeeded + - failed + description: State is the current lifecycle state of this recovery + event. + type: string + stateMessage: + description: |- + StateMessage explains, in one sentence, why this event is in the state it is in, where the + state alone does not say: which other recovery is holding the node, which spec field carries + an image that cannot be pulled, what a timed-out drain was still waiting on, which Job + failed and on which attempt, why an approved event is unapproved again. + type: string + required: + - gpuBDF + - id + - lastUpdated + - nodeName + - recoveryType + - retryCount + - state + type: object + type: array + messages: + description: Messages contains recent informational or error messages + (most recent last, capped at ~50). + items: + type: string + type: array + state: + description: State is the overall state of this recovery plan. + enum: + - idle + - error + - active + type: string + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/intel.com_gpurecoveryplans.yaml b/config/crd/bases/intel.com_gpurecoveryplans.yaml new file mode 100644 index 0000000..d92b3cd --- /dev/null +++ b/config/crd/bases/intel.com_gpurecoveryplans.yaml @@ -0,0 +1,550 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.18.0 + name: gpurecoveryplans.intel.com +spec: + group: intel.com + names: + kind: GPURecoveryPlan + listKind: GPURecoveryPlanList + plural: gpurecoveryplans + singular: gpurecoveryplan + scope: Cluster + versions: + - additionalPrinterColumns: + - jsonPath: .spec.deviceId + name: DeviceID + type: string + - jsonPath: .status.state + name: State + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: GPURecoveryPlan is the Schema for the gpurecoveryplans API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: GPURecoveryPlanSpec defines the desired state of GPURecoveryPlan. + properties: + approvals: + description: |- + Approvals contains admin-provided authorisations for specific or grouped recovery events. + The operator generates an ID for any entry that is missing one. + items: + description: |- + RecoveryApproval authorises one or more recovery events. + Exactly one of EventID or Selector must be set. + An approval is active as long as it exists in the list and Consumed is false. + properties: + comment: + description: Comment is an optional human-readable note about + why this approval was granted. + type: string + consumed: + description: |- + Consumed is set to true by the operator after a non-persistent approval has been + matched and acted upon. Persistent approvals are never consumed. + type: boolean + eventId: + description: |- + EventID references a specific event by its status.events[].id. + Mutually exclusive with Selector. + type: string + id: + description: |- + ID is the unique identifier for this approval entry. + The operator generates an ID when this field is empty. + type: string + override: + description: |- + Override substitutes a different recovery type than the system recommendation. + When set on an EventID approval, the system records the original suggestion in + status.events[].recoveryType.suggestedType for audit purposes. + properties: + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType is the recovery operation to use + instead of the system suggestion. + type: string + required: + - recoveryType + type: object + persistent: + description: |- + Persistent keeps this approval alive after matched events are processed so that + future events matching the Selector are approved automatically. + Only meaningful when Selector is set. + type: boolean + selector: + description: |- + Selector applies this approval to all currently matching events. + Mutually exclusive with EventID. + properties: + nodeName: + description: NodeName selects events on a specific node. + type: string + nodeSelector: + additionalProperties: + type: string + description: NodeSelector selects events on nodes that have + all given labels. + type: object + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType selects events whose suggested + recovery type matches. + type: string + type: object + type: object + type: array + defaultResetType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - slot + - amc + description: |- + DefaultResetType is the reset the operator runs for every reset-type recovery event it + creates on this plan. Either "slot" (PCIe slot power cycle, also called the PM reset) or + "amc" (out-of-band reset through the card's AMC). + type: string + deviceId: + description: 'DeviceID is the mandatory PCI device ID of the target + GPU. Format: ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + drain: + default: + enable: true + timeoutSeconds: 300 + description: Drain configures the node drain that precedes a reset-type + recovery. + properties: + enable: + default: true + description: Enable controls whether the node is drained before + a reset. + type: boolean + namespacesToSkip: + description: |- + NamespacesToSkip lists namespaces the drain leaves alone: their pods are neither evicted + nor waited for. Use it for cluster infrastructure that is fine to keep running through a + reset and that a whole-node drain would otherwise have to evict. + items: + maxLength: 63 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + maxItems: 64 + type: array + timeoutSeconds: + default: 300 + description: |- + TimeoutSeconds bounds how long a reset waits for the node to drain before the event is + failed. + format: int32 + minimum: 1 + type: integer + type: object + firmware: + description: |- + Firmware holds configuration for reflash-type recovery operations. + These fields are protected: changes are rejected while any reflash event is active. + properties: + file: + description: |- + File is the filename of the FDO firmware image to flash, relative to the root of the + source (no path components). + type: string + source: + description: Source specifies where the firmware file can be found. + properties: + containerSource: + description: ContainerSource specifies a container image that + holds the firmware file. + properties: + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to verify this image and the firmware file inside it. Use it where the registry + serves a self-signed or private-CA certificate. + type: boolean + name: + description: Name is the container image reference (e.g. + registry/image:tag). + type: string + required: + - name + type: object + volumeSource: + description: |- + VolumeSource specifies a Kubernetes volume that holds the firmware file. + + NOT YET SUPPORTED. The field is validated but not acted upon: a reflash event on a plan + whose source sets only volumeSource stays in the missing-firmware state, because the + reflash Job copies firmware from a container image. Use containerSource until this is + implemented. + properties: + name: + description: Name is the name of the PersistentVolumeClaim. + type: string + required: + - name + type: object + type: object + required: + - file + - source + type: object + maxRetries: + default: 3 + description: |- + MaxRetries is the maximum number of times a failed recovery event is automatically + re-queued for approval and retried while its device taint persists. Once this limit + is reached the event stays in the failed state and requires manual intervention + (e.g. delete the event entry or increase MaxRetries). Setting 0 disables automatic + retries entirely. + format: int32 + minimum: 0 + type: integer + skipImageVerification: + description: |- + SkipImageVerification disables the pre-flight registry check on the images a recovery Job + needs (spec.xpuSmi.image, and the firmware image for a reflash). + type: boolean + subDeviceId: + description: 'SubDeviceID is the optional PCI sub-device ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + subVendorId: + description: 'SubVendorID is the optional PCI sub-vendor ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + timeouts: + default: + reflashSeconds: 600 + resetSeconds: 300 + description: Timeouts bounds how long the recovery Jobs themselves + may run. + properties: + reflashSeconds: + default: 600 + description: ReflashSeconds bounds a reflash recovery Job. + format: int32 + minimum: 1 + type: integer + resetSeconds: + default: 300 + description: ResetSeconds bounds a reset-type recovery Job (SBR, + slot power cycle, AMC reset). + format: int32 + minimum: 1 + type: integer + type: object + tolerations: + description: |- + Tolerations are added to recovery Job pods on top of the blanket toleration the operator + always sets, for cases where a cluster needs an extra entry. + items: + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . + properties: + effect: + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. + type: string + operator: + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. + Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). + type: string + tolerationSeconds: + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. + format: int64 + type: integer + value: + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. + type: string + type: object + type: array + xpuSmi: + default: + pullPolicy: IfNotPresent + description: XpuSmi configures the container image providing the xpu-smi + tool. + properties: + image: + description: |- + Image is the container image providing the xpu-smi tool, used for both hardware resets + and firmware reflash operations. + type: string + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to check that Image resolves. Use it where the registry serves a self-signed or + private-CA certificate. + type: boolean + pullPolicy: + default: IfNotPresent + description: PullPolicy controls the image pull policy for the + xpu-smi container. + enum: + - Always + - IfNotPresent + - Never + type: string + type: object + required: + - defaultResetType + - deviceId + - maxRetries + type: object + status: + description: GPURecoveryPlanStatus defines the observed state of GPURecoveryPlan. + properties: + events: + description: Events is the list of active and recently completed recovery + needs (capped at 1000). + items: + description: RecoveryEvent represents a single detected GPU recovery + need and its lifecycle state. + properties: + approvalId: + description: ApprovalID is the ID of the spec.approvals entry + that matched and authorised this event. + type: string + approvalMatchedAt: + description: ApprovalMatchedAt is the timestamp when an approval + first matched this event. + format: date-time + type: string + claimsBlockingReset: + description: |- + ClaimsBlockingReset names the ResourceClaims that still reserve this GPU, in + "namespace/name" form. A non-empty list means the reset is held back because a workload + has not released the device yet. + items: + type: string + type: array + drainStartedAt: + description: |- + DrainStartedAt is when the node drain for this event began. Reset from nil on each + attempt so a retry gets a full drain timeout rather than inheriting the previous one. + format: date-time + type: string + gpuBDF: + description: GPUBDF is the PCI Bus:Device.Function address of + the affected GPU (e.g. "0000:02:00.0"). + type: string + id: + description: ID is the operator-assigned unique event identifier + (e.g. "evt-a3f2"). + type: string + imageVerifyGeneration: + description: |- + ImageVerifyGeneration is the plan's metadata.generation at the time of the last failed + pre-flight image verification for this event, and the reason the event is back in + waiting-approval with its approval still in place. + The failure itself is reported in StateMessage, which is what an admin reads; this field is + only the bookkeeping that keeps the check from repeating. Both are cleared once verification + succeeds. + format: int64 + type: integer + jobName: + description: JobName is the name of the Kubernetes Job created + to execute the recovery, if any. + type: string + lastUpdated: + description: LastUpdated is the timestamp of the most recent + state change for this event. + format: date-time + type: string + nodeName: + description: NodeName is the Kubernetes node hosting the affected + GPU. + type: string + pastJobs: + description: |- + PastJobs is the names of all Jobs created for this event across all attempts. + Jobs are retained alive until the event is removed (i.e. the device taint clears), + so their Pods remain available for diagnostics throughout the event lifecycle. + items: + type: string + type: array + podsBlockingDrain: + description: |- + PodsBlockingDrain names the pods still keeping the node from being drained, in + "namespace/name" form, so an admin can see what a stalled drain is waiting on without + reading operator logs. Capped at a handful of entries. + items: + type: string + type: array + reason: + description: Reason is the human-readable cause of this event + (e.g. "gpu-wedged", "survivability-mode"). + type: string + recoveryType: + description: RecoveryType describes the recovery operation to + perform. + properties: + suggestedType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + SuggestedType is the recovery type originally recommended by the system before + any admin override was applied. Recorded for audit purposes. + type: string + type: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + Type is the recovery operation to execute: a reset (SBR, slot power cycle, or + AMC reset) or a firmware reflash. + type: string + required: + - type + type: object + retryCount: + description: RetryCount is the number of times this recovery + has been retried after failure. + format: int32 + type: integer + state: + allOf: + - enum: + - waiting-approval + - missing-firmware + - blocked + - draining + - in-progress + - succeeded + - failed + - enum: + - waiting-approval + - missing-firmware + - blocked + - draining + - in-progress + - succeeded + - failed + description: State is the current lifecycle state of this recovery + event. + type: string + stateMessage: + description: |- + StateMessage explains, in one sentence, why this event is in the state it is in, where the + state alone does not say: which other recovery is holding the node, which spec field carries + an image that cannot be pulled, what a timed-out drain was still waiting on, which Job + failed and on which attempt, why an approved event is unapproved again. + type: string + required: + - gpuBDF + - id + - lastUpdated + - nodeName + - recoveryType + - retryCount + - state + type: object + type: array + messages: + description: Messages contains recent informational or error messages + (most recent last, capped at ~50). + items: + type: string + type: array + state: + description: State is the overall state of this recovery plan. + enum: + - idle + - error + - active + type: string + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/kustomization.yaml b/config/crd/kustomization.yaml index bae15af..ce6ed5b 100644 --- a/config/crd/kustomization.yaml +++ b/config/crd/kustomization.yaml @@ -4,6 +4,7 @@ resources: - bases/intel.com_clusterpolicies.yaml - bases/intel.com_gpufirmwareupdates.yaml +- bases/intel.com_gpurecoveryplans.yaml # +kubebuilder:scaffold:crdkustomizeresource patches: diff --git a/config/rbac/gpurecoveryplan_admin_role.yaml b/config/rbac/gpurecoveryplan_admin_role.yaml new file mode 100644 index 0000000..6bd4bd8 --- /dev/null +++ b/config/rbac/gpurecoveryplan_admin_role.yaml @@ -0,0 +1,27 @@ +# This rule is not used by the project intel-gpu-base-operator itself. +# It is provided to allow the cluster admin to help manage permissions for users. +# +# Grants full permissions ('*') over intel.com. +# This role is intended for users authorized to modify roles and bindings within the cluster, +# enabling them to delegate specific permissions to other users or groups as needed. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-admin-role +rules: +- apiGroups: + - intel.com + resources: + - gpurecoveryplans + verbs: + - '*' +- apiGroups: + - intel.com + resources: + - gpurecoveryplans/status + verbs: + - get diff --git a/config/rbac/gpurecoveryplan_editor_role.yaml b/config/rbac/gpurecoveryplan_editor_role.yaml new file mode 100644 index 0000000..d6c494c --- /dev/null +++ b/config/rbac/gpurecoveryplan_editor_role.yaml @@ -0,0 +1,33 @@ +# This rule is not used by the project intel-gpu-base-operator itself. +# It is provided to allow the cluster admin to help manage permissions for users. +# +# Grants permissions to create, update, and delete resources within the intel.com. +# This role is intended for users who need to manage these resources +# but should not control RBAC or manage permissions for others. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-editor-role +rules: +- apiGroups: + - intel.com + resources: + - gpurecoveryplans + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - intel.com + resources: + - gpurecoveryplans/status + verbs: + - get diff --git a/config/rbac/gpurecoveryplan_viewer_role.yaml b/config/rbac/gpurecoveryplan_viewer_role.yaml new file mode 100644 index 0000000..16d0dc3 --- /dev/null +++ b/config/rbac/gpurecoveryplan_viewer_role.yaml @@ -0,0 +1,29 @@ +# This rule is not used by the project intel-gpu-base-operator itself. +# It is provided to allow the cluster admin to help manage permissions for users. +# +# Grants read-only access to intel.com resources. +# This role is intended for users who need visibility into these resources +# without permissions to modify them. It is ideal for monitoring purposes and limited-access viewing. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-viewer-role +rules: +- apiGroups: + - intel.com + resources: + - gpurecoveryplans + verbs: + - get + - list + - watch +- apiGroups: + - intel.com + resources: + - gpurecoveryplans/status + verbs: + - get diff --git a/config/rbac/kustomization.yaml b/config/rbac/kustomization.yaml index 47aafc1..8f729c7 100644 --- a/config/rbac/kustomization.yaml +++ b/config/rbac/kustomization.yaml @@ -24,6 +24,9 @@ resources: # default, aiding admins in cluster management. Those roles are # not used by the intel-gpu-base-operator itself. You can comment the following lines # if you do not want those helpers be installed with your Project. +- gpurecoveryplan_admin_role.yaml +- gpurecoveryplan_editor_role.yaml +- gpurecoveryplan_viewer_role.yaml - gpufirmwareupdate_admin_role.yaml - gpufirmwareupdate_editor_role.yaml - gpufirmwareupdate_viewer_role.yaml diff --git a/config/samples/recoveryplan/gpurecoveryplan.yaml b/config/samples/recoveryplan/gpurecoveryplan.yaml new file mode 100644 index 0000000..db3ad48 --- /dev/null +++ b/config/samples/recoveryplan/gpurecoveryplan.yaml @@ -0,0 +1,88 @@ +apiVersion: intel.com/v1alpha1 +kind: GPURecoveryPlan +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-sample +spec: + deviceId: "0xe20b" + + # Mandatory. Which reset works is a property of the platform, not of the fault: use slot — the + # PCIe slot power cycle, also called the PM reset — where the slots support hot-plug, and amc + # where they do not. The DRA driver only reports "needs a reset", so this is the operator's only + # way to know which one to run, and getting it wrong is silent: the Job runs a reset the platform + # cannot perform, exits 0, and the event succeeds with the GPU still broken. + # sbr is not accepted here — it is the per-card backup, reached through an approval's override + # when the platform's normal reset does not revive one particular GPU. + defaultResetType: "slot" + + # The node drain that runs before a reset. None of it applies to a reflash: that writes firmware to + # a card which is already unusable without driving the PCIe bus, so it never drains. + drain: + # Turning the drain off means a reset can hit the bus while workloads are still running on the + # node. Prefer namespacesToSkip below if the problem is one namespace that will not drain. + # enable: true + + # How long a reset waits for the node to empty. When it expires the event fails with + # drain-timeout and status.events[].podsBlockingDrain names what it was still waiting on. + # timeoutSeconds: 300 + + # Namespaces the drain leaves alone: their pods are neither evicted nor waited for. Intended for + # node-level infrastructure that is fine to keep running through a reset and that would otherwise + # stall the drain — a PodDisruptionBudget with nowhere to reschedule, or a pod no controller + # recreates. The operator's own namespace is always skipped and does not need listing, and + # DaemonSet and static pods are skipped regardless of namespace. + # + # This exempts a namespace from the *drain*, not from the reset: its pods stay on the node + # through the PCIe reset. So do not list namespaces that run GPU workloads — including ones + # holding a GPU through a DRA claim, which is not waited for either when the only pods reserving + # it are ones this list keeps on the node. + # namespacesToSkip: + # - kube-system + # - cert-manager + + # How long a recovery Job itself may run before Kubernetes kills it (the Job's + # activeDeadlineSeconds). Separate from drain.timeoutSeconds above, which bounds only the wait for + # the node to empty. How long the hardware takes is a property of the platform — a chassis slow to + # re-enumerate a power-cycled slot, an FDO flash over a slow SPI part — and a deadline that expires + # early is not a harmless retry: the Job is killed mid-operation and the event fails over a card + # that was in fact recovering. Raising these only costs time before a genuine failure is reported, + # so err high where the numbers are uncertain. + # timeouts: + # resetSeconds: 300 # sbr, slot, amc + # reflashSeconds: 600 # includes copying the firmware out of the firmware image + + # The xpu-smi image every recovery Job runs, for both the resets and the reflash. + xpuSmi: + image: "docker.io/intel/gpu-fwupdater-mock:devel" + pullPolicy: "Always" + + # Skip registry certificate validation for the operator's pre-flight check on the image + # above. Needed where the registry serves a self-signed or private-CA certificate — but only + # for the operator's own check: the node still has to trust that registry (or have it listed + # as an insecure one) for the kubelet to pull the image. + # insecureSkipTLSVerify: false + + # The image to flash onto a GPU that has entered FDO mode. Always flashed as firmware type + # FDO — the only type that can revive such a card — so no type is configurable here. + firmware: + source: + containerSource: + name: "docker.io/intel/intel-gpu-fw-binaries:devel" + + # Same as spec.xpuSmi.insecureSkipTLSVerify, and separate from it because the known good + # firmware for one card model often lives on an internal registry while xpu-smi comes + # from a public one. + # insecureSkipTLSVerify: false + + file: "gpu_firmware.bin" + + # Both images above are resolved against their registry before any recovery Job is created, so + # an unpullable reference holds the event at waiting-approval — with its approval retained and + # the reason in that event's status.events[].stateMessage, as well as in the plan-wide + # status.messages — instead of producing a Job that reports in-progress from + # ImagePullBackOff. The check runs once per spec version: correcting the plan is what makes the + # operator look again. Set this only where the nodes hold pull credentials the operator cannot + # see, which would make the check stricter than the pull it predicts. + # skipImageVerification: false diff --git a/config/samples/recoveryplan/kustomization.yaml b/config/samples/recoveryplan/kustomization.yaml new file mode 100644 index 0000000..b6d7e89 --- /dev/null +++ b/config/samples/recoveryplan/kustomization.yaml @@ -0,0 +1,4 @@ +## Append samples of your project ## +resources: +- gpurecoveryplan.yaml +# +kubebuilder:scaffold:manifestskustomizesamples