diff --git a/Makefile b/Makefile index 5e1ac01..32c46cc 100644 --- a/Makefile +++ b/Makefile @@ -403,6 +403,7 @@ endif crdsync: generate manifests cp config/crd/bases/intel.com_clusterpolicies.yaml charts/gpu-base-operator/crds/clusterpolicies.yaml cp config/crd/bases/intel.com_gpufirmwareupdates.yaml charts/gpu-base-operator/crds/gpufirmwareupdates.yaml + cp config/crd/bases/intel.com_gpurecoveryplans.yaml charts/gpu-base-operator/crds/gpurecoveryplans.yaml .PHONY: check-generated-files check-generated-files: crdsync diff --git a/PROJECT b/PROJECT index db0b0e7..e45c2d8 100644 --- a/PROJECT +++ b/PROJECT @@ -30,4 +30,12 @@ resources: webhooks: validation: true webhookVersion: v1 +- api: + crdVersion: v1 + namespaced: false + controller: true + domain: intel.com + kind: GPURecoveryPlan + path: github.com/intel/gpu-base-operator/api/v1alpha1 + version: v1alpha1 version: "3" diff --git a/RECOVERY.md b/RECOVERY.md new file mode 100644 index 0000000..070806f --- /dev/null +++ b/RECOVERY.md @@ -0,0 +1,230 @@ +# GPU Recovery + +Intel GPUs can end up in states that a normal driver-level reset cannot fix: + +* **GPU wedged / stuck** — the device is still visible to the `xe` driver, but every operation on it + fails. A harder, PCIe-level reset is required. +* **Corrupted firmware** — the card boots only far enough for the PCIe link to come up (FDO / + survivability mode). It has to be re-flashed with a known-good "golden" firmware image. + +Recovering from either case is potentially disruptive: a PCIe reset can disturb other devices on the +same host, or wedge the host itself. The operator therefore never resets a GPU on its own — it +detects the need, reports it, and waits for a cluster admin to approve the operation. + +Recovery is driven by the cluster-scoped `GPURecoveryPlan` CRD. + +## How it works + +``` +GPU fault → Xe KMD → xpumd → GPU DRA driver → ResourceSlice device taint + ↓ + operator detects taint, adds a recovery event + ↓ + admin approves it (spec.approvals) + ↓ + node drain (resets only) → recovery Job runs xpu-smi + ↓ + taint clears → event removed, plan back to idle +``` + +The operator watches `ResourceSlice` device taints published by the GPU DRA driver: + +| Taint key | Meaning | Recovery | +|---|---|---| +| `health-xpumd-gpu.wedged` | GPU wedged/stuck at runtime | reset (`spec.defaultResetType`) | +| `health-Survivability` | card booted into survivability/FDO mode | `reflash` | +| `health-xpumd-gpu.survivability` | card fell into survivability at runtime | `reflash` | + +Each affected GPU gets one entry in `status.events` with a deterministic ID +(`evt---`). A device whose condition worsens (reset → reflash) is *escalated in +place*, which regenerates the event ID so an approval for the lighter operation cannot silently +authorise the heavier one. + +Recovery Jobs run `xpu-smi` from the image in `spec.xpuSmi`, privileged, pinned to the affected node, +and are retained after completion for post-mortem diagnostics (they are removed when the event is). + +### Reset types + +| Type | Command | Notes | +|---|---|---| +| `slot` | `xpu-smi config -d --coldreset` | PCIe slot power cycle; requires PCIe hot-plug support | +| `amc` | `xpu-smi amc --gpuReset -d ` | Out-of-band reset through the card's AMC | +| `sbr` | `xpu-smi config -d --reset` | Secondary Bus Reset; per-card backup, via an approval override only | +| `reflash` | `xpu-smi updatefw -d -t FDO -f ` | Flash the golden firmware onto a card in FDO mode | + +These are **not** a severity ladder. Exactly one of `slot` and `amc` works on a given platform — +slot where the PCIe slots do hot-plug, AMC where they do not — and the DRA driver can only say "this +device needs a reset", not which mechanism applies. That is why `spec.defaultResetType` is mandatory +with no default: a reset the platform cannot perform **exits 0**, so a wrong value produces a clean +run to `succeeded` over a GPU that was never touched. + +### Event states + +| State | Meaning | +|---|---| +| `waiting-approval` | Detected, waiting for `spec.approvals`. Also where an event lands when its pre-flight image check failed | +| `missing-goldenfw` | Reflash needed but `spec.goldenFirmware` is absent, empty or volume-sourced | +| `blocked` | Approved, but another recovery is already running on this node. The approval is kept, so it resumes on its own | +| `draining` | Node is being drained before a reset | +| `in-progress` | Recovery Job is running | +| `succeeded` / `failed` | Job finished. A failure inside `spec.maxRetries` is re-queued for approval | + +`status.events[].stateMessage` explains any state that is not self-explanatory (which recovery holds +the node, which image cannot be pulled, what a stalled drain was waiting on). An empty value means +there is nothing to add. + +The plan-level `status.state` is `idle`, `active`, or `error`. `error` means an admin is needed: an +event out of retries, or one blocked on missing golden firmware. + +## Safety mechanisms + +* **Admin approval** for every recovery, singular or by selector (see below). +* **Node drain before a reset** (`spec.drain`). The whole node is drained, not just GPU pods, because + an SBR or slot reset can wedge the host. A reflash never drains — it writes firmware without + driving the bus. The drain uses the Eviction API, so PodDisruptionBudgets are honoured, and is + bounded by `spec.drain.timeoutSeconds`. The operator's own namespace, DaemonSet pods, static pods + and already-terminated pods are always skipped; `spec.drain.namespacesToSkip` adds to that. +* **Job deadlines** (`spec.timeouts`). Each recovery Job carries an `activeDeadlineSeconds`: + `spec.timeouts.resetSeconds` (default 300) for the resets, `spec.timeouts.reflashSeconds` + (default 600) for a reflash, which also covers copying the firmware out of the golden firmware + image. They are configurable because how long the hardware takes is a property of the platform, and + a deadline that expires early kills the Job mid-operation and fails the event over a card that was + recovering — so raise them where a reset or flash is known to be slow. +* **In-use check.** A reset waits until every `ResourceClaim` reserving the GPU is released. Claims + the drain can never release are excluded (`adminAccess` claims, and claims held only by pods the + drain leaves in place) — waiting for those would deadlock the recovery. +* **One recovery per node at a time.** A second approved event on the same node is held in `blocked` + rather than run concurrently: a PCIe reset during another card's firmware write can leave that card + unrecoverable. +* **Pre-flight image verification.** `spec.xpuSmi.image` and the golden firmware image are resolved + against their registries before a Job is created, so a mistyped reference does not produce a Job + that reports `in-progress` from `ImagePullBackOff`. Checked once per spec generation; disable with + `spec.skipImageVerification` where nodes hold pull credentials the operator cannot see. +* **Retry budget.** `spec.maxRetries` bounds automatic retries; an exhausted event needs an explicit + per-event re-approval, so a standing group approval cannot loop a dying card forever. +* **Finalizer.** Deleting a plan blocks until every recovery Job is terminal, and releases all drain + taints first. +* **The operator tolerates its own drain taint**, since it is the only thing that removes it. + +## Example plan + +```yaml +apiVersion: intel.com/v1alpha1 +kind: GPURecoveryPlan +metadata: + name: recoveryplan-bmg +spec: + deviceId: "0xe20b" # mandatory; one plan per GPU model + defaultResetType: "slot" # mandatory: "slot" or "amc" + maxRetries: 3 + + drain: + enable: true + timeoutSeconds: 300 + namespacesToSkip: ["kube-system"] + + timeouts: # how long a recovery Job itself may run + resetSeconds: 300 + reflashSeconds: 600 + + xpuSmi: + image: "docker.io/intel/xpu-smi:devel" + pullPolicy: "IfNotPresent" + + goldenFirmware: # required for reflash recovery + source: + containerSource: + name: "docker.io/intel/intel-gpu-fw-binaries:devel" + file: "gpu_firmware.bin" +``` + +Heterogeneous clusters use one plan per device ID. A full, commented sample lives in +[`config/samples/recoveryplan/gpurecoveryplan.yaml`](config/samples/recoveryplan/gpurecoveryplan.yaml). + +## Approving recovery + +An approval either names one event or selects a group. Selector approvals are one-shot unless marked +`persistent: true`. `override` changes the recovery type that will actually run, and the original +suggestion is kept in `status.events[].recoveryType.suggestedType` for audit. + +```yaml +spec: + approvals: + # one specific event + - eventId: evt-node05-slot-02-00-0 + + # escalate one card to a Secondary Bus Reset + - eventId: evt-node03-slot-02-00-0 + override: + recoveryType: sbr + + # all current reset events on nodes labelled rack=rack-04-32 + - selector: + recoveryType: slot + nodeSelector: + rack: rack-04-32 + + # standing order: auto-approve future slot resets (CI/test clusters) + - selector: + recoveryType: slot + persistent: true +``` + +The operator fills in a missing `id`, and marks non-persistent approvals `consumed: true` once acted +upon; consumed entries stay as an audit trail and can be removed by hand. + +### kubectl plugin + +`kubectl-gpurecovery` wraps the patching, with tab completion for plan names, event IDs and approval +IDs: + +```sh +make install-kubectl-plugin # builds to bin/ and installs to ~/.local/bin +make setup-completion # optional: shell completion + +kubectl gpurecovery plans +kubectl gpurecovery events +kubectl gpurecovery messages +kubectl gpurecovery approvals +kubectl gpurecovery approve +kubectl gpurecovery confirm [--persistent] +kubectl gpurecovery remove +``` + +Plain `kubectl patch` works too, e.g.: + +```sh +kubectl patch gpurecoveryplan --type=json \ + -p='[{"op":"add","path":"/spec/approvals/-","value":{"eventId":""}}]' +``` + +## Metrics + +Exposed on the operator's own `/metrics` endpoint (there is deliberately no `status.stats` field — +the CR holds current state, the counters hold history): + +| Metric | Labels | +|---|---| +| `gpu_recovery_events_total` | `plan`, `type`, `reason` | +| `gpu_recovery_attempts_total` | `plan`, `type` | +| `gpu_recovery_outcomes_total` | `plan`, `type`, `result` | +| `gpu_recovery_overrides_total` | `plan`, `suggested_type`, `chosen_type` | +| `gpu_recovery_events` (gauge) | `plan`, `node`, `type`, `state` | +| `gpu_recovery_plan_state` (gauge) | `plan`, `state` | + +## Current limitations + +* **Sibling devices are not protected.** Nothing stops a slot reset or SBR on one card from + disturbing another card behind the same PCIe root port. Drain the node and check the topology + before approving. +* **Job targeting is privileged + `nodeName`**, not a DRA claim with device tolerations and CEL + expressions as originally designed. Still an open decision. +* **`spec.subDeviceId` / `spec.subVendorId` are validated but not used for matching** — the DRA + driver does not publish those attributes yet. +* **`goldenFirmware.source.volumeSource` is not implemented.** A reflash event on a + volume-only plan stays in `missing-goldenfw`; use `containerSource`. +* **Reset efficacy on BMG.** On some B580 cards a reset leaves the GPU non-working; end-to-end + validation depends on driver/firmware fixes. +* **The `health-xpumd-gpu.wedged` taint key is not yet confirmed against a shipping DRA driver.** + The survivability key and the `pciId` / `pciAddress` device attributes are. + diff --git a/api/v1alpha1/gpurecoveryplan_types.go b/api/v1alpha1/gpurecoveryplan_types.go new file mode 100644 index 0000000..4693724 --- /dev/null +++ b/api/v1alpha1/gpurecoveryplan_types.go @@ -0,0 +1,452 @@ +/* +Copyright 2026 Intel Corporation. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + core "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// RecoveryType represents the type of GPU recovery operation. +// +kubebuilder:validation:Enum=sbr;slot;amc;reflash +type RecoveryType string + +// RecoveryEventState represents the lifecycle state of a recovery event. +// +kubebuilder:validation:Enum=waiting-approval;missing-goldenfw;blocked;draining;in-progress;succeeded;failed +type RecoveryEventState string + +// PlanState represents the overall state of the recovery plan. +// +kubebuilder:validation:Enum=idle;error;active +type PlanState string + +const ( + // Secondary Bus Reset + RecoveryTypeSBR RecoveryType = "sbr" + // Power cycle/cold reset through the PCIe slot, also called the PM reset. + RecoveryTypeSlot RecoveryType = "slot" + // Out-of-band reset through the card's AMC (Advanced Management Controller). + RecoveryTypeAMC RecoveryType = "amc" + // Reflash to bring a GPU with corrupted FW into an operational state. + RecoveryTypeReflash RecoveryType = "reflash" + + // RecoveryEventStateWaitingApproval means the event is pending admin approval. This is the first state + // for an event, but an error in event processing can also return it to this state. + RecoveryEventStateWaitingApproval RecoveryEventState = "waiting-approval" + // RecoveryEventStateMissingGoldenFW means reflash cannot proceed due to missing firmware. + RecoveryEventStateMissingGoldenFW RecoveryEventState = "missing-goldenfw" + // RecoveryEventStateBlocked means the event is approved but another recovery is already + // running on the same node, so this one is held back. + RecoveryEventStateBlocked RecoveryEventState = "blocked" + // RecoveryEventStateDraining means the event is approved and the node hosting the GPU is + // being drained before the reset runs. Reset-type recoveries only: a reflash writes + // firmware without resetting the PCIe bus, so it goes straight to in-progress. + RecoveryEventStateDraining RecoveryEventState = "draining" + // RecoveryEventStateInProgress means a recovery Job is currently running. + RecoveryEventStateInProgress RecoveryEventState = "in-progress" + // RecoveryEventStateSucceeded means the recovery Job completed successfully. + RecoveryEventStateSucceeded RecoveryEventState = "succeeded" + // RecoveryEventStateFailed means the recovery Job failed. + RecoveryEventStateFailed RecoveryEventState = "failed" + + PlanStateIdle PlanState = "idle" + PlanStateError PlanState = "error" + PlanStateActive PlanState = "active" +) + +// GPURecoveryPlanSpec defines the desired state of GPURecoveryPlan. +type GPURecoveryPlanSpec struct { + // DeviceID is the mandatory PCI device ID of the target GPU. Format: '0x' followed by 4 hex digits. + // +kubebuilder:validation:Pattern=`^0x[0-9a-fA-F]{4}$` + DeviceID string `json:"deviceId"` + + // SubDeviceID is the optional PCI sub-device ID. Format: '0x' followed by 4 hex digits. + // +kubebuilder:validation:Pattern=`^0x[0-9a-fA-F]{4}$` + // +optional + SubDeviceID string `json:"subDeviceId,omitempty"` + + // SubVendorID is the optional PCI sub-vendor ID. Format: '0x' followed by 4 hex digits. + // +kubebuilder:validation:Pattern=`^0x[0-9a-fA-F]{4}$` + // +optional + SubVendorID string `json:"subVendorId,omitempty"` + + // Approvals contains admin-provided authorisations for specific or grouped recovery events. + // The operator generates an ID for any entry that is missing one. + // +optional + Approvals []RecoveryApproval `json:"approvals,omitempty"` + + // XpuSmi configures the container image providing the xpu-smi tool. + // +kubebuilder:default={pullPolicy: "IfNotPresent"} + // +optional + XpuSmi XpuSmiSpec `json:"xpuSmi,omitempty"` + + // DefaultResetType is the reset the operator runs for every reset-type recovery event it + // creates on this plan. Either "slot" (PCIe slot power cycle, also called the PM reset) or + // "amc" (out-of-band reset through the card's AMC). + // +kubebuilder:validation:Enum=slot;amc + // +kubebuilder:validation:Required + DefaultResetType RecoveryType `json:"defaultResetType"` + + // GoldenFirmware holds configuration for reflash-type recovery operations. + // These fields are protected: changes are rejected while any reflash event is active. + // +optional + GoldenFirmware *GoldenFirmwareSpec `json:"goldenFirmware,omitempty"` + + // Tolerations are added to recovery Job pods on top of the blanket toleration the operator + // always sets, for cases where a cluster needs an extra entry. + // +optional + Tolerations []core.Toleration `json:"tolerations,omitempty"` + + // Drain configures the node drain that precedes a reset-type recovery. + // +kubebuilder:default={enable: true, timeoutSeconds: 300} + // +optional + Drain DrainSpec `json:"drain,omitempty"` + + // Timeouts bounds how long the recovery Jobs themselves may run. + // +kubebuilder:default={resetSeconds: 300, reflashSeconds: 600} + // +optional + Timeouts RecoveryTimeoutsSpec `json:"timeouts,omitempty"` + + // SkipImageVerification disables the pre-flight registry check on the images a recovery Job + // needs (spec.xpuSmi.image, and the golden firmware image for a reflash). + // +optional + SkipImageVerification bool `json:"skipImageVerification,omitempty"` + + // MaxRetries is the maximum number of times a failed recovery event is automatically + // re-queued for approval and retried while its device taint persists. Once this limit + // is reached the event stays in the failed state and requires manual intervention + // (e.g. delete the event entry or increase MaxRetries). Setting 0 disables automatic + // retries entirely. + // +kubebuilder:default=3 + // +kubebuilder:validation:Minimum=0 + MaxRetries int32 `json:"maxRetries"` +} + +// DrainSpec configures the node drain a reset-type recovery performs before it touches the +// hardware. +type DrainSpec struct { + // Enable controls whether the node is drained before a reset. + // +kubebuilder:default=true + // +optional + Enable *bool `json:"enable,omitempty"` + + // TimeoutSeconds bounds how long a reset waits for the node to drain before the event is + // failed. + // +kubebuilder:default=300 + // +kubebuilder:validation:Minimum=1 + // +optional + TimeoutSeconds int32 `json:"timeoutSeconds,omitempty"` + + // NamespacesToSkip lists namespaces the drain leaves alone: their pods are neither evicted + // nor waited for. Use it for cluster infrastructure that is fine to keep running through a + // reset and that a whole-node drain would otherwise have to evict. + // +kubebuilder:validation:MaxItems=64 + // +kubebuilder:validation:items:MaxLength=63 + // +kubebuilder:validation:items:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$` + // +optional + NamespacesToSkip []string `json:"namespacesToSkip,omitempty"` +} + +// Enabled reports whether the pre-reset drain should run. A nil Enable is enabled; see the field. +func (d *DrainSpec) Enabled() bool { + return d.Enable == nil || *d.Enable +} + +// RecoveryTimeoutsSpec bounds the runtime of the recovery Jobs, i.e. the Job's activeDeadlineSeconds. +type RecoveryTimeoutsSpec struct { + // ResetSeconds bounds a reset-type recovery Job (SBR, slot power cycle, AMC reset). + // +kubebuilder:default=300 + // +kubebuilder:validation:Minimum=1 + // +optional + ResetSeconds int32 `json:"resetSeconds,omitempty"` + + // ReflashSeconds bounds a reflash recovery Job. + // +kubebuilder:default=600 + // +kubebuilder:validation:Minimum=1 + // +optional + ReflashSeconds int32 `json:"reflashSeconds,omitempty"` +} + +// XpuSmiSpec describes the xpu-smi container image every recovery Job runs. +type XpuSmiSpec struct { + // Image is the container image providing the xpu-smi tool, used for both hardware resets + // and firmware reflash operations. + // +optional + Image string `json:"image,omitempty"` + + // PullPolicy controls the image pull policy for the xpu-smi container. + // +kubebuilder:validation:Enum=Always;IfNotPresent;Never + // +kubebuilder:default="IfNotPresent" + // +optional + PullPolicy string `json:"pullPolicy,omitempty"` + + // InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + // registry to check that Image resolves. Use it where the registry serves a self-signed or + // private-CA certificate. + // +optional + InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"` +} + +// RecoveryApproval authorises one or more recovery events. +// Exactly one of EventID or Selector must be set. +// An approval is active as long as it exists in the list and Consumed is false. +type RecoveryApproval struct { + // ID is the unique identifier for this approval entry. + // The operator generates an ID when this field is empty. + // +optional + ID string `json:"id,omitempty"` + + // EventID references a specific event by its status.events[].id. + // Mutually exclusive with Selector. + // +optional + EventID string `json:"eventId,omitempty"` + + // Selector applies this approval to all currently matching events. + // Mutually exclusive with EventID. + // +optional + Selector *ApprovalSelector `json:"selector,omitempty"` + + // Override substitutes a different recovery type than the system recommendation. + // When set on an EventID approval, the system records the original suggestion in + // status.events[].recoveryType.suggestedType for audit purposes. + // +optional + Override *RecoveryOverride `json:"override,omitempty"` + + // Persistent keeps this approval alive after matched events are processed so that + // future events matching the Selector are approved automatically. + // Only meaningful when Selector is set. + // +optional + Persistent bool `json:"persistent,omitempty"` + + // Consumed is set to true by the operator after a non-persistent approval has been + // matched and acted upon. Persistent approvals are never consumed. + // +optional + Consumed bool `json:"consumed,omitempty"` + + // Comment is an optional human-readable note about why this approval was granted. + // +optional + Comment string `json:"comment,omitempty"` +} + +// ApprovalSelector filters recovery events by type, node, or node labels. +type ApprovalSelector struct { + // RecoveryType selects events whose suggested recovery type matches. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + // +optional + RecoveryType RecoveryType `json:"recoveryType,omitempty"` + + // NodeName selects events on a specific node. + // +optional + NodeName string `json:"nodeName,omitempty"` + + // NodeSelector selects events on nodes that have all given labels. + // +optional + NodeSelector map[string]string `json:"nodeSelector,omitempty"` +} + +// RecoveryOverride allows the admin to escalate or change the suggested recovery type. +type RecoveryOverride struct { + // RecoveryType is the recovery operation to use instead of the system suggestion. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + RecoveryType RecoveryType `json:"recoveryType"` +} + +// GoldenFirmwareSpec holds everything required for a firmware reflash operation. +type GoldenFirmwareSpec struct { + // Source specifies where the firmware file can be found. + Source GoldenFirmwareSource `json:"source"` + + // File is the filename of the FDO firmware image to flash, relative to the root of the + // source (no path components). + File string `json:"file"` +} + +// GoldenFirmwareSource describes where the firmware file is located. +// At least one of ContainerSource or VolumeSource must be set. +type GoldenFirmwareSource struct { + // ContainerSource specifies a container image that holds the firmware file. + // +optional + ContainerSource *ContainerFirmwareSource `json:"containerSource,omitempty"` + + // VolumeSource specifies a Kubernetes volume that holds the firmware file. + // + // NOT YET SUPPORTED. The field is validated but not acted upon: a reflash event on a plan + // whose source sets only volumeSource stays in the missing-goldenfw state, because the + // reflash Job copies firmware from a container image. Use containerSource until this is + // implemented. + // +optional + VolumeSource *VolumeFirmwareSource `json:"volumeSource,omitempty"` +} + +// ContainerFirmwareSource references a container image holding the firmware file. +type ContainerFirmwareSource struct { + // Name is the container image reference (e.g. registry/image:tag). + Name string `json:"name"` + + // InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + // registry to verify this image and the firmware file inside it. Use it where the registry + // serves a self-signed or private-CA certificate. + // +optional + InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"` +} + +// VolumeFirmwareSource references a Kubernetes volume holding the firmware file. +// See GoldenFirmwareSource.VolumeSource: not yet supported. +type VolumeFirmwareSource struct { + // Name is the name of the PersistentVolumeClaim. + Name string `json:"name"` +} + +// GPURecoveryPlanStatus defines the observed state of GPURecoveryPlan. +type GPURecoveryPlanStatus struct { + // State is the overall state of this recovery plan. + // +optional + State PlanState `json:"state,omitempty"` + + // Messages contains recent informational or error messages (most recent last, capped at ~50). + // +optional + Messages []string `json:"messages,omitempty"` + + // Events is the list of active and recently completed recovery needs (capped at 1000). + // +optional + Events []RecoveryEvent `json:"events,omitempty"` +} + +// RecoveryEvent represents a single detected GPU recovery need and its lifecycle state. +type RecoveryEvent struct { + // ID is the operator-assigned unique event identifier (e.g. "evt-a3f2"). + ID string `json:"id"` + + // ApprovalID is the ID of the spec.approvals entry that matched and authorised this event. + // +optional + ApprovalID string `json:"approvalId,omitempty"` + + // ApprovalMatchedAt is the timestamp when an approval first matched this event. + // +optional + ApprovalMatchedAt *metav1.Time `json:"approvalMatchedAt,omitempty"` + + // NodeName is the Kubernetes node hosting the affected GPU. + NodeName string `json:"nodeName"` + + // GPUBDF is the PCI Bus:Device.Function address of the affected GPU (e.g. "0000:02:00.0"). + GPUBDF string `json:"gpuBDF"` + + // Reason is the human-readable cause of this event (e.g. "gpu-wedged", "survivability-mode"). + // +optional + Reason string `json:"reason,omitempty"` + + // RecoveryType describes the recovery operation to perform. + RecoveryType RecoveryTypeSpec `json:"recoveryType"` + + // State is the current lifecycle state of this recovery event. + // +kubebuilder:validation:Enum=waiting-approval;missing-goldenfw;blocked;draining;in-progress;succeeded;failed + State RecoveryEventState `json:"state"` + + // StateMessage explains, in one sentence, why this event is in the state it is in, where the + // state alone does not say: which other recovery is holding the node, which spec field carries + // an image that cannot be pulled, what a timed-out drain was still waiting on, which Job + // failed and on which attempt, why an approved event is unapproved again. + // +optional + StateMessage string `json:"stateMessage,omitempty"` + + // ImageVerifyGeneration is the plan's metadata.generation at the time of the last failed + // pre-flight image verification for this event, and the reason the event is back in + // waiting-approval with its approval still in place. + // The failure itself is reported in StateMessage, which is what an admin reads; this field is + // only the bookkeeping that keeps the check from repeating. Both are cleared once verification + // succeeds. + // +optional + ImageVerifyGeneration int64 `json:"imageVerifyGeneration,omitempty"` + + // DrainStartedAt is when the node drain for this event began. Reset from nil on each + // attempt so a retry gets a full drain timeout rather than inheriting the previous one. + // +optional + DrainStartedAt *metav1.Time `json:"drainStartedAt,omitempty"` + + // PodsBlockingDrain names the pods still keeping the node from being drained, in + // "namespace/name" form, so an admin can see what a stalled drain is waiting on without + // reading operator logs. Capped at a handful of entries. + // +optional + PodsBlockingDrain []string `json:"podsBlockingDrain,omitempty"` + + // ClaimsBlockingReset names the ResourceClaims that still reserve this GPU, in + // "namespace/name" form. A non-empty list means the reset is held back because a workload + // has not released the device yet. + // +optional + ClaimsBlockingReset []string `json:"claimsBlockingReset,omitempty"` + + // JobName is the name of the Kubernetes Job created to execute the recovery, if any. + // +optional + JobName string `json:"jobName,omitempty"` + + // PastJobs is the names of all Jobs created for this event across all attempts. + // Jobs are retained alive until the event is removed (i.e. the device taint clears), + // so their Pods remain available for diagnostics throughout the event lifecycle. + // +optional + PastJobs []string `json:"pastJobs,omitempty"` + + // RetryCount is the number of times this recovery has been retried after failure. + RetryCount int32 `json:"retryCount"` + + // LastUpdated is the timestamp of the most recent state change for this event. + LastUpdated *metav1.Time `json:"lastUpdated"` +} + +// RecoveryTypeSpec describes which recovery operation to perform, including any +// admin override. +type RecoveryTypeSpec struct { + // Type is the recovery operation to execute: a reset (SBR, slot power cycle, or + // AMC reset) or a firmware reflash. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + Type RecoveryType `json:"type"` + + // SuggestedType is the recovery type originally recommended by the system before + // any admin override was applied. Recorded for audit purposes. + // +kubebuilder:validation:Enum=sbr;slot;amc;reflash + // +optional + SuggestedType RecoveryType `json:"suggestedType,omitempty"` +} + +// IsReflash returns true when this recovery operation is a firmware reflash rather +// than a hardware reset. +func (rt RecoveryTypeSpec) IsReflash() bool { + return rt.Type == RecoveryTypeReflash +} + +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster +// +kubebuilder:printcolumn:name="DeviceID",type=string,JSONPath=`.spec.deviceId` +// +kubebuilder:printcolumn:name="State",type=string,JSONPath=`.status.state` +// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp` + +// GPURecoveryPlan is the Schema for the gpurecoveryplans API. +type GPURecoveryPlan struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec GPURecoveryPlanSpec `json:"spec,omitempty"` + Status GPURecoveryPlanStatus `json:"status,omitempty"` +} + +// +kubebuilder:object:root=true + +// GPURecoveryPlanList contains a list of GPURecoveryPlan. +type GPURecoveryPlanList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []GPURecoveryPlan `json:"items"` +} diff --git a/api/v1alpha1/groupversion_info.go b/api/v1alpha1/groupversion_info.go index 72ff125..0dc581e 100644 --- a/api/v1alpha1/groupversion_info.go +++ b/api/v1alpha1/groupversion_info.go @@ -42,6 +42,8 @@ func addKnownTypes(s *runtime.Scheme) error { &ClusterPolicyList{}, &GPUFirmwareUpdate{}, &GPUFirmwareUpdateList{}, + &GPURecoveryPlan{}, + &GPURecoveryPlanList{}, ) metav1.AddToGroupVersion(s, GroupVersion) diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index f129114..ba7a0ba 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -25,6 +25,28 @@ import ( "k8s.io/apimachinery/pkg/runtime" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalSelector) DeepCopyInto(out *ApprovalSelector) { + *out = *in + if in.NodeSelector != nil { + in, out := &in.NodeSelector, &out.NodeSelector + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalSelector. +func (in *ApprovalSelector) DeepCopy() *ApprovalSelector { + if in == nil { + return nil + } + out := new(ApprovalSelector) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *BuildArg) DeepCopyInto(out *BuildArg) { *out = *in @@ -221,6 +243,21 @@ func (in *ClusterQueueSpec) DeepCopy() *ClusterQueueSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ContainerFirmwareSource) DeepCopyInto(out *ContainerFirmwareSource) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ContainerFirmwareSource. +func (in *ContainerFirmwareSource) DeepCopy() *ContainerFirmwareSource { + if in == nil { + return nil + } + out := new(ContainerFirmwareSource) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *DevicePluginSpec) DeepCopyInto(out *DevicePluginSpec) { *out = *in @@ -251,6 +288,31 @@ func (in *DevicePluginSpec) DeepCopy() *DevicePluginSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *DrainSpec) DeepCopyInto(out *DrainSpec) { + *out = *in + if in.Enable != nil { + in, out := &in.Enable, &out.Enable + *out = new(bool) + **out = **in + } + if in.NamespacesToSkip != nil { + in, out := &in.NamespacesToSkip, &out.NamespacesToSkip + *out = make([]string, len(*in)) + copy(*out, *in) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DrainSpec. +func (in *DrainSpec) DeepCopy() *DrainSpec { + if in == nil { + return nil + } + out := new(DrainSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *DynamicResourceAllocationSpec) DeepCopyInto(out *DynamicResourceAllocationSpec) { *out = *in @@ -481,6 +543,170 @@ func (in *GPUFirmwareUpdateSubsetStatus) DeepCopy() *GPUFirmwareUpdateSubsetStat return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlan) DeepCopyInto(out *GPURecoveryPlan) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlan. +func (in *GPURecoveryPlan) DeepCopy() *GPURecoveryPlan { + if in == nil { + return nil + } + out := new(GPURecoveryPlan) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *GPURecoveryPlan) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlanList) DeepCopyInto(out *GPURecoveryPlanList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]GPURecoveryPlan, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlanList. +func (in *GPURecoveryPlanList) DeepCopy() *GPURecoveryPlanList { + if in == nil { + return nil + } + out := new(GPURecoveryPlanList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *GPURecoveryPlanList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlanSpec) DeepCopyInto(out *GPURecoveryPlanSpec) { + *out = *in + if in.Approvals != nil { + in, out := &in.Approvals, &out.Approvals + *out = make([]RecoveryApproval, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + out.XpuSmi = in.XpuSmi + if in.GoldenFirmware != nil { + in, out := &in.GoldenFirmware, &out.GoldenFirmware + *out = new(GoldenFirmwareSpec) + (*in).DeepCopyInto(*out) + } + if in.Tolerations != nil { + in, out := &in.Tolerations, &out.Tolerations + *out = make([]v1.Toleration, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + in.Drain.DeepCopyInto(&out.Drain) + out.Timeouts = in.Timeouts +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlanSpec. +func (in *GPURecoveryPlanSpec) DeepCopy() *GPURecoveryPlanSpec { + if in == nil { + return nil + } + out := new(GPURecoveryPlanSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GPURecoveryPlanStatus) DeepCopyInto(out *GPURecoveryPlanStatus) { + *out = *in + if in.Messages != nil { + in, out := &in.Messages, &out.Messages + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.Events != nil { + in, out := &in.Events, &out.Events + *out = make([]RecoveryEvent, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GPURecoveryPlanStatus. +func (in *GPURecoveryPlanStatus) DeepCopy() *GPURecoveryPlanStatus { + if in == nil { + return nil + } + out := new(GPURecoveryPlanStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GoldenFirmwareSource) DeepCopyInto(out *GoldenFirmwareSource) { + *out = *in + if in.ContainerSource != nil { + in, out := &in.ContainerSource, &out.ContainerSource + *out = new(ContainerFirmwareSource) + **out = **in + } + if in.VolumeSource != nil { + in, out := &in.VolumeSource, &out.VolumeSource + *out = new(VolumeFirmwareSource) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GoldenFirmwareSource. +func (in *GoldenFirmwareSource) DeepCopy() *GoldenFirmwareSource { + if in == nil { + return nil + } + out := new(GoldenFirmwareSource) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GoldenFirmwareSpec) DeepCopyInto(out *GoldenFirmwareSpec) { + *out = *in + in.Source.DeepCopyInto(&out.Source) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GoldenFirmwareSpec. +func (in *GoldenFirmwareSpec) DeepCopy() *GoldenFirmwareSpec { + if in == nil { + return nil + } + out := new(GoldenFirmwareSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *HealthinessSpec) DeepCopyInto(out *HealthinessSpec) { *out = *in @@ -621,6 +847,119 @@ func (in *LocalQueueSpec) DeepCopy() *LocalQueueSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryApproval) DeepCopyInto(out *RecoveryApproval) { + *out = *in + if in.Selector != nil { + in, out := &in.Selector, &out.Selector + *out = new(ApprovalSelector) + (*in).DeepCopyInto(*out) + } + if in.Override != nil { + in, out := &in.Override, &out.Override + *out = new(RecoveryOverride) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryApproval. +func (in *RecoveryApproval) DeepCopy() *RecoveryApproval { + if in == nil { + return nil + } + out := new(RecoveryApproval) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryEvent) DeepCopyInto(out *RecoveryEvent) { + *out = *in + if in.ApprovalMatchedAt != nil { + in, out := &in.ApprovalMatchedAt, &out.ApprovalMatchedAt + *out = (*in).DeepCopy() + } + out.RecoveryType = in.RecoveryType + if in.DrainStartedAt != nil { + in, out := &in.DrainStartedAt, &out.DrainStartedAt + *out = (*in).DeepCopy() + } + if in.PodsBlockingDrain != nil { + in, out := &in.PodsBlockingDrain, &out.PodsBlockingDrain + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.ClaimsBlockingReset != nil { + in, out := &in.ClaimsBlockingReset, &out.ClaimsBlockingReset + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.PastJobs != nil { + in, out := &in.PastJobs, &out.PastJobs + *out = make([]string, len(*in)) + copy(*out, *in) + } + if in.LastUpdated != nil { + in, out := &in.LastUpdated, &out.LastUpdated + *out = (*in).DeepCopy() + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryEvent. +func (in *RecoveryEvent) DeepCopy() *RecoveryEvent { + if in == nil { + return nil + } + out := new(RecoveryEvent) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryOverride) DeepCopyInto(out *RecoveryOverride) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryOverride. +func (in *RecoveryOverride) DeepCopy() *RecoveryOverride { + if in == nil { + return nil + } + out := new(RecoveryOverride) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryTimeoutsSpec) DeepCopyInto(out *RecoveryTimeoutsSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryTimeoutsSpec. +func (in *RecoveryTimeoutsSpec) DeepCopy() *RecoveryTimeoutsSpec { + if in == nil { + return nil + } + out := new(RecoveryTimeoutsSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RecoveryTypeSpec) DeepCopyInto(out *RecoveryTypeSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecoveryTypeSpec. +func (in *RecoveryTypeSpec) DeepCopy() *RecoveryTypeSpec { + if in == nil { + return nil + } + out := new(RecoveryTypeSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RegistryTLSSpec) DeepCopyInto(out *RegistryTLSSpec) { *out = *in @@ -636,6 +975,21 @@ func (in *RegistryTLSSpec) DeepCopy() *RegistryTLSSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *VolumeFirmwareSource) DeepCopyInto(out *VolumeFirmwareSource) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new VolumeFirmwareSource. +func (in *VolumeFirmwareSource) DeepCopy() *VolumeFirmwareSource { + if in == nil { + return nil + } + out := new(VolumeFirmwareSource) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *XpuManagerSpec) DeepCopyInto(out *XpuManagerSpec) { *out = *in @@ -655,3 +1009,18 @@ func (in *XpuManagerSpec) DeepCopy() *XpuManagerSpec { in.DeepCopyInto(out) return out } + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *XpuSmiSpec) DeepCopyInto(out *XpuSmiSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new XpuSmiSpec. +func (in *XpuSmiSpec) DeepCopy() *XpuSmiSpec { + if in == nil { + return nil + } + out := new(XpuSmiSpec) + in.DeepCopyInto(out) + return out +} diff --git a/charts/gpu-base-operator/crds/gpurecoveryplans.yaml b/charts/gpu-base-operator/crds/gpurecoveryplans.yaml new file mode 100644 index 0000000..90d0e95 --- /dev/null +++ b/charts/gpu-base-operator/crds/gpurecoveryplans.yaml @@ -0,0 +1,550 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.18.0 + name: gpurecoveryplans.intel.com +spec: + group: intel.com + names: + kind: GPURecoveryPlan + listKind: GPURecoveryPlanList + plural: gpurecoveryplans + singular: gpurecoveryplan + scope: Cluster + versions: + - additionalPrinterColumns: + - jsonPath: .spec.deviceId + name: DeviceID + type: string + - jsonPath: .status.state + name: State + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: GPURecoveryPlan is the Schema for the gpurecoveryplans API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: GPURecoveryPlanSpec defines the desired state of GPURecoveryPlan. + properties: + approvals: + description: |- + Approvals contains admin-provided authorisations for specific or grouped recovery events. + The operator generates an ID for any entry that is missing one. + items: + description: |- + RecoveryApproval authorises one or more recovery events. + Exactly one of EventID or Selector must be set. + An approval is active as long as it exists in the list and Consumed is false. + properties: + comment: + description: Comment is an optional human-readable note about + why this approval was granted. + type: string + consumed: + description: |- + Consumed is set to true by the operator after a non-persistent approval has been + matched and acted upon. Persistent approvals are never consumed. + type: boolean + eventId: + description: |- + EventID references a specific event by its status.events[].id. + Mutually exclusive with Selector. + type: string + id: + description: |- + ID is the unique identifier for this approval entry. + The operator generates an ID when this field is empty. + type: string + override: + description: |- + Override substitutes a different recovery type than the system recommendation. + When set on an EventID approval, the system records the original suggestion in + status.events[].recoveryType.suggestedType for audit purposes. + properties: + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType is the recovery operation to use + instead of the system suggestion. + type: string + required: + - recoveryType + type: object + persistent: + description: |- + Persistent keeps this approval alive after matched events are processed so that + future events matching the Selector are approved automatically. + Only meaningful when Selector is set. + type: boolean + selector: + description: |- + Selector applies this approval to all currently matching events. + Mutually exclusive with EventID. + properties: + nodeName: + description: NodeName selects events on a specific node. + type: string + nodeSelector: + additionalProperties: + type: string + description: NodeSelector selects events on nodes that have + all given labels. + type: object + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType selects events whose suggested + recovery type matches. + type: string + type: object + type: object + type: array + defaultResetType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - slot + - amc + description: |- + DefaultResetType is the reset the operator runs for every reset-type recovery event it + creates on this plan. Either "slot" (PCIe slot power cycle, also called the PM reset) or + "amc" (out-of-band reset through the card's AMC). + type: string + deviceId: + description: 'DeviceID is the mandatory PCI device ID of the target + GPU. Format: ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + drain: + default: + enable: true + timeoutSeconds: 300 + description: Drain configures the node drain that precedes a reset-type + recovery. + properties: + enable: + default: true + description: Enable controls whether the node is drained before + a reset. + type: boolean + namespacesToSkip: + description: |- + NamespacesToSkip lists namespaces the drain leaves alone: their pods are neither evicted + nor waited for. Use it for cluster infrastructure that is fine to keep running through a + reset and that a whole-node drain would otherwise have to evict. + items: + maxLength: 63 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + maxItems: 64 + type: array + timeoutSeconds: + default: 300 + description: |- + TimeoutSeconds bounds how long a reset waits for the node to drain before the event is + failed. + format: int32 + minimum: 1 + type: integer + type: object + goldenFirmware: + description: |- + GoldenFirmware holds configuration for reflash-type recovery operations. + These fields are protected: changes are rejected while any reflash event is active. + properties: + file: + description: |- + File is the filename of the FDO firmware image to flash, relative to the root of the + source (no path components). + type: string + source: + description: Source specifies where the firmware file can be found. + properties: + containerSource: + description: ContainerSource specifies a container image that + holds the firmware file. + properties: + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to verify this image and the firmware file inside it. Use it where the registry + serves a self-signed or private-CA certificate. + type: boolean + name: + description: Name is the container image reference (e.g. + registry/image:tag). + type: string + required: + - name + type: object + volumeSource: + description: |- + VolumeSource specifies a Kubernetes volume that holds the firmware file. + + NOT YET SUPPORTED. The field is validated but not acted upon: a reflash event on a plan + whose source sets only volumeSource stays in the missing-goldenfw state, because the + reflash Job copies firmware from a container image. Use containerSource until this is + implemented. + properties: + name: + description: Name is the name of the PersistentVolumeClaim. + type: string + required: + - name + type: object + type: object + required: + - file + - source + type: object + maxRetries: + default: 3 + description: |- + MaxRetries is the maximum number of times a failed recovery event is automatically + re-queued for approval and retried while its device taint persists. Once this limit + is reached the event stays in the failed state and requires manual intervention + (e.g. delete the event entry or increase MaxRetries). Setting 0 disables automatic + retries entirely. + format: int32 + minimum: 0 + type: integer + skipImageVerification: + description: |- + SkipImageVerification disables the pre-flight registry check on the images a recovery Job + needs (spec.xpuSmi.image, and the golden firmware image for a reflash). + type: boolean + subDeviceId: + description: 'SubDeviceID is the optional PCI sub-device ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + subVendorId: + description: 'SubVendorID is the optional PCI sub-vendor ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + timeouts: + default: + reflashSeconds: 600 + resetSeconds: 300 + description: Timeouts bounds how long the recovery Jobs themselves + may run. + properties: + reflashSeconds: + default: 600 + description: ReflashSeconds bounds a reflash recovery Job. + format: int32 + minimum: 1 + type: integer + resetSeconds: + default: 300 + description: ResetSeconds bounds a reset-type recovery Job (SBR, + slot power cycle, AMC reset). + format: int32 + minimum: 1 + type: integer + type: object + tolerations: + description: |- + Tolerations are added to recovery Job pods on top of the blanket toleration the operator + always sets, for cases where a cluster needs an extra entry. + items: + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . + properties: + effect: + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. + type: string + operator: + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. + Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). + type: string + tolerationSeconds: + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. + format: int64 + type: integer + value: + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. + type: string + type: object + type: array + xpuSmi: + default: + pullPolicy: IfNotPresent + description: XpuSmi configures the container image providing the xpu-smi + tool. + properties: + image: + description: |- + Image is the container image providing the xpu-smi tool, used for both hardware resets + and firmware reflash operations. + type: string + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to check that Image resolves. Use it where the registry serves a self-signed or + private-CA certificate. + type: boolean + pullPolicy: + default: IfNotPresent + description: PullPolicy controls the image pull policy for the + xpu-smi container. + enum: + - Always + - IfNotPresent + - Never + type: string + type: object + required: + - defaultResetType + - deviceId + - maxRetries + type: object + status: + description: GPURecoveryPlanStatus defines the observed state of GPURecoveryPlan. + properties: + events: + description: Events is the list of active and recently completed recovery + needs (capped at 1000). + items: + description: RecoveryEvent represents a single detected GPU recovery + need and its lifecycle state. + properties: + approvalId: + description: ApprovalID is the ID of the spec.approvals entry + that matched and authorised this event. + type: string + approvalMatchedAt: + description: ApprovalMatchedAt is the timestamp when an approval + first matched this event. + format: date-time + type: string + claimsBlockingReset: + description: |- + ClaimsBlockingReset names the ResourceClaims that still reserve this GPU, in + "namespace/name" form. A non-empty list means the reset is held back because a workload + has not released the device yet. + items: + type: string + type: array + drainStartedAt: + description: |- + DrainStartedAt is when the node drain for this event began. Reset from nil on each + attempt so a retry gets a full drain timeout rather than inheriting the previous one. + format: date-time + type: string + gpuBDF: + description: GPUBDF is the PCI Bus:Device.Function address of + the affected GPU (e.g. "0000:02:00.0"). + type: string + id: + description: ID is the operator-assigned unique event identifier + (e.g. "evt-a3f2"). + type: string + imageVerifyGeneration: + description: |- + ImageVerifyGeneration is the plan's metadata.generation at the time of the last failed + pre-flight image verification for this event, and the reason the event is back in + waiting-approval with its approval still in place. + The failure itself is reported in StateMessage, which is what an admin reads; this field is + only the bookkeeping that keeps the check from repeating. Both are cleared once verification + succeeds. + format: int64 + type: integer + jobName: + description: JobName is the name of the Kubernetes Job created + to execute the recovery, if any. + type: string + lastUpdated: + description: LastUpdated is the timestamp of the most recent + state change for this event. + format: date-time + type: string + nodeName: + description: NodeName is the Kubernetes node hosting the affected + GPU. + type: string + pastJobs: + description: |- + PastJobs is the names of all Jobs created for this event across all attempts. + Jobs are retained alive until the event is removed (i.e. the device taint clears), + so their Pods remain available for diagnostics throughout the event lifecycle. + items: + type: string + type: array + podsBlockingDrain: + description: |- + PodsBlockingDrain names the pods still keeping the node from being drained, in + "namespace/name" form, so an admin can see what a stalled drain is waiting on without + reading operator logs. Capped at a handful of entries. + items: + type: string + type: array + reason: + description: Reason is the human-readable cause of this event + (e.g. "gpu-wedged", "survivability-mode"). + type: string + recoveryType: + description: RecoveryType describes the recovery operation to + perform. + properties: + suggestedType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + SuggestedType is the recovery type originally recommended by the system before + any admin override was applied. Recorded for audit purposes. + type: string + type: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + Type is the recovery operation to execute: a reset (SBR, slot power cycle, or + AMC reset) or a firmware reflash. + type: string + required: + - type + type: object + retryCount: + description: RetryCount is the number of times this recovery + has been retried after failure. + format: int32 + type: integer + state: + allOf: + - enum: + - waiting-approval + - missing-goldenfw + - blocked + - draining + - in-progress + - succeeded + - failed + - enum: + - waiting-approval + - missing-goldenfw + - blocked + - draining + - in-progress + - succeeded + - failed + description: State is the current lifecycle state of this recovery + event. + type: string + stateMessage: + description: |- + StateMessage explains, in one sentence, why this event is in the state it is in, where the + state alone does not say: which other recovery is holding the node, which spec field carries + an image that cannot be pulled, what a timed-out drain was still waiting on, which Job + failed and on which attempt, why an approved event is unapproved again. + type: string + required: + - gpuBDF + - id + - lastUpdated + - nodeName + - recoveryType + - retryCount + - state + type: object + type: array + messages: + description: Messages contains recent informational or error messages + (most recent last, capped at ~50). + items: + type: string + type: array + state: + description: State is the overall state of this recovery plan. + enum: + - idle + - error + - active + type: string + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/intel.com_gpurecoveryplans.yaml b/config/crd/bases/intel.com_gpurecoveryplans.yaml new file mode 100644 index 0000000..90d0e95 --- /dev/null +++ b/config/crd/bases/intel.com_gpurecoveryplans.yaml @@ -0,0 +1,550 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.18.0 + name: gpurecoveryplans.intel.com +spec: + group: intel.com + names: + kind: GPURecoveryPlan + listKind: GPURecoveryPlanList + plural: gpurecoveryplans + singular: gpurecoveryplan + scope: Cluster + versions: + - additionalPrinterColumns: + - jsonPath: .spec.deviceId + name: DeviceID + type: string + - jsonPath: .status.state + name: State + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: GPURecoveryPlan is the Schema for the gpurecoveryplans API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: GPURecoveryPlanSpec defines the desired state of GPURecoveryPlan. + properties: + approvals: + description: |- + Approvals contains admin-provided authorisations for specific or grouped recovery events. + The operator generates an ID for any entry that is missing one. + items: + description: |- + RecoveryApproval authorises one or more recovery events. + Exactly one of EventID or Selector must be set. + An approval is active as long as it exists in the list and Consumed is false. + properties: + comment: + description: Comment is an optional human-readable note about + why this approval was granted. + type: string + consumed: + description: |- + Consumed is set to true by the operator after a non-persistent approval has been + matched and acted upon. Persistent approvals are never consumed. + type: boolean + eventId: + description: |- + EventID references a specific event by its status.events[].id. + Mutually exclusive with Selector. + type: string + id: + description: |- + ID is the unique identifier for this approval entry. + The operator generates an ID when this field is empty. + type: string + override: + description: |- + Override substitutes a different recovery type than the system recommendation. + When set on an EventID approval, the system records the original suggestion in + status.events[].recoveryType.suggestedType for audit purposes. + properties: + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType is the recovery operation to use + instead of the system suggestion. + type: string + required: + - recoveryType + type: object + persistent: + description: |- + Persistent keeps this approval alive after matched events are processed so that + future events matching the Selector are approved automatically. + Only meaningful when Selector is set. + type: boolean + selector: + description: |- + Selector applies this approval to all currently matching events. + Mutually exclusive with EventID. + properties: + nodeName: + description: NodeName selects events on a specific node. + type: string + nodeSelector: + additionalProperties: + type: string + description: NodeSelector selects events on nodes that have + all given labels. + type: object + recoveryType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: RecoveryType selects events whose suggested + recovery type matches. + type: string + type: object + type: object + type: array + defaultResetType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - slot + - amc + description: |- + DefaultResetType is the reset the operator runs for every reset-type recovery event it + creates on this plan. Either "slot" (PCIe slot power cycle, also called the PM reset) or + "amc" (out-of-band reset through the card's AMC). + type: string + deviceId: + description: 'DeviceID is the mandatory PCI device ID of the target + GPU. Format: ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + drain: + default: + enable: true + timeoutSeconds: 300 + description: Drain configures the node drain that precedes a reset-type + recovery. + properties: + enable: + default: true + description: Enable controls whether the node is drained before + a reset. + type: boolean + namespacesToSkip: + description: |- + NamespacesToSkip lists namespaces the drain leaves alone: their pods are neither evicted + nor waited for. Use it for cluster infrastructure that is fine to keep running through a + reset and that a whole-node drain would otherwise have to evict. + items: + maxLength: 63 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + maxItems: 64 + type: array + timeoutSeconds: + default: 300 + description: |- + TimeoutSeconds bounds how long a reset waits for the node to drain before the event is + failed. + format: int32 + minimum: 1 + type: integer + type: object + goldenFirmware: + description: |- + GoldenFirmware holds configuration for reflash-type recovery operations. + These fields are protected: changes are rejected while any reflash event is active. + properties: + file: + description: |- + File is the filename of the FDO firmware image to flash, relative to the root of the + source (no path components). + type: string + source: + description: Source specifies where the firmware file can be found. + properties: + containerSource: + description: ContainerSource specifies a container image that + holds the firmware file. + properties: + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to verify this image and the firmware file inside it. Use it where the registry + serves a self-signed or private-CA certificate. + type: boolean + name: + description: Name is the container image reference (e.g. + registry/image:tag). + type: string + required: + - name + type: object + volumeSource: + description: |- + VolumeSource specifies a Kubernetes volume that holds the firmware file. + + NOT YET SUPPORTED. The field is validated but not acted upon: a reflash event on a plan + whose source sets only volumeSource stays in the missing-goldenfw state, because the + reflash Job copies firmware from a container image. Use containerSource until this is + implemented. + properties: + name: + description: Name is the name of the PersistentVolumeClaim. + type: string + required: + - name + type: object + type: object + required: + - file + - source + type: object + maxRetries: + default: 3 + description: |- + MaxRetries is the maximum number of times a failed recovery event is automatically + re-queued for approval and retried while its device taint persists. Once this limit + is reached the event stays in the failed state and requires manual intervention + (e.g. delete the event entry or increase MaxRetries). Setting 0 disables automatic + retries entirely. + format: int32 + minimum: 0 + type: integer + skipImageVerification: + description: |- + SkipImageVerification disables the pre-flight registry check on the images a recovery Job + needs (spec.xpuSmi.image, and the golden firmware image for a reflash). + type: boolean + subDeviceId: + description: 'SubDeviceID is the optional PCI sub-device ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + subVendorId: + description: 'SubVendorID is the optional PCI sub-vendor ID. Format: + ''0x'' followed by 4 hex digits.' + pattern: ^0x[0-9a-fA-F]{4}$ + type: string + timeouts: + default: + reflashSeconds: 600 + resetSeconds: 300 + description: Timeouts bounds how long the recovery Jobs themselves + may run. + properties: + reflashSeconds: + default: 600 + description: ReflashSeconds bounds a reflash recovery Job. + format: int32 + minimum: 1 + type: integer + resetSeconds: + default: 300 + description: ResetSeconds bounds a reset-type recovery Job (SBR, + slot power cycle, AMC reset). + format: int32 + minimum: 1 + type: integer + type: object + tolerations: + description: |- + Tolerations are added to recovery Job pods on top of the blanket toleration the operator + always sets, for cases where a cluster needs an extra entry. + items: + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . + properties: + effect: + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. + type: string + operator: + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. + Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). + type: string + tolerationSeconds: + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. + format: int64 + type: integer + value: + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. + type: string + type: object + type: array + xpuSmi: + default: + pullPolicy: IfNotPresent + description: XpuSmi configures the container image providing the xpu-smi + tool. + properties: + image: + description: |- + Image is the container image providing the xpu-smi tool, used for both hardware resets + and firmware reflash operations. + type: string + insecureSkipTLSVerify: + description: |- + InsecureSkipTLSVerify disables TLS certificate verification when the operator contacts the + registry to check that Image resolves. Use it where the registry serves a self-signed or + private-CA certificate. + type: boolean + pullPolicy: + default: IfNotPresent + description: PullPolicy controls the image pull policy for the + xpu-smi container. + enum: + - Always + - IfNotPresent + - Never + type: string + type: object + required: + - defaultResetType + - deviceId + - maxRetries + type: object + status: + description: GPURecoveryPlanStatus defines the observed state of GPURecoveryPlan. + properties: + events: + description: Events is the list of active and recently completed recovery + needs (capped at 1000). + items: + description: RecoveryEvent represents a single detected GPU recovery + need and its lifecycle state. + properties: + approvalId: + description: ApprovalID is the ID of the spec.approvals entry + that matched and authorised this event. + type: string + approvalMatchedAt: + description: ApprovalMatchedAt is the timestamp when an approval + first matched this event. + format: date-time + type: string + claimsBlockingReset: + description: |- + ClaimsBlockingReset names the ResourceClaims that still reserve this GPU, in + "namespace/name" form. A non-empty list means the reset is held back because a workload + has not released the device yet. + items: + type: string + type: array + drainStartedAt: + description: |- + DrainStartedAt is when the node drain for this event began. Reset from nil on each + attempt so a retry gets a full drain timeout rather than inheriting the previous one. + format: date-time + type: string + gpuBDF: + description: GPUBDF is the PCI Bus:Device.Function address of + the affected GPU (e.g. "0000:02:00.0"). + type: string + id: + description: ID is the operator-assigned unique event identifier + (e.g. "evt-a3f2"). + type: string + imageVerifyGeneration: + description: |- + ImageVerifyGeneration is the plan's metadata.generation at the time of the last failed + pre-flight image verification for this event, and the reason the event is back in + waiting-approval with its approval still in place. + The failure itself is reported in StateMessage, which is what an admin reads; this field is + only the bookkeeping that keeps the check from repeating. Both are cleared once verification + succeeds. + format: int64 + type: integer + jobName: + description: JobName is the name of the Kubernetes Job created + to execute the recovery, if any. + type: string + lastUpdated: + description: LastUpdated is the timestamp of the most recent + state change for this event. + format: date-time + type: string + nodeName: + description: NodeName is the Kubernetes node hosting the affected + GPU. + type: string + pastJobs: + description: |- + PastJobs is the names of all Jobs created for this event across all attempts. + Jobs are retained alive until the event is removed (i.e. the device taint clears), + so their Pods remain available for diagnostics throughout the event lifecycle. + items: + type: string + type: array + podsBlockingDrain: + description: |- + PodsBlockingDrain names the pods still keeping the node from being drained, in + "namespace/name" form, so an admin can see what a stalled drain is waiting on without + reading operator logs. Capped at a handful of entries. + items: + type: string + type: array + reason: + description: Reason is the human-readable cause of this event + (e.g. "gpu-wedged", "survivability-mode"). + type: string + recoveryType: + description: RecoveryType describes the recovery operation to + perform. + properties: + suggestedType: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + SuggestedType is the recovery type originally recommended by the system before + any admin override was applied. Recorded for audit purposes. + type: string + type: + allOf: + - enum: + - sbr + - slot + - amc + - reflash + - enum: + - sbr + - slot + - amc + - reflash + description: |- + Type is the recovery operation to execute: a reset (SBR, slot power cycle, or + AMC reset) or a firmware reflash. + type: string + required: + - type + type: object + retryCount: + description: RetryCount is the number of times this recovery + has been retried after failure. + format: int32 + type: integer + state: + allOf: + - enum: + - waiting-approval + - missing-goldenfw + - blocked + - draining + - in-progress + - succeeded + - failed + - enum: + - waiting-approval + - missing-goldenfw + - blocked + - draining + - in-progress + - succeeded + - failed + description: State is the current lifecycle state of this recovery + event. + type: string + stateMessage: + description: |- + StateMessage explains, in one sentence, why this event is in the state it is in, where the + state alone does not say: which other recovery is holding the node, which spec field carries + an image that cannot be pulled, what a timed-out drain was still waiting on, which Job + failed and on which attempt, why an approved event is unapproved again. + type: string + required: + - gpuBDF + - id + - lastUpdated + - nodeName + - recoveryType + - retryCount + - state + type: object + type: array + messages: + description: Messages contains recent informational or error messages + (most recent last, capped at ~50). + items: + type: string + type: array + state: + description: State is the overall state of this recovery plan. + enum: + - idle + - error + - active + type: string + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/kustomization.yaml b/config/crd/kustomization.yaml index bae15af..ce6ed5b 100644 --- a/config/crd/kustomization.yaml +++ b/config/crd/kustomization.yaml @@ -4,6 +4,7 @@ resources: - bases/intel.com_clusterpolicies.yaml - bases/intel.com_gpufirmwareupdates.yaml +- bases/intel.com_gpurecoveryplans.yaml # +kubebuilder:scaffold:crdkustomizeresource patches: diff --git a/config/rbac/gpurecoveryplan_admin_role.yaml b/config/rbac/gpurecoveryplan_admin_role.yaml new file mode 100644 index 0000000..6bd4bd8 --- /dev/null +++ b/config/rbac/gpurecoveryplan_admin_role.yaml @@ -0,0 +1,27 @@ +# This rule is not used by the project intel-gpu-base-operator itself. +# It is provided to allow the cluster admin to help manage permissions for users. +# +# Grants full permissions ('*') over intel.com. +# This role is intended for users authorized to modify roles and bindings within the cluster, +# enabling them to delegate specific permissions to other users or groups as needed. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-admin-role +rules: +- apiGroups: + - intel.com + resources: + - gpurecoveryplans + verbs: + - '*' +- apiGroups: + - intel.com + resources: + - gpurecoveryplans/status + verbs: + - get diff --git a/config/rbac/gpurecoveryplan_editor_role.yaml b/config/rbac/gpurecoveryplan_editor_role.yaml new file mode 100644 index 0000000..d6c494c --- /dev/null +++ b/config/rbac/gpurecoveryplan_editor_role.yaml @@ -0,0 +1,33 @@ +# This rule is not used by the project intel-gpu-base-operator itself. +# It is provided to allow the cluster admin to help manage permissions for users. +# +# Grants permissions to create, update, and delete resources within the intel.com. +# This role is intended for users who need to manage these resources +# but should not control RBAC or manage permissions for others. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-editor-role +rules: +- apiGroups: + - intel.com + resources: + - gpurecoveryplans + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - intel.com + resources: + - gpurecoveryplans/status + verbs: + - get diff --git a/config/rbac/gpurecoveryplan_viewer_role.yaml b/config/rbac/gpurecoveryplan_viewer_role.yaml new file mode 100644 index 0000000..16d0dc3 --- /dev/null +++ b/config/rbac/gpurecoveryplan_viewer_role.yaml @@ -0,0 +1,29 @@ +# This rule is not used by the project intel-gpu-base-operator itself. +# It is provided to allow the cluster admin to help manage permissions for users. +# +# Grants read-only access to intel.com resources. +# This role is intended for users who need visibility into these resources +# without permissions to modify them. It is ideal for monitoring purposes and limited-access viewing. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-viewer-role +rules: +- apiGroups: + - intel.com + resources: + - gpurecoveryplans + verbs: + - get + - list + - watch +- apiGroups: + - intel.com + resources: + - gpurecoveryplans/status + verbs: + - get diff --git a/config/rbac/kustomization.yaml b/config/rbac/kustomization.yaml index 47aafc1..8f729c7 100644 --- a/config/rbac/kustomization.yaml +++ b/config/rbac/kustomization.yaml @@ -24,6 +24,9 @@ resources: # default, aiding admins in cluster management. Those roles are # not used by the intel-gpu-base-operator itself. You can comment the following lines # if you do not want those helpers be installed with your Project. +- gpurecoveryplan_admin_role.yaml +- gpurecoveryplan_editor_role.yaml +- gpurecoveryplan_viewer_role.yaml - gpufirmwareupdate_admin_role.yaml - gpufirmwareupdate_editor_role.yaml - gpufirmwareupdate_viewer_role.yaml diff --git a/config/samples/recoveryplan/gpurecoveryplan.yaml b/config/samples/recoveryplan/gpurecoveryplan.yaml new file mode 100644 index 0000000..f904236 --- /dev/null +++ b/config/samples/recoveryplan/gpurecoveryplan.yaml @@ -0,0 +1,88 @@ +apiVersion: intel.com/v1alpha1 +kind: GPURecoveryPlan +metadata: + labels: + app.kubernetes.io/name: intel-gpu-base-operator + app.kubernetes.io/managed-by: kustomize + name: gpurecoveryplan-sample +spec: + deviceId: "0xe20b" + + # Mandatory. Which reset works is a property of the platform, not of the fault: use slot — the + # PCIe slot power cycle, also called the PM reset — where the slots support hot-plug, and amc + # where they do not. The DRA driver only reports "needs a reset", so this is the operator's only + # way to know which one to run, and getting it wrong is silent: the Job runs a reset the platform + # cannot perform, exits 0, and the event succeeds with the GPU still broken. + # sbr is not accepted here — it is the per-card backup, reached through an approval's override + # when the platform's normal reset does not revive one particular GPU. + defaultResetType: "slot" + + # The node drain that runs before a reset. None of it applies to a reflash: that writes firmware to + # a card which is already unusable without driving the PCIe bus, so it never drains. + drain: + # Turning the drain off means a reset can hit the bus while workloads are still running on the + # node. Prefer namespacesToSkip below if the problem is one namespace that will not drain. + # enable: true + + # How long a reset waits for the node to empty. When it expires the event fails with + # drain-timeout and status.events[].podsBlockingDrain names what it was still waiting on. + # timeoutSeconds: 300 + + # Namespaces the drain leaves alone: their pods are neither evicted nor waited for. Intended for + # node-level infrastructure that is fine to keep running through a reset and that would otherwise + # stall the drain — a PodDisruptionBudget with nowhere to reschedule, or a pod no controller + # recreates. The operator's own namespace is always skipped and does not need listing, and + # DaemonSet and static pods are skipped regardless of namespace. + # + # This exempts a namespace from the *drain*, not from the reset: its pods stay on the node + # through the PCIe reset. So do not list namespaces that run GPU workloads — including ones + # holding a GPU through a DRA claim, which is not waited for either when the only pods reserving + # it are ones this list keeps on the node. + # namespacesToSkip: + # - kube-system + # - cert-manager + + # How long a recovery Job itself may run before Kubernetes kills it (the Job's + # activeDeadlineSeconds). Separate from drain.timeoutSeconds above, which bounds only the wait for + # the node to empty. How long the hardware takes is a property of the platform — a chassis slow to + # re-enumerate a power-cycled slot, an FDO flash over a slow SPI part — and a deadline that expires + # early is not a harmless retry: the Job is killed mid-operation and the event fails over a card + # that was in fact recovering. Raising these only costs time before a genuine failure is reported, + # so err high where the numbers are uncertain. + # timeouts: + # resetSeconds: 300 # sbr, slot, amc + # reflashSeconds: 600 # includes copying the firmware out of the golden firmware image + + # The xpu-smi image every recovery Job runs, for both the resets and the reflash. + xpuSmi: + image: "docker.io/intel/gpu-fwupdater-mock:devel" + pullPolicy: "Always" + + # Skip registry certificate validation for the operator's pre-flight check on the image + # above. Needed where the registry serves a self-signed or private-CA certificate — but only + # for the operator's own check: the node still has to trust that registry (or have it listed + # as an insecure one) for the kubelet to pull the image. + # insecureSkipTLSVerify: false + + # The image to flash onto a GPU that has entered FDO mode. Always flashed as firmware type + # FDO — the only type that can revive such a card — so no type is configurable here. + goldenFirmware: + source: + containerSource: + name: "docker.io/intel/intel-gpu-fw-binaries:devel" + + # Same as spec.xpuSmi.insecureSkipTLSVerify, and separate from it because the golden + # firmware for one card model often lives on an internal registry while xpu-smi comes + # from a public one. + # insecureSkipTLSVerify: false + + file: "gpu_firmware.bin" + + # Both images above are resolved against their registry before any recovery Job is created, so + # an unpullable reference holds the event at waiting-approval — with its approval retained and + # the reason in that event's status.events[].stateMessage, as well as in the plan-wide + # status.messages — instead of producing a Job that reports in-progress from + # ImagePullBackOff. The check runs once per spec version: correcting the plan is what makes the + # operator look again. Set this only where the nodes hold pull credentials the operator cannot + # see, which would make the check stricter than the pull it predicts. + # skipImageVerification: false diff --git a/config/samples/recoveryplan/kustomization.yaml b/config/samples/recoveryplan/kustomization.yaml new file mode 100644 index 0000000..b6d7e89 --- /dev/null +++ b/config/samples/recoveryplan/kustomization.yaml @@ -0,0 +1,4 @@ +## Append samples of your project ## +resources: +- gpurecoveryplan.yaml +# +kubebuilder:scaffold:manifestskustomizesamples