Use reflector for all repeated read operations, set read timeout & a test timeout greater than that, CI stress edition

[Jakob-Naucke]

Summary by Sourcery

Consolidate controller caching and timeout handling to reduce repeated Kubernetes reads and improve retry resilience.

Bug Fixes:

• Add bounded retry coverage to the registration server so clients can tolerate operator read-timeout periods.

Enhancements:

• Centralize reflector-backed caches and controller state in a shared operator context to eliminate repeated read operations.
• Use server-side apply patches for ConfigMaps and Deployments while sourcing repeated reads from local reflector stores.
• Configure a Kubernetes API read timeout and adjust controller requeue behavior for more reliable retries.
• Filter attestation-key secret reconciliation and PCR job ownership using resource labels.

Build:

• Update the integration-test image used by the Makefile.

CI:

• Stress integration tests by retrying the test suite with an extended timeout multiplier.

Tests:

• Update unit tests to exercise reflector-backed state and patch-based API interactions.

Chores:

• Remove the shared image-PCR update macro and consolidate its update logic in the operator and compute-PCRs components.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jakob-Naucke

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sourcery-ai]

Reviewer's Guide

Refactors the operator to use a unified OperatorContext backed by reflector caches for repeated reads, switches several config/deployment mutations to server-side apply patches, adjusts controllers and tests accordingly, and tightens CI and runtime timeouts for more reliable stress testing.

Sequence diagram for keygen_reconcile using reflector-backed OperatorContext

sequenceDiagram
    actor Operator
    participant Machine
    participant KeygenController as keygen_reconcile
    participant OperatorContext
    participant Trustee as trustee
    participant TecStore as tec_store

    Operator->>Machine: create/update Machine
    Machine-->>KeygenController: event

    KeygenController->>OperatorContext: access client
    KeygenController->>Machine: inspect spec.id
    alt Event::Apply
        KeygenController->>Machine: generate_owner_reference(machine)
        KeygenController->>trustee: generate_secret(client, id, owner_reference)
        KeygenController->>trustee: mount_secret(OperatorContext, id)
        KeygenController-->>Operator: Action::await_change
    else Event::Cleanup
        KeygenController->>Machine: read owner_references
        KeygenController->>TecStore: get(ObjectRef::new(tec_name).within(ns))
        alt TEC deleting or missing
            KeygenController-->>Operator: Action::await_change
        else TEC active
            KeygenController->>trustee: unmount_secret(OperatorContext, id)
            KeygenController-->>Operator: Action::await_change
        end
    end

Loading

File-Level Changes

Change	Details	Files
Introduce a unified OperatorContext with reflector-backed stores and use it across controllers and trustee helpers instead of ad‑hoc contexts or direct GETs.	Define OperatorContext with cached stores for TrustedExecutionCluster, ConfigMap, Deployment, Machine, AttestationKey, Secret, and ApprovedImage Add helper get_opt_tec on OperatorContext to return the single TrustedExecutionCluster or error on multiple Replace AkContextData and ClusterContext usages with OperatorContext in attestation_key_register, reference_values, main, register_server, and trustee call sites	operator/src/lib.rs operator/src/main.rs operator/src/attestation_key_register.rs operator/src/reference_values.rs operator/src/register_server.rs operator/src/trustee.rs
Switch ConfigMap and Deployment mutations to use Patch::Apply with PatchParams::apply and cached objects, reducing repeated reads and improving idempotency.	Change update_reference_values to accept ImagePcrs and patch TRUSTEE_DATA_MAP via server-side apply instead of GET+replace Change do_mount_secret to operate on cached Deployment from deployment_store and patch volumes/volumeMounts via server-side apply Update update_attestation_keys to use Patch::Apply with PatchParams and cached Deployment Introduce apply_image_pcrs to patch PCR_CONFIG_MAP via server-side apply and call update_reference_values, removing the update_image_pcrs macro Adjust compute-pcrs main to inline the previous macro logic for PCR_CONFIG_MAP update	operator/src/trustee.rs operator/src/reference_values.rs compute-pcrs/src/main.rs lib/src/reference_values.rs
Refactor reference value and image controllers to consume cached data from OperatorContext stores, reduce API calls, and wire ownership between ApprovedImage and Job controllers.	Add cached_image_pcrs helper to read ImagePcrs from cm_store Refactor job_reconcile to take OperatorContext, delete completed Jobs, read PCRs from cache, and call update_reference_values with ImagePcrs Refactor launch_rv_job_controller and launch_rv_image_controller to accept OperatorContext, and have the image controller own Jobs via watcher labels Change adopt_approved_images, image_reconcile, image_add_reconcile, image_remove_reconcile, handle_new_image, and disallow_image to use OperatorContext and cached stores instead of listing or GETing resources	operator/src/reference_values.rs
Adjust keygen and attestation-key controllers to use OperatorContext caches for Machines, Secrets, Deployments, and TrustedExecutionCluster state.	Update ak_reconcile, machine_reconcile, secret_reconcile, and approve_ak to accept OperatorContext and use secret_store/deployment_store Update keygen_reconcile to accept OperatorContext, use tec_store for TEC deletion checks, and call trustee mount/unmount helpers with OperatorContext Update launch_*_controller functions to pass OperatorContext instead of raw Client	operator/src/attestation_key_register.rs operator/src/register_server.rs
Rework main initialization to create all reflectors up front, share them via OperatorContext, and configure a bounded kube write timeout.	Infer kube::Config, set write_timeout to 30 seconds, and construct Client from it Create reflector stores/writers for all relevant types (TrustedExecutionCluster, ConfigMap, Deployment, Machine, AttestationKey, Secret, ApprovedImage) and spawn_reflector for each Populate OperatorContext with these stores, perform best-effort sync_cache calls for each store via a macro, and pass Arc into all controller launch functions Start register_server, attestation_key_register, and reference_values controllers using the shared context	operator/src/main.rs operator/src/lib.rs
Update unit tests to reflect cache-based access and patch-based mutations, including new helper contexts and expectations on HTTP methods.	Add helpers to build OperatorContext with pre-populated reflector stores for Deployments, ConfigMaps, ApprovedImages, and TrustedExecutionClusters Change tests in trustee, reference_values, register_server, main, and attestation_key_register to use OperatorContext, expect PATCH instead of GET/PUT, and adjust interaction counts Ensure tests that previously relied on direct API GETs now either use cached state or assert that no API calls occur	operator/src/trustee.rs operator/src/reference_values.rs operator/src/register_server.rs operator/src/main.rs
Tighten CI integration tests for stress runs, add must-gather collection, and adjust logging/timeouts for better observability.	Change REGISTRY and TAG env defaults and update TEST_IMAGE tag in Makefile Remove workflow timeout-minutes, stop building/pushing images in CI, and run integration-tests in a loop until failure, then gather and upload must-gather artifacts Enhance test_utils logging macros to prefix messages with UTC timestamps and enable env_logger humantime feature	.github/workflows/integration-tests.yml Makefile test_utils/src/lib.rs Cargo.toml

Possibly linked issues

• #0: The PR introduces OperatorContext and reflector stores to cache resources, replacing direct get/list calls as requested.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[Jakob-Naucke]

Had one VM that didn't boot up, cause cannot be uncovered from logs (from dmesg one can only tell it has failed, and Trustee was restarted). We are working on improving observability of Trustee, but should also log failures from trustee-attester without leaking secrets.

FWIW, the failure could be explained by a missing secret which has been seen on Azure and should improve with #248. For now, rerun.

[alicefr]

is the force necessary?

[Jakob-Naucke]

Yes, this is an apply patch, not a merge patch:

[2026-08-19T13:28:45Z WARN  operator::reference_values] PCR computation for coreos-approved-primary failed: ApiError: PatchOptions.meta.k8s.io "" is invalid: fieldManager: Required value: is required for apply patch: Invalid (Status { status: Some(Failure), code: 422, message: "PatchOptions.meta.k8s.io \"\" is invalid: fieldManager: Required value: is required for apply patch", metadata: Some(ListMeta { continue_: None, remaining_item_count: None, resource_version: None, self_link: None }), reason: "Invalid", details: Some(StatusDetails { name: "", group: "meta.k8s.io", kind: "PatchOptions", uid: "", causes: [StatusCause { reason: "FieldValueRequired", message: "Required value: is required for apply patch", field: "fieldManager" }], retry_after_seconds: 0 }) })

[alicefr]

same is the force necessary?

[alicefr]

We can also set it to infinite, not sure we are ready for this though. If not can you please add a todo so we don't forget it

[alicefr]

which image is it?

[Jakob-Naucke]

I want to mention I only meant this for the CI debugging and would have removed before moving out of draft, but I built this with

[Journal]
ForwardToConsole=yes

in /usr/lib/systemd/journald.conf.d

[alicefr]

@Jakob-Naucke do we plan to merge this after #248 ? It will cause /again) a lot of rebase otherwise

[Jakob-Naucke]

@Jakob-Naucke do we plan to merge this after #248 ? It will cause /again) a lot of rebase otherwise

I'll respond to your other comments later but I have no objections against merging #248 first and rebasing this

[Jakob-Naucke]

Is the force necessary?

@alicefr I'll test without but I'd also like to ask why you added it here itfp see other response

[Jakob-Naucke]

I want to mention I only meant this for the CI debugging and would have removed before moving out of draft, but I built this with

[Journal]
ForwardToConsole=yes

in /usr/lib/systemd/journald.conf.d