Repository navigation
Converge generated-resource cleanup when a downstream is already absent #5527
Description
🤖 Generated by the Agentic Engineer
Delivered on: 2026-10-31
Measure on: 2026-11-08
Resume condition: No earlier than 2026-11-01 at 22:55 UTC, collect at least 24 hours of attributable observations on the accepted controller image and a fresh Coroot window. Rebind the image and uninterrupted observation window if the Pods or image change. A controlled or independently observed cleanup of an already-absent generated resource is still required; ordinary deletion log lines and zero errors do not establish that case. If any required evidence is absent, keep this issue open and record the remaining condition. Do not restart resources or induce a production deletion to manufacture the observation.
The controller delivery is complete through #5638, merged as 9b00677e2a52082035cf89c54486de27f1ecadc9. Both replicas were independently read back ready on the accepted image with zero restarts on 1 November. This card returns to Ready while the observation window matures; the original acceptance criteria and first-start history remain unchanged.
Problem and evidence
The current Kyverno background controller reports a failed downstream deletion when Kubernetes says the generated object is already absent. A fresh read-only observation on 28 October at 17:29:54 UTC captured this error during ordinary resource replacement; Coroot consequently reports a background-controller log error.
The deployed and latest stable release is v1.19.2. Its pkg/background/gpol/dynamic_watcher.go treats every deletion error as a failure and retains the cached downstream identity. In the already-absent case, the intended state has been reached, but that stale cache entry survives. This is a cache-reconciliation defect to reproduce, not a reason to lower log severity or hide arbitrary failures.
Hypothesis and smallest useful change
If an authenticated Kubernetes NotFound response completes downstream cleanup and removes only the matching cached identity, repeated cleanup converges without a false failure. Permission errors, conflicts and unrelated failures must remain visible and retain retryable state. Preserve policy/trigger ownership and deletion identity safety.
Acceptance
- Reproduce through the actual WatchManager/client boundary: already-absent downstreams clear their exact cache entry; wrapped NotFound is recognized; unrelated errors retain the entry and report failure; another policy/trigger is untouched. Cover each affected deletion path and repeat cleanup.
- Implement the smallest reviewed correction at the owning source/compatibility-image surface. No RBAC increase, policy exception, log filter or blanket error suppression.
- Verify the real Linux controller package and published executable, provenance and signature before any separately reviewed image adoption. Prove existing image-verification boundaries still reject unrelated publishers.
- After normal protected delivery, bind the running image and controller identities, verify retained generation/deletion behavior and observe a fresh Coroot/log window. Source tests and an absent object alone do not establish production recovery.
Part of #2043.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
chore: agent-claim #5527 nonce=afc37aac6926136cd721430d5a3ff194
devantler commentedon Oct 29, 2026
The first signed candidate from #5532 is published, but it is not accepted for image adoption. Its executable identifies the release source as a pseudo-version, and stripped symbols prevent the binary scanner from establishing package reachability. Signature, compiler and help-command checks passed; those do not complete the security acceptance.
I am correcting the producer's release-identity and executable-audit checks before any separately reviewed adoption. The candidate remains held; no advisory, signature boundary or log check is waived. Part of #5527.
devantler commentedon Oct 31, 2026
Published image read back from the registry after #5630, and accepted as an adoption candidate. It is not adopted yet.
Image:
ghcr.io/devantler-tech/platform-kyverno-background@sha256:327d72f865bb10108b19ce2510e3244104872632aa0063283da5e69034b1f1fa(amd64sha256:1024d675…), from run 41233039072.cosign verifyfinds one signature for the main-branch publisher identitygo1.26.10kyverno v1.19.2,vcs.modified=falsepkg/background/gpolGO-2025-3615 GO-2026-5337 GO-2026-5594 GO-2026-5932These are the four records the Dockerfile accepts. This read came from the image's patched layer on its own.
A wrong first reading, in case someone repeats it: my first read used
crane exportpiped totar, and it reportedgo1.26.9,v1.19.2+dirtyand no symbols. That was the upstream binary from the layer underneath. The two layers name the same file differently (ko-app/…and/ko-app/…), so the export lists both andtarkeeps the older one. The running image is not affected: the patched layer is on top.Still open on this issue: a separately reviewed adoption, then running-image identity and a fresh log window after delivery.
chore: agent-claim #5527 nonce=476b93ad641d1b8097f4c729f6681ea2
chore: agent-claim #5527 nonce=d4efaae8cbd8a8f2a9c7995b014d6685
fix(kyverno): approve the background controller image in the authoriz…
docs(kyverno): describe the background image patch accurately
Merge main into the background controller adoption
devantler commentedon Oct 31, 2026
The patched controller is running in production since #5638 (merge-group deploy run 41248257178). This issue stays open: the log window so far is too short to show the fix worked.
sha256:327d72f8…, the digest accepted abovev1.19.2-downstream-cleanup.1-aa…That the pull succeeded also shows the Talos rule from #5635 accepted this image on those two nodes. It says nothing about the other nodes.
Still owed:
devantler commentedon Nov 1, 2026
Log window read 00:18Z on 1 November, about 1.5 hours on the patched controller. The changed code path now ran, with no false failure. This issue stays open.
auto-vpa-statefulsetandauto-vpa-daemonset, the deletion path this fix changesWhat this does not show: the patched path clears an already-absent object silently, so these logs cannot tell how many of the 74 runs met that case. I did not measure the old image's error rate over a comparable window, so this is not a before-and-after comparison.
Still owed: the Coroot window, and a controlled or observed case where a generated object is already gone when cleanup runs. A clean log window of at least 24 hours can be read from about 22:45Z today.
chore: agent-claim #5527 nonce=c46907e850b41a9fb18345c2fb2e2434
chore: agent-claim #5527 nonce=ea62723967a02548fb5353dca7731473
chore: agent-claim #5527 nonce=466b8d2856525aa8e1515c085928dee4
devantler commentedon Nov 1, 2026
The controller repair is protected-merged in #5638, and a fresh read-only check finds both replicas ready on the accepted image with zero restarts. The observation period is still shorter than the required 24 hours; the earlier deletion log entries also do not prove cleanup of an already-absent resource.
I recorded the real delivery date and a continuation no earlier than 1 November, 22:55 UTC, then returned this merged check to Ready. At that checkpoint, collect an attributable log and Coroot window and the missing already-absent-resource evidence; otherwise keep the issue open. The original acceptance criteria and flow history are preserved. No source, live resource, scan schedule or security control changed.
devantler commentedon Nov 1, 2026
Checkpoint at about 23:05 UTC on 1 November. This stays open: the case it needs has not happened yet.
Still needed: one independently observed cleanup of an already-absent resource, plus a Coroot window. I did not induce a deletion, as this issue's resume condition forbids. Next read is no earlier than 8 November, and the observation window restarts if the Pods or the image change.
devantler commentedon Nov 1, 2026
Checkpoint read at 23:05 UTC on 1 November. The 24-hour Pod window is met, but this issue stays open: two required pieces of evidence are still missing.
sha256:327d72f8…, 0 restarts, ready. That is about 24 hours 20 minutes uninterruptedClosest real event: at 20:10:15 UTC the controller logged a generated resource deleted, added and deleted again in one second, with cleanup for related autoscaler policies running alongside it and no error. I did not confirm what caused it. This is a candidate only. The patched path clears an absent object silently, so the logs cannot show that any deletion actually met an absent object.
Still owed:
The 5 conflict errors are a separate finding, which I am filing as its own issue. I did not measure the old image over a comparable window, so I can't say whether they are new. No source, live resource or security control changed.
Reconciled with the checkpoint above, which another run posted within the same second: the next read stays no earlier than 8 November. On log coverage, this comment is the accurate one: the active Pod retains only the last 9 hours 17 minutes, so "the last 25 hours" holds only for the inactive Pod. Issue #5705 tracks the conflicts.