Skip to content

Clean up successful Kyverno migration jobs after upgrades #5491

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Problem

Kyverno's chart removes a successful migration hook's temporary service account but retains the completed job. The leftover job keeps a reference to an account that no longer exists and can produce misleading posture results after an upgrade.

Evidence

The published migration-hook template deletes the temporary account on success while leaving the job's deletion policy commented out. A fresh read-only observation reproduces that completed-hook lifecycle. Detailed runtime evidence remains private.

Acceptance

  • Preserve the migration's execution identity, required permissions and resource scope; do not recreate obsolete permissions to satisfy a scan.
  • Clean up successful migration jobs through their reviewed Helm lifecycle. Keep failure diagnostics available, and leave unrelated hooks and jobs unchanged.
  • Prove the actual chart and deployed Helm version apply the intended hook change, including a failing-before/passing-after regression and negative controls for unrelated jobs.
  • Deploy through the protected path, observe a successful migration and normal cleanup, and verify a current attributable scan. An orphaned report or unbound scan remains unresolved; do not delete reports, suppress the control or add an exception.

Part of #2043

Delivered on: 2026-10-31
Measure on: 2026-11-01

Resume condition: Re-enter verification when stored posture results can be attributed to the natural scan that completed on 2026-11-01. The scheduler succeeded at 09:12 UTC and the scanner recorded completion and saved results at 09:21 UTC; both stored scan surfaces still identify the 2026-10-31 report. The old migration Job and its report are currently absent, but absence without fresh attributable results does not satisfy acceptance. Do not dispatch an extra scan, delete reports, suppress findings or infer acceptance from scheduler or scanner completion alone. If current report identity or attribution remains unavailable, keep this issue open with that exact condition.

The hook lifecycle change was delivered through #5637 at merged source 72a83004aab783cde9b88c6038aadd022d9d9d3c. The recorded successful migration and cleanup remain partial acceptance; the original criteria, failure-diagnostic requirements and flow clocks are unchanged.

Activity

  1. self-assigned this
    on Oct 28, 2026
  2. added theissue type on Oct 28, 2026
  3. moved this to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Oct 28, 2026
  4. added 2 commits that reference this issue on Oct 28, 2026
  5. devantler commented on Oct 28, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The combined-stream validation prerequisite is now on protected main through #5505. The lifecycle cleanup remains a separate rollout and is natively blocked by #5542, which prepares patched migration tooling from the released implementation.

    Resume deployment after the runtime is published and independently verified and its verification policy is active. Then apply the narrowly targeted hook lifecycle change, observe the successful migration and cleanup, and obtain a fresh attributable posture scan before closing this issue.

  6. added a commit that references this issue on Oct 31, 2026
  7. devantler commented on Oct 31, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    First real run, through #5637's merge-group deploy (run 41244989833). Read live at 22:12Z on 2026-10-31.

    • The release upgraded at 22:01Z (revision 44) and reports Ready and Released, with no failures.
    • Events show the Pod pulled platform-kyverno-migrate, and the Job completed.
    • The Job is gone, along with its temporary service account and cluster role, so nothing is left naming a missing account.
    • The upgrade deleted the old retained Job too.

    Not yet met. The posture report for this Job, created 2026-06-25, still exists: it predates this change, and no scan has run since the deploy. The next scheduled scan is 2026-11-01 at 09:12Z. After it, read whether that report is gone or still orphaned. An orphaned report stays unresolved here; it is not to be deleted or suppressed by hand. Failure diagnostics and a failed hook have not been exercised on the real cluster.

    This issue stays open until that read is recorded.

  8. added 2 commits that reference this issue on Nov 1, 2026
  9. moved this from 🏃🏻‍♂️ In Progress to 🫴 Ready in 🌊 Project Boardon Nov 1, 2026
  10. added a commit that references this issue on Nov 1, 2026
  11. devantler commented on Nov 1, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Returned this delivered check to Ready with an explicit resume condition. A fresh read confirms the natural posture scan is enabled for 09:12 UTC today; its latest scheduled execution was before #5637's protected delivery. The next run after that scan completes should verify fresh attributable posture data and whether the old migration-job report has cleared.

    The issue remains open. This move does not establish scan success or waive the existing acceptance criteria; no scan was dispatched and no report was deleted. Original flow clocks are preserved.

  12. added 2 commits that reference this issue on Nov 1, 2026
  13. devantler commented on Nov 1, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Blocker: Fresh stored posture results attributable to the completed 2026-11-01 scan | outcome | last-verified 2026-11-01: The natural scan completed, and the old migration-job report is absent from both stored scan surfaces. Their report identity still names 2026-10-31, so fresh attributable acceptance remains unverified. Re-enter when current report identity and results can be read; do not dispatch another scan, delete reports or suppress findings.

  14. added a commit that references this issue on Nov 1, 2026
  15. devantler commented on Nov 1, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Measured after the natural posture scan of 2026-11-01. No scan was dispatched and no report was deleted by this run.

    • Scan: the scheduled job ran at 09:12 UTC and succeeded. Reports in the Kyverno namespace carry report-created-at 09:21:39Z today, and the report for an unchanged deployment moved to a new resource version. So this is fresh data from a completed scan.
    • Old report: there is no longer a posture report for any job or migration workload in the namespace, out of 85 reports. The orphan recorded on 2026-10-31 at 22:12Z has cleared.
    • Jobs: no job remains in the namespace, after two further Helm revisions (44 and 45, both scanned). The release reports Ready, UpgradeSucceeded.

    All four acceptance criteria now have evidence, so I am closing this as delivered.

    Limits:

    • The report was already gone in a read at about 08:44 UTC, before today's scan ran. So the scan did not remove it, and I did not observe what did. The record here shows no manual deletion, but that is the absence of a record, not proof.
    • A failed hook has still not happened on the real cluster. Keeping its diagnostics rests on the chart regression test from fix(kyverno): clean up the successful migration job on its patched runtime #5637. If a failed migration hook ever leaves no diagnostics, that is a new defect.
  16. moved this from 🫴 Ready to ✅ Done in 🌊 Project Boardon Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions