Repository navigation
Description
🤖 Generated by the Agentic Engineer
Problem
A named control-detail read can report status: passed with subStatus: w/exceptions even though all of its rules report an unsuppressed pass and every rule has no applied ignore rule. The current exception definitions do not match the affected record. The stored workload independently carries the required security fields.
This was verified through named reads and current declarations on 2026-10-05. The public example is intentionally synthetic; full receipts remain private:
{"status":{"status":"passed","subStatus":"w/exceptions"},"rules":[{"status":{"status":"passed","subStatus":""},"appliedIgnoreRules":null}]}Impact
The aggregate exception indicator cannot currently establish whether the control really passed or whether a suppression remains. Presence, rule count and a bare passed status are insufficient to clear the security acceptance.
Work
Trace where the aggregate marker is computed and persisted, using a bounded synthetic reproduction. Repair the responsible write or aggregation path rather than rewriting production scan data or weakening exception scope. Distinguish this from the earlier stale-detail investigation in #3275 and the deleted-record retention work in #4505.
Acceptance criteria
- Reproduce a stale aggregate exception marker when the final suppression is removed, or establish another verified cause.
- The repaired record's aggregate status agrees with its actual rule-level evaluation after a new scan.
- A genuinely excepted control retains its exception marker; an unsuppressed pass clears it.
- Failed or incomplete observations cannot produce a clean verdict, and storage permissions remain unchanged.
- Verify the result by named live read after delivery, without treating creation time alone as scan freshness.
Part of #2824
Remaining work: kubescape v4.0.16 (published 2026-10-14) contains kubescape/kubescape#4044; platform#4911 moves the scanner onto it. After it deploys, read a scan on the new release back live (last criterion).
Blocker: the first scheduled Kubescape posture scan after the Sigstore egress fix, due 2026-10-17 09:12 UTC | outcome | last-verified 2026-10-17: #4966 closed and its fix #4967 merged 2026-10-16 22:41 UTC; the stored posture objects still describe the last pre-upgrade scan. After the scan, confirm it completed, then re-read the named control detail for the stale aggregate exception marker
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- StatusShow more project fields✅ Done
Activity
devantler commentedon Oct 5, 2026
Read-only persistence inspection is now bound to the deployed storage base, kubescape/storage v0.0.297 (
b35788b68337134fc2514574cde1ba7f1225fd43), rather than current upstream main.The workload scan endpoint uses the generic storage backend. Its creation/update strategy leaves the supplied object unchanged, and the backend's default processor has no scan-status transformation. The virtual namespace summary sums existing workload-summary severities; it does not reconstruct a workload control's exception marker from its rules. The Platform compatibility patches inspected for this path do not introduce such a recomputation either.
This narrows the investigation: repairing the virtual namespace summary alone would not repair the contradictory marker in the named workload scan. Next proof needs the scanner's status calculation and a scan receipt that joins the resulting aggregate and rule details. A stable resourceVersion or creation time alone cannot establish that a fresh scan produced this contradiction. No stored scan data, permissions or exception definitions were changed.
Source references: endpoint assembly, workload strategy, default processor, virtual summary.
chore: agent-claim #4523 nonce=a97d1e6602b12f7abd0ec87e7519fd06
devantler commentedon Oct 5, 2026
Cause established: the scanner merges the previously stored control status into every rescan
Read from the released source of the scanner version the platform deploys (
kubescapeimagev4.0.14, which pinsopa-utils v0.0.312), not from upstreammain. No scan data, permissions or exception definitions were touched.What happens on a rescan of a workload that already has a stored record
v0.0.297) refuses a create for an existing key, so every rescan takes the update path (StoreWorkloadConfigurationScanResult).mergeWorkloadConfigurationScanSpec, which, for each control present in both records:CompareStatusAndSubStatus(existing, new, …), andruleswith the new scan's rules.CompareStatusAndSubStatusreturns the more significant of its two inputs: for twopassedinputs it returnsw/exceptionswhen either side carries it.So once a control has been stored as
passed/w/exceptions, a later scan in which the suppression no longer applies produces exactly the record this issue describes: the control keepssubStatus: w/exceptions(inherited from the old record) while its rules are the new ones, each an unsuppressed pass withappliedIgnoreRules: null. The aggregate marker and the rule details come from two different scans.The per-workload summary is merged the same way (
mergeWorkloadConfigurationScanSummarySpec), so it inherits the same marker.Consequences that follow from the same code (source-derived; not yet observed live)
failedsticky:Comparereturnsfailedwhen either side isfailed. A control that failed once and was then remediated would keep readingfailedin the stored record until that record is deleted and created afresh. This needs one named live read to confirm or refute, because it would mean stored failures can outlive their fix, which bears on the measured compliance floor (Ratchet the measured combined security compliance floor to 98 percent #4520) as an under-count, never an over-count.Not fixed upstream yet. The merge function is unchanged in the newest released scanner (
v4.0.15) apart from an unrelated metadata copy, so a version bump does not clear it.Acceptance criteria against this finding
CompareStatusAndSubStatus(passed, passed, "w/exceptions", "")returnspassed,w/exceptions.Repair options
failedcase unless it also runs after remediationRecommendation: A, with the sticky-
failedlive read done first — if that read confirms stored failures outlive their fix, the under-count is worth the interim patch in B; if it refutes it, the contradiction is cosmetic on the safe side and A alone is enough. Until one of them lands, aw/exceptionsmarker on a stored control means "was excepted in some earlier scan", and only the rule-levelappliedIgnoreRulessays whether an exception applies now.devantler commentedon Oct 5, 2026
Live read: the sticky-
failedconsequence is not observed; the stale exception marker is, and it is rareFollow-up to the cause recorded above. Read-only named reads of 45 stored workload scans (Deployments, StatefulSets, DaemonSets and CronJobs across namespaces, excluding the scanner's own), 2026-10-05 ~15:45Z. A list was not used, since it does not return control detail. Nothing was changed.
passedpassed/w/exceptionsskipped/configurationfailedfailedwith every rule passing: 0 of 100. The source allows a remediated control to stayfailed, but this sample shows no instance. That is evidence against it mattering in practice here, not proof it cannot happen; a control remediated and rescanned inside the sample would have shown it.w/exceptionswith every rule an unsuppressed pass and no applied ignore rule: 2 of 271, one each onC-0077andC-0211, in two different records. This is the contradiction this issue describes, reproduced by named read, and it matches the merge behaviour: the marker survives from an earlier scan while the rules are the current ones.An earlier pass over 120 records of other kinds (Secrets, Endpoints, volume claims) found only plain
passedcontrols, so the effect is confined to workload records that have carried an exception.What this changes
appliedIgnoreRulesat rule level to decide whether an exception applies now, as the exception guard already documents forC-0211.Remaining acceptance criteria (repaired record, marker cleared on an unsuppressed pass, live read after delivery) stay open behind that fix.
devantler commentedon Oct 6, 2026
Maintainer answer, given in the Slack thread under the ask on 2026-10-05 and missed until now (no run was reading thread replies; the maintainer pointed this out on 2026-10-06):
Meaning: option 2 of the ask — an upstream issue and a fix pull request on the scanner are approved. This is no longer blocked on a decision. Next: write the upstream issue from the released source with a synthetic example (nothing about this cluster), then the fix pull request, following the third-party contribution rules; link both here.
devantler commentedon Oct 6, 2026
The upstream report and a tested fix are written and ready to submit
The maintainer approved an upstream issue and a fix pull request (option 2, recorded above). Both are prepared below. Neither has been opened yet: this scheduled run's dispatch lists only
devantler-techpull requests and issues as write actions, so it stopped at preparation. Any run or session whose dispatch covers third-party contributions can submit them as they stand.What changed in my understanding of the cause
Keeping the more significant status is deliberate upstream: a table test named existing failed wins over new passed pins it. So "replace instead of merge" would be rejected as written. What is missing is a distinction between results from the same report and results from a later one. The newest scanner release (
v4.0.15) records each report's generation time on the stored record, which makes that distinction possible.The fix
When the incoming scan comes from a newer report than the stored record, its status replaces the stored one. Results from the same report keep today's behaviour. A scan with no timestamp, or one older than the stored record, also keeps today's behaviour, so a late or unplaceable result can never clear a stored failure.
Proof, run against the released
v4.0.15source in a disposable copy:go vetcleanThis covers acceptance criteria 2 and 3 at source level. The fourth still holds: nothing can produce a clean verdict from a scan that cannot be placed in time.
Two things the fix does not settle
v4.0.14with the platform-built storage based onv0.0.297; the timestamp field arrives withv4.0.15and a newer storage API. The fix only takes effect once the platform's storage image is rebased far enough to persist that field. That rebase is separate work and should be filed when the upstream fix is accepted.Before submitting
go test ./httphandler/storage) and the related issue.Upstream issue text
Title: Rescans merge the previously stored control status, so a remediated control stays
failedand a removed exception staysw/exceptionsDescription
When the in-cluster scanner stores a
WorkloadConfigurationScanfor a workload that already has a stored record, it does not write the new result. It merges the stored control status into the new one and keeps the more significant of the two, while replacing the control's rules with the new scan's rules.The result is a stored control whose status and rules come from two different scans:
status: failed, although every rule in the same record now passes;subStatus: w/exceptionsafter the exception is removed, although no rule in the same record has an applied ignore rule.The per-workload
WorkloadConfigurationScanSummaryis merged the same way, so its severity counts inherit the stale status.Where
httphandler/storage/apiserver.go, atv4.0.15:StoreWorkloadConfigurationScanResulttakes the update path whenever the record already exists and callsmergeWorkloadConfigurationScanSpec.mergeWorkloadConfigurationScanSpeccomputes the status withapis.CompareStatusAndSubStatus(existing, new, …)and setsRules: newCtrl.Rules.mergeWorkloadConfigurationScanSummarySpecdoes the same for the summary.CompareStatusAndSubStatus(opa-utilsv0.0.312) returnsfailedwhen either side isfailed, and for twopassedinputs returnsw/exceptionswhen either side carries it.Reproduction
No cluster is needed. Against the current function:
The existing table test
existing failed wins over new passed, rules replaced by new scanpins this behaviour.In a cluster I have observed the exception variant: reading stored
WorkloadConfigurationScanrecords by name, a small share of controls markedpassed/w/exceptionshad only unsuppressed passing rules withappliedIgnoreRules: null. I have not observed thefailedvariant in a cluster; it follows from the same code path and the snippet above.Expected behaviour
A stored control's status agrees with the rules stored beside it. A scan from a newer report replaces the status from an older one.
Suggested fix
Keeping the more significant status looks intended for results that belong to the same report.
v4.0.15records the report generation time inspec.metadata.report.createdAt, which makes that distinction available: keep the current merge when both sides come from the same report, and let the new scan's status replace the stored one when its report is newer. A new scan without a timestamp, or one older than the stored record, keeps the current behaviour, so a late or unplaceable result can never clear a stored failure.I have a small patch for
mergeWorkloadConfigurationScanSpecalong those lines, with table tests, and can open a pull request. Two things I would like your view on first:WorkloadConfigurationScanSummarySpechas no report timestamp, so the summary cannot make the same decision by itself. Would you prefer the summary to be rebuilt from the merged scan, or to carry the timestamp too?Environment
v4.0.14in-cluster (operator-triggered continuous scanning); merge code read atv4.0.14andv4.0.15, unchanged in substance.Patch against v4.0.15
New test file: httphandler/storage/apiserver_merge_report_test.go
devantler commentedon Oct 7, 2026
Maintainer confirmation, given in an interactive session on 2026-10-07: open the approved upstream contribution now, from this session. No AI attribution unless the target project requires it.
Next (this session): re-check the target project's contribution and AI policy and that the prepared change still applies upstream, open it, and link it here.
devantler commentedon Oct 7, 2026
The upstream contribution is not opened, because upstream fixed this defect on 2026-10-06. Checked on 2026-10-07 before submitting anything.
failedforever in WorkloadConfigurationScanSummary (merge keeps the existingfailedstatus) kubescape/kubescape#4041 and closed by fix(storage): clear remediated findings on later scans kubescape/kubescape#4044, merged 2026-10-06.What this issue waits for now: a Kubescape release that contains the fix. The newest release is v4.0.15 from 2026-09-29, which predates it. The blocker line is updated to that.
One thing to re-check when the release arrives. The earlier note says production cannot use a fix by a scanner upgrade alone, because the platform-built storage is older than the report-time field. The merged fix also writes the report time as an annotation from the scanner side, which may make the scanner upgrade sufficient without rebasing the storage image. That is unverified: confirm it against the released code and a named live read before closing the acceptance criteria.
chore(policy): re-approve the kubescape release after its scanner ima…
devantler commentedon Oct 15, 2026
Pre-deployment verification baseline
A fresh read-only sample retrieved 105 workload scan records by name (6,669 controls), not through the spec-stripping list response. It found 17 controls carrying an aggregate exception marker despite only unsuppressed passing rules, including both original affected controls. It also found 803 controls with genuinely applied exceptions. This is a wider sample than the earlier investigation, not a like-for-like regression-rate comparison.
No sampled record carries a usable report identity. The released scanner writes
kubescape.io/report-created-atas an annotation as well as the newer spec metadata, so the post-deployment check can bind a genuinely new report even when older storage drops the spec field. That storage compatibility remains a live verification requirement, not a source-derived success claim.The scanner was still on the pre-fix release at collection. #4911 remains queued; keep this issue open until a new-release report clears the stale markers while retaining genuine exceptions. No stored results, exception definitions or permissions were changed.
devantler commentedon Oct 15, 2026
PR #4911 is now actually merged:
7cde9dcfb748e5d867d48f56be39b13d8a17ca10at 11:06:32 UTC. OIDC readback confirms scanner v4.0.16 is running, Deployment generation 42 is observed with 1/1 ready, and the HelmRelease reports UpgradeSucceeded. The new server started at 10:57:38 UTC during the queue deployment.Verification remains open: all 2,801 stored posture objects still lack
kubescape.io/report-created-at; the original C-0077 object has the same resourceVersion and contradiction as the pre-upgrade sample. The daily scan completed before this deployment, so existing records are not evidence that the new scanner has rescanned them. No reports were deleted or patched and no local production scan was started. Wait for a fresh scan before assessing the 17-control baseline and preserving genuine exception results.devantler commentedon Oct 16, 2026
The fresh scan this verification waits for has not happened. The first scheduled scan on the new scanner (16 October, 09:12 UTC) aborted before scanning. The release added Sigstore verification of its rule bundle, and the namespace's egress policy blocks the trust-root host that verification needs. The scheduler Job still reported
Completed. The stored posture objects therefore still describe the last pre-upgrade scan, and nothing here can be assessed yet.That cause is #4966, with the one-host egress fix in #4967. This issue is now natively blocked by #4966. Once #4967 deploys and a scan completes, the 17-control baseline above can be re-read.
devantler commentedon Oct 24, 2026
Completed the remaining live verification after #4911 and the scan-egress repair. The current scanner runs v4.0.16, is Ready at its observed generation, and its running image predates the fresh report stamped 2026-10-24 09:22:04 UTC. This uses the report annotation retained by the deployed storage, not object creation time or a scheduler Job's completion flag.
Read 105 current report objects individually by name: 1,070 control details, zero contradictions of the recorded stale-marker shape, and 102 aggregate exception markers backed by actual applied rule exceptions. Five unsuppressed passing controls in the originally affected control class correctly have no exception marker. Two named records were re-read unchanged afterward. The list projection has empty controls and was used only to select names; it was not treated as assessment data.
The owning upstream repair is merged, and #4911 is merged. Its report-boundary and incomplete/late-report protections remain in place; this verification changed no scans, exceptions, storage, permissions or production configuration. The evidence supports the deployed defect correction and closes this issue's named-read acceptance. It is a bounded live sample, not a claim that all 17 historical baseline cases were matched or that the portfolio's entire security posture is clean. Broader posture and coverage remain under #2824. Private receipt: R69-SCAN-4523.