OCPBUGS-88732: Treat missing cloud config outages as transient#472
Open
RadekManak wants to merge 4 commits into
Open
OCPBUGS-88732: Treat missing cloud config outages as transient#472RadekManak wants to merge 4 commits into
RadekManak wants to merge 4 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (406)
📒 Files selected for processing (7)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (6)
WalkthroughCloud config reconciliation now fills in minimal managed config when the source ConfigMap is missing, or returns a namespaced not-found error. A new disruptive serial operator suite was added with registration, docs, and dependency updates. ChangesCloud config reconciliation
Disruptive operator test suite
Estimated code review effort: 3 (Moderate) | ~25 minutes ✨ Finishing Touches🧪 Generate unit tests (beta)
|
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
RadekManak
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@RadekManak: This pull request references Jira Issue OCPBUGS-88732, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@RadekManak: This pull request references Jira Issue OCPBUGS-88732, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
RadekManak
force-pushed
the
regression-ci-failure-investigation
branch
from
b53992f to
042c11c
Compare
Contributor
Author
|
/testwith openshift/cloud-provider-aws/main/regression-clusterinfra-aws-ipi-ccm openshift/cloud-provider-aws#158 |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/controllers/cloud_config_sync_controller_test.go`:
- Around line 431-438: The code in the deletion loop deletes ConfigMaps but does
not wait for the deletions to be observed before proceeding, which can cause
race conditions with subsequent creates. After the deletion loop that calls
cl.Delete() on each ConfigMap in existingCMs.Items, add an Eventually block that
polls to verify all ConfigMaps have been deleted from the namespace by listing
the ConfigMaps again and confirming the list is empty. This ensures the
deletions are fully observed before the test continues, preventing flaky
AlreadyExists or conflict errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ddd7f818-5aa2-4d7a-8264-0e8a3d087a27
📒 Files selected for processing (2)
pkg/controllers/cloud_config_sync_controller.gopkg/controllers/cloud_config_sync_controller_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- pkg/controllers/cloud_config_sync_controller.go
Comment on lines
+431
to
+438
| deleteOptions := &client.DeleteOptions{ | ||
| GracePeriodSeconds: ptr.To[int64](0), | ||
| } | ||
| existingCMs := &corev1.ConfigMapList{} | ||
| Expect(cl.List(ctx, existingCMs, &client.ListOptions{Namespace: targetNamespaceName})).To(Succeed()) | ||
| for _, cm := range existingCMs.Items { | ||
| Expect(cl.Delete(ctx, cm.DeepCopy(), deleteOptions)).To(Succeed()) | ||
| } |
There was a problem hiding this comment.
Wait for ConfigMap deletions to be observed before proceeding.
Line 431-438 deletes resources but doesn’t verify they’re gone; this can race with subsequent creates in the same namespace and cause flakes (AlreadyExists/conflicts).
As per coding guidelines, “Operations interacting with clusters must include timeouts” and tests should keep setup/cleanup reliable in BeforeEach/AfterEach.
Suggested fix
existingCMs := &corev1.ConfigMapList{}
Expect(cl.List(ctx, existingCMs, &client.ListOptions{Namespace: targetNamespaceName})).To(Succeed())
for _, cm := range existingCMs.Items {
Expect(cl.Delete(ctx, cm.DeepCopy(), deleteOptions)).To(Succeed())
+ Eventually(func() error {
+ return cl.Get(ctx, client.ObjectKeyFromObject(cm.DeepCopy()), &corev1.ConfigMap{})
+ }, timeout).Should(MatchError(apierrors.IsNotFound, "IsNotFound"))
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/controllers/cloud_config_sync_controller_test.go` around lines 431 - 438,
The code in the deletion loop deletes ConfigMaps but does not wait for the
deletions to be observed before proceeding, which can cause race conditions with
subsequent creates. After the deletion loop that calls cl.Delete() on each
ConfigMap in existingCMs.Items, add an Eventually block that polls to verify all
ConfigMaps have been deleted from the namespace by listing the ConfigMaps again
and confirming the list is empty. This ensures the deletions are fully observed
before the test continues, preventing flaky AlreadyExists or conflict errors.
Source: Coding guidelines
Keep brief source ConfigMap outages on the retry path so CCCMO does not immediately degrade during recovery, while preserving terminal behavior for real bad-key misconfigurations.
RadekManak
force-pushed
the
regression-ci-failure-investigation
branch
from
042c11c to
c566c55
Compare
Contributor
Author
|
/testwith openshift/cloud-provider-aws/main/regression-clusterinfra-aws-ipi-ccm openshift/cloud-provider-aws#158 https://github.com/openshift/openshift-tests-private/pull/30029 |
Contributor
Author
|
/testwith openshift/cluster-cloud-controller-manager-operator/main/regression-clusterinfra-aws-ipi-ccm https://github.com/openshift/openshift-tests-private/pull/30029 |
Migrate the disruptive CCM regression coverage into the operator OTE binary so this fix can be rehearsed with its tests from the same PR.
Refresh the workspace module files and vendor tree so the migrated OTE tests build with the updated client dependencies and the vendor presubmit stays green.
Contributor
Author
|
/test e2e-aws-ipi-ccm-operator-serial-disruptive-ote |
Contributor
Author
|
/test e2e-aws-ipi-ccm-operator-serial-disruptive-ote |
Contributor
|
@RadekManak: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
openshift-tests-privateinto the CCCMO operator OTE binaryWhy OTE
openshift-tests-private, which/testwithcannot rebuild together with component PRsTest plan
GOFLAGS=-buildvcs=false make cluster-cloud-controller-manager-operator-tests-ext./openshift-tests/bin/cloud-controller-manager-operator-tests-ext listRelated
Summary by CodeRabbit
cloud.conf.