OCPEDGE-2700: fix flaky container-kill recovery test with migration-threshold by lucaconsalvi · Pull Request #31276 · openshift/origin

lucaconsalvi · 2026-06-09T16:26:44Z

Problem

The test should coordinate recovery with peer when local etcd container is killed flakes at ~1.9% (4 failures / 212 CI runs, seen on both PR payload runs and periodic nightlies).

All failures are identical: 600s timeout waiting for etcd-clone to be Started on both nodes. The failure log shows:

Failed Resource Actions:
  * etcd start on master-1 returned 'error' after 2m1.131s
  * etcd 30s-interval monitor on master-0 returned 'error' (master-0 must force a new cluster)

Root cause

When podman kill etcd is run on one node, the podman-etcd resource agent coordinates recovery via the force_new_cluster protocol:

The peer node's monitor detects a leadership loss → sets force_new_cluster → monitor fails
The peer restarts etcd with --force-new-cluster
The killed node's start action retries until it can rejoin as a learner

The race: step 3 (start action) can fail multiple times before step 1–2 completes. If it hits Pacemaker's migration-threshold (default: 3 failures) before the peer is ready, Pacemaker stops scheduling the resource entirely. The longRecoveryTimeout (10 min) then expires waiting for a recovery that Pacemaker won't attempt.

Fix

Raise migration-threshold to INFINITY on the etcd resource before killing the container, so Pacemaker never stops retrying during the transient coordination window. The original threshold is saved and restored via DeferCleanup.

save original migration-threshold
  → set migration-threshold=INFINITY
  → kill etcd
  → wait etcd cluster healthy          (first Eventually)
  → wait pcs etcd-clone Started on both nodes  (second Eventually)
  → assert Failed Resource Actions on both nodes
  → verifyFinalClusterHealth
  → DeferCleanup: restore original migration-threshold

This approach is better than the previous pcs resource cleanup mid-test because:

The test purely observes recovery without intervening
If the resource agent has a genuine bug, etcd-clone will never start and the test times out — no false passes
pcs resource cleanup could mask real start-action failures

Verification

Passes locally on TNF cluster (~400s)
Payload CI: container-kill test passed in every run where the cluster provisioned successfully
Unrelated CI failures across runs were all infra flakes (cluster setup SSH timeouts, oc debug namespace issues in other tests, MonitorTest noise)

Fixes: https://redhat.atlassian.net/browse/OCPEDGE-2700

Summary by CodeRabbit

Tests
- Improved test infrastructure for cluster edge scenarios to enhance stability and reliability during resource management disruption testing.

When a local etcd container is killed, the peer sets force_new_cluster and etcd recovers. In rare cases (~1.9% of runs) the killed node's repeated start failures hit Pacemaker's migration-threshold before coordination completes, causing Pacemaker to stop scheduling the resource and the 10-minute Eventually to expire. The "Failed Resource Actions" evidence (which the test asserts) is captured immediately after the cluster self-heals — at that point both nodes are guaranteed to appear since the peer's force_new_cluster monitor failure and the killed node's start failures have already been recorded. After capturing the evidence, pcs resource cleanup resets the failure count so Pacemaker can retry the start action against the now-healthy peer cluster. Fixes: https://redhat.atlassian.net/browse/OCPEDGE-2700 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

openshift-merge-bot · 2026-06-09T16:26:47Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci · 2026-06-09T16:26:56Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coderabbitai · 2026-06-09T16:27:08Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Test adds helpers to read, set, and restore Pacemaker's migration-threshold for etcd-clone. The coordinated local-etcd-kill test now captures the original threshold, sets it to INFINITY before killing the container, and defers restoring the prior value during cleanup.

Changes

Etcd disruption test robustness improvements

Layer / File(s)	Summary
migration-threshold helper functions `test/extended/edge_topologies/tnf_etcd_disruption.go`	Adds internal helpers to retrieve the current Pacemaker `migration-threshold` value for `etcd-clone`, update it via `pcs resource meta`, and restore it afterward (deleting the override if it was originally unset).
Test coordinated recovery with migration-threshold management `test/extended/edge_topologies/tnf_etcd_disruption.go`	The coordinated "local etcd container is killed" test captures the original `migration-threshold`, sets it to `INFINITY` before killing the container to prevent Pacemaker abandonment, and defers restoration. Global `AfterEach` cleanup removes any lingering override.

Sequence Diagram(s)

sequenceDiagram
  participant Test
  participant PacemakerCLI as Pacemaker<br/>(pcs)
  participant TargetNode as Target Node<br/>(etcd)
  Test->>PacemakerCLI: read migration-threshold(etcd-clone)
  Test->>PacemakerCLI: set migration-threshold=INFINITY
  Test->>TargetNode: kill local etcd container
  TargetNode->>Test: etcd recovered
  Test->>PacemakerCLI: verify etcd-clone Started on both nodes
  Test->>PacemakerCLI: restore migration-threshold (original or delete)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: fixing a flaky container-kill recovery test by adjusting the migration-threshold parameter.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	All test names (It, Describe) are static strings with no dynamic content, format specifiers, or generated identifiers. Test titles are deterministic and will remain stable across runs.
Test Structure And Quality	✅ Passed	Test code meets all quality requirements: single responsibility (tests container-kill recovery coordination), proper BeforeEach/AfterEach setup and cleanup with DeferCleanup for resource restoratio...
Microshift Test Compatibility	✅ Passed	No new Ginkgo e2e tests are added. The PR modifies an existing test and adds internal helper functions. All existing tests are already protected by the parent Describe block's [apigroup:config.open...
Single Node Openshift (Sno) Test Compatibility	✅ Passed	This PR does not add new Ginkgo e2e tests; it modifies an existing test already protected by `utils.SkipIfNotTopology(oc, v1.DualReplicaTopologyMode)` for TNF (two-node fencing) clusters, not SNO.
Topology-Aware Scheduling Compatibility	✅ Passed	PR modifies only a test file (tnf_etcd_disruption.go) with test utilities; no deployment manifests, operator code, or controllers are added or modified, so topology-aware scheduling check does not...
Ote Binary Stdout Contract	✅ Passed	The PR modifies test-only code with no process-level stdout writes. All logging uses framework.Logf (safe for Ginkgo). No fmt.Print, log.Print, or klog to stdout violations found.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR modifies existing test without adding new Ginkgo test definitions (It, Describe, Context, etc.). No IPv4 hardcoding, IPv4-specific parsing, or external connectivity requirements found. Check app...
No-Weak-Crypto	✅ Passed	No weak cryptography (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto implementations, or non-constant-time secret comparisons found. Changes are limited to cluster configuration managemen...
Container-Privileges	✅ Passed	The PR modifies only a Go test file (tnf_etcd_disruption.go) with no container/K8s manifests, security contexts, or privilege escalation declarations present.
No-Sensitive-Data-In-Logs	✅ Passed	No sensitive data (passwords, tokens, API keys, PII, session IDs, internal hostnames, or customer data) is exposed in logging added by the PR. All logged values are non-sensitive operational metric...

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci · 2026-06-09T16:27:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lucaconsalvi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~test/extended/edge_topologies/OWNERS~~ [lucaconsalvi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lucaconsalvi · 2026-06-09T16:30:57Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-06-09T16:31:01Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/98519310-6420-11f1-85a8-bef173c7a909-0

coderabbitai

🧹 Nitpick comments (1)

test/extended/edge_topologies/tnf_etcd_disruption.go (1)

535-543: ⚡ Quick win

Consider logging cleanup output on success for consistency and diagnostics.

The AfterEach cleanup at Line 337-343 logs the pcs resource cleanup output even on success (Line 342). For consistency and better observability when investigating test runs, consider logging the output here too.

📝 Suggested improvement

 g.By("Running pcs resource cleanup to unblock etcd-clone if Pacemaker hit migration-threshold")
 if output, cleanupErr := exutil.DebugNodeRetryWithOptionsAndChroot(
     oc, execNode.Name, "default", "bash", "-c", "sudo pcs resource cleanup etcd-clone"); cleanupErr != nil {
     framework.Logf("Warning: pcs resource cleanup failed: %v\noutput: %s", cleanupErr, output)
+} else {
+    framework.Logf("PCS resource cleanup output: %s", output)
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/edge_topologies/tnf_etcd_disruption.go` around lines 535 - 543,
Update the pcs resource cleanup block that calls
exutil.DebugNodeRetryWithOptionsAndChroot (the call using oc, execNode.Name and
"sudo pcs resource cleanup etcd-clone") to log the command output on success as
well as on error; i.e., after the call, always call framework.Logf with a
concise success message and the captured output when cleanupErr is nil, and keep
the existing warning log when cleanupErr is non-nil so diagnostics are
consistent.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/extended/edge_topologies/tnf_etcd_disruption.go`:
- Around line 535-543: Update the pcs resource cleanup block that calls
exutil.DebugNodeRetryWithOptionsAndChroot (the call using oc, execNode.Name and
"sudo pcs resource cleanup etcd-clone") to log the command output on success as
well as on error; i.e., after the call, always call framework.Logf with a
concise success message and the captured output when cleanupErr is nil, and keep
the existing warning log when cleanupErr is non-nil so diagnostics are
consistent.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4ffeaed2-43be-4d3e-819c-258eef5d5aab

📥 Commits

Reviewing files that changed from the base of the PR and between 6b33871 and a69bbb6.

📒 Files selected for processing (1)

test/extended/edge_topologies/tnf_etcd_disruption.go

lucaconsalvi · 2026-06-10T07:56:16Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-06-10T07:56:23Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/dbf33f20-64a1-11f1-99be-5db787e1433a-0

openshift-merge-bot · 2026-06-10T13:59:44Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

openshift-ci-robot · 2026-06-10T19:28:23Z

@lucaconsalvi: This pull request references OCPEDGE-2700 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Problem

The test should coordinate recovery with peer when local etcd container is killed flakes at ~1.9% (4 failures / 212 CI runs, seen on both PR payload runs and periodic nightlies).

All failures are identical: 600s timeout waiting for etcd-clone to be Started on both nodes. The failure log shows:
Failed Resource Actions:
 * etcd start on master-1 returned 'error' after 2m1.131s
 * etcd 30s-interval monitor on master-0 returned 'error' (master-0 must force a new cluster)
Root cause

When podman kill etcd is run on one node, the podman-etcd resource agent coordinates recovery via the force_new_cluster protocol:

The peer node's monitor detects a leadership loss → sets force_new_cluster → monitor fails

The peer restarts etcd with --force-new-cluster

The killed node's start action retries until it can rejoin as a learner

The race: step 3 (start action) can fail multiple times before step 1–2 completes. If it hits Pacemaker's migration-threshold (default: 3 failures) before the peer is ready, Pacemaker stops scheduling the resource entirely. The longRecoveryTimeout (10 min) then expires waiting for a recovery that Pacemaker won't attempt.

Fix

Restructure the test to capture the "Failed Resource Actions" evidence before running pcs resource cleanup, then let cleanup unblock Pacemaker:
kill etcd
 → wait cluster healthy          (first Eventually)
 → assert Failed Resource Actions on both nodes   ← moved here
 → pcs resource cleanup etcd-clone               ← new step
 → wait pcs Started on both nodes (second Eventually)
 → verifyFinalClusterHealth
By the time the cluster health check passes, the coordinated recovery has already run — both the peer's force_new_cluster monitor failure and the killed node's start failures are recorded in pcs status. We capture that evidence, then reset the failure count so Pacemaker can retry against the now-healthy peer cluster.

In the non-flaky path (98.1% of runs), cleanup is a no-op and the timing of the existing assertion is simply moved earlier.

Verification

Passes locally on TNF cluster (~400s)

Payload CI: first run 2/3 PASS, 1/3 infra flake (504 timeout during cluster setup — test never ran); second run 2/3 PASS, 1/3 FAIL due to unrelated Monitor test failures (missing SCC annotations, kubelet lease backoffs — our test passed in that shard)

Fixes: https://redhat.atlassian.net/browse/OCPEDGE-2700

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

lucaconsalvi · 2026-06-11T09:11:08Z

/retest

openshift-trt · 2026-06-11T14:47:39Z

Job Failure Risk Analysis for sha: ff7fde9

Job Name	Failure Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift	High install should succeed: other This test has passed 99.39% of 3782 runs on release 5.0 [Overall] in the last week.

lucaconsalvi · 2026-06-11T15:09:55Z

/retest-required

dhensel-rh · 2026-06-11T20:31:25Z

@lucaconsalvi We are capturing the failure, and running the necessary command to clean up the cluster so it is not left in a bad state. How do we make sure this does not result in missing failures if a real problem introduced ?

lucaconsalvi · 2026-06-12T09:17:26Z

@dhensel-rh Thanks for the observation! The cleanup could mask a real start-action failure. I've reworked the approach: instead of running pcs resource cleanup mid-test, I now raise migration-threshold to INFINITY on etcd-clone before the container kill, and restore it in DeferCleanup/AfterEach.
This way Pacemaker never stops retrying during the transient coordination window, so the test purely observes recovery without intervening. If a future code change genuinely breaks the start action, etcd-clone will never reach Started and the test times out — no cleanup to hide it.
I'm testing it locally to see if this could be a better solution!

- Parse migration-threshold in Go instead of shell grep pipeline to stop masking pcs failures behind || true - Add migration-threshold=INFINITY to test 5 (etcd process crash) which had the same podman kill as test 4 but lacked the protection - Remove unconditional restoreMigrationThreshold("") from AfterEach so DeferCleanup can restore to the actual original value - Drop verifyFinalClusterHealth from tests 1-6 (AfterEach covers it) - Remove redundant recovery polling in tests 2, 3 - Simplify Failed Resource Actions check in test 4 - Extract sshTestContext struct from 58-line inline SSH setup in test 7 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

lucaconsalvi · 2026-07-06T12:25:44Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-06T12:25:48Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/cfe00a30-7935-11f1-82e4-4a0a664b2c88-0

openshift-merge-bot · 2026-07-06T12:48:53Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

pcs stonith fence prints its confirmation ("Node: X fenced") to stderr, not stdout. runOnNode was discarding stderr, so the fence assertion retried for 3 minutes against empty stdout then timed out — re-fencing an already-fenced node on every retry. Combine stdout and stderr to match oc debug behavior, which merges both streams. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

lucaconsalvi · 2026-07-07T08:32:21Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-07T08:32:32Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5fa6e5f0-79de-11f1-8826-c62118e8b5f2-0

openshift-merge-bot · 2026-07-07T09:05:21Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

lucaconsalvi · 2026-07-07T09:57:34Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-07T09:57:38Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/472aa000-79ea-11f1-865a-99de944fca83-0

lucaconsalvi · 2026-07-07T12:54:26Z

/retest

lucaconsalvi · 2026-07-07T12:54:28Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-07T12:54:35Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fe7bb740-7a02-11f1-8188-b03bb8928b49-0

openshift-trt · 2026-07-07T18:14:06Z

Job Failure Risk Analysis for sha: 74acd2b

Job Name	Failure Risk
pull-ci-openshift-origin-main-e2e-metal-ovn-two-node-arbiter	Medium [sig-sippy] infrastructure should work This test has passed 80.85% of 8327 runs on release 5.0 [Overall] in the last week. --- install should succeed: other This test has passed 95.50% of 6180 runs on release 5.0 [Overall] in the last week. --- install should succeed: overall This test has passed 91.43% of 6263 runs on release 5.0 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-metal-ovn-two-node-fencing	Medium install should succeed: other This test has passed 95.50% of 6180 runs on release 5.0 [Overall] in the last week. --- install should succeed: overall This test has passed 91.43% of 6263 runs on release 5.0 [Overall] in the last week. --- [sig-sippy] infrastructure should work This test has passed 80.85% of 8327 runs on release 5.0 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-metal-ovn-two-node-fencing-recovery	Medium install should succeed: other This test has passed 95.50% of 6180 runs on release 5.0 [Overall] in the last week. --- install should succeed: overall This test has passed 91.43% of 6263 runs on release 5.0 [Overall] in the last week. --- [sig-sippy] infrastructure should work This test has passed 80.85% of 8327 runs on release 5.0 [Overall] in the last week.

openshift-trt · 2026-07-07T19:11:02Z

Job Failure Risk Analysis for sha: 74acd2b

Job Name	Failure Risk
pull-ci-openshift-origin-main-e2e-metal-ovn-two-node-arbiter	Medium [sig-sippy] infrastructure should work This test has passed 80.82% of 8375 runs on release 5.0 [Overall] in the last week. --- install should succeed: other This test has passed 95.45% of 6220 runs on release 5.0 [Overall] in the last week. --- install should succeed: overall This test has passed 91.40% of 6303 runs on release 5.0 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-metal-ovn-two-node-fencing	Medium install should succeed: other This test has passed 95.45% of 6220 runs on release 5.0 [Overall] in the last week. --- install should succeed: overall This test has passed 91.40% of 6303 runs on release 5.0 [Overall] in the last week. --- [sig-sippy] infrastructure should work This test has passed 80.82% of 8375 runs on release 5.0 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-metal-ovn-two-node-fencing-recovery	Medium install should succeed: other This test has passed 95.45% of 6220 runs on release 5.0 [Overall] in the last week. --- install should succeed: overall This test has passed 91.40% of 6303 runs on release 5.0 [Overall] in the last week. --- [sig-sippy] infrastructure should work This test has passed 80.82% of 8375 runs on release 5.0 [Overall] in the last week.

lucaconsalvi · 2026-07-08T15:40:15Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-08T15:40:24Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/51171c70-7ae3-11f1-8870-750edcbb4e67-0

lucaconsalvi · 2026-07-09T12:37:26Z

/retest

lucaconsalvi · 2026-07-09T12:37:39Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-09T12:37:46Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f94004c0-7b92-11f1-999b-7c2e23773d32-0

The is_standalone test (test 7) fails at the log grep assertion because pacemaker takes 4-5 minutes after unstandby to schedule a podman_start action, exceeding the 3-minute timeout. Additionally, the first start attempt after fence/reboot runs before CRM attributes are spoofed and may leave a FAILED state that delays subsequent starts. Add pcs resource cleanup before the standby/unstandby cycle to clear stale failure history, and increase the grep timeout from 3 minutes to longRecoveryTimeout (10 minutes) to match all other recovery waits. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

lucaconsalvi · 2026-07-10T09:50:14Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-10T09:50:18Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c07772b0-7c44-11f1-8e84-5d2ae4b525fb-0

openshift-merge-bot · 2026-07-10T10:14:42Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-10T17:13:11Z

@lucaconsalvi: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-metal-ovn-two-node-fencing-recovery	`a94cfc1`	link	false	`/test e2e-metal-ovn-two-node-fencing-recovery`
ci/prow/e2e-aws-ovn-microshift-serial	`a94cfc1`	link	true	`/test e2e-aws-ovn-microshift-serial`
ci/prow/e2e-metal-ovn-two-node-fencing	`a94cfc1`	link	false	`/test e2e-metal-ovn-two-node-fencing`
ci/prow/e2e-vsphere-ovn-upi	`a94cfc1`	link	true	`/test e2e-vsphere-ovn-upi`
ci/prow/e2e-aws-ovn-microshift	`a94cfc1`	link	true	`/test e2e-aws-ovn-microshift`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

lucaconsalvi · 2026-07-13T05:49:23Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-13T05:49:27Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9a291a00-7e7e-11f1-801d-76e7db5ff51f-0

After fencing a node in a two-node cluster, the Kubernetes API may be unreachable for several minutes while force-new-cluster runs on the survivor. The 3-minute timeout for the NotReady check is too short when the API is temporarily unavailable — increase to longRecoveryTimeout (10 min) to match other recovery waits in the file. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

lucaconsalvi · 2026-07-13T10:50:50Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery

openshift-ci · 2026-07-13T10:50:54Z

@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b71b2110-7ea8-11f1-8f10-e5ba67306ea2-0

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 9, 2026

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 9, 2026

openshift-ci Bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 9, 2026

coderabbitai Bot reviewed Jun 9, 2026

View reviewed changes

coderabbitai Bot approved these changes Jun 9, 2026

View reviewed changes

openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 9, 2026

lucaconsalvi marked this pull request as ready for review June 10, 2026 13:36

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2026

openshift-ci Bot requested review from eggfoobar and jeff-roche June 10, 2026 13:38

lucaconsalvi changed the title ~~test/etcd: fix flaky container-kill recovery test by adding pcs cleanup~~ OCPEDGE-2700: test/etcd: fix flaky container-kill recovery test by adding pcs cleanup Jun 10, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 10, 2026

lucaconsalvi changed the title ~~OCPEDGE-2700: test/etcd: fix flaky container-kill recovery test by adding pcs cleanup~~ OCPEDGE-2700: fix flaky container-kill recovery test by adding pcs cleanup Jun 10, 2026

Uh oh!

Conversation

lucaconsalvi commented Jun 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix

Verification

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jun 9, 2026

Uh oh!

openshift-ci Bot commented Jun 9, 2026

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Uh oh!

openshift-ci Bot commented Jun 9, 2026

Uh oh!

lucaconsalvi commented Jun 9, 2026

Uh oh!

openshift-ci Bot commented Jun 9, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

lucaconsalvi commented Jun 10, 2026

Uh oh!

openshift-ci Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-ci-robot commented Jun 10, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix

Verification

Uh oh!

lucaconsalvi commented Jun 11, 2026

Uh oh!

openshift-trt Bot commented Jun 11, 2026

Uh oh!

lucaconsalvi commented Jun 11, 2026

Uh oh!

dhensel-rh commented Jun 11, 2026

Uh oh!

lucaconsalvi commented Jun 12, 2026

Uh oh!

lucaconsalvi commented Jul 6, 2026

Uh oh!

openshift-ci Bot commented Jul 6, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 6, 2026

Uh oh!

lucaconsalvi commented Jul 7, 2026

Uh oh!

openshift-ci Bot commented Jul 7, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 7, 2026

Uh oh!

lucaconsalvi commented Jul 7, 2026

Uh oh!

openshift-ci Bot commented Jul 7, 2026

Uh oh!

lucaconsalvi commented Jul 7, 2026

Uh oh!

lucaconsalvi commented Jul 7, 2026

Uh oh!

openshift-ci Bot commented Jul 7, 2026

Uh oh!

openshift-trt Bot commented Jul 7, 2026

Uh oh!

openshift-trt Bot commented Jul 7, 2026

Uh oh!

lucaconsalvi commented Jun 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

openshift-ci-robot commented Jun 10, 2026 •

edited by openshift-ci Bot

Loading