Skip to content

50 hack debug - #82599

Open
sergiordlr wants to merge 2 commits into
openshift:mainfrom
sergiordlr:50_hack_debug
Open

50 hack debug#82599
sergiordlr wants to merge 2 commits into
openshift:mainfrom
sergiordlr:50_hack_debug

Conversation

@sergiordlr

@sergiordlr sergiordlr commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

wip

Summary by CodeRabbit

  • Adds weekly disconnected AWS IPI pin-target-release rehearsals for MCO releases 4.23 and 5.0.
  • Moves pin-MCO coverage out of the release-5.0 automated nightly upgrade jobs.
  • Simplifies the disconnected AWS IPI provision workflow.
  • Adds an option to skip QE catalog source deployment.
  • Ensures required ICSP or IDMS mirror policies for disconnected sample-operator checks.
  • Improves image mirroring reliability with up to 10 retries.
  • Keeps failed upgrade steps available for debugging until /tmp/continue is provided.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Changes

Disconnected MCO rehearsal

Layer / File(s) Summary
Periodic MCO pinning jobs
ci-operator/config/openshift/machine-config-operator/*periodics.yaml, ci-operator/config/openshift/openshift-tests-private/*automated-release-nightly*.yaml
Adds periodic AWS disconnected MCO pinning jobs for releases 4.23 and 5.0. Removes two older scheduled entries.
Private disconnected provisioning
ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/*chain.yaml, ci-operator/step-registry/enable-qe-catalogsource/disconnected/*, ci-operator/step-registry/set-sample-operator/disconnected/*
Removes selected provisioning steps, adds an optional QE catalog source skip, and creates version-specific mirror policies before sample operator checks.
Image mirroring retries
ci-operator/step-registry/cucushift/upgrade/mirror-images/by-oc-adm/*, ci-operator/step-registry/mirror-images/by-oc-adm/*
Increases mirroring retries to ten attempts and adds bounded retry handling with delays between failures.
Upgrade failure debugging
ci-operator/step-registry/cucushift/upgrade/toimage/*
Preserves upgrade and status-check return codes and pauses for manual debugging until /tmp/continue exists.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: do-not-merge/work-in-progress

Suggested reviewers: bandrade, jechen0648

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title indicates a debugging work-in-progress but does not clearly summarize the periodic test and disconnected workflow changes. Use a concise title that names the main change, such as adding disconnected private AWS IPI MCO rehearsal periodics and related debugging updates.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes only YAML job/step configuration and shell scripts; searches found no Ginkgo It, Describe, Context, When, or Specify declarations in changed files.
Test Structure And Quality ✅ Passed The full diff against origin/main contains only CI YAML and shell changes; no Ginkgo test files or It/BeforeEach/AfterEach/Eventually constructs were modified.
Microshift Test Compatibility ✅ Passed The commit adds no Go files, Ginkgo declarations, or new e2e tests; it changes only YAML job definitions and shell scripts, so MicroShift test compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The pull request changes only CI YAML and shell scripts; no Go or Ginkgo test files were added or modified, so SNO compatibility review is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The full PR diff adds CI jobs and image-policy resources only; no deployment/controller scheduling fields such as affinity, topology spread, node selectors, tolerations, replicas, or PDBs were added.
Ote Binary Stdout Contract ✅ Passed The PR changes only CI YAML and shell step files; no OTE binary source or main/suite setup code changed, and no process-level OTE stdout violation is present.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR adds CI YAML jobs and shell-step changes only; it adds no Go files or Ginkgo It/Describe/Context/When declarations, so this check is not applicable.
No-Weak-Crypto ✅ Passed The full PR diff adds no MD5, SHA1, DES, RC4, Blowfish, ECB, custom crypto, or secret/token comparisons; changes only add workflow, retry, and status logic.
Container-Privileges ✅ Passed The full PR diff adds no privileged:true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or runAsUser: 0; new Job containers have no securityContext.
No-Sensitive-Data-In-Logs ✅ Passed Added logs contain only retry counts and generic failure messages; no added line prints passwords, tokens, credentials, PII, hostnames, or customer data.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
openshift-ci Bot requested review from bandrade and jechen0648 July 29, 2026 07:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml`:
- Line 437: Replace the mutable latest release image override with an immutable
digest or durable promoted release image in both
CUSTOM_OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE entries:
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml:437
and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml:472.
Keep the machine-config-operator periodic test configuration unchanged
otherwise.

In
`@ci-operator/step-registry/set-sample-operator/disconnected/set-sample-operator-disconnected-commands.sh`:
- Around line 95-99: Update the new-API guard around the
ImageDigestMirrorSet/ImageTagMirrorSet creation flow to skip only when both
resources exist; retain the legacy ImageContentSourcePolicy handling as
appropriate. Replace the non-idempotent creation path with oc apply so a missing
ImageTagMirrorSet is created when ImageDigestMirrorSet already exists, including
the corresponding logic in the later resource-creation block.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: f5aeb3c0-bd4e-4bb2-941c-964e76db8490

📥 Commits

Reviewing files that changed from the base of the PR and between b43de5a and ce4c714.

⛔ Files ignored due to path filters (3)
  • ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (8)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-4.22.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-5.0.yaml
  • ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-provision-chain.yaml
  • ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh
  • ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml
  • ci-operator/step-registry/set-sample-operator/disconnected/set-sample-operator-disconnected-commands.sh
💤 Files with no reviewable changes (3)
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-5.0.yaml
  • ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-provision-chain.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-4.22.yaml

dependencies:
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE: release:initial
env:
CUSTOM_OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE: registry.build10.ci.openshift.org/ci-ln-xitccvk/release:latest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

if rg -n 'CUSTOM_OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE:.*:latest' ci-operator/config; then
  echo "Mutable release-image override remains" >&2
  exit 1
fi

Repository: openshift/release

Length of output: 1835


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

for f in \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
do
  echo "===== $f ====="
  rg -n 'name:|cron:|interval:|periodics:|test:|CUSTOM_OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE:|CUSTOM_OPENSHIFT_INSTALL_RELEASE_IMAGE:' "$f" | sed -n '1,180p'
done

Repository: openshift/release

Length of output: 2335


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

for f in \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
do
  echo "===== contexts around line 437/472 in $f ====="
  sed -n '412,475p' "$f" | nl -ba -v412
done

Repository: openshift/release

Length of output: 355


Do not use a mutable build image for recurring pinning tests.

Both machine-config-operator periodics run on 72h/weekly schedules but override the release image with registry.build${N}.ci.openshift.org/ci-ln-xitccvk/release:latest. Move these to an immutable digest or durable promoted release image; otherwise the release-under-test can change between runs or fail if the build namespace image is removed.

  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L437
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml#L472
📍 Affects 2 files
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L437-L437 (this comment)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml#L472-L472
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml`
at line 437, Replace the mutable latest release image override with an immutable
digest or durable promoted release image in both
CUSTOM_OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE entries:
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml:437
and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml:472.
Keep the machine-config-operator periodic test configuration unchanged
otherwise.

Comment on lines +95 to +99
if oc get ImageContentSourcePolicy image-policy-aosqe &>/dev/null || \
oc get ImageDigestMirrorSet image-policy-aosqe &>/dev/null; then
echo "ICSP/IDMS already exists, skip creating."
return 0
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Require both IDMS and ITMS before skipping.

For the new-API path, an existing ImageDigestMirrorSet alone is insufficient: tag-based pulls require ImageTagMirrorSet. A partial prior oc create leaves IDMS present and ITMS absent; this guard exits successfully and cannot repair it. Check for both resources, then use oc apply so a missing ITMS is created.

Proposed fix
-    if oc get ImageContentSourcePolicy image-policy-aosqe &>/dev/null || \
-       oc get ImageDigestMirrorSet image-policy-aosqe &>/dev/null; then
-        echo "ICSP/IDMS already exists, skip creating."
-        return 0
-    fi
-
-    echo "ICSP/IDMS not found, creating..."
     kube_minor=$(oc version -o json | grep -o '"minor": *"[^"]*"' | tail -1 | grep -o '[0-9]*')
     icsp_num=$(oc get ImageContentSourcePolicy -o name 2>/dev/null | wc -l)
 
     if [[ $icsp_num -gt 0 || $kube_minor -lt 26 ]]; then
+        oc get ImageContentSourcePolicy image-policy-aosqe &>/dev/null && return 0
         cat <<EOF | oc apply -f -
+    elif oc get ImageDigestMirrorSet image-policy-aosqe &>/dev/null && \
+         oc get ImageTagMirrorSet image-policy-aosqe &>/dev/null; then
+        return 0
+    else
+        cat <<EOF | oc apply -f -

This matches the upstream mirror-policy contract, which creates IDMS and ITMS together.

Also applies to: 130-174

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/set-sample-operator/disconnected/set-sample-operator-disconnected-commands.sh`
around lines 95 - 99, Update the new-API guard around the
ImageDigestMirrorSet/ImageTagMirrorSet creation flow to skip only when both
resources exist; retain the legacy ImageContentSourcePolicy handling as
appropriate. Replace the non-idempotent creation path with oc apply so a missing
ImageTagMirrorSet is created when ImageDigestMirrorSet already exists, including
the corresponding logic in the later resource-creation block.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: pj-rehearse could not automatically process this event because the request waited in queue for longer than 5 minutes. Use /pj-rehearse to trigger rehearsals manually.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/cucushift/upgrade/mirror-images/by-oc-adm/cucushift-upgrade-mirror-images-by-oc-adm-commands.sh`:
- Around line 94-97: Update the retry failure branch in the mirroring loop to
log “Trying again...” and sleep only when attempt is less than max_attempts. On
the final failed attempt, skip both actions so the script terminates
immediately.

In
`@ci-operator/step-registry/mco/conf/day2/pintargetrelease/mco-conf-day2-pintargetrelease-commands.sh`:
- Around line 24-39: Align the upgrade-edge consumer with the producer’s
“intermediate,target” format in the edge_count validation and TARGET assignment.
Do not reject the two-entry value as a serial upgrade, and select the target
(second comma-separated entry) for pinning while preserving the existing
override behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 980faf7c-b67e-4ce2-95b4-ce30a437a3fa

📥 Commits

Reviewing files that changed from the base of the PR and between a5f75d5 and 4ce35ec.

📒 Files selected for processing (6)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
  • ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-provision-chain.yaml
  • ci-operator/step-registry/cucushift/upgrade/mirror-images/by-oc-adm/cucushift-upgrade-mirror-images-by-oc-adm-commands.sh
  • ci-operator/step-registry/mco/conf/day2/pintargetrelease/mco-conf-day2-pintargetrelease-commands.sh
  • ci-operator/step-registry/mirror-images/by-oc-adm/mirror-images-by-oc-adm-commands.sh
💤 Files with no reviewable changes (1)
  • ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-provision-chain.yaml

Comment on lines +94 to +97
else
echo "Mirroring images attempt ${attempt} failed. Trying again..."
sleep 120
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win

Avoid sleeping after the final failed attempt.

On attempt 10, the script still logs “Trying again” and sleeps for 120 seconds before the loop terminates. Gate the log and sleep on (( attempt < max_attempts )) to fail immediately when retries are exhausted.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/cucushift/upgrade/mirror-images/by-oc-adm/cucushift-upgrade-mirror-images-by-oc-adm-commands.sh`
around lines 94 - 97, Update the retry failure branch in the mirroring loop to
log “Trying again...” and sleep only when attempt is less than max_attempts. On
the final failed attempt, skip both actions so the script terminates
immediately.

Comment on lines +24 to +39
# If those file exists with multiple releases we fail the execution since serial upgrades are not supported in this step
if [[ -f "${SHARED_DIR}/upgrade-edge" ]]; then
echo "ERROR: Serial upgrades are not supported!!"
exit 255
edge_count=$(tr ',' '\n' < "${SHARED_DIR}/upgrade-edge" | grep -c .)
if (( edge_count > 1 )); then
echo "ERROR: Serial upgrades are not supported!!"
exit 255
fi
echo "Single upgrade target override found in upgrade-edge, continuing..."
fi


TARGET="${OPENSHIFT_UPGRADE_RELEASE_IMAGE_OVERRIDE}"
if [[ -f "${SHARED_DIR}/upgrade-edge" ]]; then
TARGET=$(head -n 1 "${SHARED_DIR}/upgrade-edge" | tr ',' '\n' | head -1)
echo "Overriding upgrade target from upgrade-edge: ${TARGET}"
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Align upgrade-edge parsing with its producer.

The supplied producer writes ${RELEASE_IMAGE_INTERMEDIATE},${RELEASE_IMAGE_TARGET}. This code counts two entries and exits at Line 29, so that flow cannot reach pinning. If the check is relaxed, Line 37 selects the intermediate rather than the target. Align the producer/consumer contract or select the target entry here; otherwise disconnected MCO pinning will fail or pin the wrong release.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/mco/conf/day2/pintargetrelease/mco-conf-day2-pintargetrelease-commands.sh`
around lines 24 - 39, Align the upgrade-edge consumer with the producer’s
“intermediate,target” format in the edge_count validation and TARGET assignment.
Do not reject the two-entry value as a serial upgrade, and select the target
(second comma-separated entry) for pinning while preserving the existing
override behavior.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sergiordlr
Once this PR has been reviewed and has the lgtm label, please assign aguidirh, bandrade for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr, pj-rehearse: unable prepare a candidate for rehearsal; rehearsals will not be run. This could be due to a branch that needs to be rebased. ERROR:

couldn't checkout base SHA cdb530e220c01bd33972d2bc459be9599a811ac0: error checking out "cdb530e220c01bd33972d2bc459be9599a811ac0": exit status 128 fatal: unable to read tree (cdb530e220c01bd33972d2bc459be9599a811ac0)

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sergiordlr

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sergiordlr: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@sergiordlr: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-priv-openshift-tests-private-release-4.22-debug-winc-vsphere-ipi-disconnected openshift-priv/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-priv-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi-disconnected openshift-priv/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-priv-openshift-tests-private-release-4.20-debug-winc-vsphere-ipi-disconnected openshift-priv/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-vsphere-ipi-disconnected openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi-disconnected openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-vsphere-ipi-disconnected openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-vsphere-ipi-disconnected openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.18-debug-winc-vsphere-ipi-disconnected openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-installer-main-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-5.1-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-5.0-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-4.23-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-4.22-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-4.21-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-4.20-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-4.19-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-installer-release-4.18-e2e-openstack-singlestackv6 openshift/installer presubmit Registry content changed
pull-ci-openshift-machine-config-operator-main-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed
pull-ci-openshift-machine-config-operator-release-5.1-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed
pull-ci-openshift-machine-config-operator-release-5.0-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed
pull-ci-openshift-machine-config-operator-release-4.23-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed
pull-ci-openshift-machine-config-operator-release-4.22-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed
pull-ci-openshift-machine-config-operator-release-4.21-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed
pull-ci-openshift-machine-config-operator-release-4.20-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed
pull-ci-openshift-machine-config-operator-release-4.19-e2e-openstack-singlestackv6 openshift/machine-config-operator presubmit Registry content changed

A total of 3631 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/cucushift/upgrade/toimage/cucushift-upgrade-toimage-commands.sh`:
- Around line 665-667: Update the upgrade function so non-expected failure paths
use return-based propagation instead of exit 1, including failed run_command
calls around the command execution paths, allowing upgrade_rc to always capture
the function status. Preserve intentional negative-test failures with explicit
statuses and comments, and keep successful execution behavior unchanged.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 3505b9d4-b357-4f8d-9fd1-816a4e9785a0

📥 Commits

Reviewing files that changed from the base of the PR and between f1945d4 and 80493f8.

⛔ Files ignored due to path filters (3)
  • ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (11)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-4.22.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-5.0.yaml
  • ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-provision-chain.yaml
  • ci-operator/step-registry/cucushift/upgrade/mirror-images/by-oc-adm/cucushift-upgrade-mirror-images-by-oc-adm-commands.sh
  • ci-operator/step-registry/cucushift/upgrade/toimage/cucushift-upgrade-toimage-commands.sh
  • ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh
  • ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml
  • ci-operator/step-registry/mirror-images/by-oc-adm/mirror-images-by-oc-adm-commands.sh
  • ci-operator/step-registry/set-sample-operator/disconnected/set-sample-operator-disconnected-commands.sh
💤 Files with no reviewable changes (3)
  • ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-provision-chain.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-4.22.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__automated-release-nightly-5.0-upgrade-from-stable-5.0.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
  • ci-operator/step-registry/cucushift/upgrade/mirror-images/by-oc-adm/cucushift-upgrade-mirror-images-by-oc-adm-commands.sh
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
  • ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-commands.sh
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
  • ci-operator/step-registry/enable-qe-catalogsource/disconnected/enable-qe-catalogsource-disconnected-ref.yaml
  • ci-operator/step-registry/mirror-images/by-oc-adm/mirror-images-by-oc-adm-commands.sh

Comment on lines +665 to +667
set +e
upgrade
upgrade_rc=$?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="ci-operator/step-registry/cucushift/upgrade/toimage/cucushift-upgrade-toimage-commands.sh"

printf 'File exists: '
test -f "$file" && printf 'yes\n' || printf 'no\n'

printf '\nRelevant function/header sections:\n'
sed -n '1,80p' "$file"
printf '\n'
sed -n '380,455p' "$file"
printf '\n'
sed -n '655,690p' "$file"

printf '\nRun command definitions/usages:\n'
rg -n '^[[:space:]]*(run_command|function run_command|upgrade_rc|check_upgrade_status|exit 1)\b' "$file"

Repository: openshift/release

Length of output: 10600


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="ci-operator/step-registry/cucushift/upgrade/toimage/cucushift-upgrade-toimage-commands.sh"

printf 'Candidate files matching name:\n'
fd -a 'cucushift-upgrade-toimage-commands\.sh$' /

printf '\nAll run_command definitions in repo:\n'
rg -n '^[[:space:]]*(run_command|function run_command)\b' ci-operator/step-registry || true

Repository: openshift/release

Length of output: 50374


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="ci-operator/step-registry/cucushift/upgrade/toimage/cucushift-upgrade-toimage-commands.sh"

printf 'run_command definition:\n'
sed -n '280,315p' "$file"
printf '\n'

printf 'Relevant run_command callers around upgrade():\n'
sed -n '370,450p' "$file"

printf '\nBash set -e status probe for function return vs exit:\n'
bash -c '
set -e
run() {
  local cmd
  cmd="false"
  (eval "$cmd")
  :
}
set +e
run
rc=$?
echo "shell_rc=$rc"
'
printf '\n'

printf 'Bash set -e probe for expected failures using exit 1 and uncheck command in function:\n'
bash -c '
set -e
function upgrade() {
  if false; then
    run_command "echo fail"
    exit 1
  fi
  run_command "false"
  echo "started"
}
run_command() {
  eval "$1"
}
set +e
upgrade
upgrade_rc=$?
echo "upgrade_rc=${upgrade_rc:-set-to-undefined}"
'

Repository: openshift/release

Length of output: 5042


Make upgrade() return explicit failures.

upgrade() uses exit 1, so failures inside the function can terminate the script before upgrade_rc is assigned. run_command also returns the eval status, so a failed command at Lines 436 or 441 can continue and fall through to Line 442. Replace non-expected exit 1 paths with return/|| return $? so the caller receives the status reliably.

Keep intentional negative-test failures handled with explicit status and comments.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/cucushift/upgrade/toimage/cucushift-upgrade-toimage-commands.sh`
around lines 665 - 667, Update the upgrade function so non-expected failure
paths use return-based propagation instead of exit 1, including failed
run_command calls around the command execution paths, allowing upgrade_rc to
always capture the function status. Preserve intentional negative-test failures
with explicit statuses and comments, and keep successful execution behavior
unchanged.

Source: MCP tools

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@sergiordlr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco ce4c714 link unknown /pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco
ci/rehearse/periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco 80493f8 link unknown /pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-ipi-disc-priv-tp-pin-mco

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant