Skip to content

Regulus: Add periodic jobs for metal-telco-x86 regulus-jetlag - #81496

Open
HughNhan wants to merge 14 commits into
openshift:mainfrom
HughNhan:metal-telco-x86-nightly
Open

Regulus: Add periodic jobs for metal-telco-x86 regulus-jetlag#81496
HughNhan wants to merge 14 commits into
openshift:mainfrom
HughNhan:metal-telco-x86-nightly

Conversation

@HughNhan

@HughNhan HughNhan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Add nightly periodic for metal-telco-x86 regulus-jetlag

Summary

Adds a new ci-operator config variant metal-telco-x86-nightly that runs the regulus-jetlag workload nightly against OCP 4.22 nightly candidate builds on bare metal telco hardware.

Changes

  • ci-operator/config/openshift-eng/ocp-qe-perfscale-ci/openshift-eng-ocp-qe-perfscale-ci-main__metal-telco-x86-nightly.yaml — new config targeting OCP 4.22 nightly candidate, cron 0 9 * * * (9AM UTC daily)
  • ci-operator/jobs/openshift-eng/ocp-qe-perfscale-ci/openshift-eng-ocp-qe-perfscale-ci-main-periodics.yaml — auto-generated by make jobs

Why

The existing metal-telco-x86 config targets stable 4.18 and is on-demand only. This nightly variant enables continuous regression detection on the latest OCP nightly builds using the Crucible/Regulus benchmarking stack on the metal-perfscale-jetlag cluster profile.

Testing

  • make jobs run locally, generated periodics YAML included
  • /pj-rehearse to validate before merge

Summary by CodeRabbit

This PR extends ocp-qe-perfscale-ci with a metal-telco-x86-regulus variant for bare-metal telco hardware. It runs Regulus benchmarks and Orion fingerprint analysis with the metal-perfscale-jetlag profile.

Key changes:

  • Adds the openshift-qe-orion-regulus CI step for Regulus regression detection.
  • Changes LOOKBACK to retain 15 samples and supports sample counts or duration strings.
  • Mounts Elasticsearch credentials and protects them during shell execution.
  • Adds deploy, no-jetlag, and Orion-only workflows.
  • Targets OCP 4.22 nightly candidate builds.
  • Schedules the nightly job at 0 9 * * * (09:00 UTC).
  • Adds generated periodic, pull, and rehearsal job configuration.

For Regulus ORION references

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 57cb2a6f-3a84-4169-b0ec-edd32eda563e

📥 Commits

Reviewing files that changed from the base of the PR and between 42151de and 6e00886.

📒 Files selected for processing (1)
  • ci-operator/step-registry/openshift-qe/orion/regulus/openshift-qe-orion-regulus-ref.yaml

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


Walkthrough

Adds the openshift-qe-orion-regulus CI step, Elasticsearch credential propagation for Regulus installation, and a new metal-telco-x86-regulus configuration with scheduled, no-jetlag, and Orion-only workflows.

Changes

Regulus Orion CI integration

Layer / File(s) Summary
Step contract and execution
ci-operator/step-registry/openshift-qe/orion/regulus/...
Registers the step, defines its inputs and resources, assigns ownership metadata, clones Orion and Regulus, installs dependencies, runs prow-entry.sh, and preserves the analysis exit status.
Elasticsearch credential propagation
ci-operator/step-registry/openshift-qe/installer/bm/regulus/...
Mounts Elasticsearch credentials and writes them to the generated lab configuration with shell tracing disabled and restrictive file permissions.
Metal telco workflow configuration
ci-operator/config/openshift-eng/ocp-qe-perfscale-ci/...
Adds images, the OpenShift 4.22 nightly release, resource defaults, and three Regulus-related workflows using the metal-perfscale-jetlag profile.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 6e008

The PR adds a scheduled CI configuration and generated job definitions; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant CIWorkflow
  participant OrionRegulusStep
  participant OrionRepository
  participant RegulusRepository
  CIWorkflow->>OrionRegulusStep: Execute configured step
  OrionRegulusStep->>OrionRepository: Resolve tag and clone repository
  OrionRegulusStep->>RegulusRepository: Clone configured branch
  OrionRegulusStep->>OrionRegulusStep: Install dependencies and run prow-entry.sh
  OrionRegulusStep-->>CIWorkflow: Return analysis exit status
Loading

Possibly related PRs

  • openshift/release#83659: Adds a similar CI Operator step-registry entry with scripts, reference YAML, metadata, and ownership files.

Suggested reviewers: chaitanyaenr, sachinninganure


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The new Orion step echoes ${ORION_REPO} and ${REGULUS_REPO} directly; configured URLs can contain access tokens or internal hostnames. Do not log repository URLs. Log fixed repository labels, or sanitize URLs to remove credentials and restrict host details before output.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding periodic Regulus jobs for the metal-telco-x86 variant.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR adds CI YAML and shell steps, not Ginkgo tests; searches of all 9 changed files found no It(), Describe(), Context(), or When() titles.
Test Structure And Quality ✅ Passed The PR diff adds only YAML, shell, JSON, and ownership files; it adds no Ginkgo test code or It/BeforeEach/AfterEach/Eventually calls to assess.
Microshift Test Compatibility ✅ Passed The pull request adds CI YAML, generated jobs, and shell steps only; the diff contains no new Ginkgo e2e tests or It/Describe/Context/When constructs.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The complete PR diff adds or modifies only YAML, shell, JSON, and OWNERS files; it adds no Ginkgo e2e tests or multi-node test assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed The complete diff adds CI step scripts, registry metadata, and Prow job/config YAML; scans found no deployment/controller code or topology scheduling constraints.
Ote Binary Stdout Contract ✅ Passed The aggregate PR diff contains only YAML, JSON, and Bash; no Go or OTE/Ginkgo process-level code. Shell echo/printf output is CI-step logging, not OTE binary stdout.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The complete PR diff adds only CI YAML, shell, JSON, and ownership files; it adds no Ginkgo e2e tests or IPv4 test logic, so this check does not apply.
No-Weak-Crypto ✅ Passed The PR diff adds no MD5, SHA1, DES, RC4, Blowfish, ECB, crypto API, custom crypto, or secret/token comparison; it only reads ES credentials over HTTPS.
Container-Privileges ✅ Passed The PR diff adds no privileged:true, hostPID/Network/IPC, SYS_ADMIN, allowPrivilegeEscalation:true, or runAsUser:0; root references are pre-existing or remote SSH targets.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@HughNhan

HughNhan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-nightly-regulus-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

HughNhan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

@sraviteja-maker, can you add /pj-rehearse network-access-allowed so I can rehearse this.

@HughNhan
HughNhan force-pushed the metal-telco-x86-nightly branch from 32f9bf9 to 1ca2cb1 Compare July 6, 2026 15:17
@sraviteja-maker

Copy link
Copy Markdown
Contributor

/pj-rehearse network-access-allowed

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sraviteja-maker: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

HughNhan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-nightly-regulus-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

HughNhan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-nightly-regulus-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@afcollins

Copy link
Copy Markdown
Contributor

/pj-rehearse network-access-allowed

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@afcollins: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

HughNhan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-nightly-regulus-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

HughNhan commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-nightly-regulus-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 8, 2026
@sraviteja-maker

Copy link
Copy Markdown
Contributor

/pj-rehearse network-access-allowed

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sraviteja-maker: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan
HughNhan force-pushed the metal-telco-x86-nightly branch from fb7a737 to 8d9d581 Compare July 9, 2026 16:01
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 9, 2026
@afcollins

Copy link
Copy Markdown
Contributor

/pj-rehearse network-access-allowed

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: pj-rehearse could not automatically process this event because the request waited in queue for longer than 5 minutes. Use /pj-rehearse to trigger rehearsals manually.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-orion-only

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-jetlag-no-deploy

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

@HughNhan: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift-eng/ocp-qe-perfscale-ci/main/metal-telco-x86-nightly-regulus-jetlag-without-deploy 51373f5 link unknown /pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-nightly-regulus-jetlag-without-deploy
ci/rehearse/openshift-eng/ocp-perfscale/main/metal-telco-x86-regulus-jetlag a0426a7 link unknown /pj-rehearse pull-ci-openshift-eng-ocp-perfscale-main-metal-telco-x86-regulus-jetlag

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@HughNhan

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

… from tracing

- Rename job as: jetlag-no-deploy → no-jetlag
- Add set -o errexit to orion-regulus step; wrap prow-entry.sh in if/else
  to preserve exit status capture
- Disable set -x tracing around credential reads and lab.config generation
  in bm-regulus step to prevent leaking secrets in CI logs
- Set umask 077 for lab.config (mode 600)
- Use pip install -r requirements.txt instead of inline package list

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@HughNhan

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Already reviewed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sample count is better suited for weekly jobs where a time-based
lookback of 15d would yield only ~2 samples.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@HughNhan: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-no-jetlag openshift-eng/ocp-qe-perfscale-ci presubmit Presubmit changed
pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-orion-only openshift-eng/ocp-qe-perfscale-ci presubmit Presubmit changed
pull-ci-openshift-eng-ocp-perfscale-main-metal-telco-x86-regulus-cpt-without-deploy openshift-eng/ocp-perfscale presubmit Registry content changed
pull-ci-openshift-eng-ocp-perfscale-main-metal-telco-x86-regulus-jetlag-without-deploy openshift-eng/ocp-perfscale presubmit Registry content changed
pull-ci-openshift-eng-ocp-perfscale-main-metal-telco-x86-regulus-cpt openshift-eng/ocp-perfscale presubmit Registry content changed
pull-ci-openshift-eng-ocp-perfscale-main-metal-telco-x86-regulus-jetlag openshift-eng/ocp-perfscale presubmit Registry content changed
periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-jetlag N/A periodic Periodic changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@vishnuchalla

Copy link
Copy Markdown
Contributor

/pj-rehearse network-access-allowed

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@vishnuchalla: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-no-jetlag

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-orion-only

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

Copy link
Copy Markdown
Contributor Author

@HughNhan Thanks for putting this together. In general I have some concerns with how you are using Orion here.

We shouldn't need a new way to create and use orion configs -- this PR uses aspects of a python library which generates Orion configs from python. https://github.com/redhat-performance/regulus/blob/main/ORION/scripts/analyze-batch.py#L310

As mentioned over email, this shouldn't be necessary, we should define our KPIs upfront and not have any sort of dynamic discovery -- meaning, from test execution to test execution, the tests should stay consistent -- and what we compare shouldn't change either. Now, this doesn't mean we don't update tests overtime -- turn on fips, try new features, etc - but when it comes to regression detection, we want stability.

We typically default to a 15 day lookback, you have 90 days defined here ( https://github.com/openshift/release/pull/81496/changes#diff-1af550afed65863a5c4d1f935618f80edc4e553a8cccabb78993d74d03b3fa31R34 ), I would recommend to keeping a similar lookback -- or not using the lookback. It appears you won't run this workload but once a week?

@chentex can you review the changes here to the orion execution and provide some feedback too?

@HughNhan Thanks for putting this together. In general I have some concerns with how you are using Orion here.

We shouldn't need a new way to create and use orion configs -- this PR uses aspects of a python library which generates Orion configs from python. https://github.com/redhat-performance/regulus/blob/main/ORION/scripts/analyze-batch.py#L310

As mentioned over email, this shouldn't be necessary, we should define our KPIs upfront and not have any sort of dynamic discovery -- meaning, from test execution to test execution, the tests should stay consistent -- and what we compare shouldn't change either. Now, this doesn't mean we don't update tests overtime -- turn on fips, try new features, etc - but when it comes to regression detection, we want stability.

We typically default to a 15 day lookback, you have 90 days defined here ( https://github.com/openshift/release/pull/81496/changes#diff-1af550afed65863a5c4d1f935618f80edc4e553a8cccabb78993d74d03b3fa31R34 ), I would recommend to keeping a similar lookback -- or not using the lookback. It appears you won't run this workload but once a week?

@chentex can you review the changes here to the orion execution and provide some feedback too?

@jtaleric - I have made the necessary changes to address your comments:

  1. use template method. So there is no in-flight config creations.
  2. use loopback-size=15 so that we will have an acceptable number of (15) samples. worked for both daily and weekly jobs.

@HughNhan
HughNhan requested a review from jtaleric August 20, 2026 15:40
@HughNhan

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-telco-x86-regulus-orion-only

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@HughNhan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@HughNhan

Copy link
Copy Markdown
Contributor Author

@jtaleric @afcollins @vishnuchalla - I have addressed all comments. It is ready to merge if you don't see any new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants