Skip to content

CNTRLPLANE-3846: Multiple CI jobs failing for 4.19. - #82614

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
amogh-redhat:CNTRLPLANE-3946-4.19
Aug 4, 2026
Merged

CNTRLPLANE-3846: Multiple CI jobs failing for 4.19.#82614
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
amogh-redhat:CNTRLPLANE-3946-4.19

Conversation

@amogh-redhat

@amogh-redhat amogh-redhat commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:
Multiple periodic CI jobs in the openshift/hypershift repository have been consistently failing across releases 4.19 with near-zero pass rates, consuming significant CI infrastructure resources with no useful signal.

e2e-ibmcloud-ovn-iks
e2e-ibmcloud-ovn-roks
e2e-powervs-ovn
e2e-agent-disconnected-ovn-ipv6-metal-conformance
e2e-ibmz-ovn-conformance
e2e-kubevirt-metal-ovn
e2e-kubevirt-metal-ovn-multinet
e2e-kubevirt-metal-ovn-multinet-default-net
e2e-power-ovn-conformance
e2e-x86-power-heterogeneous-conformance

I have updated the cron schedule to trigger once a year on 1st Dec every year (cron: 0 4 1 12 *) and for few jobs, changed the minimal interval to 3 months (2190d) so we get enough time to fix the errors and the jobs wont trigger till then.
The job definition itself stays in the configuration file, so the YAML remains syntactically valid.
If these jobs had presubmit/postsubmit triggers defined elsewhere (like running on PRs via /test jobname), those manual/PR triggers would still work.

Which issue(s) this PR fixes: Failing periodic jobs.

Fixes - https://redhat.atlassian.net/browse/CNTRLPLANE-3946

Special notes for your reviewer:

Checklist:
Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

Summary by CodeRabbit

Updates Hypershift periodic CI jobs across release 4.19–4.22 configurations to run annually on December 1 at 04:00 UTC, reducing repeated failures while retaining manual and presubmit/postsubmit triggers. Selected conformance jobs now also enforce a 2,190-day minimum interval; job definitions and workflows remain intact.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 29, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@amogh-redhat: This pull request references CNTRLPLANE-3846 which is a valid jira issue.

Details

In response to this:

What this PR does / why we need it:
Multiple periodic CI jobs in the openshift/hypershift repository have been consistently failing across releases 4.19 with near-zero pass rates, consuming significant CI infrastructure resources with no useful signal.

e2e-ibmcloud-ovn-iks
e2e-ibmcloud-ovn-roks
e2e-powervs-ovn
e2e-agent-disconnected-ovn-ipv6-metal-conformance
e2e-ibmz-ovn-conformance
e2e-kubevirt-metal-ovn
e2e-kubevirt-metal-ovn-multinet
e2e-kubevirt-metal-ovn-multinet-default-net
e2e-power-ovn-conformance
e2e-x86-power-heterogeneous-conformance

I have updated the cron schedule to trigger once a year on 1st Dec every year (cron: 0 4 1 12 *) and for few jobs, changed the minimal interval to 3 months (2190d) so we get enough time to fix the errors and the jobs wont trigger till then.
The job definition itself stays in the configuration file, so the YAML remains syntactically valid.
If these jobs had presubmit/postsubmit triggers defined elsewhere (like running on PRs via /test jobname), those manual/PR triggers would still work.

Which issue(s) this PR fixes: Failing periodic jobs.

Fixes - https://redhat.atlassian.net/browse/CNTRLPLANE-3946

Special notes for your reviewer:

Checklist:
Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@amogh-redhat, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 50 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 5aada1e8-1497-46a5-a8a1-9c333fcdc0ce

📥 Commits

Reviewing files that changed from the base of the PR and between 6ece02c and 9da35ca.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.19-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (2)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19__periodics-mce.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19__periodics.yaml

Walkthrough

HyperShift periodic configurations for releases 4.19–4.22 align multiple MCE and standard E2E jobs to 0 4 1 12 *, increase selected minimum_interval values to 2190h, and add a PowerVS job in release 4.21.

Changes

HyperShift periodic schedules

Layer / File(s) Summary
MCE cron schedule updates
ci-operator/config/openshift/hypershift/*periodics-mce.yaml
Multiple MCE conformance jobs now use 0 4 1 12 *.
MCE minimum interval updates
ci-operator/config/openshift/hypershift/*periodics-mce.yaml
Selected IBM Z, power, and other conformance jobs use minimum_interval: 2190h.
Standard periodic job schedules
ci-operator/config/openshift/hypershift/*periodics.yaml
Cloud and PowerVS periodic schedules use 0 4 1 12 *; release 4.21 adds an e2e-powervs-ovn entry.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: simonpasquier

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is related to the change set, describing failing 4.19 CI jobs that the PR updates.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo test titles were changed; the PR only updates ci-operator YAML cron/minimum_interval schedules.
Test Structure And Quality ✅ Passed PASS: PR only changes ci-operator periodic YAML schedules; no Ginkgo test code or It/BeforeEach/Eventually patterns were modified.
Microshift Test Compatibility ✅ Passed PR only updates CI schedule YAML; no new Ginkgo tests or API-using test code were added, so MicroShift compatibility is unaffected.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: This PR only updates periodic cron/minimum_interval YAML; it adds no new Ginkgo tests or test logic, so there’s no SNO compatibility risk to review.
Topology-Aware Scheduling Compatibility ✅ Passed Only ci-operator periodic cron/minimum_interval updates were changed; no deployment manifests, controllers, or topology-sensitive scheduling logic were added.
Ote Binary Stdout Contract ✅ Passed PASS: The PR only updates ci-operator YAML schedules/minimum intervals; it does not touch OTE binary code or any main/init/TestMain stdout paths.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Only periodic YAML schedules changed; no new Ginkgo test code, IPv4 literals, or external connectivity requirements were added.
No-Weak-Crypto ✅ Passed Touched files are schedule-only YAML; searches found no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed Touched YAMLs are periodic job configs, and scans found 0 hits for privileged/hostPID/hostNetwork/hostIPC/allowPrivilegeEscalation/securityContext.
No-Sensitive-Data-In-Logs ✅ Passed Diff only changes cron/minimum_interval in periodic YAMLs; no new logging or secret/PII-bearing data was added.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…the minimum interval to 3 months (2190 days)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@amogh-redhat
amogh-redhat force-pushed the CNTRLPLANE-3946-4.19 branch from 6ece02c to 9da35ca Compare July 29, 2026 16:45
@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@amogh-redhat: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@csrwng

csrwng commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 3, 2026
@openshift-ci

openshift-ci Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amogh-redhat, csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 3, 2026
@amogh-redhat

Copy link
Copy Markdown
Contributor Author

/pj-rehearse skip

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@amogh-redhat: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Aug 4, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit 100f3d7 into openshift:main Aug 4, 2026
16 checks passed
amogh-redhat added a commit to amogh-redhat/release that referenced this pull request Aug 5, 2026
…the minimum interval to 3 months (2190 days) (openshift#82614)

Co-authored-by: openshift-prow[bot] <openshift-prow[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants