Skip to content

CNTRLPLANE-3846: Multiple CI jobs failing for 4.20. - #82612

Open
amogh-redhat wants to merge 1 commit into
openshift:mainfrom
amogh-redhat:CNTRLPLANE-3946-4.20
Open

CNTRLPLANE-3846: Multiple CI jobs failing for 4.20.#82612
amogh-redhat wants to merge 1 commit into
openshift:mainfrom
amogh-redhat:CNTRLPLANE-3946-4.20

Conversation

@amogh-redhat

@amogh-redhat amogh-redhat commented Jul 29, 2026

Copy link
Copy Markdown

What this PR does / why we need it:
Multiple periodic CI jobs in the openshift/hypershift repository have been consistently failing across releases 4.20 with near-zero pass rates, consuming significant CI infrastructure resources with no useful signal.

e2e-aws-ovn-proxy-conformance
e2e-ibmcloud-ovn-iks
e2e-ibmcloud-ovn-roks
e2e-powervs-ovn
e2e-agent-disconnected-ovn-ipv6-metal-conformance
e2e-ibmz-ovn-conformance
e2e-ibmz-s390x-mgmt-ovn-conformance
e2e-kubevirt-metal-ovn
e2e-kubevirt-metal-ovn-multinet
e2e-kubevirt-metal-ovn-multinet-default-net
e2e-power-ovn-conformance
e2e-x86-power-heterogeneous-conformance

I have updated the cron schedule to trigger once a year on 1st Dec every year (cron: 0 4 1 12 *) and for few jobs, changed the minimal interval to 3 months (2190d) so we get enough time to fix the errors and the jobs wont trigger till then.
The job definition itself stays in the configuration file, so the YAML remains syntactically valid.
If these jobs had presubmit/postsubmit triggers defined elsewhere (like running on PRs via /test jobname), those manual/PR triggers would still work.

Which issue(s) this PR fixes: Failing periodic jobs.

Fixes - https://redhat.atlassian.net/browse/CNTRLPLANE-3946

Special notes for your reviewer:

Checklist:
Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

Summary by CodeRabbit

  • Updated 12 failing periodic CI jobs in openshift/hypershift release 4.20 to run annually on December 1 at 04:00 UTC.
  • Increased minimum intervals for several longer-running conformance jobs.
  • Preserved job definitions and existing presubmit, postsubmit, and manual triggers.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 29, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@amogh-redhat: This pull request references CNTRLPLANE-3846 which is a valid jira issue.

Details

In response to this:

What this PR does / why we need it:
Multiple periodic CI jobs in the openshift/hypershift repository have been consistently failing across releases 4.20 with near-zero pass rates, consuming significant CI infrastructure resources with no useful signal.

e2e-aws-ovn-proxy-conformance
e2e-ibmcloud-ovn-iks
e2e-ibmcloud-ovn-roks
e2e-powervs-ovn
e2e-agent-disconnected-ovn-ipv6-metal-conformance
e2e-ibmz-ovn-conformance
e2e-ibmz-s390x-mgmt-ovn-conformance
e2e-kubevirt-metal-ovn
e2e-kubevirt-metal-ovn-multinet
e2e-kubevirt-metal-ovn-multinet-default-net
e2e-power-ovn-conformance
e2e-x86-power-heterogeneous-conformance

I have updated the cron schedule to trigger once a year on 1st Dec every year (cron: 0 4 1 12 *) and for few jobs, changed the minimal interval to 3 months (2190d) so we get enough time to fix the errors and the jobs wont trigger till then.
The job definition itself stays in the configuration file, so the YAML remains syntactically valid.
If these jobs had presubmit/postsubmit triggers defined elsewhere (like running on PRs via /test jobname), those manual/PR triggers would still work.

Which issue(s) this PR fixes: Failing periodic jobs.

Fixes - https://redhat.atlassian.net/browse/CNTRLPLANE-3946

Special notes for your reviewer:

Checklist:
Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
openshift-ci Bot requested review from cblecker and muraee July 29, 2026 13:15
@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: amogh-redhat
Once this PR has been reviewed and has the lgtm label, please assign jparrill for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 680d0795-5d55-4435-9cd8-d066effe3632

📥 Commits

Reviewing files that changed from the base of the PR and between 4883e9e and 95b272e.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.20-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (2)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics-mce.yaml

Walkthrough

Updated Hypershift periodic conformance job timing. Several cron schedules now run at 0 4 1 12 *, while three MCE jobs have minimum_interval increased from 168h to 2190h.

Changes

Hypershift periodic schedule updates

Layer / File(s) Summary
Conformance schedule and interval declarations
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics*.yaml
OVN and kubevirt-related periodic jobs use the annual cron schedule, and IBM Z, Power, and heterogeneous conformance jobs increase minimum_interval to 2190h where specified.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

Suggested reviewers: cblecker, muraee

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is relevant to the PR’s purpose: fixing multiple failing CI jobs in release 4.20.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS: The PR only edits ci-operator periodic cron/minimum_interval YAML; no test code or Ginkgo titles changed.
Test Structure And Quality ✅ Passed PR only changes periodic CI YAML schedules; no Ginkgo test code or test logic was modified, so this check is not applicable.
Microshift Test Compatibility ✅ Passed No Ginkgo tests were added or changed; the PR only updates periodic CI YAML schedules.
Single Node Openshift (Sno) Test Compatibility ✅ Passed Only CI YAML schedule metadata changed; no Ginkgo/e2e source files or new tests were added, so the SNO compatibility check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed Only periodic CI cron/minimum_interval YAML changed; no manifests, controllers, or pod scheduling logic were modified.
Ote Binary Stdout Contract ✅ Passed PASS: The PR only changes CI config and step-registry scripts; no Go/main/TestMain or OTE binary code is touched, so stdout contract isn’t implicated.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests were added; the diff only changes periodic CI cron/minimum_interval YAML entries.
No-Weak-Crypto ✅ Passed Patch only changes CI YAML cron/minimum_interval fields; no crypto primitives, custom crypto, or secret comparisons were added.
Container-Privileges ✅ Passed Only cron/minimum_interval values changed; no privileged, hostPID/Network/IPC, SYS_ADMIN, runAsRoot, or allowPrivilegeEscalation fields were added.
No-Sensitive-Data-In-Logs ✅ Passed The PR only changes job schedules/minimum intervals in YAML; no new log statements or secret/PII-bearing fields were added.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…c CI jobs

Changed cron schedule to run on 1st Dec and increased minimum_interval
to 2190h for persistently failing jobs. Excludes IBM Cloud tests
already removed upstream.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@amogh-redhat, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

couldn't prepare candidate: couldn't checkout base SHA 073940e9279f89195e0ac60567e2109603ab51e0: error checking out "073940e9279f89195e0ac60567e2109603ab51e0": exit status 128 fatal: unable to read tree (073940e9279f89195e0ac60567e2109603ab51e0)

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@amogh-redhat: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants