CNTRLPLANE-3846: Multiple CI jobs failing for 4.21. - #82600
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@amogh-redhat: This pull request references CNTRLPLANE-3846 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: amogh-redhat The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughUpdated scheduling cadence for selected Hypershift periodic E2E jobs, including monthly cron expressions and longer minimum intervals for IBM Z and power-related conformance jobs. ChangesHypershift periodic cadence
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics-mce.yaml (1)
153-153: 🚀 Performance & Scalability | 🔵 TrivialConfirm capacity for the synchronized annual window.
These jobs now all run at
04:00 UTCon December 1, alongside the corresponding 4.22 and standard jobs. Consider staggering platforms/releases or confirm that the CI capacity can absorb the burst.Based on the supplied PR objective, the synchronized schedule appears intentional; this is an operational capacity check.
Also applies to: 209-209, 252-252, 267-267, 397-397
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics-mce.yaml` at line 153, Confirm that CI capacity can absorb the synchronized annual burst of jobs scheduled at 04:00 UTC on December 1 across the affected platform and release configurations, preserving the intentional schedule unless capacity validation identifies a required staggering change.ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml (1)
161-161: 🚀 Performance & Scalability | 🔵 TrivialConfirm capacity for the synchronized annual window.
These standard 4.21 jobs share the same
04:00 UTCDecember 1 schedule as the MCE and 4.22 variants. Consider staggering them or validate the capacity impact.Based on the supplied PR objective, the synchronized schedule appears intentional; this is an operational capacity check.
Also applies to: 225-230, 240-240
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml` at line 161, Validate that the shared 04:00 UTC December 1 schedule for the standard 4.21 jobs and the referenced MCE and 4.22 variants fits available operational capacity; if not, stagger the cron schedules while preserving the intended annual execution window.ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics-mce.yaml (1)
169-169: 🚀 Performance & Scalability | 🔵 TrivialConfirm capacity for the synchronized annual window.
These jobs share the same
04:00 UTCDecember 1 schedule as the other changed release and platform variants. Consider staggering them or explicitly validate the expected capacity burst.Based on the supplied PR objective, the synchronized schedule appears intentional; this is an operational capacity check.
Also applies to: 229-229, 248-248, 272-272, 303-303, 318-318, 438-438
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics-mce.yaml` at line 169, Validate that the shared 04:00 UTC December 1 schedules across the affected periodic jobs have sufficient capacity for the synchronized annual burst. If capacity is insufficient, stagger the cron schedules; otherwise preserve the intentional synchronized timing and document or otherwise confirm the capacity validation for these job entries.ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics.yaml (1)
180-180: 🚀 Performance & Scalability | 🔵 TrivialConfirm capacity for the synchronized annual window.
These standard 4.22 jobs share the same
04:00 UTCDecember 1 schedule as the other changed variants. Consider staggering them or validate the capacity impact.Based on the supplied PR objective, the synchronized schedule appears intentional; this is an operational capacity check.
Also applies to: 243-248, 258-258
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics.yaml` at line 180, Validate the capacity impact of the synchronized 04:00 UTC December 1 schedules for the standard 4.22 jobs, including the variants referenced by the review. If capacity is insufficient, stagger the cron schedules; otherwise preserve the intentional synchronized annual window.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics-mce.yaml`:
- Line 153: Confirm that CI capacity can absorb the synchronized annual burst of
jobs scheduled at 04:00 UTC on December 1 across the affected platform and
release configurations, preserving the intentional schedule unless capacity
validation identifies a required staggering change.
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml`:
- Line 161: Validate that the shared 04:00 UTC December 1 schedule for the
standard 4.21 jobs and the referenced MCE and 4.22 variants fits available
operational capacity; if not, stagger the cron schedules while preserving the
intended annual execution window.
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics-mce.yaml`:
- Line 169: Validate that the shared 04:00 UTC December 1 schedules across the
affected periodic jobs have sufficient capacity for the synchronized annual
burst. If capacity is insufficient, stagger the cron schedules; otherwise
preserve the intentional synchronized timing and document or otherwise confirm
the capacity validation for these job entries.
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics.yaml`:
- Line 180: Validate the capacity impact of the synchronized 04:00 UTC December
1 schedules for the standard 4.22 jobs, including the variants referenced by the
review. If capacity is insufficient, stagger the cron schedules; otherwise
preserve the intentional synchronized annual window.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 17687155-4e4e-4eb5-8fb7-ffca21949287
⛔ Files ignored due to path filters (2)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.21-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.22-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (4)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics-mce.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics-mce.yamlci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics.yaml
![openshift-prow[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
…c CI jobs Changed cron schedule to run on 1st Dec and increased minimum_interval to 2190h for persistently failing jobs. Excludes IBM Cloud tests already removed upstream. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
d054a81 to
6c1c581
Compare
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@amogh-redhat: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
What this PR does / why we need it:
Multiple periodic CI jobs in the openshift/hypershift repository have been consistently failing across releases 4.21 with near-zero pass rates, consuming significant CI infrastructure resources with no useful signal.
e2e-aws-ovn-proxy-conformance
e2e-ibmcloud-ovn-iks
e2e-ibmcloud-ovn-roks
e2e-powervs-ovn
e2e-agent-disconnected-ovn-ipv6-metal-conformance
e2e-ibmz-ovn-conformance
e2e-ibmz-s390x-mgmt-ovn-conformance
e2e-kubevirt-metal-ovn
e2e-kubevirt-metal-ovn-multinet
e2e-kubevirt-metal-ovn-multinet-default-net
e2e-power-ovn-conformance
e2e-x86-power-heterogeneous-conformance
I have updated the cron schedule to trigger once a year on 1st Dec every year (cron: 0 4 1 12 *) and for few jobs, changed the minimal interval to 3 months (2190d) so we get enough time to fix the errors and the jobs wont trigger till then.
The job definition itself stays in the configuration file, so the YAML remains syntactically valid.
If these jobs had presubmit/postsubmit triggers defined elsewhere (like running on PRs via /test jobname), those manual/PR triggers would still work.
Which issue(s) this PR fixes: Failing periodic jobs.
Fixes - https://redhat.atlassian.net/browse/CNTRLPLANE-3946
Special notes for your reviewer:
Checklist:
Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.
Summary by CodeRabbit
Updates 12 consistently failing periodic Hypershift release 4.21 CI jobs in
openshift/hypershiftto run annually on December 1 at 04:00. Selected IBM Z and Power conformance jobs now have a 2190-day minimum interval, while existing job definitions and separately configured presubmit, postsubmit, and manual triggers remain available.