Skip to content

Allow openshift/conformance/serial tests 50 min to complete - #31627

Open
jacobsee wants to merge 1 commit into
openshift:mainfrom
jacobsee:bump-openshift/conformance/serial-to-50-min-timeout
Open

Allow openshift/conformance/serial tests 50 min to complete#31627
jacobsee wants to merge 1 commit into
openshift:mainfrom
jacobsee:bump-openshift/conformance/serial-to-50-min-timeout

Conversation

@jacobsee

@jacobsee jacobsee commented Sep 9, 2026

Copy link
Copy Markdown
Member

The bump from 40 to 50 min is because we have observed significant slowdowns in AWS loadbalancer/healthcheck behavior, causing kube-apiserver to take longer to wait for network loadbalancers to drain so that it can safely shut down. In observation, this means that a KAS rollout now takes ~20 minutes. A test that does a rollout and then reverts its change needs ~40 minutes just for rollouts.

Summary by CodeRabbit

  • Tests
    • Increased the timeout for the OpenShift conformance test suite from 40 to 50 minutes.

The bump from 40 to 50 min is because we have observed significant slowdowns in AWS loadbalancer/healthcheck behavior, causing kube-apiserver to take longer to wait for network loadbalancers to drain so that it can safely shut down. In observation, this means that a KAS rollout now takes ~20 minutes. A test that does a rollout and then reverts its change needs ~40 minutes just for rollouts.
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci

openshift-ci Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jacobsee
Once this PR has been reviewed and has the lgtm label, please assign xueqzhan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Sep 9, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 31c70221-526e-4159-8b15-56ee23ef5928

📥 Commits

Reviewing files that changed from the base of the PR and between 5540cb9 and cfb3e10.

📒 Files selected for processing (1)
  • pkg/testsuites/standard_suites.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The openshift/conformance/serial test suite timeout increases from 40 to 50 minutes.

Changes

Conformance timeout

Layer / File(s) Summary
Serial suite timeout
pkg/testsuites/standard_suites.go
The openshift/conformance/serial suite timeout increases from 40 to 50 minutes.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to cfb3e

The serial conformance suite receives an additional ten minutes to accommodate slower rollout scenarios, reducing avoidable timeout failures without changing test behavior. No current merge-readiness risk remains.

Suggested reviewers: dgoodwin

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly states the main change: the openshift/conformance/serial test timeout increases to 50 minutes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS. The pull request changes only TestTimeout for openshift/conformance/serial from 40 to 50 minutes. The exact diff adds no Ginkgo test title or dynamic value. The affected file contains no `It…
Test Structure And Quality ✅ Passed PASS. The pull request changes only pkg/testsuites/standard_suites.go, increasing the openshift/conformance/serial suite-level TestTimeout from 40 to 50 minutes. The diff adds no Ginkgo It blo…
Microshift Test Compatibility ✅ Passed The pull request changes only TestTimeout for the existing openshift/conformance/serial suite from 40 to 50 minutes. It adds no Ginkgo tests and references no MicroShift-incompatible APIs, namespa…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The pull request changes only the existing openshift/conformance/serial suite timeout from 40 to 50 minutes in pkg/testsuites/standard_suites.go. The diff adds no It(), Describe(), `Cont…
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The pull request changes only pkg/testsuites/standard_suites.go, changing openshift/conformance/serial TestTimeout from 40 to 50 minutes. The diff adds or modifies no deployment manifest, …
Ote Binary Stdout Contract ✅ Passed The pull request changes only TestTimeout for openshift/conformance/serial from 40 * time.Minute to 50 * time.Minute in pkg/testsuites/standard_suites.go. It adds no stdout write, logging co…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS. The pull request changes only TestTimeout for openshift/conformance/serial from 40 to 50 minutes. The exact diff adds no Ginkgo test declarations, IP handling, URLs, or external connectivity…
No-Weak-Crypto ✅ Passed PASS. The pull request changes only pkg/testsuites/standard_suites.go, increasing openshift/conformance/serial TestTimeout from 40 to 50 minutes. The diff adds no cryptographic algorithm, custom…
Container-Privileges ✅ Passed The pull request changes only pkg/testsuites/standard_suites.go. It changes openshift/conformance/serial TestTimeout from 40 to 50 minutes. The patch adds no container or Kubernetes manifest fie…
No-Sensitive-Data-In-Logs ✅ Passed PASS: The pull request changes only TestTimeout for openshift/conformance/serial from 40 to 50 minutes. The commit adds no logging statements, sensitive-data output, or handling of passwords, toke…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Sep 9, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-ovn
/test e2e-metal-ipi-ovn-ipv6

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

/override-sticky ci/prow/e2e-aws-ovn-serial-2of2

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit e2e/integration job. The generated job uses the openshift-org-aws cluster profile and the AWS OVN serial workflow; the run lasted 4h31m22s.

Revision check: Run cfb3e1040dce189e3e9e1d7d930641e99d946b1d; current PR HEAD cfb3e1040dce189e3e9e1d7d930641e99d946b1d; match.

Execution status: Tests executed. The log shows openshift-tests run openshift/conformance/serial on shard 2. The e2e suite completed without test failures; the blocking result was a monitor invariant failure: etcdMemberCommunicationSlow.

Completed supporting jobs: ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-microshift, ci/prow/e2e-aws-ovn-microshift-serial, ci/prow/e2e-gcp-ovn, and ci/prow/e2e-metal-ipi-ovn-ipv6 passed on this SHA. Overall live PR checks are 15 passed, 1 failed, and 1 pending (tide). Pending checks are not counted as positive signal.

Overlap assessment: The PR changes one line in pkg/testsuites/standard_suites.go, increasing only the openshift/conformance/serial TestTimeout from 40 to 50 minutes. It does not change etcd, monitor invariants, cluster configuration, networking, dependencies, or test logic. The failed monitor invariant has no plausible direct or indirect overlap with this timeout-only change.

Missing-coverage risk: Low. The changed conformance serial surface executed and passed, the monitor suite executed and isolated the failure to cluster health, and multiple other e2e presubmits passed. This override removes an infrastructure-derived blocking result; it does not substitute for unexecuted coverage.

Rationale: The failed invariant observed sustained etcd communication degradation, including the alert firing across etcd members and corroborating API/etcd instability. Historical CI analysis shows this job passed 18 of its last 20 runs, with another occurrence on PR #31593. No overlapping SHIP outage record was found for the run window.

If you disagree with this assessment, rerun the current job with /test e2e-aws-ovn-serial-2of2.


AI-generated. Review for accuracy.

@openshift-ci

openshift-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-aws-ovn-serial-2of2

These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use /override-cancel to remove them.

Details

In response to this:

/override-sticky ci/prow/e2e-aws-ovn-serial-2of2

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit e2e/integration job. The generated job uses the openshift-org-aws cluster profile and the AWS OVN serial workflow; the run lasted 4h31m22s.

Revision check: Run cfb3e1040dce189e3e9e1d7d930641e99d946b1d; current PR HEAD cfb3e1040dce189e3e9e1d7d930641e99d946b1d; match.

Execution status: Tests executed. The log shows openshift-tests run openshift/conformance/serial on shard 2. The e2e suite completed without test failures; the blocking result was a monitor invariant failure: etcdMemberCommunicationSlow.

Completed supporting jobs: ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-microshift, ci/prow/e2e-aws-ovn-microshift-serial, ci/prow/e2e-gcp-ovn, and ci/prow/e2e-metal-ipi-ovn-ipv6 passed on this SHA. Overall live PR checks are 15 passed, 1 failed, and 1 pending (tide). Pending checks are not counted as positive signal.

Overlap assessment: The PR changes one line in pkg/testsuites/standard_suites.go, increasing only the openshift/conformance/serial TestTimeout from 40 to 50 minutes. It does not change etcd, monitor invariants, cluster configuration, networking, dependencies, or test logic. The failed monitor invariant has no plausible direct or indirect overlap with this timeout-only change.

Missing-coverage risk: Low. The changed conformance serial surface executed and passed, the monitor suite executed and isolated the failure to cluster health, and multiple other e2e presubmits passed. This override removes an infrastructure-derived blocking result; it does not substitute for unexecuted coverage.

Rationale: The failed invariant observed sustained etcd communication degradation, including the alert firing across etcd members and corroborating API/etcd instability. Historical CI analysis shows this job passed 18 of its last 20 runs, with another occurrence on PR #31593. No overlapping SHIP outage record was found for the run window.

If you disagree with this assessment, rerun the current job with /test e2e-aws-ovn-serial-2of2.


AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@jacobsee: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants