PROJQUAY-5850: ci: add shared QE STS test steps

[sudipshil9862]

Summary

Add reusable provisioning, test and cleanup steps for Quay Operator STS testing on the existing QA-created Quay QE cluster, with per-run S3 storage and an IAM role in the Quay DEV AWS account. Supports quay/quay-operator#1324.

This is a draft for design review, not an enabled E2E job. The target is confirmed: the existing shared Quay QE cluster. These steps are intended for a separate e2e-sts job, keeping existing jobs unchanged. Job wiring and the isolated OLM installation/teardown adapter remain to be completed and reviewed with Nayan/Brady. Credential collection names are proposed and are not registered by this PR.

Rationale and changes

• New WebIdentity provisioning exercises the operator/CCO service-account-token flow; the old IAM-user/AssumeRole provisioning does not test that path.
• Unique namespaces, buckets and roles isolate shared-cluster runs. Cluster UID, AWS account and resource ownership checks protect other installations during cleanup.
• The test adapter validates the operator's ROLEARN and namespace scope, rejects overlapping OLM Quay operators, and invokes the existing make test-e2e-sts from the operator PR source. It does not mount DEV AWS provisioning credentials.
• Cleanup requires registry/workload and test-operator teardown first; it removes only owned namespace, role and S3 resources. OIDC registration is shared and retained to avoid deleting a provider another run is using.
• Step refs, OWNERS and generated metadata make these reusable registry steps reviewable. The README documents images, proposed credentials, permissions, execution order, failure recovery and the reason for each scope decision. Offline tests cover identity/ownership rejection, partial failures, provider reuse/races and cleanup.

No temporary-cluster workflow is added. The runnable e2e-sts job is pending the shared-cluster access, helper images and OLM adapter described below. No shared operator or cluster-wide CRD is installed, patched or deleted. The caller's approved OLM adapter must account for CRD compatibility and unmanaged competing operators before enablement.

Review decisions before enablement

1. Review the new, separate e2e-sts job integration using these Chainsaw steps on the confirmed shared QE cluster; existing E2E jobs must remain unchanged.
2. What is the supported isolated operator install/registry teardown/operator uninstall procedure on that shared cluster?
3. Confirm the DEV account, direct GSM collection/group references and field names, permission boundary and QE cluster access. No synchronized GSM bundle is required for the ordinary collection/group approach used here.
4. Agree the shared OIDC provider's retained lifecycle and owner.
5. Supply the documented helper/source images and wire cleanup into the job's failure/post path.

Validation

• 15 offline behavior tests passed (python3 hack/test-quay-sts.py).
• ShellCheck and pylint (10/10), Bash syntax, embedded Python compilation, YAML parsing and YAML indentation checks passed.
• Official registry metadata generation and make ci-operator-checkconfig passed.
• git diff --cached --check passed.
• Live provisioning/OLM/push-pull has not been run for these steps; it requires the integration decisions and credentials above.

Jira

https://issues.redhat.com/browse/PROJQUAY-5850

Backport

Not required; this adds CI step-registry entries.

Summary by CodeRabbit

This PR adds reusable OpenShift CI steps for Quay Operator STS testing on the shared Quay QE cluster. The steps provision run-owned S3 and IAM resources, validate cluster and operator scope, run the existing STS E2E test, and clean up resources with ownership checks.

The PR also adds offline tests for success, failure, collision, and cleanup paths. It does not enable an E2E job. OLM integration, credentials, helper images, and failure-path wiring remain pending. Live provisioning and OLM testing were not performed.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@sudipshil9862: This pull request references [Jira Issue PROJQUAY-5850](https://redhat.atlassian.net/browse/PROJQUAY-5850), which is invalid:

• expected the feature to target either version "5.1.0." or "openshift-5.1.0.", but it targets "quay-v3.19.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is In Progress instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Summary

Add reusable provisioning, test and cleanup steps for Quay Operator STS testing on the existing QA-created Quay QE cluster, with per-run S3 storage and an IAM role in the Quay DEV AWS account. Supports quay/quay-operator#1324.

This is a draft for design review, not an enabled E2E job. The target is confirmed: the existing shared Quay QE cluster. These steps are intended for a separate e2e-sts job, keeping existing jobs unchanged. Job wiring and the isolated OLM installation/teardown adapter remain to be completed and reviewed with Nayan/Brady. Credential collection names are proposed and are not registered by this PR.

Rationale and changes

• New WebIdentity provisioning exercises the operator/CCO service-account-token flow; the old IAM-user/AssumeRole provisioning does not test that path.
• Unique namespaces, buckets and roles isolate shared-cluster runs. Cluster UID, AWS account and resource ownership checks protect other installations during cleanup.
• The test adapter validates the operator's ROLEARN and namespace scope, rejects overlapping OLM Quay operators, and invokes the existing make test-e2e-sts from the operator PR source. It does not mount DEV AWS provisioning credentials.
• Cleanup requires registry/workload and test-operator teardown first; it removes only owned namespace, role and S3 resources. OIDC registration is shared and retained to avoid deleting a provider another run is using.
• Step refs, OWNERS and generated metadata make these reusable registry steps reviewable. The README documents images, proposed credentials, permissions, execution order, failure recovery and the reason for each scope decision. Offline tests cover identity/ownership rejection, partial failures, provider reuse/races and cleanup.

No temporary-cluster workflow is added. The runnable e2e-sts job is pending the shared-cluster access, helper images and OLM adapter described below. No shared operator or cluster-wide CRD is installed, patched or deleted. The caller's approved OLM adapter must account for CRD compatibility and unmanaged competing operators before enablement.

Review decisions before enablement

1. Review the new, separate e2e-sts job integration using these Chainsaw steps on the confirmed shared QE cluster; existing E2E jobs must remain unchanged.
2. What is the supported isolated operator install/registry teardown/operator uninstall procedure on that shared cluster?
3. Confirm the DEV account, direct GSM collection/group references and field names, permission boundary and QE cluster access. No synchronized GSM bundle is required for the ordinary collection/group approach used here.
4. Agree the shared OIDC provider's retained lifecycle and owner.
5. Supply the documented helper/source images and wire cleanup into the job's failure/post path.

Validation

• 15 offline behavior tests passed (python3 hack/test-quay-sts.py).
• ShellCheck and pylint (10/10), Bash syntax, embedded Python compilation, YAML parsing and YAML indentation checks passed.
• Official registry metadata generation and make ci-operator-checkconfig passed.
• git diff --cached --check passed.
• Live provisioning/OLM/push-pull has not been run for these steps; it requires the integration decisions and credentials above.

Jira

https://issues.redhat.com/browse/PROJQUAY-5850

Backport

Not required; this adds CI step-registry entries.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sudipshil9862
Once this PR has been reviewed and has the lgtm label, please assign droslean for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: ec4575f1-da09-446a-96da-c4e5ad9471af

📥 Commits

Reviewing files that changed from the base of the PR and between 1653540 and c7b334b.

📒 Files selected for processing (14)

• ci-operator/step-registry/quay/sts/README.md
• ci-operator/step-registry/quay/sts/cleanup/OWNERS
• ci-operator/step-registry/quay/sts/cleanup/quay-sts-cleanup-commands.sh
• ci-operator/step-registry/quay/sts/cleanup/quay-sts-cleanup-ref.metadata.json
• ci-operator/step-registry/quay/sts/cleanup/quay-sts-cleanup-ref.yaml
• ci-operator/step-registry/quay/sts/provision/OWNERS
• ci-operator/step-registry/quay/sts/provision/quay-sts-provision-commands.sh
• ci-operator/step-registry/quay/sts/provision/quay-sts-provision-ref.metadata.json
• ci-operator/step-registry/quay/sts/provision/quay-sts-provision-ref.yaml
• ci-operator/step-registry/quay/sts/test/OWNERS
• ci-operator/step-registry/quay/sts/test/quay-sts-test-commands.sh
• ci-operator/step-registry/quay/sts/test/quay-sts-test-ref.metadata.json
• ci-operator/step-registry/quay/sts/test/quay-sts-test-ref.yaml
• hack/test-quay-sts.py

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

Walkthrough

The PR adds Quay STS provision, test, and cleanup steps. The steps validate identities and ownership, create isolated AWS and OpenShift resources, run the Chainsaw STS test, remove owned resources, and include offline coverage.

Changes

Quay STS workflow

Layer / File(s)	Summary
Step contract and registry wiring ci-operator/step-registry/quay/sts/README.md, ci-operator/step-registry/quay/sts/*/quay-sts-*-ref.yaml, ci-operator/step-registry/quay/sts/*/quay-sts-*-ref.metadata.json, ci-operator/step-registry/quay/sts/*/OWNERS	Documents and registers the provision, test, and cleanup steps, including credential mounts, environment variables, tool requirements, ownership, and execution order.
Provision isolated STS resources ci-operator/step-registry/quay/sts/provision/quay-sts-provision-commands.sh	Validates the cluster and AWS account, creates run-owned namespace, bucket, OIDC-linked IAM role, and policy, then persists state and outputs.
Validate and run the STS test ci-operator/step-registry/quay/sts/test/quay-sts-test-commands.sh	Validates state, namespace ownership, operator role and watch scope, required tools, and overlapping operators before running make test-e2e-sts.
Safely clean up run-owned resources ci-operator/step-registry/quay/sts/cleanup/quay-sts-cleanup-commands.sh	Checks identities and ownership before deleting the namespace, IAM role, and versioned S3 contents. The shared OIDC provider remains.
Offline workflow validation hack/test-quay-sts.py	Runs the embedded scripts with fake AWS and OpenShift clients and checks success, failure, collision, ownership, operator, and cleanup paths.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant CI as CI job
  participant Provision as quay-sts-provision
  participant Test as quay-sts-test
  participant Cleanup as quay-sts-cleanup
  participant AWS as DEV AWS account
  participant QE as QE OpenShift cluster
  CI->>Provision: Start provisioning
  Provision->>AWS: Validate account and create run-owned resources
  Provision->>QE: Create namespace and save state
  CI->>Test: Start STS test
  Test->>QE: Validate operator and namespace scope
  Test->>AWS: Use provisioned role for STS test
  CI->>Cleanup: Run post-test cleanup
  Cleanup->>QE: Validate ownership and delete namespace
  Cleanup->>AWS: Delete owned role and bucket contents

Loading

Merge Risk: ⚪ Minimal · up to c7b33

No concrete code risk blocks this design-only change; the runnable job and live enablement remain explicitly out of scope.

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 2 warnings)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	The new provision and cleanup commands write provider_arn to CI stdout. That value is constructed from the cluster's authentication/cluster.spec.serviceAccountIssuer (`issuer.removeprefix("https:/…	Remove the two provider_arn print statements, or replace them with a fixed message that contains no issuer, ARN, account, or hostname. Keep error reporting sanitized so raw client or command output cannot emit credential material or inter…
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 4 files. (10 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.
Ipv6 And Disconnected Network Test Compatibility	⚠️ Warning	The new STS test step requires external network access. Its script runs go install github.com/google/go-containerregistry/cmd/crane@v0.20.3 and go install github.com/itchyny/gojq/cmd/gojq@v0.12.17…	IPv6 and disconnected network compatibility notice: This test may contain IPv4 assumptions or external connectivity requirements that will fail in IPv6-only disconnected environments. Please verify your test works on IPv6 by running an …

✅ Passed checks (12 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding shared QE STS test steps for CI. It is concise and specific.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	No changed file defines a Ginkgo test declaration or test title. The added hack/test-quay-sts.py uses Python unittest, and the added quay-sts-test-commands.sh only invokes the existing external …
Test Structure And Quality	✅ Passed	PASS: The pull request adds no Ginkgo test code. The changed test-related files are a shell step that invokes the existing make test-e2e-sts target and a Python unittest offline suite. The changed…
Microshift Test Compatibility	✅ Passed	The pull request adds CI shell/Python wrappers and a Python unittest suite. It adds no Go or Ginkgo test code and contains no new It, Describe, Context, or When constructs. The test step only …
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo e2e tests are added in the reviewed range. The diff contains shell scripts, Python offline tests, YAML, JSON, OWNERS, and documentation, with no changed or added Go files or It/`Descri…
Topology-Aware Scheduling Compatibility	✅ Passed	PASS: The PR adds ci-operator step references, shell wrappers, documentation, and offline tests. The added YAML files use the ci-operator ref: schema and contain no Kubernetes workload manifests or …
Ote Binary Stdout Contract	✅ Passed	No changed OTE binary or OTE suite code exists in the pull request. The only added stdout writes are top-level Python status prints in CI shell wrappers. The test step invokes the operator's Chainsaw …
No-Weak-Crypto	✅ Passed	No explicit weak-crypto condition is introduced. The authoritative PR diff contains no MD5, SHA-1, DES, 3DES, RC4, Blowfish, or ECB usage. The only encryption algorithm is AWS S3 server-side AES256 …
Container-Privileges	✅ Passed	The changed step references do not set privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or any securityContext/root user field. The added scripts use oc,…

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 4 files. (10 skipped: 10 unsupported.)

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

The new STS test step requires external network access. Its script runs go install github.com/google/go-containerregistry/cmd/crane@v0.20.3 and go install github.com/itchyny/gojq/cmd/gojq@v0.12.17 without an internal mirror or proxy. The README also states that the runner needs network access to download tools and reach AWS endpoints. These requirements fail in disconnected CI. No hardcoded IPv4 assumption was found in the changed files.

Resolution

IPv6 and disconnected network compatibility notice: This test may contain IPv4 assumptions or external connectivity requirements that will fail in IPv6-only disconnected environments. Please verify your test works on IPv6 by running an additional CI job: For parallel tests: /payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv6 For serial tests (test name contains [Serial]): /payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-serial-ovn-ipv6 In the openshift/origin repo, use GetIPAddressFamily() to detect the cluster's IP family and adapt accordingly. Or use GetIPFamilyForCluster() or InIPv4ClusterContext() when the test is IPv4-only. For CIDRs, use correctCIDRFamily(). To support disconnected environments, provide crane and gojq in the helper image or obtain them from an approved internal mirror. Also confirm that AWS and registry endpoints are reachable through approved internal or cluster-local paths. If external connectivity is required and cannot be adapted, add [Skipped:Disconnected] to the test name.

Full details: No-Sensitive-Data-In-Logs

Explanation

The new provision and cleanup commands write provider_arn to CI stdout. That value is constructed from the cluster's authentication/cluster.spec.serviceAccountIssuer (issuer.removeprefix("https://")), so it includes the OIDC issuer hostname and AWS account identifier. The issuer can be an internal cluster hostname. The cleanup step repeats the same output from persisted state. set +x does not suppress these explicit print calls. This is pull-request-introduced logging that may expose an internal hostname.

Resolution

Remove the two provider_arn print statements, or replace them with a fixed message that contains no issuer, ARN, account, or hostname. Keep error reporting sanitized so raw client or command output cannot emit credential material or internal endpoints.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci-robot]

@sudipshil9862: This pull request references [Jira Issue PROJQUAY-5850](https://redhat.atlassian.net/browse/PROJQUAY-5850), which is invalid:

• expected the feature to target either version "5.1.0." or "openshift-5.1.0.", but it targets "quay-v3.19.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is In Progress instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Summary

Add reusable provisioning, test and cleanup steps for Quay Operator STS testing on the existing QA-created Quay QE cluster, with per-run S3 storage and an IAM role in the Quay DEV AWS account. Supports quay/quay-operator#1324.

This is a draft for design review, not an enabled E2E job. The target is confirmed: the existing shared Quay QE cluster. These steps are intended for a separate e2e-sts job, keeping existing jobs unchanged. Job wiring and the isolated OLM installation/teardown adapter remain to be completed and reviewed with Nayan/Brady. Credential collection names are proposed and are not registered by this PR.

Rationale and changes

• New WebIdentity provisioning exercises the operator/CCO service-account-token flow; the old IAM-user/AssumeRole provisioning does not test that path.
• Unique namespaces, buckets and roles isolate shared-cluster runs. Cluster UID, AWS account and resource ownership checks protect other installations during cleanup.
• The test adapter validates the operator's ROLEARN and namespace scope, rejects overlapping OLM Quay operators, and invokes the existing make test-e2e-sts from the operator PR source. It does not mount DEV AWS provisioning credentials.
• Cleanup requires registry/workload and test-operator teardown first; it removes only owned namespace, role and S3 resources. OIDC registration is shared and retained to avoid deleting a provider another run is using.
• Step refs, OWNERS and generated metadata make these reusable registry steps reviewable. The README documents images, proposed credentials, permissions, execution order, failure recovery and the reason for each scope decision. Offline tests cover identity/ownership rejection, partial failures, provider reuse/races and cleanup.

No temporary-cluster workflow is added. The runnable e2e-sts job is pending the shared-cluster access, helper images and OLM adapter described below. No shared operator or cluster-wide CRD is installed, patched or deleted. The caller's approved OLM adapter must account for CRD compatibility and unmanaged competing operators before enablement.

Review decisions before enablement

1. Review the new, separate e2e-sts job integration using these Chainsaw steps on the confirmed shared QE cluster; existing E2E jobs must remain unchanged.
2. What is the supported isolated operator install/registry teardown/operator uninstall procedure on that shared cluster?
3. Confirm the DEV account, direct GSM collection/group references and field names, permission boundary and QE cluster access. No synchronized GSM bundle is required for the ordinary collection/group approach used here.
4. Agree the shared OIDC provider's retained lifecycle and owner.
5. Supply the documented helper/source images and wire cleanup into the job's failure/post path.

Validation

• 15 offline behavior tests passed (python3 hack/test-quay-sts.py).
• ShellCheck and pylint (10/10), Bash syntax, embedded Python compilation, YAML parsing and YAML indentation checks passed.
• Official registry metadata generation and make ci-operator-checkconfig passed.
• git diff --cached --check passed.
• Live provisioning/OLM/push-pull has not been run for these steps; it requires the integration decisions and credentials above.

Jira

https://issues.redhat.com/browse/PROJQUAY-5850

Backport

Not required; this adds CI step-registry entries.

Summary by CodeRabbit

This PR adds reusable OpenShift CI steps for Quay Operator STS testing on the shared Quay QE cluster. The steps provision run-owned S3 and IAM resources, validate cluster and operator scope, run the existing STS E2E test, and clean up resources with ownership checks.

The PR also adds offline tests for success, failure, collision, and cleanup paths. It does not enable an E2E job. OLM integration, credentials, helper images, and failure-path wiring remain pending. Live provisioning and OLM testing were not performed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@sudipshil9862: no rehearsable tests are affected by this change

Note: If this PR includes changes to step registry files (ci-operator/step-registry/) and you expected jobs to be found, try rebasing your PR onto the base branch. This helps pj-rehearse accurately detect changes when the base branch has moved forward.