UPSTREAM: <carry>: e2e test isolation - dynamic catalogs and godog framework #700
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,118 @@ | ||||||
| # E2E Test Isolation: Per-Scenario Catalogs via Dynamic OCI Image Building | ||||||
|
|
||||||
| ## Problem | ||||||
|
|
||||||
| E2E test scenarios previously shared cluster-scoped resources (ClusterCatalogs, CRDs, packages), | ||||||
| causing cascading failures when one scenario left state behind. Parallelism was impossible because | ||||||
| scenarios conflicted on shared resource names. | ||||||
|
|
||||||
| ## Solution | ||||||
|
|
||||||
| Each scenario dynamically builds and pushes its own bundle and catalog OCI images at test time, | ||||||
| parameterized by scenario ID. All cluster-scoped resource names include the scenario ID, making | ||||||
| conflicts structurally impossible. | ||||||
|
|
||||||
| ``` | ||||||
| Scenario starts | ||||||
| -> Generate parameterized bundle manifests (CRD names, deployments, etc. include scenario ID) | ||||||
| -> Build + push bundle OCI images to e2e registry via go-containerregistry | ||||||
| -> Generate FBC catalog config referencing those bundle image refs | ||||||
| -> Build + push catalog OCI image to e2e registry | ||||||
| -> Create ClusterCatalog pointing at the catalog image | ||||||
| -> Run scenario steps | ||||||
| -> Cleanup all resources (including catalog) | ||||||
| ``` | ||||||
|
|
||||||
| ### Key Properties | ||||||
|
|
||||||
| - Every cluster-scoped resource name includes the scenario ID -- no conflicts by construction. | ||||||
| - Failed scenario state is preserved for debugging without affecting other scenarios. | ||||||
| - Parallelism (`Concurrency > 1`) is safe without further changes. | ||||||
| - Adding new scenarios requires zero coordination with existing ones. | ||||||
|
|
||||||
| ## Builder API (`test/e2e/catalog/`) | ||||||
|
There was a problem hiding this comment. Point the Builder API section at the current package path. The implementation in this PR lives under 📝 Minimal fix-## Builder API (`test/e2e/catalog/`)
+## Builder API (`test/internal/catalog/`)📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||
|
|
||||||
| Bundles are defined as components of a catalog. A single `Build()` call builds and pushes | ||||||
| all bundle images, generates the FBC, and pushes the catalog image: | ||||||
|
|
||||||
| ```go | ||||||
| cat := catalog.NewCatalog("test", scenarioID, | ||||||
| catalog.WithPackage("test", | ||||||
| catalog.Bundle("1.0.0", catalog.WithCRD(), catalog.WithDeployment(), catalog.WithConfigMap()), | ||||||
| catalog.Bundle("1.2.0", catalog.WithCRD(), catalog.WithDeployment()), | ||||||
| catalog.Channel("beta", catalog.Entry("1.0.0"), catalog.Entry("1.2.0")), | ||||||
| ), | ||||||
| ) | ||||||
| result, err := cat.Build(ctx, "v1", localRegistry, clusterRegistry) | ||||||
| // result.CatalogName = "test-catalog-{scenarioID}" | ||||||
| // result.CatalogImageRef = "{clusterRegistry}/e2e/test-catalog-{scenarioID}:v1" | ||||||
| // result.PackageNames = {"test": "test-{scenarioID}"} | ||||||
| ``` | ||||||
|
|
||||||
| ### Bundle Options | ||||||
|
|
||||||
| - `WithCRD()` -- CRD with group `e2e-{id}.e2e.operatorframework.io` | ||||||
| - `WithDeployment()` -- Deployment named `test-operator-{id}` (includes CSV, script ConfigMap, NetworkPolicy) | ||||||
| - `WithConfigMap()` -- additional test ConfigMap | ||||||
| - `WithInstallMode(modes...)` -- sets supported install modes on the CSV | ||||||
| - `WithLargeCRD(fieldCount)` -- CRD with many fields for large bundle testing | ||||||
| - `WithClusterRegistry(host)` -- overrides the cluster-side registry host (for mirror testing) | ||||||
| - `StaticBundleDir(dir)` -- reads pre-built bundle manifests without parameterization (e.g. webhook-operator) | ||||||
| - `BadImage()` -- uses an invalid container image to trigger ImagePullBackOff | ||||||
| - `WithBundleProperty(type, value)` -- adds a property to bundle metadata | ||||||
|
|
||||||
| ## Feature File Conventions | ||||||
|
|
||||||
| Feature files define catalogs inline via data tables: | ||||||
|
|
||||||
| ```gherkin | ||||||
| Background: | ||||||
| Given OLM is available | ||||||
| And an image registry is available | ||||||
| And a catalog "test" with packages: | ||||||
| | package | version | channel | replaces | contents | | ||||||
| | test | 1.0.0 | alpha | | CRD, Deployment, ConfigMap | | ||||||
| | test | 1.0.1 | alpha | 1.0.0 | CRD, Deployment, ConfigMap | | ||||||
| | test | 1.2.0 | beta | | CRD, Deployment | | ||||||
| ``` | ||||||
|
|
||||||
| ### Variable Substitution | ||||||
|
|
||||||
| Templates in feature file YAML use these variables: | ||||||
|
|
||||||
| | Variable | Expansion | Example | | ||||||
| |----------|-----------|---------| | ||||||
| | `${NAME}` | ClusterExtension name | `ce-abc123` | | ||||||
| | `${TEST_NAMESPACE}` | Scenario namespace | `ns-abc123` | | ||||||
| | `${SCENARIO_ID}` | Unique scenario identifier | `abc123` | | ||||||
| | `${PACKAGE:<name>}` | Parameterized package name | `test-abc123` | | ||||||
| | `${CATALOG:<name>}` | ClusterCatalog resource name | `test-catalog-abc123` | | ||||||
| | `${COS_NAME}` | ClusterObjectSet name | `cos-abc123` | | ||||||
|
|
||||||
| ### Naming Conventions | ||||||
|
|
||||||
| | Resource | Pattern | | ||||||
| |----------|---------| | ||||||
| | CRD group | `e2e-{id}.e2e.operatorframework.io` | | ||||||
| | Deployment | `test-operator-{id}` | | ||||||
| | Package name (FBC) | `{package}-{id}` | | ||||||
| | Bundle image | `{registry}/bundles/{package}-{id}:v{version}` | | ||||||
| | Catalog image | `{registry}/e2e/{name}-catalog-{id}:{tag}` | | ||||||
| | ClusterCatalog | `{name}-catalog-{id}` | | ||||||
| | Namespace | `ns-{id}` | | ||||||
| | ClusterExtension | `ce-{id}` | | ||||||
|
|
||||||
| ## Registry Access | ||||||
|
|
||||||
| An in-cluster OCI registry (`test/internal/registry/`) stores bundle and catalog images. | ||||||
| The registry runs as a ClusterIP Service; there is no NodePort or kind `extraPortMappings`. | ||||||
|
|
||||||
| The test runner reaches the registry via **Kubernetes port-forward** (SPDY through the API | ||||||
| server), which works regardless of the cluster's network topology. A `sync.OnceValues` in the | ||||||
| step definitions starts the port-forward once and returns the dynamically assigned | ||||||
| `localhost:<port>` address used for all `crane.Push` / `crane.Tag` calls. | ||||||
|
|
||||||
| In-cluster components (e.g. the catalog unpacker) pull images using the Service DNS name | ||||||
| (`docker-registry.operator-controller-e2e.svc.cluster.local:5000`), resolved by CoreDNS. | ||||||
| Containerd on the node is never involved because the registry only holds OCI artifacts | ||||||
| consumed by Go code, not container images for pods. | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -90,6 +90,7 @@ require ( | |
| github.com/containerd/typeurl/v2 v2.2.3 // indirect | ||
| github.com/containers/libtrust v0.0.0-20230121012942-c1716e8a8d01 // indirect | ||
| github.com/containers/ocicrypt v1.2.1 // indirect | ||
| github.com/creack/pty v1.1.24 // indirect | ||
| github.com/cucumber/gherkin/go/v26 v26.2.0 // indirect | ||
| github.com/cucumber/messages/go/v21 v21.0.1 // indirect | ||
| github.com/cyberphone/json-canonicalization v0.0.0-20241213102144-19d51d7fe467 // indirect | ||
|
|
@@ -141,6 +142,7 @@ require ( | |
| github.com/google/gnostic-models v0.7.1 // indirect | ||
| github.com/google/uuid v1.6.0 // indirect | ||
| github.com/gorilla/mux v1.8.1 // indirect | ||
| github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect | ||
| github.com/gosuri/uitable v0.0.4 // indirect | ||
| github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect | ||
| github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.7 // indirect | ||
|
|
@@ -172,6 +174,7 @@ require ( | |
| github.com/mitchellh/go-wordwrap v1.0.1 // indirect | ||
| github.com/mitchellh/reflectwalk v1.0.2 // indirect | ||
| github.com/moby/locker v1.0.1 // indirect | ||
| github.com/moby/spdystream v0.5.0 // indirect | ||
|
There was a problem hiding this comment. 🧩 Analysis chain🌐 Web query:
💡 Result: Versions of github.com/moby/spdystream prior to v0.5.1 are affected by GHSA-pc3f-x583-g7j2 (CVE-2026-35469). There is a patched release at v0.5.1. Kubernetes client-go/portforward in the 0.35.x line uses v0.5.0, which is vulnerable. Citations:
🌐 Web query:
💡 Result: No, Kubernetes client-go v0.35 releases (v0.35.0 to v0.35.3) do not patch the spdystream v0.5.1 port-forward vulnerability GHSA-pc3f-x583-g7j2. The vulnerability GHSA-pc3f-x583-g7j2 (CVE-2026-35469) affects moby/spdystream before v0.5.1 and was fixed in spdystream v0.5.1. It relates to port-forwarding via SPDY streams used in Kubernetes components like kubectl port-forward. client-go v0.35.x releases (v0.35.0 on ~2026-02, v0.35.1 on 2026-02-10, v0.35.2 on 2026-02-26, v0.35.3 on 2026-03-19) update api/apimachinery dependencies but retain github.com/moby/spdystream v0.5.0 as an indirect dependency (confirmed in master go.mod post-v0.35). The spdystream update to v0.5.1 was backported to Kubernetes core via PRs Citations:
🏁 Script executed: # Verify the dependency chain in the current file
cat -n go.mod | grep -A5 -B5 "client-go\|spdystream"Repository: openshift/operator-framework-operator-controller Length of output: 1520 The spdystream v0.5.0 dependency is vulnerable; upgrade client-go or pin spdystream to v0.5.1. Line 177 adds 🧰 Tools🪛 OSV Scanner (2.3.5)[HIGH] 177-177: github.com/moby/spdystream 0.5.0: SpdyStream: DOS on CRI 🤖 Prompt for AI Agents |
||
| github.com/moby/sys/capability v0.4.0 // indirect | ||
| github.com/moby/sys/mountinfo v0.7.2 // indirect | ||
| github.com/moby/sys/sequential v0.6.0 // indirect | ||
|
|
@@ -182,6 +185,7 @@ require ( | |
| github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect | ||
| github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect | ||
| github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect | ||
| github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect | ||
| github.com/nxadm/tail v1.4.11 // indirect | ||
| github.com/onsi/gomega v1.39.1 // indirect | ||
| github.com/opencontainers/runtime-spec v1.3.0 // indirect | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a language tag to this fenced block.
markdownlintwill keep flagging this section with MD040 until the fence is annotated, e.g. astext.📝 Minimal fix
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)
[warning] 15-15: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents