Fix tracing-ui QE agent skill's FLAKY misdiagnosis of a suppressed crash

ci-operator/step-registry/openshift-observability/qe-agent/resources/skills/tracing-ui/SKILL.md

@@ -29,10 +29,10 @@ Read `${SHARED_DIR}/qe-agent-context.json`, written by the test step at exit:
 }
 ```
 
-- `step_script_ref` — path relative to `ci-operator/step-registry/` in openshift/release
+- `step_script_ref` — path relative to `ci-operator/step-registry/`
 - `env` — job-time env values needed to reproduce setup; `CYPRESS_SKIP_TESTS` is the job's `@cypress/grep` pattern (empty = all tests), applied to reruns in Step 3
 
-Fetch `https://raw.githubusercontent.com/openshift/release/main/ci-operator/step-registry/<step_script_ref>`. Everything before the first `npx cypress run` / `npm run` is **setup** (repo clone, IDP/htpasswd, env vars, `npm install`); the rest is **test execution**.
+Fetch `https://raw.githubusercontent.com/openshift/release/main/ci-operator/step-registry/<step_script_ref>`. Everything before the first `npx cypress run` / `npm run` is **setup** (clone, IDP/htpasswd, env vars, `npm install`); the rest is **test execution**.
 
 ## Step 0a — Verify Cluster Stability
 
@@ -94,12 +94,13 @@ Export the `env` vars, then run the script's setup section (up to the first `npx
 
 | Script pattern | Adaptation |
 |---|---|
-| `cp -R /tmp/<name>` (image mount) | `git clone <repo> <dest>` — repo URL from the step script or CI config |
-| `kubectl create -f <url>` (CRDs) | `kubectl apply -f <url>` — `create` fails if the CRD exists |
+| `cp -R /tmp/<name>` (image mount) | `git clone --branch main --single-branch <repo> <dest>` — URL/branch from the step script |
+| `kubectl create -f <url>` (CRDs) | `kubectl apply -f <url>` — `create` fails if it exists |
 | `oc patch csv ...` | Skip — already patched; verify with `oc get csv -n openshift-cluster-observability-operator` |
 | `CYPRESS_SKIP_TESTS` block | Keep it; reruns use it (Step 3) |
-| htpasswd / oauth setup | Create the secret only if `oc get secret htpass-secret -n openshift-config` fails; patch oauth only if the htpasswd IDP is missing |
-| Operator installs / OperatorGroups | Already installed by the original run — check `oc get csv -A`, do not reinstall. Check `oc get operatorgroup -n <ns>` before creating one; a second OperatorGroup fails the CSV ("csv created in namespace with multiple operatorgroups") |
+| htpasswd / oauth setup | Create `uiauto-htpass-secret` only if `oc get secret uiauto-htpass-secret -n openshift-config` fails; patch oauth only if `uiauto-htpasswd-idp` is missing |
+| Operator installs / OperatorGroups | Verify readiness, not just presence — require CSV phase `Succeeded`; (re)install if missing or not Succeeded (`after()` can delete or fail COO/OTel/Tempo/Lightspeed). Check `oc get operatorgroup -n <ns>` first; a second one fails the CSV |
+| Cypress binary | `npx cypress version` can pass while the binary is only under `/root/.cache/Cypress/`, not `$CYPRESS_CACHE_FOLDER`. If empty: `cp -r /root/.cache/Cypress/*/ /tmp/Cypress/` |
 
 Then continue with Steps 1–6 in the cloned repo. If `qe-agent-context.json` is missing, infer the suite from the JUnit prefix, skip the rerun, and diagnose from the JUnit content and cluster state.
 
@@ -109,7 +110,7 @@ Read `${SHARED_DIR}/qe-agent-junit-*.xml`. For each file extract the suite name
 
 ### High-failure triage: more than 5 failures total
 
-More than 5 failures usually share one root cause (console plugin not loaded, auth failure, UI not responding, network error, or a failed `before` hook). Look for a common pattern: the same error string (`Cannot read properties of null`, `element not found`, `401 Unauthorized`, `plugin not enabled`), the same failing Cypress command (`cy.visit`, `cy.get`, `cy.findByText`), or tightly clustered failure times.
+More than 5 failures usually share one root cause (plugin not loaded, auth failure, UI not responding, network error, or a failed `before` hook). Look for a common pattern: the same error string (`Cannot read properties of null`, `401 Unauthorized`), the same failing Cypress command (`cy.visit`, `cy.get`), or tightly clustered failure times.
 
 - **Clear pattern**: pick the simplest failing test as the representative and run Steps 2–5 for it only.
 - **No clear pattern**: process failures individually, cap at 3 tests, and note this in the summary.
@@ -120,7 +121,7 @@ Write the pattern conclusion near the top of `${ARTIFACT_DIR}/qe-agent-analysis.
 
 ## Step 2 — Locate Test Source Files
 
-The repo root is the clone destination from the step script — do not scan `/tmp/` broadly. All tests are in one spec, `tests/e2e/dt-plugin-tests.cy.ts`: a single `describe` whose `before` hook installs/verifies the operators, sets up Lightspeed and creates the UIPlugin, then one `it` per capability. A `before` hook failure skips every test. Locate the `it` block with `grep -n "<test-name>"`. Supporting files under `tests/`: Cypress config, `cypress/support/` custom commands (e.g. `cy.runChainsawTest`), `views/` page objects, `fixtures/` chainsaw tests.
+The repo root is the clone destination — do not scan `/tmp/` broadly. All tests are in one spec, `tests/e2e/dt-plugin-tests.cy.ts`: a single `describe` whose `before` hook installs/verifies the operators, sets up Lightspeed and creates the UIPlugin, then one `it` per capability. A `before` hook failure skips every test. Locate the `it` block with `grep -n "<test-name>"`. Supporting files under `tests/`: Cypress config, `cypress/support/` custom commands (e.g. `cy.runChainsawTest`), `views/` page objects, `fixtures/` chainsaw tests.
 
 ---
 
@@ -133,7 +134,11 @@ Before rerunning, inspect the RBAC test's `chainsaw-*` namespaces and `verify-tr
 ```bash
 cd "<repo root>/tests"
 export NO_COLOR=1 CYPRESS_CACHE_FOLDER=/tmp/Cypress CYPRESS_SKIP_COO_INSTALL=true
-# Fresh shell: also re-export the CYPRESS_* vars from the step script setup (base URL, login, kubeconfig, Lightspeed)
+# Fresh shell — these don't persist, set them in every rerun command:
+export CYPRESS_KUBECONFIG_PATH="${KUBECONFIG}"
+export CYPRESS_BASE_URL="https://$(oc get route console -n openshift-console -o jsonpath='{.spec.host}')"
+export CYPRESS_LOGIN_IDP="kube:admin"
+export CYPRESS_LOGIN_USERS="kubeadmin:$(cat /tmp/secret/kubeadmin-password)"
 CYPRESS_SKIP_TESTS=$(jq -r '.env.CYPRESS_SKIP_TESTS // ""' "${SHARED_DIR}/qe-agent-context.json" 2>/dev/null)
 GREP="<unique part of the failing test title>"
 [[ -n "${CYPRESS_SKIP_TESTS}" ]] && GREP="${GREP}; ${CYPRESS_SKIP_TESTS}"
@@ -145,10 +150,10 @@ npx cypress run --browser chrome --headless --spec "e2e/dt-plugin-tests.cy.ts" \
 
 ### Selecting what to rerun
 
-- Keep `CYPRESS_SKIP_TESTS` in the grep (`;` separates patterns, `-` excludes): the `before` hook reads it too, e.g. `-Lightspeed` skips the Lightspeed install on OCP versions where Lightspeed is not published.
-- Tests are order-dependent: `Capability:RBAC` creates the Tempo instances (`chainsaw-rbac / simplst`, `chainsaw-mmo-rbac / mmo-rbac`) and traces that every later test uses except `Capability:TLSCertRotation` and `Capability:Installation`. Include it for those tests (`GREP="Capability:RBAC; Capability:TraceLimits"`), otherwise the rerun fails on `input[placeholder="Select a Tempo instance"]`.
-- For a `before` hook failure, set `GREP` to only the `CYPRESS_SKIP_TESTS` value (empty runs every test).
-- `CYPRESS_SKIP_COO_INSTALL=true` skips the OperatorHub install path (`CYPRESS_COO_UI_INSTALL`, the job default). A passing rerun does not verify an install-path fix — mark it "not re-verified" in `CHANGES.md`.
+- Keep `CYPRESS_SKIP_TESTS` in the grep (`;` separates, `-` excludes): the `before` hook reads it too, e.g. `-Lightspeed` skips the Lightspeed install where it's not published.
+- Tests are order-dependent: `Capability:RBAC` creates the Tempo instances (`chainsaw-rbac / simplst`, `chainsaw-mmo-rbac / mmo-rbac`) and traces every later test uses except `Capability:TLSCertRotation`/`Capability:Installation`. Include it for others (`GREP="Capability:RBAC; Capability:TraceLimits"`), otherwise the rerun fails on `input[placeholder="Select a Tempo instance"]`.
+- For a `before` hook failure, set `GREP` to just `CYPRESS_SKIP_TESTS` (empty runs every test).
+- `CYPRESS_SKIP_COO_INSTALL=true` skips the OperatorHub install path (`CYPRESS_COO_UI_INSTALL`, the job default). A passing rerun doesn't verify an install-path fix — mark it "not re-verified" in `CHANGES.md`.
 
 Read the rerun JUnit XML:
 - **Same failure** → Step 4
@@ -157,18 +162,18 @@ Read the rerun JUnit XML:
 
 ### Flakiness confirmation loop
 
-Run the rerun block 3 more times with `RUN=2`, `3` and `4` (one JUnit file each) and record the pass/fail pattern (e.g. `PFPP`). Look for missing `cy.intercept()` or condition-based waits before asserting UI state, `cy.get()` without a visibility wait, or a changed URL path. A failure in even 1 of 4 runs is `FLAKY` → Step 5c.
+Run the rerun block 3 more times with `RUN=2`, `3` and `4` (one JUnit file each) and record the pass/fail pattern (e.g. `PFPP`). Look for missing `cy.intercept()` or condition-based waits before asserting UI state, `cy.get()` without a visibility wait, or a changed URL path. A failure in even 1 of 4 runs still goes through Step 4 first — call it `FLAKY` → Step 5c only if that finds no other explanation. An incomplete loop is tentative, never `FLAKY`.
 
 ---
 
 ## Step 4 — Diagnose: Product Bug vs Test Issue
 
-Run the diagnostics below before classifying — logs and resource status, read with the failure message and test source, are the primary evidence.
+Run the diagnostics below first; logs and resource status are the primary evidence alongside the failure message and test source.
 
 ### Cluster Observability Operator Diagnostics
 
 ```bash
-# Auto-detect COO namespace (depends on install mode)
+# Auto-detect COO namespace
 COO_NS="$(oc get pods --all-namespaces -l app.kubernetes.io/name=observability-operator -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null)"
 COO_NS="${COO_NS:-openshift-cluster-observability-operator}"
 
@@ -177,30 +182,30 @@ oc get pods -n "${COO_NS}"
 oc logs -n "${COO_NS}" deploy/observability-operator --tail=150 2>/dev/null || true
 oc logs -n "${COO_NS}" deploy/observability-operator --previous --tail=50 2>/dev/null || true
 
-# UIPlugins (cluster-scoped; control console plugin registration) and MonitoringStacks
+# UIPlugins and MonitoringStacks
 oc get uiplugins -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.status.conditions[*].type}={.status.conditions[*].status} {.status.conditions[*].message}{"\n"}{end}' 2>/dev/null || true
 oc get monitoringstacks --all-namespaces -o wide 2>/dev/null || true
 
 # Console plugin registration status
 oc get consoleplugin distributed-tracing-console-plugin -o jsonpath='{.status}{"\n"}' 2>/dev/null || true
 oc get consoles.operator.openshift.io cluster -o jsonpath='{.spec.plugins}{"\n"}' 2>/dev/null || true
 
-# Events and CSV status in the COO namespace
+# Events and CSV status
 oc get events -n "${COO_NS}" --sort-by='.lastTimestamp' | tail -20
 oc get csv -n "${COO_NS}" -o jsonpath='{range .items[*]}{.metadata.name}: {.status.phase} — {.status.message}{"\n"}{end}'
 ```
 
 ### CRD and API availability check
 
 ```bash
-# Missing CRDs cause plugin registration failures
+# Missing CRDs cause registration failures
 oc get crd | grep -E 'observability|uiplugin|monitoringstack'
 oc api-resources | grep observability
 ```
 
 ### Operator catalog availability check
 
-The `before` hook installs COO, OpenTelemetry, Tempo and Lightspeed from `redhat-operators`. Pre-GA OCP versions may not ship all of them yet; OperatorHub then never renders the install form (`[data-test="install-operator"]` times out). Check before blaming the test or product:
+The `before` hook installs COO, OpenTelemetry, Tempo and Lightspeed from `redhat-operators`. Pre-GA OCP versions may lack one, so OperatorHub never renders the install form (`[data-test="install-operator"]` times out). Check before blaming the test or product:
 
 ```bash
 oc get clusterversion version -o jsonpath='{.status.desired.version}{"\n"}'
@@ -211,6 +216,10 @@ done
 
 A package missing from the catalog on a pre-GA OCP version is `JOB_CONFIG` (Step 5e): do not add catalog auto-detection or skip logic to the test (it would silently skip the capability where the operator must exist), and do not file a product bug.
 
+### Suppressed exception check
+
+`e2e.js`'s `uncaught:exception` handler swallows crashes like `'Cannot read prop'` — they surface as a timeout, invisible to `qe-agent-commands.log`/JUnit. Before `FLAKY`/`TEST_ISSUE`, re-run with the filter commented out, or diff the API response against what the frontend expects.
+
 ### Product Bug indicators
 Classify as `PRODUCT_BUG` when the operator, plugin, or console itself misbehaved:
 - COO operator pod in `CrashLoopBackOff` or `OOMKilled`
@@ -226,7 +235,7 @@ Classify as `TEST_ISSUE` when the test itself is wrong or stale:
 
 ### Cluster Instability indicators
 
-Before classifying as `CLUSTER_INSTABILITY`, rule out a tight COO reconciliation loop with debug logging:
+Before `CLUSTER_INSTABILITY`, rule out a tight COO reconciliation loop with debug logging:
 
 ```bash
 COO_NS="$(oc get pods --all-namespaces -l app.kubernetes.io/name=observability-operator \
@@ -359,19 +368,19 @@ The description must NOT contain raw credentials, tokens, passwords, or SHA-256
 
 ## Step 5c — If FLAKY: Fix and Export
 
-Fix the race itself, not with blanket retries. Typical fixes: `cy.intercept()` plus `cy.wait('@alias')` before asserting UI state after an API call, `.should('be.visible')` with a timeout instead of an immediate assertion, and condition-based waits instead of `cy.wait(<ms>)`. Export the changed files and `CHANGES.md` as in Step 5a, with the 4-run pass/fail pattern as evidence.
+Fix the race itself, not with blanket retries: `cy.intercept()` + `cy.wait('@alias')` before asserting post-API UI state, `.should('be.visible')` with a timeout, condition-based waits instead of `cy.wait(<ms>)`. Export as in Step 5a, with the 4-run pattern as evidence.
 
 ---
 
 ## Step 5d — If CLUSTER_INSTABILITY: Write Incident Note
 
-Write `${ARTIFACT_DIR}/cluster-instability-report.md` with: a one-sentence summary; a table of affected tests (suite / test case / original duration / rerun duration); root cause (MCP updates, node evictions, COO pod restarts — include the MCP status snapshot from Step 0a); evidence (MCP output, relevant pod events); and a recommendation to rerun the CI job. Begin it with the same AI-Generated Content banner as the other reports.
+Write `${ARTIFACT_DIR}/cluster-instability-report.md`: a one-sentence summary; affected-tests table (suite / test case / original vs rerun duration); root cause (MCP updates, node evictions, COO restarts — include the Step 0a MCP snapshot); evidence (MCP output, pod events); and a recommendation to rerun the job. Same AI-Generated Content banner as the other reports.
 
 ---
 
 ## Step 5e — If JOB_CONFIG: Recommend a Job Configuration Change
 
-The test and product are fine, but the job needs something this cluster cannot provide (typically an operator not yet published for this OCP version). Do not modify the tests or write `jira-payload.json`. In `qe-agent-analysis.md`, give the missing package, OCP version and catalog evidence, and recommend the config change. For Lightspeed, add `CYPRESS_SKIP_TESTS: -Lightspeed` to the e2e `env` in `ci-operator/config/openshift/distributed-tracing-console-plugin/<variant>.yaml`; the spec then skips the Lightspeed install, setup and test. If COO, OpenTelemetry or Tempo is missing, recommend disabling or re-pointing the job instead.
+The test and product are fine, but the job needs something the cluster can't provide (typically an operator unpublished for this OCP version). Don't modify tests or write `jira-payload.json`. In `qe-agent-analysis.md`, give the missing package, OCP version and catalog evidence, and recommend the config change. For Lightspeed, add `CYPRESS_SKIP_TESTS: -Lightspeed` to the e2e `env` in `ci-operator/config/openshift/distributed-tracing-console-plugin/<variant>.yaml`. If COO, OpenTelemetry or Tempo is missing, recommend disabling or re-pointing the job.
 
 ---
 
@@ -431,9 +440,9 @@ Write `${ARTIFACT_DIR}/qe-agent-analysis.md` after each diagnosis and overwrite
 ## Notes for CI context
 
 - The cluster is already provisioned with COO and the Tracing UI console plugin installed — do not reinstall them
-- The qe-agent runs in a fresh pod, so `/tmp/` is empty at start; Step 0b clones the test repo there
-- `$KUBECONFIG` points to the test cluster; `oc`, `kubectl`, `jq` and `npm`/`npx` are in PATH
-- Do **not** copy Cypress screenshots or videos to `$SHARED_DIR` (1 MiB Secret limit); only JUnit XML is safe there. Write all output to `$ARTIFACT_DIR` (uploaded to GCS) or `$SHARED_DIR` (shared with other steps)
-- **Namespace restriction**: You MUST NOT access, read, list, or modify any resource in the `kube-system` namespace (cloud provider credentials, platform-critical components) — no `oc` or `kubectl` command may target it. Filter `kube-system` out of all-namespace output (e.g. `oc get pods -A`) before analysis
-- Do not call external APIs (Jira, GitHub API, Slack, etc.); the wrapper script handles integrations after the agent exits
-- This step runs `best_effort: true` — always exit 0 even if analysis is incomplete
+- The qe-agent runs in a fresh pod — `/tmp/` is empty at start; Step 0b clones the test repo there
+- `$KUBECONFIG` points to the test cluster; `oc`, `kubectl`, `jq`, `npm`/`npx` are in PATH
+- Do **not** copy screenshots/videos to `$SHARED_DIR` (1 MiB Secret limit); only JUnit XML is safe there. Write output to `$ARTIFACT_DIR` (GCS) or `$SHARED_DIR` (shared with other steps)
+- **Namespace restriction**: MUST NOT access, read, list, or modify any `kube-system` resource — an all-namespaces query filtered afterward still fetches it, so scope every command to named namespaces or a label selector that excludes it (e.g. `-l app.kubernetes.io/name=...`, as used above), not `-A` piped to `grep -v kube-system`
+- Do not call external APIs (Jira, GitHub, Slack); the wrapper script handles integrations after exit
+- This step runs `best_effort: true` — always exit 0, even incomplete