fix(cluster-install): version-gate OIDC install order; add OCI VMDK and cnpg webhook gotchas

plugins/cozystack/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "cozystack",
-  "version": "1.15.0",
+  "version": "1.15.1",
   "description": "Cozystack platform skills bundle. Start with cozystack:wizard — it begins with a free-form 'tell me about your setup and goal' question, parses hints, then asks Talos / Ubuntu / Existing, builds a chain, and dispatches downstream skills via a cluster config directory the operator picks (every artifact lives there: inventory.yml, kubeconfig, .state.yaml, cozystack-platform-package.yaml — operator manages git on their own; optional sops opt-in encrypts secret files in-tree). Skills, invoked as cozystack:<name>: wizard (orchestrator + 3-route dispatcher + Phase 4.5 active research + auto-dispatches debug on any failed_at), talos-bootstrap (Talos node prep via talm — Talos-1.12-aware maintenance probe, NAT-provider cert-SAN guardrail before first talm apply, multidoc machine-config with per-node VIP-link IPv4 stubs, etcd bootstrap, kubeconfig fetch, cozystack-tuned shape verification with Phase 11.5 auto-upgrade), talos-reset (cloud-provider terminate+relaunch helper for OCI/AWS/GCP/Hetzner when nodes are unrecoverable from inside; preserves block volumes + secondary VNICs + NSG memberships), ubuntu-bootstrap (wraps cozystack/ansible-cozystack — OS prep + k3s install in one go), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning via privileged DaemonSet on Talos with hostNetwork, extractedprism for kube-apiserver HA, OCI-tag-normalized cozy-installer chart, Platform Package, inline tenants/root ingress patch + LINSTOR pool registration during watch loop with combined HRs-Ready + pools-registered gate, Phase 8.6 default StorageClasses for v1.3.x, Phase 9.1 end-to-end reachability probe), debug (investigate a stuck or broken install — gathers symptoms, classifies operator error / config drift / upstream bug / not-yet-supported, applies fixes or workarounds, drafts upstream issues on approval; never opens PRs or files silently), cluster-upgrade (v1.x patch/minor upgrade with release-notes analysis), package-deploy (dev-loop deploy of a single package with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (bootstrap a UI dev sandbox — Playwright + Vite dev server for cozystack-ui pointed at a chosen kubeconfig; worktree, @playwright/test, Chromium, playwright.config.ts, dev:e2e / test:e2e scripts, kubectl proxy + Vite on a free port). All skills match the operator's natural language detected from conversation context — code identifiers, commands, file paths, and GitHub-public text stay canonical. All skills follow the same gate-and-confirm discipline: read-only lookups run freely; any mutation needs explicit per-step approval.",
   "author": {
     "name": "Cozystack",

plugins/cozystack/skills/cluster-install/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: cluster-install
-description: Use when installing Cozystack on an existing Kubernetes cluster (kubeadm / k3s / RKE2 / managed). Discovers cluster facts, validates node readiness via kubectl debug, recommends an installer + platform variant, gathers values interactively, creates the ZFS pool on each storage node through kubectl debug (cozystack standardises on ZFS for LINSTOR — LVM / LVM-thin paths are not supported), installs extractedprism (per-node kube-apiserver HA proxy, generic variant default), installs the cozy-installer chart, applies the Platform Package, patches the root Tenant for ingress (breaks the OIDC chicken-and-egg), waits until every HelmRelease is Ready, prints a NOTES-style access summary, and offers an issue-template handoff to cozystack/* on fatal failure. Not for Kubernetes bootstrap and not for upgrades — see `cozystack:cluster-upgrade` for that. Talos node-prep is out of scope — `cozystack:cluster-install` refuses if Talos nodes are missing cozystack-tuned extensions and points at `cozystack:talos-bootstrap`.
+description: Use when installing Cozystack on an existing Kubernetes cluster (kubeadm / k3s / RKE2 / managed). Discovers cluster facts, validates node readiness via kubectl debug, recommends an installer + platform variant, gathers values interactively, creates the ZFS pool on each storage node through kubectl debug (cozystack standardises on ZFS for LINSTOR — LVM / LVM-thin paths are not supported), installs extractedprism (per-node kube-apiserver HA proxy, generic variant default), installs the cozy-installer chart, applies the Platform Package (on current releases OIDC is enabled after convergence, not at install), patches the root Tenant host + ingress, waits until every HelmRelease is Ready, prints a NOTES-style access summary, and offers an issue-template handoff to cozystack/* on fatal failure. Not for Kubernetes bootstrap and not for upgrades — see `cozystack:cluster-upgrade` for that. Talos node-prep is out of scope — `cozystack:cluster-install` refuses if Talos nodes are missing cozystack-tuned extensions and points at `cozystack:talos-bootstrap`.
 argument-hint: "[--config-dir=<path>] [--context=<kube-context>] [--installer-version=<vX.Y.Z>] [--installer-variant=<talos|generic|hosted>] [--platform-variant=<isp-full|isp-full-generic|isp-hosted|default>] [--no-extractedprism] [--api-host=<ip>] [--dry-run]"
 ---
 
@@ -411,10 +411,11 @@ actions on Continue:
   3. (if cozy-system namespace exists but unowned) adopt namespace into cozy-system/cozy-installer
   4. helm upgrade --install cozy-installer ... --namespace cozy-system --create-namespace  (~2 min)
   5. wait deploy/cozystack-operator Available; wait CRD packages.cozystack.io Established
-  6. kubectl apply --filename /tmp/.../platform-package.yaml
-  7. wait root Tenant CR, patch spec.host + spec.ingress=true (~3 min — required for Phase 8 to ever finish; breaks the OIDC chicken-and-egg)
+  6. kubectl apply --filename /tmp/.../platform-package.yaml  (OIDC OFF on installer ≥1.5.0; ON on v1.4.x — see Phase 8 version gate)
+  7. wait root Tenant CR, patch spec.host + spec.ingress=true (~3 min — brings up the root ingress with the right domain)
   8. poll HRs every 30s until all Ready=True (~30–60 min)
-  9. print access summary
+  9. (installer ≥1.5.0 + dashboard auth = OIDC; on v1.4.x OIDC is already on from step 6) once HRs are green and root-ingress-controller is up, enable authentication.oidc.enabled + expose keycloak, then wait for keycloak + gatekeeper rollout (~5–10 min) — see Phase 8
+  10. print access summary
 
 options: Continue / Edit values / Cancel
 ```
@@ -678,14 +679,22 @@ See `references/helmrelease-monitoring.md` for the full polling pattern and stuc
 
 This phase merges what used to be Phase 7.5 (root Tenant ingress patch) into the watch loop. The motivation: on a real install the `tenants.apps.cozystack.io` CRD can take longer to install than the wait deadline of a separate phase — sometimes the CRD doesn't exist until after most HRs are Ready, so a fixed 5-minute wait-for-Tenant expires before the CRD lands. Folding the patch into the watch loop makes it event-driven: as soon as the CR appears, the skill patches it and continues monitoring, regardless of where Phase 8 is in its own progression.
 
-**Why the patch is needed**: cozystack's dashboard ships gatekeeper (oauth2-proxy) which, on startup, does OIDC discovery against the **public FQDN** `https://keycloak.${HOST}/realms/cozy/.well-known/openid-configuration` — not an in-cluster service. Without the root ingress controller running, nothing listens on 443, gatekeeper CrashLoopBackOffs, the `cozy-dashboard/dashboard` HR sits in `Unknown: Running 'install' action with timeout of 10m0s` and then `InstallFailed: context deadline exceeded`, Flux remediates and retries forever. `cozy-fluxcd/flux-plunger` has a hard dependency on `cozy-dashboard/dashboard` and stays `False: dependency is not ready`. The phase would never go green.
+**Why the patch is needed**: the root ingress controller only starts once `tenants.apps.cozystack.io/root` is patched with `spec.host` + `spec.ingress: true`. Until then nothing serves the public wildcard `*.${HOST}` on 443, so every per-tenant ingress object (`dashboard.${HOST}`, `keycloak.${HOST}`, `api.${HOST}`) renders against an empty/parked domain and no public URL works. The root tenant ships with `spec.host: ""` and does NOT inherit `publishing.host` from the Platform Package (verified on 1.6.x), so an ingress-only patch with an empty host is not enough — set both keys. The CR can appear at any point in the watch loop; patch it event-driven as soon as it lands (loop below). This patch is required for correct public URLs and is a precondition for the OIDC flip described next — but it is NOT what unblocks a stalled OIDC-at-install (that is avoided by installing OIDC-off, see below).
 
-**The dashboard requires OIDC/Keycloak — it is not optional on the supported path.** The "Why the patch is needed" note above describes gatekeeper doing OIDC discovery against Keycloak. But Keycloak only deploys when `authentication.oidc.enabled: true` in the Platform Package — and that key defaults to `false`, with the `isp-full*` overlays NOT turning it on. If the skill enables `ingress` + sets `host` but never enables OIDC, the result on v1.4.2 is: no `cozy-keycloak` namespace, no Keycloak HR, and the dashboard falls back to its non-OIDC `token-proxy` container, which is **broken on v1.4.2** — the container starts, never binds `:8000` (connection refused, zero logs), and is killed by its own `/ping` liveness probe every ~45 s → CrashLoopBackOff → the `cozy-dashboard/dashboard` HR fails install → `flux-plunger` and the rest of the chain hang. The cluster reports 88/90 HRs Ready and looks "almost done" forever.
+**Version gate.** The OIDC-off-then-flip sequence below is for `installer_version` **≥ 1.5.0**, where the OIDC-off dashboard converges on a working `token-proxy` container (its `/ping` liveness CrashLoop was fixed by a `startupProbe` backstop in v1.5.0). On **v1.4.x** the token-proxy is broken (`/ping` liveness kills the container → CrashLoopBackOff → the dashboard HR fails install), so a working web dashboard there needs OIDC enabled at install — keep `authentication.oidc.enabled: true` in the Package for v1.4.x and, if the install stalls, fall back to the recovery in `references/known-failures.md`. v1.3.x ships no Keycloak in `isp-full` and the dashboard uses a bundled self-issued OIDC (see `references/provider-pitfalls.md`). The rest of this section is the v1.5.0+ path.
 
-So for a usable dashboard the skill must enable OIDC. This is exactly what cozystack's own e2e (`hack/e2e-install-cozystack.bats`) does — patch the root tenant host, then enable OIDC and expose Keycloak:
+**Install with OIDC OFF, converge, THEN flip it on (installer_version ≥ 1.5.0).** On 1.6.x, setting `authentication.oidc.enabled: true` in the Platform Package before the platform has converged DEADLOCKS the install. The flag switches the system bundle to the `oidc` engine variant (`packages/core/platform/templates/bundles/system.yaml` selects `cozystack.cozystack-engine` variant `oidc` when `authentication.oidc.enabled`), which pulls Keycloak into the dashboard's dependency chain: on the `oidc` variant the `dashboard` component `dependsOn: keycloak-configure` and the engine itself `dependsOn cozystack.keycloak, cozystack.keycloak-operator` (`packages/core/platform/sources/cozystack-engine.yaml`); on the `default` variant the dashboard depends only on `cozystack-api`/`cozystack-controller`/`application-definition-crd`. The dashboard's gatekeeper (oauth2-proxy) then also needs to reach Keycloak's public URL, which is only served once the root ingress is up — but the root ingress and the root Tenant CR sit behind the dashboard in the reconcile graph. Net cycle: dashboard needs Keycloak + ingress, ingress needs the dashboard-gated chain, so the install plateaus partway (e.g. ~65% of HRs) and never converges. Patching a tenant CR does NOT rescue this — on the `oidc` variant the tenant CR does not even appear until the dashboard-gated chain unblocks, which with OIDC on it never does.
+
+The underlying deadlock is fixed on `main` (post-v1.6.0-rc.2) by defaulting the gatekeeper's OIDC backend to the in-cluster Keycloak URL, so the gatekeeper skips external discovery and no longer needs the root ingress at startup; this is not yet in a released tag. OIDC-off-then-flip remains the recommended, e2e-verified sequence on all versions.
+
+**Supported sequence — exactly what cozystack's own e2e (`hack/e2e-install-cozystack.bats`) does:**
+
+1. Apply the Platform Package with OIDC **off** — the default (`authentication.oidc.enabled` unset / `false`; the `isp-full*` overlays do not turn it on). The whole platform converges and the dashboard comes up on its non-OIDC `token-proxy` container.
+2. Patch `tenants/root` `spec.host` + `spec.ingress: true` (the inline watch-loop patch below) and wait until `root-ingress-controller` is Available (`2/2`) in `tenant-root`.
+3. Only THEN enable OIDC, in a second step:
 
 ```bash
-# Enable OIDC and expose keycloak (do this once, after the Package exists).
+# Step 3 — after all HRs are Ready AND root-ingress-controller is Available.
 kubectl --context $CTX patch package cozystack.cozystack-platform --type merge \
   --patch '{"spec":{"components":{"platform":{"values":{"authentication":{"oidc":{"enabled":true}}}}}}}'
 
@@ -695,7 +704,21 @@ kubectl --context $CTX patch package cozystack.cozystack-platform --type merge \
   --patch '{"spec":{"components":{"platform":{"values":{"publishing":{"exposedServices":["api","dashboard","keycloak"]}}}}}}'
 ```
 
-Better: bake both into the Platform Package CR written in Phase 4/7 from the start (`authentication.oidc.enabled: true` and `keycloak` in `exposedServices`) so there is no second reconcile. The Phase 4 intake should collect a **dashboard auth** decision — OIDC/Keycloak (recommended; the only path with a working dashboard on 1.4.2) vs none (API-only, no web dashboard) — and only enable OIDC when the operator wants the dashboard. When OIDC is enabled, Keycloak needs a working LE cert for `keycloak.${HOST}`, so the same DNS/port-80 preconditions as the dashboard host apply (Phase 4 publishing gate already covers this — just make sure `keycloak.${HOST}` is inside the wildcard).
+With the root ingress already up, enabling OIDC now brings up Keycloak, swaps the dashboard's `token-proxy` container for the oauth2-proxy gatekeeper, and gatekeeper reaches Keycloak — by default via the in-cluster keycloak Service (`keycloakInternalUrl`), and even when it is pointed at the public `keycloak.${HOST}` URL those in-cluster requests hairpin through the now-running ingress (verified HTTP 200) — so the gatekeeper rolls out cleanly instead of CrashLooping. Gate the flip on the `auth-proxy` container appearing in `deploy/incloud-web-gatekeeper` and its `rollout status`, not on `hr/dashboard` (which stays stale-Ready from the token-proxy install).
+
+The Phase 4 intake collects a **dashboard auth** decision — OIDC/Keycloak (recommended; a browser SSO dashboard) vs none (API-only; dashboard stays on the token-proxy container). On installer_version ≥ 1.5.0, even when the operator wants OIDC the skill installs OIDC-off first and performs the flip as a post-convergence step (Phase 5 plan item 9), NOT by baking `authentication.oidc.enabled: true` into the Platform Package from the start; on v1.4.x it bakes OIDC on at install per the version gate above. When OIDC is enabled, Keycloak needs a working LE cert for `keycloak.${HOST}`, so the same DNS/port-80 preconditions as the dashboard host apply — make sure `keycloak.${HOST}` is inside the wildcard (Phase 4 publishing gate already covers this).
+
+**off→on toggle hazard — flip OIDC on exactly once; do not toggle it back off.** Disabling OIDC after it was enabled marks the `keycloak-configure` HR for deletion; its Helm pre-delete teardown Job fails on RBAC and the HR sticks in `Terminating`, with the Helm release secret stuck in status `uninstalling`. Recovery:
+
+```bash
+kubectl --context $CTX --namespace cozy-keycloak patch hr keycloak-configure \
+  --type merge --patch '{"metadata":{"finalizers":null}}'
+kubectl --context $CTX --namespace cozy-keycloak delete secret \
+  --selector owner=helm,name=keycloak-configure,status=uninstalling
+flux --context $CTX --namespace cozy-keycloak reconcile hr keycloak-configure --force
+```
+
+Installing OIDC-off and flipping it on once avoids this churn entirely.
 
 Skip the root-Tenant patch entirely on `isp-hosted` or when the `system` bundle was disabled in Phase 4 — there is no root Tenant CR in those modes.
 
@@ -1086,7 +1109,8 @@ If any phase hits a fatal failure that looks like an upstream bug or doc gap, fo
 - NEVER bootstrap Talos nodes or invoke `boot-to-talos` / `talm` from inside this skill — that flow lives in `/cozystack:talos-bootstrap`. Refuse and hand off.
 - NEVER auto-rollback a partially provisioned storage state — print backout commands and let the operator decide.
 - NEVER accept a custom `publishing.host` without an explicit operator confirmation that they own the domain and will configure wildcard DNS — the HTTP-01 cert solver fails silently otherwise. nip.io patterns skip this gate because nip.io is publicly hosted DNS.
-- ALWAYS patch `tenants/root` with BOTH `spec.host=$HOST` and `spec.ingress=true` from inside the Phase 8 watch loop as soon as the CR appears, on `system`-bundle installs. The root tenant ships with `spec.host: ""` and does not inherit `publishing.host`, so an ingress-only patch leaves every per-tenant ingress object rendering against an empty domain. The OIDC chicken-and-egg makes Phase 8 unreachable otherwise — dashboard / keycloak / flux-plunger loop forever, every other downstream HR stalls on the missing root ingress. The CR can appear at any point during the watch loop; do not gate the patch behind a fixed pre-Phase-8 wait.
+- On `installer_version` ≥ 1.5.0, do NOT set `authentication.oidc.enabled: true` in the Platform Package at install time. On 1.6.x it switches the system bundle to the `oidc` engine variant, which makes the dashboard `dependsOn` Keycloak; the dashboard then can't converge until the root ingress is up, but the root ingress sits behind the dashboard in the reconcile graph — the install deadlocks partway. Install OIDC-off, let the platform converge and `root-ingress-controller` come up, THEN `kubectl patch package … authentication.oidc.enabled=true` exactly once. Flipping OIDC back off strands `keycloak-configure` in `Terminating` (see Phase 8 recovery). Exception: on v1.4.x the OIDC-off token-proxy dashboard is broken, so OIDC must be enabled at install there (Phase 8 version gate).
+- ALWAYS patch `tenants/root` with BOTH `spec.host=$HOST` and `spec.ingress=true` from inside the Phase 8 watch loop as soon as the CR appears, on `system`-bundle installs. The root tenant ships with `spec.host: ""` and does not inherit `publishing.host`, so an ingress-only patch leaves every per-tenant ingress object rendering against an empty domain, and the root ingress controller never serves any public URL (`dashboard.${HOST}`, `keycloak.${HOST}`, …). The CR can appear at any point during the watch loop; do not gate the patch behind a fixed pre-Phase-8 wait. This patch is required for correct public URLs and is a precondition for the post-convergence OIDC flip — it is NOT itself the fix for the OIDC-at-install deadlock (installing OIDC-off is).
 - ALWAYS read variant overlays and `requirements.md` before declaring "this looks fine" — variant-specific checks (CP-label value, ZFS availability, KubeOVN MASTER_NODES) are easy to miss.
 - ALWAYS pull live data over cached assumption: `kubectl get` over "I think this is …".
 - ALWAYS write Phase 4 collected values to disk in `<config-dir>/cozystack-platform-package.yaml` before applying — the file is part of the diagnostic bundle if Phase 8 fails. ZFS pool registration is stored separately under `<config-dir>/.state.yaml` `cozystack.storage.nodes[]` and replayed by the Phase 8 post-Ready hook (there is no `LinstorSatelliteConfiguration` CR for the ZFS path).