fix(cluster-install): version-gate OIDC install order; add OCI VMDK and cnpg webhook gotchas#19
Open
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Conversation
…nd cnpg gotchas Correct the OIDC install-order guidance and add two provider gotchas that each cost a long real-world debug session. OIDC install order, version-gated. On installer_version >= 1.5.0, enabling authentication.oidc.enabled in the Platform Package at install time deadlocks the install: the flag switches the system bundle to the oidc engine variant, which makes the dashboard dependsOn Keycloak. The dashboard's gatekeeper then needs the root ingress, but the root ingress and the root Tenant CR sit behind the dashboard in the reconcile graph, so the platform plateaus partway and never converges; the root Tenant CR does not even appear, so patching it cannot rescue the stall. Reframe Phase 4 / Phase 8 to the supported sequence (matching cozystack's own e2e): apply the Platform Package with OIDC off, let every HelmRelease converge, patch the root Tenant host + ingress, wait for the root ingress controller, and only then flip OIDC on. Flip the canonical Platform Package template default to OIDC-off to match (it previously baked OIDC on at install, which was the actual landmine). Present the root Tenant patch as what brings up the root ingress with the right domain, not as the deadlock fix. Document the off->on toggle hazard (disabling OIDC strands keycloak-configure in Terminating) with revision-agnostic recovery, rewrite the stalled-install recovery to disable OIDC first, and correct the gatekeeper Deployment name to incloud-web-gatekeeper. Keep the v1.4.x path intact: on v1.4.x the OIDC-off token-proxy dashboard is broken (/ping liveness CrashLoop, fixed by a startupProbe backstop in v1.5.0), so OIDC must be enabled at install there. The guidance, the values-template default, and the provider-pitfalls Keycloak note are all version-gated (v1.3.x self-issued OIDC / v1.4.x OIDC-at-install / v1.5.0+ OIDC-off-then-flip) rather than stated absolutely. OCI Talos image boot. Current Talos disk images (1.11.5+) do not boot on OCI when imported from qcow2 (UEFI shell / empty BIOS console, API never answers). The same disk imported as a streamOptimized VMDK with firmware BIOS boots fine. Add the workaround and the practice of verifying boot on a throwaway instance before recreating the whole cluster. cnpg webhook unreachable from the apiserver. On Cilium + KubeOVN the cloudnative-pg operator flaps on leader-election during early KubeOVN convergence and restarts with a new pod IP; Cilium's host-side service BPF keeps the stale endpoint, so the apiserver gets connection refused on the webhook ClusterIP (failurePolicy=Fail blocks keycloak and every postgres-backed release) while pods reach it fine. Add the host-network-vs-pod-network diagnostic and the rollout-restart recovery. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
📝 WalkthroughWalkthroughChangesThe cluster-install documentation now describes deferred OIDC enablement for installer versions 1.5.0 and newer, including root-tenant ingress preparation and recovery procedures. It also adds OCI image troubleshooting, CNPG webhook diagnostics, updated Keycloak guidance, and a plugin version bump. Cluster installation guidance
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Installer
participant PlatformPackage
participant RootTenant as tenants/root
participant Ingress as root-ingress-controller
participant AuthProxy as auth-proxy
Installer->>PlatformPackage: apply with OIDC disabled
Installer->>RootTenant: patch host and ingress
Installer->>Ingress: wait for availability
Installer->>PlatformPackage: enable OIDC once
PlatformPackage->>AuthProxy: roll out and verify auth-proxy
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Tools execution failed with the following error: Failed to run tools: 14 UNAVAILABLE: read ECONNRESET Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Aleksei Sviridkin (lexfrei)
marked this pull request as ready for review
July 20, 2026 13:35
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the cozystack plugin to version 1.15.1 and significantly refines the cluster-install skill documentation. It introduces a critical installation sequence change for installer versions >= 1.5.0, recommending that OIDC be disabled during initial install to avoid deadlocks and enabled only after convergence. It also adds troubleshooting sections for CNPG webhook issues on Cilium and OCI Talos boot failures. The review feedback is highly accurate, pointing out that the --force flag is invalid for flux reconcile hr and that the standard busybox version of nc does not support the -z and -v flags.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| --type merge --patch '{"metadata":{"finalizers":null}}' | ||
| kubectl --context $CTX --namespace cozy-keycloak delete secret \ | ||
| --selector owner=helm,name=keycloak-configure,status=uninstalling | ||
| flux --context $CTX --namespace cozy-keycloak reconcile hr keycloak-configure --force |
There was a problem hiding this comment.
The flux reconcile helmrelease (or flux reconcile hr) command does not support a --force flag. Running this command with --force will result in an unknown flag: --force error. Remove the --force flag to ensure the command executes successfully.
Suggested change
| flux --context $CTX --namespace cozy-keycloak reconcile hr keycloak-configure --force | |
| flux --context $CTX --namespace cozy-keycloak reconcile hr keycloak-configure |
Comment on lines
+101
to
+105
| kubectl --context $CTX run nc-pod --rm --stdin --tty --restart=Never \ | ||
| --image=busybox -- nc -zvw3 $WEBHOOK_CIP 443 # from pod network → OK | ||
| kubectl --context $CTX run nc-host --rm --stdin --tty --restart=Never \ | ||
| --overrides='{"spec":{"hostNetwork":true}}' \ | ||
| --image=busybox -- nc -zvw3 $WEBHOOK_CIP 443 # from host network → refused |
There was a problem hiding this comment.
The standard busybox image's nc (netcat) utility does not support the -z (zero-I/O mode) or -v (verbose) flags. Running nc -zvw3 in a standard busybox container will fail with nc: invalid option -- 'z'. To perform a reliable port check using busybox, redirect /dev/null to stdin and use the -w flag for timeout.
Suggested change
| kubectl --context $CTX run nc-pod --rm --stdin --tty --restart=Never \ | |
| --image=busybox -- nc -zvw3 $WEBHOOK_CIP 443 # from pod network → OK | |
| kubectl --context $CTX run nc-host --rm --stdin --tty --restart=Never \ | |
| --overrides='{"spec":{"hostNetwork":true}}' \ | |
| --image=busybox -- nc -zvw3 $WEBHOOK_CIP 443 # from host network → refused | |
| kubectl --context $CTX run nc-pod --rm --stdin --tty --restart=Never \ | |
| --image=busybox -- sh -c "nc -w3 $WEBHOOK_CIP 443 < /dev/null" # from pod network → OK | |
| kubectl --context $CTX run nc-host --rm --stdin --tty --restart=Never \ | |
| --overrides='{"spec":{"hostNetwork":true}}' \ | |
| --image=busybox -- sh -c "nc -w3 $WEBHOOK_CIP 443 < /dev/null" # from host network → refused |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/cozystack/skills/cluster-install/references/known-failures.md`:
- Line 74: The documented recovery flow incorrectly skips OIDC re-enablement
when OIDC is already disabled. Update the post-convergence instructions in the
“apply the tenant patch alone” path to skip only disabling OIDC, then enable it
exactly once after root-ingress-controller is Available and wait for the
incloud-web-gatekeeper rollout when dashboard SSO is requested.
In `@plugins/cozystack/skills/cluster-install/SKILL.md`:
- Line 688: Update the OIDC-off-then-flip recommendation in the cluster
installation guidance to apply only to installer versions v1.5.0 and newer,
replacing the “all versions” wording while preserving the existing v1.4.x
exception.
- Line 3: Normalize all internal skill references to slash-prefixed links:
update the references in plugins/cozystack/skills/cluster-install/SKILL.md:3-3
to /cozystack:cluster-upgrade, /cozystack:cluster-install, and
/cozystack:talos-bootstrap; update
plugins/cozystack/skills/cluster-install/references/provider-pitfalls.md:20-20
to /cozystack:talos-bootstrap and :116-116 to /cozystack:cluster-install; update
plugins/cozystack/skills/cluster-install/references/known-failures.md:43-43 to
/cozystack:cluster-install. Ensure each link resolves to an existing skill
directory.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 6ba221a6-f082-4009-98e7-369beae808b6
📒 Files selected for processing (5)
plugins/cozystack/.claude-plugin/plugin.jsonplugins/cozystack/skills/cluster-install/SKILL.mdplugins/cozystack/skills/cluster-install/references/known-failures.mdplugins/cozystack/skills/cluster-install/references/provider-pitfalls.mdplugins/cozystack/skills/cluster-install/references/values-template.md
| - the dashboard (token-proxy container) and the rest of the chain reach Ready; the cluster converges to `N/N Ready`. | ||
|
|
||
| The whole cluster typically reaches `N/N Ready` within 5 min of the patch. | ||
| Once every HR is Ready and `root-ingress-controller` is Available, re-enable OIDC exactly once (SKILL.md Phase 8 step 3) and wait for the `incloud-web-gatekeeper` rollout. If the cluster was already OIDC-off (the operator followed the skill and only the tenant patch was missing), skip the disable/re-enable steps and apply the tenant patch alone. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not skip the post-convergence OIDC flip when OIDC is already off.
If the operator followed the new flow and only the tenant patch was missing, “apply the tenant patch alone” leaves OIDC disabled even when dashboard SSO was requested. Skip only the disable step; still enable OIDC exactly once after ingress is ready.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/cozystack/skills/cluster-install/references/known-failures.md` at
line 74, The documented recovery flow incorrectly skips OIDC re-enablement when
OIDC is already disabled. Update the post-convergence instructions in the “apply
the tenant patch alone” path to skip only disabling OIDC, then enable it exactly
once after root-ingress-controller is Available and wait for the
incloud-web-gatekeeper rollout when dashboard SSO is requested.
| --- | ||
| name: cluster-install | ||
| description: Use when installing Cozystack on an existing Kubernetes cluster (kubeadm / k3s / RKE2 / managed). Discovers cluster facts, validates node readiness via kubectl debug, recommends an installer + platform variant, gathers values interactively, creates the ZFS pool on each storage node through kubectl debug (cozystack standardises on ZFS for LINSTOR — LVM / LVM-thin paths are not supported), installs extractedprism (per-node kube-apiserver HA proxy, generic variant default), installs the cozy-installer chart, applies the Platform Package, patches the root Tenant for ingress (breaks the OIDC chicken-and-egg), waits until every HelmRelease is Ready, prints a NOTES-style access summary, and offers an issue-template handoff to cozystack/* on fatal failure. Not for Kubernetes bootstrap and not for upgrades — see `cozystack:cluster-upgrade` for that. Talos node-prep is out of scope — `cozystack:cluster-install` refuses if Talos nodes are missing cozystack-tuned extensions and points at `cozystack:talos-bootstrap`. | ||
| description: Use when installing Cozystack on an existing Kubernetes cluster (kubeadm / k3s / RKE2 / managed). Discovers cluster facts, validates node readiness via kubectl debug, recommends an installer + platform variant, gathers values interactively, creates the ZFS pool on each storage node through kubectl debug (cozystack standardises on ZFS for LINSTOR — LVM / LVM-thin paths are not supported), installs extractedprism (per-node kube-apiserver HA proxy, generic variant default), installs the cozy-installer chart, applies the Platform Package (on current releases OIDC is enabled after convergence, not at install), patches the root Tenant host + ingress, waits until every HelmRelease is Ready, prints a NOTES-style access summary, and offers an issue-template handoff to cozystack/* on fatal failure. Not for Kubernetes bootstrap and not for upgrades — see `cozystack:cluster-upgrade` for that. Talos node-prep is out of scope — `cozystack:cluster-install` refuses if Talos nodes are missing cozystack-tuned extensions and points at `cozystack:talos-bootstrap`. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Normalize all internal skill references to slash-prefixed links.
plugins/cozystack/skills/cluster-install/SKILL.md#L3-L3: use/cozystack:cluster-upgrade,/cozystack:cluster-install, and/cozystack:talos-bootstrap.plugins/cozystack/skills/cluster-install/references/provider-pitfalls.md#L20-L20: use/cozystack:talos-bootstrap.plugins/cozystack/skills/cluster-install/references/provider-pitfalls.md#L116-L116: use/cozystack:cluster-install.plugins/cozystack/skills/cluster-install/references/known-failures.md#L43-L43: use/cozystack:cluster-install.
As per coding guidelines, internal skill references must follow /<plugin>:<skill> and resolve to an actual skill directory.
📍 Affects 3 files
plugins/cozystack/skills/cluster-install/SKILL.md#L3-L3(this comment)plugins/cozystack/skills/cluster-install/references/provider-pitfalls.md#L20-L20plugins/cozystack/skills/cluster-install/references/provider-pitfalls.md#L116-L116plugins/cozystack/skills/cluster-install/references/known-failures.md#L43-L43
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/cozystack/skills/cluster-install/SKILL.md` at line 3, Normalize all
internal skill references to slash-prefixed links: update the references in
plugins/cozystack/skills/cluster-install/SKILL.md:3-3 to
/cozystack:cluster-upgrade, /cozystack:cluster-install, and
/cozystack:talos-bootstrap; update
plugins/cozystack/skills/cluster-install/references/provider-pitfalls.md:20-20
to /cozystack:talos-bootstrap and :116-116 to /cozystack:cluster-install; update
plugins/cozystack/skills/cluster-install/references/known-failures.md:43-43 to
/cozystack:cluster-install. Ensure each link resolves to an existing skill
directory.
Source: Coding guidelines
| So for a usable dashboard the skill must enable OIDC. This is exactly what cozystack's own e2e (`hack/e2e-install-cozystack.bats`) does — patch the root tenant host, then enable OIDC and expose Keycloak: | ||
| **Install with OIDC OFF, converge, THEN flip it on (installer_version ≥ 1.5.0).** On 1.6.x, setting `authentication.oidc.enabled: true` in the Platform Package before the platform has converged DEADLOCKS the install. The flag switches the system bundle to the `oidc` engine variant (`packages/core/platform/templates/bundles/system.yaml` selects `cozystack.cozystack-engine` variant `oidc` when `authentication.oidc.enabled`), which pulls Keycloak into the dashboard's dependency chain: on the `oidc` variant the `dashboard` component `dependsOn: keycloak-configure` and the engine itself `dependsOn cozystack.keycloak, cozystack.keycloak-operator` (`packages/core/platform/sources/cozystack-engine.yaml`); on the `default` variant the dashboard depends only on `cozystack-api`/`cozystack-controller`/`application-definition-crd`. The dashboard's gatekeeper (oauth2-proxy) then also needs to reach Keycloak's public URL, which is only served once the root ingress is up — but the root ingress and the root Tenant CR sit behind the dashboard in the reconcile graph. Net cycle: dashboard needs Keycloak + ingress, ingress needs the dashboard-gated chain, so the install plateaus partway (e.g. ~65% of HRs) and never converges. Patching a tenant CR does NOT rescue this — on the `oidc` variant the tenant CR does not even appear until the dashboard-gated chain unblocks, which with OIDC on it never does. | ||
|
|
||
| The underlying deadlock is fixed on `main` (post-v1.6.0-rc.2) by defaulting the gatekeeper's OIDC backend to the in-cluster Keycloak URL, so the gatekeeper skips external discovery and no longer needs the root ingress at startup; this is not yet in a released tag. OIDC-off-then-flip remains the recommended, e2e-verified sequence on all versions. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Limit the OIDC-off sequence to installer versions ≥ 1.5.0.
Line 688 says this remains recommended “on all versions,” contradicting the preceding v1.4.x exception, where OIDC must be enabled at install. Change this to “v1.5.0+” to avoid breaking v1.4.x installations.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/cozystack/skills/cluster-install/SKILL.md` at line 688, Update the
OIDC-off-then-flip recommendation in the cluster installation guidance to apply
only to installer versions v1.5.0 and newer, replacing the “all versions”
wording while preserving the existing v1.4.x exception.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corrects the
cluster-installskill's OIDC install-order guidance and adds two provider gotchas that each cost a long real-world debug session.OIDC at install deadlocks on current releases. The skill previously told the operator to bake
authentication.oidc.enabled: trueinto the Platform Package from the start (its canonical values template hard-set it on). On installer versions ≥ 1.5.0 that deadlocks the install: the flag switches the system bundle to theoidcengine variant, which makes the dashboarddependsOnKeycloak. The dashboard's gatekeeper then needs the root ingress, but the root ingress and the root Tenant CR sit behind the dashboard in the reconcile graph, so the platform plateaus partway (~65% of HelmReleases) and never converges — the root Tenant CR never even appears, so patching it cannot rescue the stall.Corrected to the supported sequence (matching the project's own e2e,
hack/e2e-install-cozystack.bats): apply the Platform Package with OIDC off, let every HelmRelease converge, patch the root Tenant host + ingress, wait for the root ingress controller, and only then flip OIDC on. The root Tenant patch is reframed as what brings up the root ingress with the right domain, not as the deadlock fix, and the values-template default flips to OIDC-off.Version-gated, not absolute. On v1.4.x the OIDC-off token-proxy dashboard is genuinely broken (its
/pingliveness CrashLoops; fixed by astartupProbebackstop in v1.5.0), so OIDC must be enabled at install there; v1.3.x uses a bundled self-issued OIDC with no Keycloak inisp-full. The guidance, the template default, and the provider-pitfalls Keycloak note all carry the three-band gate (v1.3.x / v1.4.x / v1.5.0+) rather than an absolute rule. The underlying deadlock is also fixed onmain(post-v1.6.0-rc.2) by defaulting the gatekeeper's OIDC backend to the in-cluster Keycloak URL, so the guidance notes that OIDC-off-then-flip remains the recommended, e2e-verified sequence on all versions rather than presenting the deadlock as permanent.Off→on toggle hazard. Disabling OIDC after it was enabled strands
keycloak-configureinTerminating(its Helm pre-delete teardown fails on RBAC, the release secret sticks inuninstalling); recovery steps are included. Installing OIDC-off and flipping it on once avoids the churn.New failure-mode docs. OCI Talos image boot: current Talos disk images (1.11.5+) do not boot on OCI when imported from qcow2 (UEFI shell / empty BIOS console, API never answers; siderolabs/talos#12557) — the same disk imported as a stream-optimized VMDK with BIOS firmware boots fine; added with the practice of verifying boot on a throwaway instance before recreating the cluster. cnpg webhook: on Cilium + KubeOVN the cloudnative-pg operator flaps on leader-election during early convergence and restarts with a new pod IP, and Cilium's host-side service BPF keeps the stale endpoint, so the apiserver gets connection refused on the webhook ClusterIP (
failurePolicy: Failblocks Keycloak and every postgres-backed release) while pods reach it fine; added the host-network-vs-pod-network diagnostic and the operator rollout-restart recovery.The claims were checked against the monorepo source (engine variant selection, the oidc-variant dashboard→keycloak-configure dependency, the e2e install sequence,
incloud-web-gatekeepernaming, and the v1.5.0 token-proxy startupProbe fix).Summary by CodeRabbit
Documentation
Chores