APS-4720: Stage edge signing secrets separately from Kong restart

[Elson9]

Summary

• Split edge key bootstrap from Kong restart: bootstrap.stageSecret writes {release}-client-next without restarting.
• rotation.promote copies the staged secret to live client/server TLS secrets and rolling-restarts Kong (chart 0.3.6).

Test plan

• stageSecret=true creates -client-next and skip Kong restart (helm template)
• rotation.promote=true promotes next → live and restarts (helm template)
• Default renewal still restarts (helm template)
• Cluster apply of stage/promote on a real runtime group

[phowells]

Could we clear bootstrap.tls.token as part of this promotion upgrade, or otherwise remove and recreate the bootstrap Job? --reuse-values keeps the staged token, so the same *-boot-<token hash> Job remains rendered while stageSecret changes its pod template from staging to live-secret writes. Kubernetes Job pod templates are immutable, so a stage-to-promote upgrade will be rejected while that completed Job still exists. If TTL has already removed it, Helm can instead recreate the bootstrap Job with the consumed one-time token. A consecutive stage/promote test would help pin this down.

[phowells]

Thanks for tackling this. I traced the --reuse-values merge in Helm 3.20, and null unfortunately does not clear a value stored by the previous release: the null entry is removed during coalescing, then the old token is supplied again when Helm renders. That means this command still renders the same bootstrap Job with stageSecret=false and runs into the immutable pod template. Could we use an explicit empty string instead (--set-string bootstrap.tls.token="") and cover a real stage-to-promote reuse-values sequence? An empty string remains an explicit override and rendered correctly in my check.

[phowells]

Thanks for tackling this. I traced the --reuse-values merge in Helm 3.20, and null unfortunately does not clear a value stored by the previous release: the null entry is removed during coalescing, then the old token is supplied again when Helm renders. That means this command still renders the same bootstrap Job with stageSecret=false and runs into the immutable pod template. Could we use an explicit empty string instead (--set-string bootstrap.tls.token="") and cover a real stage-to-promote reuse-values sequence? An empty string remains an explicit override and rendered correctly in my check.

[phowells]

The staging Job already runs step ca sign and stores only tls.crt and tls.key in client-next; it does not leave a CSR for an operator to sign afterward. I think this should explain how to extract tls.crt and pass it as certificatePem to operation=rotate, or point to the automation that performs that handoff.