Skip to content

fix(cilium): limit concurrent agent and Envoy rollouts - #4065

Merged
devantler merged 2 commits into
mainfrom
codex/4064-cilium-rollout
Sep 23, 2026
Merged

devantler merged 2 commits into
mainfrom
codex/4064-cilium-rollout

Conversation

@devantler

@devantler devantler commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

During the Cilium 1.20.2 rollout, production observability jobs hit DNS and service timeouts and left failed-job warnings. The chart currently permits two networking agents and two Envoy pods to be unavailable at once. That concurrency increases the disruption risk during planned upgrades, even though the exact packet path of each timeout is not proven.

What

Keep normal Helm-managed rolling upgrades, but limit each Cilium agent and Envoy DaemonSet to one unavailable pod at a time. Replace the old test that required the chart default with a production-render assertion for the new bounds. The old temporary rollout handoff and disableWait stay absent; no Coroot error checks are weakened.

The production authorization gate correctly detected the changed Cilium HelmRelease. Its approved CI renderer reported exactly that one changed resource and no added or removed resource; the aggregate fingerprint is re-approved with the previous digest recorded. No identity or grant changes.

Fixes #4064

@devantler devantler left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated by the Agentic Engineer

Exact-head review of aaa8511d6db480aadd64d8c8eb9e23261e13502b: the Cilium 1.20.2 chart exposes both updateStrategy and envoy.updateStrategy, each defaulting to maxUnavailable: 2; the production render now sets both to RollingUpdate/1. The existing CI-registered activation test failed against the baseline and passed after the change. Its assertions still reject the old Helm disableWait handoff, and the separate autoscaler-gate test passed. Shellcheck and git diff --check passed. No RBAC, policy, Coroot suppression, or image change appears in this diff.

No blocking code findings. This is a concurrency reduction, not proof of zero packet loss: the two DaemonSets can still overlap on different nodes, so post-deployment readiness and Coroot readback remain required.

@devantler devantler left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated by the Agentic Engineer

Exact-head review of f53276dc1c9f6a08eff8f484763e04fa8919de88 against bad3342cdaaaaa70510847ae942c2d70b9a9cdf0: the authored manifest changes only the agent and Envoy DaemonSet rollout values (each RollingUpdate/maxUnavailable=1); the test checks both fields in the production render and still rejects the old disableWait handoff. The first CI run's authorization gate correctly failed on the changed HelmRelease. Its approved renderer measured aggregate 33eb2b3a1201e8c6c304eab0bb65feed702d8244ed96ebb315b90f3901d95792 and diagnosed only HelmRelease kube-system/cilium as changed; the new commit records that digest and the exact prior digest 65fecc82a527f6b5fc9c2154b58d2268d4eb4cfa49bb2ec9dfdfd65789621250. Local authorization Go tests now pass with the new value. No grant-bearing object, service account, binding, policy, or AWS identity changed.

No blocking code findings. Await the rerun of the required CI gate and production readiness/readback before treating the risk reduction as delivered. The two DaemonSets can still roll on different nodes simultaneously; this does not establish zero packet loss.

@devantler
devantler marked this pull request as ready for review September 23, 2026 05:33
@devantler
devantler added this pull request to the merge queue Sep 23, 2026
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

@devantler I’ll review the changes in #4065.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 0faf039d-e194-4a9c-b783-53c38baf3e88

📥 Commits

Reviewing files that changed from the base of the PR and between bad3342 and f53276d.

📒 Files selected for processing (3)
  • k8s/bases/infrastructure/controllers/cilium/helm-release.yaml
  • scripts/tests/test-cilium-homogeneous-devices-activation.sh
  • scripts/validate-eks-ci-role-policy/main.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details
🔇 Additional comments (3)
k8s/bases/infrastructure/controllers/cilium/helm-release.yaml (1)

42-46: LGTM!

Also applies to: 48-51, 233-236

scripts/tests/test-cilium-homogeneous-devices-activation.sh (1)

202-211: LGTM!

scripts/validate-eks-ci-role-policy/main.go (1)

3150-3160: LGTM!

Also applies to: 3173-3173


📝 Walkthrough

Walkthrough

The Cilium HelmRelease now sets RollingUpdate with maxUnavailable: 1 for the agent and Envoy DaemonSets. The activation test verifies both rendered settings. The EKS CI policy validator updates the approved rendered-surface hash and the superseded hash record.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to f5327

Both Cilium DaemonSets are configured for bounded rollouts, and no actionable issue remains before normal merge checks.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: limiting concurrent Cilium agent and Envoy rollouts.
Description check ✅ Passed The description directly explains the rollout limit, test updates, authorization approval, and scope of the Cilium changes.
Linked Issues check ✅ Passed [#4064] The production HelmRelease sets updateStrategy.type: RollingUpdate and rollingUpdate.maxUnavailable: 1 for both the Cilium agent and Envoy DaemonSets. The production-render test asserts bo…
Out of Scope Changes check ✅ Passed The changes are limited to the Cilium rollout settings, their production-render test, and the rendered-surface approval fingerprint. The fingerprint update supports authorization of the changed HelmRe…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Merged via the queue into main with commit c4fb12c Sep 23, 2026
30 checks passed
@devantler
devantler deleted the codex/4064-cilium-rollout branch September 23, 2026 09:01
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

Keep observability reachable during Cilium rollouts

1 participant