Merge dind mode template overrides for generated runner containers#4567
Open
kiarashazarnia wants to merge 2 commits into
Open
Merge dind mode template overrides for generated runner containers#4567kiarashazarnia wants to merge 2 commits into
kiarashazarnia wants to merge 2 commits into
Conversation
kiarashazarnia
requested review from
Steve-Glass,
mumoshu,
nikola-jokic,
rentziass and
toast-gear
as code owners
July 12, 2026 20:49
…nals and filter duplicate initContainers
edbzn
added a commit
to edbzn/k8s-bare-metal-ci
that referenced
this pull request
Jul 24, 2026
A concurrent-load test (20 workflow dispatches, both scale-sets at maxRunners:3 = 6 concurrent kata-fc pods) genuinely OOM'd ci-worker2: every runner container had zero memory requests/limits, so the scheduler had no way to know 3 microVMs wouldn't fit on a 2.8GB node - it kept stacking them until the guest kernel itself started thrashing (load average 114 inside a 2-vCPU VM, 83MB free out of 2.8GB, zero swap). The node went fully unresponsive (NotReady, SSH timing out, even the QEMU guest agent disappearing) and needed a hard virsh reset, after which the devmapper thin-pool needed manually recreating again before kubelet would start. Fixed at the root: added resources.requests/limits to arc-runner-set's runner container (500m/1Gi) and arc-runner-set-dind's runner container (500m/512Mi), plus a namespace-wide LimitRange in arc-runners (500m/1Gi default) for the dind scale-set's auto-generated dind/ init-dind-externals containers - which cannot receive resources via the chart's own values.yaml at all in gha-runner-scale-set 0.14.2 (confirmed real, open upstream bug: actions/actions-runner-controller#4567 - only the runner container gets field-level override merging; any dind-named entry we supply gets appended as a raw duplicate instead, failing server-side-apply). Re-ran the identical 20-dispatch wave after the fix: jobs correctly queued/drained instead of stacking, all 5 nodes stayed Ready throughout, host load stayed under 5 instead of spiking past 11. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When using
containerMode.type: dind, thegha-runner-scale-setchart auto-generatesrunner,dind, andinit-dind-externalscontainers. Previously, user template overrides were inconsistent:runneroverrides fromtemplate.spec.containersalready workeddindoverrides (e.g.resources) were ignoredinit-dind-externalsoverrides were ignoredtemplate.spec.initContainersentries for generateddind/init-dind-externalscould be appended as duplicatesChanges
dindcontainer from:init-dind-externalsfrom:dind/init-dind-externalsout of appended user initContainers to avoid duplicatesmergeOverwritewhen combining dind overrides from both containers and initContainersvalues.yamlExample
Test plan
go test ./charts/gha-runner-scale-set/... -vNotes