Skip to content

Merge dind mode template overrides for generated runner containers#4567

Open
kiarashazarnia wants to merge 2 commits into
actions:masterfrom
kiarashazarnia:kiarashazarnia/dind-resources-merge-spec
Open

Merge dind mode template overrides for generated runner containers#4567
kiarashazarnia wants to merge 2 commits into
actions:masterfrom
kiarashazarnia:kiarashazarnia/dind-resources-merge-spec

Conversation

@kiarashazarnia

@kiarashazarnia kiarashazarnia commented Jul 12, 2026

Copy link
Copy Markdown

Summary

When using containerMode.type: dind, the gha-runner-scale-set chart auto-generates runner, dind, and init-dind-externals containers. Previously, user template overrides were inconsistent:

  • runner overrides from template.spec.containers already worked
  • dind overrides (e.g. resources) were ignored
  • init-dind-externals overrides were ignored
  • user template.spec.initContainers entries for generated dind / init-dind-externals could be appended as duplicates

Changes

  • Merge user overrides into generated dind container from:
  • Merge user overrides into generated init-dind-externals from:
  • Filter generated dind / init-dind-externals out of appended user initContainers to avoid duplicates
  • Use mergeOverwrite when combining dind overrides from both containers and initContainers
  • Document customization paths in values.yaml
  • Add regression tests for resource merge, default-empty resources, init-dind-externals merge, and duplicate filtering

Example

containerMode:
  type: dind

template:
  spec:
    initContainers:
    - name: init-dind-externals
      resources:
        requests:
          cpu: "50m"
          memory: "128Mi"
    containers:
    - name: runner
      resources:
        requests:
          cpu: "500m"
    - name: dind
      resources:
        requests:
          cpu: "100m"
          memory: "256Mi"
        limits:
          cpu: "200m"

Test plan

  • go test ./charts/gha-runner-scale-set/... -v
  • (gha) Validate Helm Charts CI passes

Notes

  • No chart version bump (per CONTRIBUTING.md)
  • No controller/runtime changes; ARC already passes the rendered pod template through to runner pods
  • Backward compatible: if no override entries are provided, rendered pod specs are unchanged

@kiarashazarnia kiarashazarnia changed the title Merge dind container resources from template in dind mode Merge dind mode template overrides for generated runner containers Jul 12, 2026
edbzn added a commit to edbzn/k8s-bare-metal-ci that referenced this pull request Jul 24, 2026
A concurrent-load test (20 workflow dispatches, both scale-sets at
maxRunners:3 = 6 concurrent kata-fc pods) genuinely OOM'd ci-worker2:
every runner container had zero memory requests/limits, so the
scheduler had no way to know 3 microVMs wouldn't fit on a 2.8GB node -
it kept stacking them until the guest kernel itself started thrashing
(load average 114 inside a 2-vCPU VM, 83MB free out of 2.8GB, zero
swap). The node went fully unresponsive (NotReady, SSH timing out,
even the QEMU guest agent disappearing) and needed a hard virsh reset,
after which the devmapper thin-pool needed manually recreating again
before kubelet would start.

Fixed at the root: added resources.requests/limits to arc-runner-set's
runner container (500m/1Gi) and arc-runner-set-dind's runner container
(500m/512Mi), plus a namespace-wide LimitRange in arc-runners
(500m/1Gi default) for the dind scale-set's auto-generated dind/
init-dind-externals containers - which cannot receive resources via
the chart's own values.yaml at all in gha-runner-scale-set 0.14.2
(confirmed real, open upstream bug:
actions/actions-runner-controller#4567 - only the runner container
gets field-level override merging; any dind-named entry we supply
gets appended as a raw duplicate instead, failing server-side-apply).

Re-ran the identical 20-dispatch wave after the fix: jobs correctly
queued/drained instead of stacking, all 5 nodes stayed Ready
throughout, host load stayed under 5 instead of spiking past 11.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant