From 976fc105d3c3ce2f56cb0552939ee9136d435d72 Mon Sep 17 00:00:00 2001 From: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Date: Thu, 20 Aug 2026 21:43:59 +0000 Subject: [PATCH 1/7] Add DatadogPodAutoscaler manifest reference page and update autoscaling overview Co-authored-by: clamoriniere --- hugo/config/_default/menus/main.en.yaml | 5 + .../en/containers/autoscaling/_index.md | 84 ++- .../en/containers/autoscaling/manifest.md | 690 ++++++++++++++++++ 3 files changed, 767 insertions(+), 12 deletions(-) create mode 100644 hugo/content/en/containers/autoscaling/manifest.md diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index a2761dfa87f..e08bf16b02a 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -3312,6 +3312,11 @@ menu: parent: containers identifier: containers_autoscaling weight: 2 + - name: Manifest reference + url: containers/autoscaling/manifest + parent: containers_autoscaling + identifier: containers_autoscaling_manifest + weight: 200 - name: Cluster url: containers/autoscaling/cluster parent: containers_autoscaling diff --git a/hugo/content/en/containers/autoscaling/_index.md b/hugo/content/en/containers/autoscaling/_index.md index 418ce15adbe..ef0536d2a4e 100644 --- a/hugo/content/en/containers/autoscaling/_index.md +++ b/hugo/content/en/containers/autoscaling/_index.md @@ -170,6 +170,8 @@ helm upgrade -f datadog-values.yaml datadog/datadog {{% /tab %}} {{< /tabs >}} +**Note**: Vertical scaling recommendations are applied to new pods through the [Admission Controller](/containers/cluster_agent/admission_controller/) mutating webhook. The Admission Controller is enabled by default. If you disable it, horizontal scaling continues to work but vertical scaling has no effect. + ### Idle cost and savings estimates {{< tabs >}} @@ -210,6 +212,63 @@ _Fixed cost values are subject to refinement over time._ {{% /tab %}} {{< /tabs >}} +### In-place vertical scaling + +By default, applying a vertical recommendation requires a full pod rollout: the pod template is updated, Kubernetes recreates the pods, and the new resources take effect as those pods are admitted. For slow-starting or latency-sensitive services, that is a meaningful disruption. + +In-place vertical scaling instead updates container resources on the running pods through the Kubernetes [pod resize subresource](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/), so most resizes happen with no restart. This requires Datadog Cluster Agent 7.78+. + +With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling for a workload, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. To tune how long the controller waits before evicting a pending resize or falling back to a rollout, see the [DatadogPodAutoscaler manifest reference][15]. + +#### Kubernetes support + +The `InPlacePodVerticalScaling` feature gate must be enabled on **both** the API server and the kubelet; a cluster can have one without the other. + +| Kubernetes version | Feature gate | +|---|---| +| 1.27 - 1.32 | Alpha, disabled by default | +| 1.33 - 1.34 | Beta, enabled by default | +| 1.35+ | GA, gate removed | + +#### Enable in-place vertical scaling + +Enable the feature on the Cluster Agent. This also grants the Cluster Agent the permissions it needs to resize and, where necessary, evict pods. + +{{< tabs >}} +{{% tab "Datadog Operator" %}} + +```yaml +spec: + features: + autoscaling: + workload: + enabled: true + inPlaceVerticalScaling: + enabled: true +``` + +{{% /tab %}} +{{% tab "Helm" %}} + +```yaml +datadog: + autoscaling: + workload: + enabled: true + inPlaceVerticalScaling: + enabled: true +``` + +{{% /tab %}} +{{< /tabs >}} + +#### Behavior and limitations + +- **`resizePolicy` stays under your control.** Datadog never sets or overrides the container-level `resizePolicy`; it is immutable after pod creation and is an application-level decision. When unset, Kubernetes defaults to `NotRequired` for CPU and memory, meaning resize without restart. Set `RestartContainer` per resource on containers that cannot absorb a live change. +- **Kubernetes limitations apply.** Only CPU and memory can be resized; QoS class cannot change; requests and limits can be changed but not removed entirely; Windows pods and pods under static CPU or memory manager policies are excluded. See the [upstream limitations](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#limitations). +- **Burstable mode still requires a rollout.** Burstable mode removes the CPU limit, and in-place resize can change a limit but not remove one. See the [DatadogPodAutoscaler manifest reference][15]. +- If a resize is infeasible or stays pending, the pod is evicted through the Kubernetes Eviction API, which respects PodDisruptionBudgets. + ## Usage ### Identify resources to rightsize @@ -249,7 +308,7 @@ The Setup wizard is best for trying autoscaling on a single workload, getting ha #### Path B: GitOps -Define a `DatadogPodAutoscaler` custom resource that targets your workload and apply it through whatever tooling you already use to ship Kubernetes manifests, whether that's `kubectl apply`, Helm, ArgoCD, Terraform, or another GitOps tool. Authoring the manifest is the same regardless of delivery mechanism. See the [example configurations](#example-datadogpodautoscaler-configurations) below for ready-to-edit starting points covering cost optimization, balanced scaling, vertical-only resizing, and custom-query horizontal scaling. +Define a `DatadogPodAutoscaler` custom resource that targets your workload and apply it through whatever tooling you already use to ship Kubernetes manifests, whether that's `kubectl apply`, Helm, ArgoCD, Terraform, or another GitOps tool. Authoring the manifest is the same regardless of delivery mechanism. See the [example configurations](#example-datadogpodautoscaler-configurations) below for ready-to-edit starting points covering cost optimization, balanced scaling, vertical-only resizing, and custom-query horizontal scaling. For a complete reference of the manifest fields and options, including options the UI does not expose, see the [DatadogPodAutoscaler manifest reference][15]. For tool-specific guides, see: @@ -258,7 +317,7 @@ For tool-specific guides, see: ### Example DatadogPodAutoscaler configurations -The following examples demonstrate common `DatadogPodAutoscaler` configurations for different scaling strategies. Use them as starting points and adjust the values to match your workload's requirements. If you would rather pick a template in the UI, follow [Path A](#path-a-datadog-ui-setup-wizard) above. +The following examples demonstrate common `DatadogPodAutoscaler` configurations for different scaling strategies. Use them as starting points and adjust the values to match your workload's requirements. If you would rather pick a template in the UI, follow [Path A](#path-a-datadog-ui-setup-wizard) above. For the full range of manifest fields and options not shown here, see the [DatadogPodAutoscaler manifest reference][15]. {{< tabs >}} {{% tab "Optimize Cost" %}} @@ -361,15 +420,7 @@ spec: Pick this template when a workload can't be scaled horizontally, or when you want pure rightsizing without changing replica counts. Common cases are singleton services, stateful workloads, and leader-elected components. The defining setting is `scaleDown.strategy: Disabled` and `scaleUp.strategy: Disabled`, which leaves only `update.strategy: Auto` to apply CPU and memory recommendations. -By default, the controller applies vertical recommendations by triggering a rollout (evict and recreate pods). Cluster Agent **7.78+** also supports **in-place pod resizing**, which updates a pod's CPU and memory requests and limits without restarting it. In-place resize is opt-in: set `autoscaling.workload.in_place_vertical_scaling.enabled: true` on the Cluster Agent (or set the environment variable `DD_AUTOSCALING_WORKLOAD_IN_PLACE_VERTICAL_SCALING_ENABLED=true`). - -Your cluster must also expose the `pods/resize` subresource. This is the default in Kubernetes 1.33+ where the `InPlacePodVerticalScaling` feature gate is beta. On Kubernetes 1.27 to 1.32, the feature gate must be enabled on `kube-apiserver` and every `kubelet`. - -When both prerequisites are met: - -- **Default**: Workloads with `applyPolicy.update.strategy: Auto` (the default) resize in place. -- **Fallback**: If the kubelet reports a resize as `Infeasible`, the controller falls back to a rollout. -- **Opt-out**: To force a workload to always use rollout-based vertical scaling regardless of the cluster setting, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. +By default, the controller applies vertical recommendations by triggering a rollout (evict and recreate pods). Cluster Agent **7.78+** also supports **in-place pod resizing**, which updates a pod's CPU and memory requests and limits without restarting it. With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling, set `applyPolicy.update.strategy: TriggerRollout`. For enablement, Kubernetes requirements, and behavior, see [In-place vertical scaling](#in-place-vertical-scaling). ```yaml apiVersion: datadoghq.com/v1alpha2 @@ -469,6 +520,14 @@ spec: {{% /tab %}} {{< /tabs >}} +The examples above cover the most common strategies. The manifest supports additional options that the templates don't show, including: + +- **CPU rightsizing** alongside memory (`constraints.containers[].controlledResources`). When a workload combines horizontal and vertical scaling, vertical recommendations cover memory only by default. +- **Burstable mode** (`spec.options.burstable`) to remove CPU limits on spiky workloads while still right-sizing CPU requests. +- **Per-container bounds** (`minAllowed` and `maxAllowed`), request-only rightsizing (`controlledValues: RequestsOnly`), and per-container targeting for horizontal scaling (`ContainerResource` objectives). + +For the full field reference, see the [DatadogPodAutoscaler manifest reference][15]. + ### Cluster profiles A `DatadogPodAutoscalerClusterProfile` is a cluster-scoped resource that holds a `DatadogPodAutoscaler` template. The Cluster Agent watches `Deployment` and `StatefulSet` resources (and, on 7.79+, the namespaces that contain them) for the `autoscaling.datadoghq.com/profile` label, and creates a managed `DatadogPodAutoscaler` for every matching workload. One profile applies to many workloads; one workload still maps to one `DatadogPodAutoscaler`. @@ -636,7 +695,7 @@ Datadog computes vertical scaling recommendations for CPU and memory by analyzin - **8-day lookback window**: All recommendations consider usage data from the past 8 days, providing enough history to capture weekly traffic patterns while remaining responsive to changes. - **Decaying weights**: For Burstable-class request recommendations (CPU or memory), older samples are weighted less heavily, so the recommendation adapts faster to recent usage shifts. - **Safety margins**: Every recommendation includes a margin above observed usage (5 to 10%) to provide a buffer against unexpected spikes. -- **OOMKill response**: When memory is Guaranteed-class (request equals limit) and an OOMKill occurs, a 20% bump is applied to reduce the likelihood of repeated out-of-memory failures. +- **OOMKill response**: When an OOMKill occurs, the memory limit is raised (by 20% by default) and re-applied on each subsequent OOMKill until the workload stabilizes, reducing the likelihood of repeated out-of-memory failures. Change the ratio with `spec.options.outOfMemory.bumpUpRatio`; see the [DatadogPodAutoscaler manifest reference][15]. - **Guaranteed-class preservation**: When a resource has request equal to limit, Datadog uses the more conservative (limit-level) computation for both, ensuring recommendations do not introduce a gap between request and limit. ## Further reading @@ -657,3 +716,4 @@ Datadog computes vertical scaling recommendations for CPU and memory by analyzin [12]: /containers/guide/manage-datadogpodautoscaler-with-argocd/ [13]: /containers/guide/manage-datdadogpodautoscaler-with-terraform/ [14]: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ +[15]: /containers/autoscaling/manifest/ diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md new file mode 100644 index 00000000000..a40d63d7fef --- /dev/null +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -0,0 +1,690 @@ +--- +title: DatadogPodAutoscaler manifest reference +description: Configure DatadogPodAutoscaler custom resources in YAML to access options that the Datadog UI does not expose. +further_reading: +- link: "/containers/autoscaling/" + tag: "Documentation" + text: "Kubernetes Autoscaling" +- link: "/containers/guide/manage-datadogpodautoscaler-with-argocd/" + tag: "Documentation" + text: "Manage DatadogPodAutoscaler with ArgoCD" +- link: "/containers/guide/manage-datdadogpodautoscaler-with-terraform/" + tag: "Documentation" + text: "Manage DatadogPodAutoscaler with Terraform" +--- + +The `DatadogPodAutoscaler` (DPA) custom resource defines autoscaling behavior for a single Kubernetes workload. The [Autoscaling UI][1] with {{< ui >}}Export Recommendation{{< /ui >}} is a good place to start: configure a workload, then copy the generated manifest. Editing the manifest directly gives you access to every field in the custom resource definition (CRD) and makes the DPA a normal part of your GitOps workflow, where the manifest is the reviewed, versioned source of truth. + +This page covers the configuration options available in the manifest. It uses API version `datadoghq.com/v1alpha2`. + +For setup and prerequisites—enabling Workload Autoscaling and the Admission Controller on the Datadog Cluster Agent, required Agent versions, and enabling [in-place vertical scaling][3]—see [Kubernetes Autoscaling][2]. + +**Feature status labels:** features on this page are generally available unless marked **Preview**, **Beta**, or **Experimental**. Preview and Beta features are available and supported, but their field names and behavior may still change. Experimental features are available but not recommended for production without contacting [Datadog Support][9] first. + +## Anatomy of a manifest + +The following annotated skeleton shows the structure of a `DatadogPodAutoscaler`. Every field is optional except `targetRef`. + +```yaml +apiVersion: datadoghq.com/v1alpha2 +kind: DatadogPodAutoscaler +metadata: + name: my-app # conventionally the workload name + namespace: my-namespace # must match the target workload + annotations: + ad.datadoghq.com/tags: '{"team": "my-team"}' # optional: tags on this DPA's telemetry +spec: + owner: Local # Local = this manifest is the source of truth (use for GitOps) + # Remote = created and managed from the Datadog UI + + targetRef: # the workload being autoscaled - one DPA per workload + apiVersion: apps/v1 + kind: Deployment + name: my-app + + applyPolicy: + mode: Apply # Apply | Preview (Preview = compute recommendations, change nothing) + + scaleUp: # horizontal, upward + strategy: Max # Max | Min | Disabled + stabilizationWindowSeconds: 600 + rules: + - type: Percent # Percent | Pods + value: 50 + periodSeconds: 120 # 1..3600 + + scaleDown: # horizontal, downward + strategy: Max + stabilizationWindowSeconds: 600 + rules: + - type: Percent + value: 10 + periodSeconds: 1800 + + update: # vertical + strategy: Auto # Auto | Disabled | TriggerRollout + # resizePendingPeriod: 600 # see Vertical rollout timing + # rolloutFallbackDelay: 900 # see Vertical rollout timing + + constraints: + minReplicas: 3 + maxReplicas: 100 + containers: # per-container vertical configuration + - name: "*" # "*" matches all containers + enabled: true + controlledResources: [cpu, memory] + controlledValues: RequestsAndLimits # RequestsAndLimits | RequestsOnly + minAllowed: + cpu: "500m" + memory: 1Gi + maxAllowed: + cpu: "4" + memory: 8Gi + + objectives: # HORIZONTAL only. Exactly one entry. + - type: ContainerResource # PodResource | ContainerResource | CustomQuery + containerResource: + container: my-app + name: cpu # cpu | memory + value: + type: Utilization # Utilization | AbsoluteValue + utilization: 65 + + fallback: # in-cluster horizontal fallback if recommendations go stale + horizontal: + enabled: true + direction: ScaleUp # ScaleUp | ScaleDown | All (default ScaleUp) + triggers: + staleRecommendationThresholdSeconds: 600 # 100..3600, default 600 + + options: + burstable: false # true = remove CPU limits, keep CPU request recommendations + outOfMemory: + bumpUpRatio: "1.2" # +20% memory limit after an OOMKill (default) +``` + +### Supported target workloads + +| `targetRef.kind` | `apiVersion` | Status | +|---|---|---| +| `Deployment` | `apps/v1` | Supported | +| `Rollout` (Argo Rollouts) | `argoproj.io/v1alpha1` | Supported | +| `StatefulSet` | `apps/v1` | Supported | + +For an Argo Rollout, point `targetRef` at the Rollout itself rather than at any Deployment it manages: + +```yaml + targetRef: + apiVersion: argoproj.io/v1alpha1 + kind: Rollout + name: my-app +``` + +### Choose a scaling mode + +The combination of `objectives` and `applyPolicy.update.strategy` determines whether a DPA scales horizontally, vertically, or both: + +| `objectives` set | `applyPolicy.update.strategy` | Resulting mode | +|---|---|---| +| yes | `Disabled` (or unset) | Horizontal only | +| no | `Auto` | Vertical only | +| yes | `Auto` | Multidimensional (both) | + +## Container constraints + +Most vertical options are expressed through `spec.constraints.containers[]`: + +| Field | Type | Default | Meaning | +|---|---|---|---| +| `name` | string, **required** | — | Container name, or `"*"` to match every container that has no entry of its own (see [Exclude a container](#exclude-a-container)) | +| `enabled` | bool | `true` | `false` disables resource autoscaling for this container | +| `controlledResources` | list of `cpu`, `memory` | `[cpu, memory]` | Which resources receive vertical recommendations. An empty list is equivalent to `enabled: false` | +| `controlledValues` | enum | `RequestsAndLimits` | Whether recommendations write both requests _and_ limits, or requests only | +| `minAllowed` | resource map | — | Lower bound for the container's requests | +| `maxAllowed` | resource map | — | Upper bound for the container's requests | + +If `constraints.containers` is omitted entirely, resource scaling is enabled for **all** containers, with no bounds. + +## Right-size CPU and memory + +**Status: Preview.** + +When a DPA combines horizontal scaling (`objectives`) with vertical scaling (`update.strategy: Auto`), the default behavior is to produce vertical recommendations for **memory only**. CPU requests and limits are left untouched, and the `VerticalAbleToRecommend` condition may show as `Unknown`. + +If you have seen a DPA right-size memory while leaving CPU unchanged, this is the reason. It is not related to Quality of Service (QoS) class preservation. + +Use `controlledResources` to declare which resources receive vertical recommendations: + +| `controlledResources` | Vertical recommendations produced | +|---|---| +| unset | memory only (default) | +| `[memory]` | memory only, same as unset | +| `[cpu, memory]` | **memory and CPU** | +| `[cpu]` | CPU only | + +Full example: + +```yaml +apiVersion: datadoghq.com/v1alpha2 +kind: DatadogPodAutoscaler +metadata: + name: my-app + namespace: my-namespace +spec: + owner: Local + targetRef: + apiVersion: apps/v1 + kind: Deployment + name: my-app + applyPolicy: + mode: Apply + update: + strategy: Auto # required - without it, nothing vertical is applied + constraints: + minReplicas: 3 + maxReplicas: 60 + containers: + - name: "*" + controlledResources: + - cpu # opts CPU into vertical rightsizing + - memory + controlledValues: RequestsAndLimits + objectives: + - type: ContainerResource + containerResource: + container: my-app + name: cpu + value: + type: Utilization + utilization: 65 +``` + +This requires a recent Agent version; see the requirements in [Kubernetes Autoscaling][2]. On older Agents, the field is accepted by the CRD but has no effect. + +## Remove CPU limits with burstable mode + +**Status: Beta.** + +CPU limit recommendations are derived from sustained usage percentiles over a multi-day window. A short warm-up spike—a JVM starting up, for example—is statistically invisible in that window, so the recommended CPU limit can land too low and the application is throttled at the wrong moment. Memory is not affected in the same way, because peak memory usage gives a reliable ceiling. + +Burstable mode **removes CPU limits entirely** while still applying CPU _request_ recommendations: + +```yaml +apiVersion: datadoghq.com/v1alpha2 +kind: DatadogPodAutoscaler +metadata: + name: my-java-app + namespace: my-namespace +spec: + owner: Local + targetRef: + apiVersion: apps/v1 + kind: Deployment + name: my-java-app + applyPolicy: + mode: Apply + update: + strategy: Auto + options: + burstable: true +``` + +If `options.burstable` is left unset, the Cluster Agent's cluster-wide default applies. Set it explicitly to `false` to opt a single workload out of that default. + +Effect on the pod: + +| | Before | After | +|---|---|---| +| CPU request | `1` | `400m` (recommendation) | +| CPU limit | `2` | **removed** | +| Memory request | `500Mi` | `450Mi` (recommendation) | +| Memory limit | `2Gi` | `2Gi`, preserved | + +Before enabling it: + +- Pods that were **Guaranteed** QoS become **Burstable** QoS. This changes their eviction priority under node pressure. +- Without a CPU limit, a container can consume available node CPU. Kernel CPU shares still apply. +- Burstable mode **takes precedence over** `controlledValues` for CPU limits. If both are set, burstable wins. + +### Apply burstable mode to many workloads + +To onboard a whole category of workloads (every JVM service, for example), set the option in a `DatadogPodAutoscalerClusterProfile` template. It propagates to every DPA the profile generates, and removing it propagates the removal. For more information about profiles, see [Cluster profiles][4]. + +```yaml +apiVersion: datadoghq.com/v1alpha2 +kind: DatadogPodAutoscalerClusterProfile +metadata: + name: java-burstable +spec: + template: + options: + burstable: true + applyPolicy: + mode: Apply + update: + strategy: Auto + constraints: + minReplicas: 2 + maxReplicas: 25 + containers: + - name: "*" + enabled: true + objectives: + - type: PodResource + podResource: + name: cpu + value: + type: Utilization + utilization: 80 +``` + +Then label the workloads that use it: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: order-service + namespace: production + labels: + autoscaling.datadoghq.com/profile: java-burstable +``` + +To verify, run `kubectl -n production get dpa -o yaml` and confirm the generated DPA carries `spec.options.burstable: true`. + +## Tune the OOMKill memory bump + +After an OOMKill, the memory limit is raised by **20%** relative to the limit in force at the time. The bump is applied immediately and re-applied on each subsequent OOMKill until the workload stabilizes—a self-correcting ratchet rather than a one-time adjustment. + +To change the ratio: + +```yaml +spec: + options: + outOfMemory: + bumpUpRatio: "1.5" # 1.2 = +20% (default), 1.5 = +50% +``` + +Quote the value: it is a Kubernetes quantity, not a floating-point number. + +**When to raise it.** If a container runs out of memory very fast (roughly **under 30 seconds**), the Agent may not capture the memory peak before the restart, so recommendations cannot ratchet upward and the crash loop persists. + +If you hit that situation: + +1. Confirm the OOM check is enabled on the Agent. +2. Raise `bumpUpRatio`. +3. Set a `minAllowed` memory floor (see [Set per-container bounds](#set-per-container-bounds)) so the limit cannot fall back below a safe value between recommendation cycles. + +## Right-size requests only + +If you have deliberately tuned limits—for burst headroom, a platform requirement, or a QoS guarantee—and want Datadog to right-size only **requests**, use `controlledValues: RequestsOnly`. + +```yaml +spec: + constraints: + containers: + - name: my-app + controlledResources: [cpu, memory] + controlledValues: RequestsOnly # limits are not right-sized +``` + +| `controlledValues` | Requests | Limits | +|---|---|---| +| `RequestsAndLimits` (default) | recommended | recommended | +| `RequestsOnly` | recommended | not right-sized, **except** where a limit must move to keep the pod spec valid (see below) | + +Interactions to be aware of: + +- On a container where `request == limit`, lowering the request breaks the Guaranteed QoS class. If you need Guaranteed, keep `RequestsAndLimits`; the recommender handles `request == limit` containers explicitly. +- Burstable mode overrides this for CPU limits (see [Remove CPU limits with burstable mode](#remove-cpu-limits-with-burstable-mode)). +- **OOMKill handling still adjusts the memory limit.** `RequestsOnly` does not suppress the memory bump, and cannot: Kubernetes rejects a container whose request exceeds its limit, so raising the request without raising the limit would produce an invalid pod. Read `RequestsOnly` as "limits are not _right-sized_", not "limits are never modified". See [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump). + +Choosing a combination: + +| Goal | Configuration | +|---|---| +| Right-size everything | `controlledResources: [cpu, memory]` + `controlledValues: RequestsAndLimits` | +| Right-size requests, leave limits as written | `controlledValues: RequestsOnly` | +| Right-size memory only, leave CPU alone | `controlledResources: [memory]` | +| Right-size CPU requests, no CPU limit at all | `options.burstable: true` | +| Leave a container entirely alone | `enabled: false` | + +## Set per-container bounds + +`minAllowed` and `maxAllowed` are guard-rails on the requests the recommender may produce. They are recommended for latency-sensitive workloads, and advisable whenever you change the OOM bump ratio (see [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump)), so memory cannot drift below a safe floor between recommendation cycles. + +```yaml +spec: + constraints: + minReplicas: 2 + maxReplicas: 100 + containers: + - name: api + enabled: true + minAllowed: + cpu: "1" + memory: 1Gi + maxAllowed: + cpu: "4" + memory: 5Gi + - name: worker + enabled: true # no bounds - recommendations are unconstrained +``` + +## Exclude a container + +There are two distinct things you may want to exclude a container from. + +### Exclude a container from vertical recommendations + +```yaml +spec: + constraints: + containers: + - name: my-app + enabled: true + - name: istio-proxy + enabled: false # resources for this container are never modified +``` + +Equivalent, if you prefer to be explicit: + +```yaml + - name: istio-proxy + controlledResources: [] # empty list is equivalent to enabled: false +``` + +A common pattern is to autoscale everything except a known sidecar: + +```yaml +spec: + constraints: + containers: + - name: "*" + enabled: true + controlledResources: [cpu, memory] + - name: istio-proxy + enabled: false +``` + +**How `"*"` and named entries combine:** the `"*"` entry applies to every container that does **not** have a named entry. A container with its own named entry takes only the settings declared under that name—the two are **not merged**, so the wildcard contributes nothing to it. + +In the example above, `istio-proxy` is governed solely by `enabled: false` and does not inherit `controlledResources` from the wildcard. Every other container in the pod uses the wildcard entry. + +**Note**: if you add a named entry only to set a bound, repeat any wildcard settings you still want. In the example below, `my-app` falls back to the default `RequestsAndLimits` rather than the `RequestsOnly` set on the wildcard: + +```yaml + - name: "*" + controlledValues: RequestsOnly + - name: my-app + maxAllowed: + memory: 8Gi # controlledValues is NOT inherited - repeat it if you want it +``` + +### Exclude a container from the horizontal signal + +`enabled: false` governs _vertical_ behavior only. The horizontal objective is chosen separately, and this is where sidecars most often distort scaling decisions: + +```yaml + objectives: + # Recommended: scale on the application container's CPU + - type: ContainerResource + containerResource: + container: my-app + name: cpu + value: + type: Utilization + utilization: 65 +``` + +compared with: + +```yaml + objectives: + # Risky when sidecars are present: pod-level utilization is diluted by + # sidecar requests, so a busy application container can appear idle. + - type: PodResource + podResource: + name: cpu + value: + type: Utilization + utilization: 65 +``` + +**Rule of thumb:** if the pod has any sidecar, use `ContainerResource` scoped to the main container. Reserve `PodResource` for genuinely single-container pods. + +## Configure sidecars + +### Ordinary sidecars (`spec.containers`) + +Nothing special is required. They appear as containers and can be bounded, excluded, or targeted like any other; see [Exclude a container](#exclude-a-container). + +### Native sidecars (`spec.initContainers` with `restartPolicy: Always`) + +Kubernetes 1.29+ allows a long-running sidecar to be declared in `initContainers` with `restartPolicy: Always`, the [native sidecar][7] pattern. It runs for the pod's whole lifetime but lives in a different field of the pod spec: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app +spec: + template: + spec: + initContainers: + - name: log-shipper + image: log-shipper:1.2 + restartPolicy: Always # this is what makes it a native sidecar + resources: + requests: {cpu: 100m, memory: 128Mi} + limits: {memory: 256Mi} + containers: + - name: my-app + image: my-app:4.5 + resources: + requests: {cpu: "1", memory: 2Gi} + limits: {cpu: "2", memory: 4Gi} +``` + +**Native sidecars are fully supported.** They are treated as ordinary containers: recommendations are produced and applied for them, and they appear in the workload's container list where they can be configured or excluded. Reference them in `constraints.containers[]` by **name**, exactly like any other container. There is no separate `initContainers` block in the DPA spec, and a `"*"` entry covers them too. + +```yaml +spec: + constraints: + containers: + - name: my-app + controlledResources: [cpu, memory] + controlledValues: RequestsAndLimits + - name: log-shipper # native sidecar, referenced by name + enabled: false +``` + +Points to be aware of: + +- `restartPolicy: Always` is what distinguishes them. Ordinary init containers—those that run to completion before the application starts—are not native sidecars and are not managed by a DPA. +- **Cost and savings figures may under-count native sidecars.** Their resource requests are reported under a separate aggregation, so the cost figures shown for a workload with native sidecars can look inconsistent with its observed usage. This is a known limitation that affects the cost display only; recommendations are unaffected. +- **Injected sidecars** (such as Istio's) are added by a mutating admission webhook at pod level and never appear in the Deployment manifest. They are still picked up, because the container list is reconciled from running pods rather than from the workload manifest alone. +- **Agent-level container exclusions take precedence.** If a sidecar is filtered out of collection on the Agent, no metrics exist for it and no recommendation can be produced. Recommendations for the rest of the pod are unaffected, but that container is reported as missing data. + +## Additional manifest options + +### Preview (dry-run) mode + +```yaml +spec: + applyPolicy: + mode: Preview # recommendations are computed and visible in .status, but nothing is applied +``` + +Useful as a temporary stop switch. If a horizontal configuration is invalid, vertical rightsizing continues to run; setting `mode: Preview` freezes both while you correct it. + +### Disable one scaling direction + +```yaml +spec: + applyPolicy: + scaleUp: + strategy: Max + scaleDown: + strategy: Disabled # never scale in + update: + strategy: Auto +``` + +For vertical-only scaling, set both horizontal strategies to `Disabled`, set `update.strategy: Auto`, and omit `objectives`. + +### Vertical rollout timing + +The controller applies a vertical change by the least disruptive route available and escalates if that route stalls. Two fields control how long it waits at each step: + +```yaml +spec: + applyPolicy: + update: + strategy: Auto + resizePendingPeriod: 600 # 1..3600 seconds + rolloutFallbackDelay: 900 # 1..3600 seconds +``` + +| Field | Controls | +|---|---| +| `resizePendingPeriod` | How long to wait before evicting a pod when the kubelet reports the resize as pending—the resize was accepted but is not progressing, often because the node lacks headroom | +| `rolloutFallbackDelay` | How long to wait before falling back to a full rollout when evictions are blocked, typically by a PodDisruptionBudget | + +Both are optional and accept 1 to 3600 seconds. Leaving them unset uses the controller's built-in defaults. + +- **Raise** `resizePendingPeriod` where eviction is expensive (long warm-up, large caches, slow drain). You trade a longer period at the old size for fewer restarts. +- **Raise** `rolloutFallbackDelay` on workloads with a tight PodDisruptionBudget, so a temporary budget constraint does not immediately escalate to a full rollout. +- **Lower either** where restarts are cheap and you want recommendations to take effect faster. + +These control _how_ a change is delivered, not _whether_ one happens. To stop changes entirely, use `update.strategy: Disabled` or `applyPolicy.mode: Preview`. These fields apply to [in-place vertical scaling][3]; see the overview for cluster-level enablement and Kubernetes requirements. + +### Local fallback tuning + +```yaml +spec: + fallback: + horizontal: + enabled: true + direction: ScaleUp # default; use All to allow fallback scale-in as well + triggers: + staleRecommendationThresholdSeconds: 600 # 100..3600 +``` + +Fallback recommendations are computed inside the cluster from Agent-collected metrics, so scaling continues if Datadog cannot deliver a recommendation within the threshold. + +This feature also requires cluster-side configuration on both the Cluster Agent and the node Agents. See [Kubernetes Autoscaling][2] or contact [Datadog Support][9]. + +### Absolute-value objectives + +Instead of a utilization percentage, target an absolute value: + +```yaml + objectives: + - type: ContainerResource + containerResource: + container: my-app + name: cpu + value: + type: AbsoluteValue + absoluteValue: "1.5" # target cores per pod +``` + +### Custom query objectives + +Scale on any Datadog metric rather than CPU or memory: + +```yaml + objectives: + - type: CustomQuery + customQuery: + window: 5m + request: + queries: + - name: a + source: Metrics + metrics: + query: "avg:my.queue.depth{service:my-app}" + value: + type: AbsoluteValue + absoluteValue: "100" +``` + +`source` may also be `ApmMetrics`, with fields such as `service`, `resourceName`, `operationName`, and `stat`. + +Custom queries are supported for **horizontal-only** scaling. Combining a custom query **with vertical scaling is experimental**, because the autoscaler cannot infer which dimension an arbitrary query should act on. Contact [Datadog Support][9] before relying on it. + +### Tag a DPA's telemetry + +```yaml +metadata: + annotations: + ad.datadoghq.com/tags: '{"team": "my-team", "tier": "critical"}' +``` + +This adds the tags to the autoscaling telemetry emitted for this DPA. + +## Troubleshooting + +### `objectives length must be exactly 1, got 2` + +```text +Invalid configuration: invalid horizontal policy: objectives length must be exactly 1, got 2 +reason: RecommendationError +``` + +`objectives` configures **horizontal** scaling only and accepts exactly one entry. The CRD accepts multiple entries syntactically, but the controller does not run. The most common cause is trying to combine a CPU objective and a memory objective; choose one. + +Two things worth knowing when this occurs: + +- The single-objective limit applies to horizontal scaling only. Vertical rightsizing still covers CPU and memory independently. +- When the horizontal policy is invalid, vertical rightsizing **continues to run**. Set `applyPolicy.mode: Preview` if you want everything frozen while you fix the configuration. + +### Nothing vertical is happening + +Check that `applyPolicy.update.strategy` is set to `Auto`. Without it, no vertical recommendation is applied regardless of what `constraints.containers` says. Then confirm the Admission Controller is enabled; see [Kubernetes Autoscaling][2]. + +### Only memory is being right-sized + +This is expected default behavior. See [Right-size CPU and memory](#right-size-cpu-and-memory). + +### The DPA exists but `.status` stays empty + +Workload Autoscaling is most likely not enabled on the Cluster Agent. See [Kubernetes Autoscaling][2]. + +### Deleting a DPA + +- `spec.owner: Local` (created in-cluster): delete it from the cluster. +- Created from the Datadog UI (`spec.owner: Remote`): delete it from the UI. Deleting it in-cluster only causes it to be recreated. +- **Delete the DPA before deleting the target workload**, otherwise it is left orphaned and is harder to clean up. + +### Multi-region deployments + +A DPA has no cross-cluster awareness. Each cluster scales independently within the same `minReplicas` and `maxReplicas`. Clusters under different load diverge, which is expected. + +### Manifest managers (Helm, Argo CD, Flux) + +- Horizontal scaling writes `replicas` through the `scale` subresource—the equivalent of `kubectl scale`, which manifest managers tolerate well. Remove any static `replicas:` from your manifest after you switch to Apply mode. +- Vertical scaling can reach pods two ways. In-place resize and the admission controller's mutating webhook apply resources to pods without rewriting the workload manifest, so no drift is reported. Rollout-based vertical scaling updates the target Deployment or StatefulSet, which a GitOps tool can detect as drift. If you use Argo CD, configure it to ignore the fields the Cluster Agent owns; see [Manage DatadogPodAutoscaler with ArgoCD][5]. + +## Rollout checklist + +1. Confirm the cluster prerequisites—Workload Autoscaling and the Admission Controller enabled on the Cluster Agent (see [Kubernetes Autoscaling][2]). Without this, the DPA applies cleanly and then does nothing. +2. Generate a baseline in the UI ({{< ui >}}Configure Recommendation{{< /ui >}} > {{< ui >}}Export Recommendation{{< /ui >}}). +3. Add the manifest options you need from this page. +4. Commit with `applyPolicy.mode: Preview`, leaving any existing HPA or VPA in place. +5. Watch the recommendations in the Autoscaling UI for at least one full traffic cycle. +6. Switch to `mode: Apply`, disable the previous autoscaler, and remove any static `replicas`. +7. If you use multidimensional scaling, confirm that both CPU and memory recommendations are being applied. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: https://app.datadoghq.com/orchestration/scaling/workload +[2]: /containers/autoscaling/ +[3]: /containers/autoscaling/#in-place-vertical-scaling +[4]: /containers/autoscaling/#cluster-profiles +[5]: /containers/guide/manage-datadogpodautoscaler-with-argocd/ +[7]: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ +[9]: /help/ From 7437852289862d38b4ebfbebd375dc126e903167 Mon Sep 17 00:00:00 2001 From: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Date: Fri, 21 Aug 2026 08:51:20 +0000 Subject: [PATCH 2/7] Remove em dashes from autoscaling manifest docs Co-authored-by: clamoriniere --- .../en/containers/autoscaling/_index.md | 4 ++-- .../en/containers/autoscaling/manifest.md | 24 +++++++++---------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/hugo/content/en/containers/autoscaling/_index.md b/hugo/content/en/containers/autoscaling/_index.md index ef0536d2a4e..86ea344c45c 100644 --- a/hugo/content/en/containers/autoscaling/_index.md +++ b/hugo/content/en/containers/autoscaling/_index.md @@ -478,7 +478,7 @@ spec: type: Percent value: 50 stabilizationWindowSeconds: 130 - # Vertical updates disabled — horizontal only + # Vertical updates disabled, horizontal only update: strategy: Disabled constraints: @@ -617,7 +617,7 @@ The template body accepts the same fields as a `DatadogPodAutoscaler` spec, minu #### Activation precedence -Cluster Agent 7.79.0+ adds namespace-level activation, the `excluded` opt-out, and the precedence rule between them. On Cluster Agent 7.78.0, only the workload-level label is read — the rules below that involve namespaces or the `excluded` value do not apply. +Cluster Agent 7.79.0+ adds namespace-level activation, the `excluded` opt-out, and the precedence rule between them. On Cluster Agent 7.78.0, only the workload-level label is read. The rules below that involve namespaces or the `excluded` value do not apply. - **Workload labels take precedence over namespace labels.** If a namespace is labeled `autoscaling.datadoghq.com/profile=ns-profile` and a workload inside it is labeled `autoscaling.datadoghq.com/profile=workload-profile`, the workload uses `workload-profile`. - **Opt out with `excluded`.** Set `autoscaling.datadoghq.com/profile: excluded` on a workload to exempt it when its namespace is labeled. This is useful for stateful or critical workloads in an otherwise opted-in namespace. diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md index a40d63d7fef..5984eb358a2 100644 --- a/hugo/content/en/containers/autoscaling/manifest.md +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -17,7 +17,7 @@ The `DatadogPodAutoscaler` (DPA) custom resource defines autoscaling behavior fo This page covers the configuration options available in the manifest. It uses API version `datadoghq.com/v1alpha2`. -For setup and prerequisites—enabling Workload Autoscaling and the Admission Controller on the Datadog Cluster Agent, required Agent versions, and enabling [in-place vertical scaling][3]—see [Kubernetes Autoscaling][2]. +For setup and prerequisites, see [Kubernetes Autoscaling][2]. That page covers enabling Workload Autoscaling and the Admission Controller on the Datadog Cluster Agent, required Agent versions, and enabling [in-place vertical scaling][3]. **Feature status labels:** features on this page are generally available unless marked **Preview**, **Beta**, or **Experimental**. Preview and Beta features are available and supported, but their field names and behavior may still change. Experimental features are available but not recommended for production without contacting [Datadog Support][9] first. @@ -136,12 +136,12 @@ Most vertical options are expressed through `spec.constraints.containers[]`: | Field | Type | Default | Meaning | |---|---|---|---| -| `name` | string, **required** | — | Container name, or `"*"` to match every container that has no entry of its own (see [Exclude a container](#exclude-a-container)) | +| `name` | string, **required** | N/A | Container name, or `"*"` to match every container that has no entry of its own (see [Exclude a container](#exclude-a-container)) | | `enabled` | bool | `true` | `false` disables resource autoscaling for this container | | `controlledResources` | list of `cpu`, `memory` | `[cpu, memory]` | Which resources receive vertical recommendations. An empty list is equivalent to `enabled: false` | | `controlledValues` | enum | `RequestsAndLimits` | Whether recommendations write both requests _and_ limits, or requests only | -| `minAllowed` | resource map | — | Lower bound for the container's requests | -| `maxAllowed` | resource map | — | Upper bound for the container's requests | +| `minAllowed` | resource map | None | Lower bound for the container's requests | +| `maxAllowed` | resource map | None | Upper bound for the container's requests | If `constraints.containers` is omitted entirely, resource scaling is enabled for **all** containers, with no bounds. @@ -205,7 +205,7 @@ This requires a recent Agent version; see the requirements in [Kubernetes Autosc **Status: Beta.** -CPU limit recommendations are derived from sustained usage percentiles over a multi-day window. A short warm-up spike—a JVM starting up, for example—is statistically invisible in that window, so the recommended CPU limit can land too low and the application is throttled at the wrong moment. Memory is not affected in the same way, because peak memory usage gives a reliable ceiling. +CPU limit recommendations are derived from sustained usage percentiles over a multi-day window. A short warm-up spike (a JVM starting up, for example) is statistically invisible in that window, so the recommended CPU limit can land too low and the application is throttled at the wrong moment. Memory is not affected in the same way, because peak memory usage gives a reliable ceiling. Burstable mode **removes CPU limits entirely** while still applying CPU _request_ recommendations: @@ -294,7 +294,7 @@ To verify, run `kubectl -n production get dpa -o yaml` and confirm the generated ## Tune the OOMKill memory bump -After an OOMKill, the memory limit is raised by **20%** relative to the limit in force at the time. The bump is applied immediately and re-applied on each subsequent OOMKill until the workload stabilizes—a self-correcting ratchet rather than a one-time adjustment. +After an OOMKill, the memory limit is raised by **20%** relative to the limit in force at the time. The bump is applied immediately and re-applied on each subsequent OOMKill until the workload stabilizes. This is a self-correcting ratchet rather than a one-time adjustment. To change the ratio: @@ -317,7 +317,7 @@ If you hit that situation: ## Right-size requests only -If you have deliberately tuned limits—for burst headroom, a platform requirement, or a QoS guarantee—and want Datadog to right-size only **requests**, use `controlledValues: RequestsOnly`. +If you have deliberately tuned limits (for burst headroom, a platform requirement, or a QoS guarantee) and want Datadog to right-size only **requests**, use `controlledValues: RequestsOnly`. ```yaml spec: @@ -407,7 +407,7 @@ spec: enabled: false ``` -**How `"*"` and named entries combine:** the `"*"` entry applies to every container that does **not** have a named entry. A container with its own named entry takes only the settings declared under that name—the two are **not merged**, so the wildcard contributes nothing to it. +**How `"*"` and named entries combine:** the `"*"` entry applies to every container that does **not** have a named entry. A container with its own named entry takes only the settings declared under that name. The two are **not merged**, so the wildcard contributes nothing to it. In the example above, `istio-proxy` is governed solely by `enabled: false` and does not inherit `controlledResources` from the wildcard. Every other container in the pod uses the wildcard entry. @@ -501,7 +501,7 @@ spec: Points to be aware of: -- `restartPolicy: Always` is what distinguishes them. Ordinary init containers—those that run to completion before the application starts—are not native sidecars and are not managed by a DPA. +- `restartPolicy: Always` is what distinguishes them. Ordinary init containers (those that run to completion before the application starts) are not native sidecars and are not managed by a DPA. - **Cost and savings figures may under-count native sidecars.** Their resource requests are reported under a separate aggregation, so the cost figures shown for a workload with native sidecars can look inconsistent with its observed usage. This is a known limitation that affects the cost display only; recommendations are unaffected. - **Injected sidecars** (such as Istio's) are added by a mutating admission webhook at pod level and never appear in the Deployment manifest. They are still picked up, because the container list is reconciled from running pods rather than from the workload manifest alone. - **Agent-level container exclusions take precedence.** If a sidecar is filtered out of collection on the Agent, no metrics exist for it and no recommendation can be produced. Recommendations for the rest of the pod are unaffected, but that container is reported as missing data. @@ -548,7 +548,7 @@ spec: | Field | Controls | |---|---| -| `resizePendingPeriod` | How long to wait before evicting a pod when the kubelet reports the resize as pending—the resize was accepted but is not progressing, often because the node lacks headroom | +| `resizePendingPeriod` | How long to wait before evicting a pod when the kubelet reports the resize as pending (accepted but not progressing, often because the node lacks headroom) | | `rolloutFallbackDelay` | How long to wait before falling back to a full rollout when evictions are blocked, typically by a PodDisruptionBudget | Both are optional and accept 1 to 3600 seconds. Leaving them unset uses the controller's built-in defaults. @@ -664,12 +664,12 @@ A DPA has no cross-cluster awareness. Each cluster scales independently within t ### Manifest managers (Helm, Argo CD, Flux) -- Horizontal scaling writes `replicas` through the `scale` subresource—the equivalent of `kubectl scale`, which manifest managers tolerate well. Remove any static `replicas:` from your manifest after you switch to Apply mode. +- Horizontal scaling writes `replicas` through the `scale` subresource (the equivalent of `kubectl scale`), which manifest managers tolerate well. Remove any static `replicas:` from your manifest after you switch to Apply mode. - Vertical scaling can reach pods two ways. In-place resize and the admission controller's mutating webhook apply resources to pods without rewriting the workload manifest, so no drift is reported. Rollout-based vertical scaling updates the target Deployment or StatefulSet, which a GitOps tool can detect as drift. If you use Argo CD, configure it to ignore the fields the Cluster Agent owns; see [Manage DatadogPodAutoscaler with ArgoCD][5]. ## Rollout checklist -1. Confirm the cluster prerequisites—Workload Autoscaling and the Admission Controller enabled on the Cluster Agent (see [Kubernetes Autoscaling][2]). Without this, the DPA applies cleanly and then does nothing. +1. Confirm the cluster prerequisites: Workload Autoscaling and the Admission Controller enabled on the Cluster Agent (see [Kubernetes Autoscaling][2]). Without this, the DPA applies cleanly and then does nothing. 2. Generate a baseline in the UI ({{< ui >}}Configure Recommendation{{< /ui >}} > {{< ui >}}Export Recommendation{{< /ui >}}). 3. Add the manifest options you need from this page. 4. Commit with `applyPolicy.mode: Preview`, leaving any existing HPA or VPA in place. From ffa96946873fa810abbd14f8afb4b4e39c4e9f43 Mon Sep 17 00:00:00 2001 From: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Date: Fri, 21 Aug 2026 09:14:28 +0000 Subject: [PATCH 3/7] Address review feedback on manifest reference page Co-authored-by: clamoriniere --- .../en/containers/autoscaling/manifest.md | 120 +----------------- 1 file changed, 7 insertions(+), 113 deletions(-) diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md index 5984eb358a2..e2e3130bfd8 100644 --- a/hugo/content/en/containers/autoscaling/manifest.md +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -136,12 +136,12 @@ Most vertical options are expressed through `spec.constraints.containers[]`: | Field | Type | Default | Meaning | |---|---|---|---| -| `name` | string, **required** | N/A | Container name, or `"*"` to match every container that has no entry of its own (see [Exclude a container](#exclude-a-container)) | +| `name` | string, **required** | - | Container name, or `"*"` to match every container that has no entry of its own (see [Exclude a container](#exclude-a-container)) | | `enabled` | bool | `true` | `false` disables resource autoscaling for this container | | `controlledResources` | list of `cpu`, `memory` | `[cpu, memory]` | Which resources receive vertical recommendations. An empty list is equivalent to `enabled: false` | | `controlledValues` | enum | `RequestsAndLimits` | Whether recommendations write both requests _and_ limits, or requests only | -| `minAllowed` | resource map | None | Lower bound for the container's requests | -| `maxAllowed` | resource map | None | Upper bound for the container's requests | +| `minAllowed` | resource map | - | Lower bound for the container's requests | +| `maxAllowed` | resource map | - | Upper bound for the container's requests | If `constraints.containers` is omitted entirely, resource scaling is enabled for **all** containers, with no bounds. @@ -199,7 +199,7 @@ spec: utilization: 65 ``` -This requires a recent Agent version; see the requirements in [Kubernetes Autoscaling][2]. On older Agents, the field is accepted by the CRD but has no effect. +This feature requires Datadog Cluster Agent 7.78.0+. On older versions, the `controlledResources` field is accepted by the CRD but has no effect. ## Remove CPU limits with burstable mode @@ -246,52 +246,6 @@ Before enabling it: - Without a CPU limit, a container can consume available node CPU. Kernel CPU shares still apply. - Burstable mode **takes precedence over** `controlledValues` for CPU limits. If both are set, burstable wins. -### Apply burstable mode to many workloads - -To onboard a whole category of workloads (every JVM service, for example), set the option in a `DatadogPodAutoscalerClusterProfile` template. It propagates to every DPA the profile generates, and removing it propagates the removal. For more information about profiles, see [Cluster profiles][4]. - -```yaml -apiVersion: datadoghq.com/v1alpha2 -kind: DatadogPodAutoscalerClusterProfile -metadata: - name: java-burstable -spec: - template: - options: - burstable: true - applyPolicy: - mode: Apply - update: - strategy: Auto - constraints: - minReplicas: 2 - maxReplicas: 25 - containers: - - name: "*" - enabled: true - objectives: - - type: PodResource - podResource: - name: cpu - value: - type: Utilization - utilization: 80 -``` - -Then label the workloads that use it: - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: order-service - namespace: production - labels: - autoscaling.datadoghq.com/profile: java-burstable -``` - -To verify, run `kubectl -n production get dpa -o yaml` and confirm the generated DPA carries `spec.options.burstable: true`. - ## Tune the OOMKill memory bump After an OOMKill, the memory limit is raised by **20%** relative to the limit in force at the time. The bump is applied immediately and re-applied on each subsequent OOMKill until the workload stabilizes. This is a self-correcting ratchet rather than a one-time adjustment. @@ -307,13 +261,7 @@ spec: Quote the value: it is a Kubernetes quantity, not a floating-point number. -**When to raise it.** If a container runs out of memory very fast (roughly **under 30 seconds**), the Agent may not capture the memory peak before the restart, so recommendations cannot ratchet upward and the crash loop persists. - -If you hit that situation: - -1. Confirm the OOM check is enabled on the Agent. -2. Raise `bumpUpRatio`. -3. Set a `minAllowed` memory floor (see [Set per-container bounds](#set-per-container-bounds)) so the limit cannot fall back below a safe value between recommendation cycles. +**When to raise it.** Raise the ratio for workloads whose memory usage can peak sharply above previous peaks. A larger bump reaches the right memory limit faster and avoids several successive bumps before the workload stabilizes. When you raise it, also set a `minAllowed` memory floor (see [Set per-container bounds](#set-per-container-bounds)) so the limit cannot fall back below a safe value between recommendation cycles. ## Right-size requests only @@ -622,60 +570,7 @@ metadata: ad.datadoghq.com/tags: '{"team": "my-team", "tier": "critical"}' ``` -This adds the tags to the autoscaling telemetry emitted for this DPA. - -## Troubleshooting - -### `objectives length must be exactly 1, got 2` - -```text -Invalid configuration: invalid horizontal policy: objectives length must be exactly 1, got 2 -reason: RecommendationError -``` - -`objectives` configures **horizontal** scaling only and accepts exactly one entry. The CRD accepts multiple entries syntactically, but the controller does not run. The most common cause is trying to combine a CPU objective and a memory objective; choose one. - -Two things worth knowing when this occurs: - -- The single-objective limit applies to horizontal scaling only. Vertical rightsizing still covers CPU and memory independently. -- When the horizontal policy is invalid, vertical rightsizing **continues to run**. Set `applyPolicy.mode: Preview` if you want everything frozen while you fix the configuration. - -### Nothing vertical is happening - -Check that `applyPolicy.update.strategy` is set to `Auto`. Without it, no vertical recommendation is applied regardless of what `constraints.containers` says. Then confirm the Admission Controller is enabled; see [Kubernetes Autoscaling][2]. - -### Only memory is being right-sized - -This is expected default behavior. See [Right-size CPU and memory](#right-size-cpu-and-memory). - -### The DPA exists but `.status` stays empty - -Workload Autoscaling is most likely not enabled on the Cluster Agent. See [Kubernetes Autoscaling][2]. - -### Deleting a DPA - -- `spec.owner: Local` (created in-cluster): delete it from the cluster. -- Created from the Datadog UI (`spec.owner: Remote`): delete it from the UI. Deleting it in-cluster only causes it to be recreated. -- **Delete the DPA before deleting the target workload**, otherwise it is left orphaned and is harder to clean up. - -### Multi-region deployments - -A DPA has no cross-cluster awareness. Each cluster scales independently within the same `minReplicas` and `maxReplicas`. Clusters under different load diverge, which is expected. - -### Manifest managers (Helm, Argo CD, Flux) - -- Horizontal scaling writes `replicas` through the `scale` subresource (the equivalent of `kubectl scale`), which manifest managers tolerate well. Remove any static `replicas:` from your manifest after you switch to Apply mode. -- Vertical scaling can reach pods two ways. In-place resize and the admission controller's mutating webhook apply resources to pods without rewriting the workload manifest, so no drift is reported. Rollout-based vertical scaling updates the target Deployment or StatefulSet, which a GitOps tool can detect as drift. If you use Argo CD, configure it to ignore the fields the Cluster Agent owns; see [Manage DatadogPodAutoscaler with ArgoCD][5]. - -## Rollout checklist - -1. Confirm the cluster prerequisites: Workload Autoscaling and the Admission Controller enabled on the Cluster Agent (see [Kubernetes Autoscaling][2]). Without this, the DPA applies cleanly and then does nothing. -2. Generate a baseline in the UI ({{< ui >}}Configure Recommendation{{< /ui >}} > {{< ui >}}Export Recommendation{{< /ui >}}). -3. Add the manifest options you need from this page. -4. Commit with `applyPolicy.mode: Preview`, leaving any existing HPA or VPA in place. -5. Watch the recommendations in the Autoscaling UI for at least one full traffic cycle. -6. Switch to `mode: Apply`, disable the previous autoscaler, and remove any static `replicas`. -7. If you use multidimensional scaling, confirm that both CPU and memory recommendations are being applied. +This adds the tags to the autoscaling telemetry emitted for this DPA. For the list of metrics the Cluster Agent emits, see the [Datadog Cluster Agent integration][8]. ## Further reading @@ -684,7 +579,6 @@ A DPA has no cross-cluster awareness. Each cluster scales independently within t [1]: https://app.datadoghq.com/orchestration/scaling/workload [2]: /containers/autoscaling/ [3]: /containers/autoscaling/#in-place-vertical-scaling -[4]: /containers/autoscaling/#cluster-profiles -[5]: /containers/guide/manage-datadogpodautoscaler-with-argocd/ [7]: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ +[8]: /integrations/datadog-cluster-agent/#metrics [9]: /help/ From c93aa6618ac75bd78b94fa8e2737a60fd1bf3daa Mon Sep 17 00:00:00 2001 From: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Date: Fri, 21 Aug 2026 13:33:43 +0000 Subject: [PATCH 4/7] Address second-round review feedback on manifest reference Co-authored-by: clamoriniere --- .../en/containers/autoscaling/_index.md | 14 ++----- .../en/containers/autoscaling/manifest.md | 41 ++++++++----------- 2 files changed, 21 insertions(+), 34 deletions(-) diff --git a/hugo/content/en/containers/autoscaling/_index.md b/hugo/content/en/containers/autoscaling/_index.md index 86ea344c45c..c474be35dab 100644 --- a/hugo/content/en/containers/autoscaling/_index.md +++ b/hugo/content/en/containers/autoscaling/_index.md @@ -216,19 +216,11 @@ _Fixed cost values are subject to refinement over time._ By default, applying a vertical recommendation requires a full pod rollout: the pod template is updated, Kubernetes recreates the pods, and the new resources take effect as those pods are admitted. For slow-starting or latency-sensitive services, that is a meaningful disruption. -In-place vertical scaling instead updates container resources on the running pods through the Kubernetes [pod resize subresource](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/), so most resizes happen with no restart. This requires Datadog Cluster Agent 7.78+. +In-place vertical scaling instead updates container resources on the running pods through the Kubernetes [pod resize subresource](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/), so most resizes happen with no restart. In-place vertical scaling is supported on Kubernetes 1.33+, where the `InPlacePodVerticalScaling` feature gate is enabled by default. It requires Datadog Cluster Agent 7.78+. -With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling for a workload, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. To tune how long the controller waits before evicting a pending resize or falling back to a rollout, see the [DatadogPodAutoscaler manifest reference][15]. - -#### Kubernetes support - -The `InPlacePodVerticalScaling` feature gate must be enabled on **both** the API server and the kubelet; a cluster can have one without the other. +A resize takes effect on the running container without restarting the application process. Applications that read their CPU or memory requests and limits only at startup do not see the new values until they restart, and environment variables populated from resource fields through the [downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) are not refreshed on an in-place resize. Keep this in mind for workloads that size internal components (thread pools, heap, or caches) from their resource requests or limits. -| Kubernetes version | Feature gate | -|---|---| -| 1.27 - 1.32 | Alpha, disabled by default | -| 1.33 - 1.34 | Beta, enabled by default | -| 1.35+ | GA, gate removed | +With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling for a workload, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. To tune how long the controller waits before evicting a pending resize or falling back to a rollout, see the [DatadogPodAutoscaler manifest reference][15]. #### Enable in-place vertical scaling diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md index e2e3130bfd8..0f6e7170d9b 100644 --- a/hugo/content/en/containers/autoscaling/manifest.md +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -19,8 +19,6 @@ This page covers the configuration options available in the manifest. It uses AP For setup and prerequisites, see [Kubernetes Autoscaling][2]. That page covers enabling Workload Autoscaling and the Admission Controller on the Datadog Cluster Agent, required Agent versions, and enabling [in-place vertical scaling][3]. -**Feature status labels:** features on this page are generally available unless marked **Preview**, **Beta**, or **Experimental**. Preview and Beta features are available and supported, but their field names and behavior may still change. Experimental features are available but not recommended for production without contacting [Datadog Support][9] first. - ## Anatomy of a manifest The following annotated skeleton shows the structure of a `DatadogPodAutoscaler`. Every field is optional except `targetRef`. @@ -29,23 +27,23 @@ The following annotated skeleton shows the structure of a `DatadogPodAutoscaler` apiVersion: datadoghq.com/v1alpha2 kind: DatadogPodAutoscaler metadata: - name: my-app # conventionally the workload name - namespace: my-namespace # must match the target workload - annotations: + name: my-app # required: conventionally the workload name + namespace: my-namespace # required: must match the target workload + annotations: # optional ad.datadoghq.com/tags: '{"team": "my-team"}' # optional: tags on this DPA's telemetry spec: - owner: Local # Local = this manifest is the source of truth (use for GitOps) + owner: Local # optional: Local = this manifest is the source of truth (use for GitOps) # Remote = created and managed from the Datadog UI - targetRef: # the workload being autoscaled - one DPA per workload + targetRef: # required: the workload being autoscaled - one DPA per workload apiVersion: apps/v1 kind: Deployment name: my-app - applyPolicy: + applyPolicy: # optional mode: Apply # Apply | Preview (Preview = compute recommendations, change nothing) - scaleUp: # horizontal, upward + scaleUp: # optional: horizontal, upward strategy: Max # Max | Min | Disabled stabilizationWindowSeconds: 600 rules: @@ -53,7 +51,7 @@ spec: value: 50 periodSeconds: 120 # 1..3600 - scaleDown: # horizontal, downward + scaleDown: # optional: horizontal, downward strategy: Max stabilizationWindowSeconds: 600 rules: @@ -61,15 +59,15 @@ spec: value: 10 periodSeconds: 1800 - update: # vertical + update: # optional: vertical strategy: Auto # Auto | Disabled | TriggerRollout # resizePendingPeriod: 600 # see Vertical rollout timing # rolloutFallbackDelay: 900 # see Vertical rollout timing - constraints: + constraints: # optional minReplicas: 3 maxReplicas: 100 - containers: # per-container vertical configuration + containers: # optional: per-container vertical configuration - name: "*" # "*" matches all containers enabled: true controlledResources: [cpu, memory] @@ -81,7 +79,7 @@ spec: cpu: "4" memory: 8Gi - objectives: # HORIZONTAL only. Exactly one entry. + objectives: # optional: configures horizontal scaling (also used by multidimensional). Exactly one entry. - type: ContainerResource # PodResource | ContainerResource | CustomQuery containerResource: container: my-app @@ -90,14 +88,14 @@ spec: type: Utilization # Utilization | AbsoluteValue utilization: 65 - fallback: # in-cluster horizontal fallback if recommendations go stale + fallback: # optional: in-cluster horizontal fallback if recommendations go stale horizontal: enabled: true direction: ScaleUp # ScaleUp | ScaleDown | All (default ScaleUp) triggers: staleRecommendationThresholdSeconds: 600 # 100..3600, default 600 - options: + options: # optional burstable: false # true = remove CPU limits, keep CPU request recommendations outOfMemory: bumpUpRatio: "1.2" # +20% memory limit after an OOMKill (default) @@ -147,8 +145,6 @@ If `constraints.containers` is omitted entirely, resource scaling is enabled for ## Right-size CPU and memory -**Status: Preview.** - When a DPA combines horizontal scaling (`objectives`) with vertical scaling (`update.strategy: Auto`), the default behavior is to produce vertical recommendations for **memory only**. CPU requests and limits are left untouched, and the `VerticalAbleToRecommend` condition may show as `Unknown`. If you have seen a DPA right-size memory while leaving CPU unchanged, this is the reason. It is not related to Quality of Service (QoS) class preservation. @@ -203,8 +199,6 @@ This feature requires Datadog Cluster Agent 7.78.0+. On older versions, the `con ## Remove CPU limits with burstable mode -**Status: Beta.** - CPU limit recommendations are derived from sustained usage percentiles over a multi-day window. A short warm-up spike (a JVM starting up, for example) is statistically invisible in that window, so the recommended CPU limit can land too low and the application is throttled at the wrong moment. Memory is not affected in the same way, because peak memory usage gives a reliable ceiling. Burstable mode **removes CPU limits entirely** while still applying CPU _request_ recommendations: @@ -452,7 +446,7 @@ Points to be aware of: - `restartPolicy: Always` is what distinguishes them. Ordinary init containers (those that run to completion before the application starts) are not native sidecars and are not managed by a DPA. - **Cost and savings figures may under-count native sidecars.** Their resource requests are reported under a separate aggregation, so the cost figures shown for a workload with native sidecars can look inconsistent with its observed usage. This is a known limitation that affects the cost display only; recommendations are unaffected. - **Injected sidecars** (such as Istio's) are added by a mutating admission webhook at pod level and never appear in the Deployment manifest. They are still picked up, because the container list is reconciled from running pods rather than from the workload manifest alone. -- **Agent-level container exclusions take precedence.** If a sidecar is filtered out of collection on the Agent, no metrics exist for it and no recommendation can be produced. Recommendations for the rest of the pod are unaffected, but that container is reported as missing data. +- **Do not exclude autoscaled containers from Agent collection.** A `DatadogPodAutoscaler` relies on the metrics the Agent collects for the containers it manages. If a container in an autoscaled workload is filtered out through the Agent's container discovery configuration, no metrics exist for it and it cannot be right-sized. Confirm that no container in an autoscaled workload is excluded from collection. For how inclusion and exclusion rules work, see [Container Discovery Management][10]. ## Additional manifest options @@ -474,7 +468,7 @@ spec: scaleUp: strategy: Max scaleDown: - strategy: Disabled # never scale in + strategy: Disabled # never scale down update: strategy: Auto ``` @@ -560,7 +554,7 @@ Scale on any Datadog metric rather than CPU or memory: `source` may also be `ApmMetrics`, with fields such as `service`, `resourceName`, `operationName`, and `stat`. -Custom queries are supported for **horizontal-only** scaling. Combining a custom query **with vertical scaling is experimental**, because the autoscaler cannot infer which dimension an arbitrary query should act on. Contact [Datadog Support][9] before relying on it. +Custom queries are supported for **horizontal scaling only**. Combining a custom query with vertical scaling is **not supported**, because the autoscaler cannot infer which dimension an arbitrary query should act on. ### Tag a DPA's telemetry @@ -582,3 +576,4 @@ This adds the tags to the autoscaling telemetry emitted for this DPA. For the li [7]: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ [8]: /integrations/datadog-cluster-agent/#metrics [9]: /help/ +[10]: /containers/guide/container-discovery-management/ From 0ed0e06c5c0816aaf5ee3d4c400d0a0bcde1c4ca Mon Sep 17 00:00:00 2001 From: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Date: Mon, 24 Aug 2026 12:43:12 +0000 Subject: [PATCH 5/7] Apply automated review suggestions on autoscaling docs Co-authored-by: clamoriniere --- .../en/containers/autoscaling/_index.md | 13 ++++++++++--- .../en/containers/autoscaling/manifest.md | 18 +++++++++--------- 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/hugo/content/en/containers/autoscaling/_index.md b/hugo/content/en/containers/autoscaling/_index.md index c474be35dab..3654873ae89 100644 --- a/hugo/content/en/containers/autoscaling/_index.md +++ b/hugo/content/en/containers/autoscaling/_index.md @@ -216,9 +216,9 @@ _Fixed cost values are subject to refinement over time._ By default, applying a vertical recommendation requires a full pod rollout: the pod template is updated, Kubernetes recreates the pods, and the new resources take effect as those pods are admitted. For slow-starting or latency-sensitive services, that is a meaningful disruption. -In-place vertical scaling instead updates container resources on the running pods through the Kubernetes [pod resize subresource](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/), so most resizes happen with no restart. In-place vertical scaling is supported on Kubernetes 1.33+, where the `InPlacePodVerticalScaling` feature gate is enabled by default. It requires Datadog Cluster Agent 7.78+. +In-place vertical scaling instead updates container resources on the running pods through the Kubernetes [pod resize subresource][16], so most resizes happen with no restart. In-place vertical scaling is supported on Kubernetes 1.33+, where the `InPlacePodVerticalScaling` feature gate is enabled by default. It requires Datadog Cluster Agent 7.78+. -A resize takes effect on the running container without restarting the application process. Applications that read their CPU or memory requests and limits only at startup do not see the new values until they restart, and environment variables populated from resource fields through the [downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) are not refreshed on an in-place resize. Keep this in mind for workloads that size internal components (thread pools, heap, or caches) from their resource requests or limits. +A resize takes effect on the running container without restarting the application process. Applications that read their CPU or memory requests and limits only at startup do not see the new values until they restart, and environment variables populated from resource fields through the [downward API][17] are not refreshed on an in-place resize. Keep this in mind for workloads that size internal components (thread pools, heap, or caches) from their resource requests or limits. With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling for a workload, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. To tune how long the controller waits before evicting a pending resize or falling back to a rollout, see the [DatadogPodAutoscaler manifest reference][15]. @@ -257,7 +257,11 @@ datadog: #### Behavior and limitations - **`resizePolicy` stays under your control.** Datadog never sets or overrides the container-level `resizePolicy`; it is immutable after pod creation and is an application-level decision. When unset, Kubernetes defaults to `NotRequired` for CPU and memory, meaning resize without restart. Set `RestartContainer` per resource on containers that cannot absorb a live change. -- **Kubernetes limitations apply.** Only CPU and memory can be resized; QoS class cannot change; requests and limits can be changed but not removed entirely; Windows pods and pods under static CPU or memory manager policies are excluded. See the [upstream limitations](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#limitations). +- **Kubernetes limitations apply.** See the [upstream limitations][18]: + - Only CPU and memory can be resized. + - QoS class cannot change. + - Requests and limits can be changed but not removed entirely. + - Windows pods and pods under static CPU or memory manager policies are excluded. - **Burstable mode still requires a rollout.** Burstable mode removes the CPU limit, and in-place resize can change a limit but not remove one. See the [DatadogPodAutoscaler manifest reference][15]. - If a resize is infeasible or stays pending, the pod is evicted through the Kubernetes Eviction API, which respects PodDisruptionBudgets. @@ -709,3 +713,6 @@ Datadog computes vertical scaling recommendations for CPU and memory by analyzin [13]: /containers/guide/manage-datdadogpodautoscaler-with-terraform/ [14]: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ [15]: /containers/autoscaling/manifest/ +[16]: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/ +[17]: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/ +[18]: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#limitations diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md index 0f6e7170d9b..d2abb86fd4c 100644 --- a/hugo/content/en/containers/autoscaling/manifest.md +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -403,7 +403,7 @@ Nothing special is required. They appear as containers and can be bounded, exclu ### Native sidecars (`spec.initContainers` with `restartPolicy: Always`) -Kubernetes 1.29+ allows a long-running sidecar to be declared in `initContainers` with `restartPolicy: Always`, the [native sidecar][7] pattern. It runs for the pod's whole lifetime but lives in a different field of the pod spec: +Kubernetes 1.29+ allows a long-running sidecar to be declared in `initContainers` with `restartPolicy: Always`, the [native sidecar][4] pattern. It runs for the pod's whole lifetime but lives in a different field of the pod spec: ```yaml apiVersion: apps/v1 @@ -446,7 +446,7 @@ Points to be aware of: - `restartPolicy: Always` is what distinguishes them. Ordinary init containers (those that run to completion before the application starts) are not native sidecars and are not managed by a DPA. - **Cost and savings figures may under-count native sidecars.** Their resource requests are reported under a separate aggregation, so the cost figures shown for a workload with native sidecars can look inconsistent with its observed usage. This is a known limitation that affects the cost display only; recommendations are unaffected. - **Injected sidecars** (such as Istio's) are added by a mutating admission webhook at pod level and never appear in the Deployment manifest. They are still picked up, because the container list is reconciled from running pods rather than from the workload manifest alone. -- **Do not exclude autoscaled containers from Agent collection.** A `DatadogPodAutoscaler` relies on the metrics the Agent collects for the containers it manages. If a container in an autoscaled workload is filtered out through the Agent's container discovery configuration, no metrics exist for it and it cannot be right-sized. Confirm that no container in an autoscaled workload is excluded from collection. For how inclusion and exclusion rules work, see [Container Discovery Management][10]. +- **Do not exclude autoscaled containers from Agent collection.** A `DatadogPodAutoscaler` relies on the metrics the Agent collects for the containers it manages. If a container in an autoscaled workload is filtered out through the Agent's container discovery configuration, no metrics exist for it and it cannot be right-sized. Confirm that no container in an autoscaled workload is excluded from collection. For how inclusion and exclusion rules work, see [Container Discovery Management][7]. ## Additional manifest options @@ -473,7 +473,7 @@ spec: strategy: Auto ``` -For vertical-only scaling, set both horizontal strategies to `Disabled`, set `update.strategy: Auto`, and omit `objectives`. +For vertical-only scaling, omit `objectives` and set `update.strategy: Auto`, as in the [Choose a scaling mode](#choose-a-scaling-mode) table. With no `objectives`, horizontal scaling has no target to act on, so you do not also need to set `scaleUp` and `scaleDown` to `Disabled`. ### Vertical rollout timing @@ -515,7 +515,7 @@ spec: Fallback recommendations are computed inside the cluster from Agent-collected metrics, so scaling continues if Datadog cannot deliver a recommendation within the threshold. -This feature also requires cluster-side configuration on both the Cluster Agent and the node Agents. See [Kubernetes Autoscaling][2] or contact [Datadog Support][9]. +This feature also requires cluster-side configuration on both the Cluster Agent and the node Agents. See [Kubernetes Autoscaling][2] or contact [Datadog Support][6]. ### Absolute-value objectives @@ -564,7 +564,7 @@ metadata: ad.datadoghq.com/tags: '{"team": "my-team", "tier": "critical"}' ``` -This adds the tags to the autoscaling telemetry emitted for this DPA. For the list of metrics the Cluster Agent emits, see the [Datadog Cluster Agent integration][8]. +This adds the tags to the autoscaling telemetry emitted for this DPA. For the list of metrics the Cluster Agent emits, see the [Datadog Cluster Agent integration][5]. ## Further reading @@ -573,7 +573,7 @@ This adds the tags to the autoscaling telemetry emitted for this DPA. For the li [1]: https://app.datadoghq.com/orchestration/scaling/workload [2]: /containers/autoscaling/ [3]: /containers/autoscaling/#in-place-vertical-scaling -[7]: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ -[8]: /integrations/datadog-cluster-agent/#metrics -[9]: /help/ -[10]: /containers/guide/container-discovery-management/ +[4]: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ +[5]: /integrations/datadog-cluster-agent/#metrics +[6]: /help/ +[7]: /containers/guide/container-discovery-management/ From 9b9381a24012daf2301964b5a8aa3127a720e7c5 Mon Sep 17 00:00:00 2001 From: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Date: Mon, 24 Aug 2026 13:41:32 +0000 Subject: [PATCH 6/7] Apply automated review suggestions on autoscaling docs Co-authored-by: clamoriniere --- hugo/content/en/containers/autoscaling/_index.md | 2 +- hugo/content/en/containers/autoscaling/manifest.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/hugo/content/en/containers/autoscaling/_index.md b/hugo/content/en/containers/autoscaling/_index.md index 3654873ae89..b3cbc281476 100644 --- a/hugo/content/en/containers/autoscaling/_index.md +++ b/hugo/content/en/containers/autoscaling/_index.md @@ -519,7 +519,7 @@ spec: The examples above cover the most common strategies. The manifest supports additional options that the templates don't show, including: - **CPU rightsizing** alongside memory (`constraints.containers[].controlledResources`). When a workload combines horizontal and vertical scaling, vertical recommendations cover memory only by default. -- **Burstable mode** (`spec.options.burstable`) to remove CPU limits on spiky workloads while still right-sizing CPU requests. +- **Burstable mode** (`spec.options.burstable`) to remove CPU limits on spiky workloads while still rightsizing CPU requests. - **Per-container bounds** (`minAllowed` and `maxAllowed`), request-only rightsizing (`controlledValues: RequestsOnly`), and per-container targeting for horizontal scaling (`ContainerResource` objectives). For the full field reference, see the [DatadogPodAutoscaler manifest reference][15]. diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md index d2abb86fd4c..b49b7bc97b2 100644 --- a/hugo/content/en/containers/autoscaling/manifest.md +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -15,7 +15,7 @@ further_reading: The `DatadogPodAutoscaler` (DPA) custom resource defines autoscaling behavior for a single Kubernetes workload. The [Autoscaling UI][1] with {{< ui >}}Export Recommendation{{< /ui >}} is a good place to start: configure a workload, then copy the generated manifest. Editing the manifest directly gives you access to every field in the custom resource definition (CRD) and makes the DPA a normal part of your GitOps workflow, where the manifest is the reviewed, versioned source of truth. -This page covers the configuration options available in the manifest. It uses API version `datadoghq.com/v1alpha2`. +This page covers the configuration options available in the manifest. Examples on this page use API version `datadoghq.com/v1alpha2`. For setup and prerequisites, see [Kubernetes Autoscaling][2]. That page covers enabling Workload Autoscaling and the Admission Controller on the Datadog Cluster Agent, required Agent versions, and enabling [in-place vertical scaling][3]. @@ -279,7 +279,7 @@ Interactions to be aware of: - On a container where `request == limit`, lowering the request breaks the Guaranteed QoS class. If you need Guaranteed, keep `RequestsAndLimits`; the recommender handles `request == limit` containers explicitly. - Burstable mode overrides this for CPU limits (see [Remove CPU limits with burstable mode](#remove-cpu-limits-with-burstable-mode)). -- **OOMKill handling still adjusts the memory limit.** `RequestsOnly` does not suppress the memory bump, and cannot: Kubernetes rejects a container whose request exceeds its limit, so raising the request without raising the limit would produce an invalid pod. Read `RequestsOnly` as "limits are not _right-sized_", not "limits are never modified". See [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump). +- **OOMKill handling still adjusts the memory limit.** `RequestsOnly` does not suppress the memory bump. If a request is raised close to the current limit, Kubernetes rejects any pod whose request exceeds its limit, so the limit is bumped to keep the spec valid. Read `RequestsOnly` as "limits are not _right-sized_", not "limits are never modified". See [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump). Choosing a combination: @@ -353,7 +353,7 @@ spec: In the example above, `istio-proxy` is governed solely by `enabled: false` and does not inherit `controlledResources` from the wildcard. Every other container in the pod uses the wildcard entry. -**Note**: if you add a named entry only to set a bound, repeat any wildcard settings you still want. In the example below, `my-app` falls back to the default `RequestsAndLimits` rather than the `RequestsOnly` set on the wildcard: +**Note**: If you add a named entry only to set a bound, repeat any wildcard settings you still want. In the example below, `my-app` falls back to the default `RequestsAndLimits` rather than the `RequestsOnly` set on the wildcard: ```yaml - name: "*" From 015d9f9a4965a10ec6678b475470063ba8a4b611 Mon Sep 17 00:00:00 2001 From: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Date: Mon, 24 Aug 2026 15:16:26 +0000 Subject: [PATCH 7/7] Clarify OOMKill raises the memory limit and potentially the request --- hugo/content/en/containers/autoscaling/manifest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md index b49b7bc97b2..fe92603837f 100644 --- a/hugo/content/en/containers/autoscaling/manifest.md +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -279,7 +279,7 @@ Interactions to be aware of: - On a container where `request == limit`, lowering the request breaks the Guaranteed QoS class. If you need Guaranteed, keep `RequestsAndLimits`; the recommender handles `request == limit` containers explicitly. - Burstable mode overrides this for CPU limits (see [Remove CPU limits with burstable mode](#remove-cpu-limits-with-burstable-mode)). -- **OOMKill handling still adjusts the memory limit.** `RequestsOnly` does not suppress the memory bump. If a request is raised close to the current limit, Kubernetes rejects any pod whose request exceeds its limit, so the limit is bumped to keep the spec valid. Read `RequestsOnly` as "limits are not _right-sized_", not "limits are never modified". See [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump). +- **OOMKill handling still adjusts the memory limit.** `RequestsOnly` does not suppress the memory bump. After an OOMKill, the memory limit is raised, and the request is potentially raised with it (Kubernetes rejects any pod whose request exceeds its limit). Read `RequestsOnly` as "limits are not _right-sized_", not "limits are never modified". See [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump). Choosing a combination: