diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 3dc8f396f44..0da3cf6402a 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -3317,6 +3317,11 @@ menu: parent: containers identifier: containers_autoscaling weight: 2 + - name: Manifest reference + url: containers/autoscaling/manifest + parent: containers_autoscaling + identifier: containers_autoscaling_manifest + weight: 200 - name: Cluster url: containers/autoscaling/cluster parent: containers_autoscaling diff --git a/hugo/content/en/containers/autoscaling/_index.md b/hugo/content/en/containers/autoscaling/_index.md index 418ce15adbe..b3cbc281476 100644 --- a/hugo/content/en/containers/autoscaling/_index.md +++ b/hugo/content/en/containers/autoscaling/_index.md @@ -170,6 +170,8 @@ helm upgrade -f datadog-values.yaml datadog/datadog {{% /tab %}} {{< /tabs >}} +**Note**: Vertical scaling recommendations are applied to new pods through the [Admission Controller](/containers/cluster_agent/admission_controller/) mutating webhook. The Admission Controller is enabled by default. If you disable it, horizontal scaling continues to work but vertical scaling has no effect. + ### Idle cost and savings estimates {{< tabs >}} @@ -210,6 +212,59 @@ _Fixed cost values are subject to refinement over time._ {{% /tab %}} {{< /tabs >}} +### In-place vertical scaling + +By default, applying a vertical recommendation requires a full pod rollout: the pod template is updated, Kubernetes recreates the pods, and the new resources take effect as those pods are admitted. For slow-starting or latency-sensitive services, that is a meaningful disruption. + +In-place vertical scaling instead updates container resources on the running pods through the Kubernetes [pod resize subresource][16], so most resizes happen with no restart. In-place vertical scaling is supported on Kubernetes 1.33+, where the `InPlacePodVerticalScaling` feature gate is enabled by default. It requires Datadog Cluster Agent 7.78+. + +A resize takes effect on the running container without restarting the application process. Applications that read their CPU or memory requests and limits only at startup do not see the new values until they restart, and environment variables populated from resource fields through the [downward API][17] are not refreshed on an in-place resize. Keep this in mind for workloads that size internal components (thread pools, heap, or caches) from their resource requests or limits. + +With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling for a workload, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. To tune how long the controller waits before evicting a pending resize or falling back to a rollout, see the [DatadogPodAutoscaler manifest reference][15]. + +#### Enable in-place vertical scaling + +Enable the feature on the Cluster Agent. This also grants the Cluster Agent the permissions it needs to resize and, where necessary, evict pods. + +{{< tabs >}} +{{% tab "Datadog Operator" %}} + +```yaml +spec: + features: + autoscaling: + workload: + enabled: true + inPlaceVerticalScaling: + enabled: true +``` + +{{% /tab %}} +{{% tab "Helm" %}} + +```yaml +datadog: + autoscaling: + workload: + enabled: true + inPlaceVerticalScaling: + enabled: true +``` + +{{% /tab %}} +{{< /tabs >}} + +#### Behavior and limitations + +- **`resizePolicy` stays under your control.** Datadog never sets or overrides the container-level `resizePolicy`; it is immutable after pod creation and is an application-level decision. When unset, Kubernetes defaults to `NotRequired` for CPU and memory, meaning resize without restart. Set `RestartContainer` per resource on containers that cannot absorb a live change. +- **Kubernetes limitations apply.** See the [upstream limitations][18]: + - Only CPU and memory can be resized. + - QoS class cannot change. + - Requests and limits can be changed but not removed entirely. + - Windows pods and pods under static CPU or memory manager policies are excluded. +- **Burstable mode still requires a rollout.** Burstable mode removes the CPU limit, and in-place resize can change a limit but not remove one. See the [DatadogPodAutoscaler manifest reference][15]. +- If a resize is infeasible or stays pending, the pod is evicted through the Kubernetes Eviction API, which respects PodDisruptionBudgets. + ## Usage ### Identify resources to rightsize @@ -249,7 +304,7 @@ The Setup wizard is best for trying autoscaling on a single workload, getting ha #### Path B: GitOps -Define a `DatadogPodAutoscaler` custom resource that targets your workload and apply it through whatever tooling you already use to ship Kubernetes manifests, whether that's `kubectl apply`, Helm, ArgoCD, Terraform, or another GitOps tool. Authoring the manifest is the same regardless of delivery mechanism. See the [example configurations](#example-datadogpodautoscaler-configurations) below for ready-to-edit starting points covering cost optimization, balanced scaling, vertical-only resizing, and custom-query horizontal scaling. +Define a `DatadogPodAutoscaler` custom resource that targets your workload and apply it through whatever tooling you already use to ship Kubernetes manifests, whether that's `kubectl apply`, Helm, ArgoCD, Terraform, or another GitOps tool. Authoring the manifest is the same regardless of delivery mechanism. See the [example configurations](#example-datadogpodautoscaler-configurations) below for ready-to-edit starting points covering cost optimization, balanced scaling, vertical-only resizing, and custom-query horizontal scaling. For a complete reference of the manifest fields and options, including options the UI does not expose, see the [DatadogPodAutoscaler manifest reference][15]. For tool-specific guides, see: @@ -258,7 +313,7 @@ For tool-specific guides, see: ### Example DatadogPodAutoscaler configurations -The following examples demonstrate common `DatadogPodAutoscaler` configurations for different scaling strategies. Use them as starting points and adjust the values to match your workload's requirements. If you would rather pick a template in the UI, follow [Path A](#path-a-datadog-ui-setup-wizard) above. +The following examples demonstrate common `DatadogPodAutoscaler` configurations for different scaling strategies. Use them as starting points and adjust the values to match your workload's requirements. If you would rather pick a template in the UI, follow [Path A](#path-a-datadog-ui-setup-wizard) above. For the full range of manifest fields and options not shown here, see the [DatadogPodAutoscaler manifest reference][15]. {{< tabs >}} {{% tab "Optimize Cost" %}} @@ -361,15 +416,7 @@ spec: Pick this template when a workload can't be scaled horizontally, or when you want pure rightsizing without changing replica counts. Common cases are singleton services, stateful workloads, and leader-elected components. The defining setting is `scaleDown.strategy: Disabled` and `scaleUp.strategy: Disabled`, which leaves only `update.strategy: Auto` to apply CPU and memory recommendations. -By default, the controller applies vertical recommendations by triggering a rollout (evict and recreate pods). Cluster Agent **7.78+** also supports **in-place pod resizing**, which updates a pod's CPU and memory requests and limits without restarting it. In-place resize is opt-in: set `autoscaling.workload.in_place_vertical_scaling.enabled: true` on the Cluster Agent (or set the environment variable `DD_AUTOSCALING_WORKLOAD_IN_PLACE_VERTICAL_SCALING_ENABLED=true`). - -Your cluster must also expose the `pods/resize` subresource. This is the default in Kubernetes 1.33+ where the `InPlacePodVerticalScaling` feature gate is beta. On Kubernetes 1.27 to 1.32, the feature gate must be enabled on `kube-apiserver` and every `kubelet`. - -When both prerequisites are met: - -- **Default**: Workloads with `applyPolicy.update.strategy: Auto` (the default) resize in place. -- **Fallback**: If the kubelet reports a resize as `Infeasible`, the controller falls back to a rollout. -- **Opt-out**: To force a workload to always use rollout-based vertical scaling regardless of the cluster setting, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. +By default, the controller applies vertical recommendations by triggering a rollout (evict and recreate pods). Cluster Agent **7.78+** also supports **in-place pod resizing**, which updates a pod's CPU and memory requests and limits without restarting it. With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling, set `applyPolicy.update.strategy: TriggerRollout`. For enablement, Kubernetes requirements, and behavior, see [In-place vertical scaling](#in-place-vertical-scaling). ```yaml apiVersion: datadoghq.com/v1alpha2 @@ -427,7 +474,7 @@ spec: type: Percent value: 50 stabilizationWindowSeconds: 130 - # Vertical updates disabled — horizontal only + # Vertical updates disabled, horizontal only update: strategy: Disabled constraints: @@ -469,6 +516,14 @@ spec: {{% /tab %}} {{< /tabs >}} +The examples above cover the most common strategies. The manifest supports additional options that the templates don't show, including: + +- **CPU rightsizing** alongside memory (`constraints.containers[].controlledResources`). When a workload combines horizontal and vertical scaling, vertical recommendations cover memory only by default. +- **Burstable mode** (`spec.options.burstable`) to remove CPU limits on spiky workloads while still rightsizing CPU requests. +- **Per-container bounds** (`minAllowed` and `maxAllowed`), request-only rightsizing (`controlledValues: RequestsOnly`), and per-container targeting for horizontal scaling (`ContainerResource` objectives). + +For the full field reference, see the [DatadogPodAutoscaler manifest reference][15]. + ### Cluster profiles A `DatadogPodAutoscalerClusterProfile` is a cluster-scoped resource that holds a `DatadogPodAutoscaler` template. The Cluster Agent watches `Deployment` and `StatefulSet` resources (and, on 7.79+, the namespaces that contain them) for the `autoscaling.datadoghq.com/profile` label, and creates a managed `DatadogPodAutoscaler` for every matching workload. One profile applies to many workloads; one workload still maps to one `DatadogPodAutoscaler`. @@ -558,7 +613,7 @@ The template body accepts the same fields as a `DatadogPodAutoscaler` spec, minu #### Activation precedence -Cluster Agent 7.79.0+ adds namespace-level activation, the `excluded` opt-out, and the precedence rule between them. On Cluster Agent 7.78.0, only the workload-level label is read — the rules below that involve namespaces or the `excluded` value do not apply. +Cluster Agent 7.79.0+ adds namespace-level activation, the `excluded` opt-out, and the precedence rule between them. On Cluster Agent 7.78.0, only the workload-level label is read. The rules below that involve namespaces or the `excluded` value do not apply. - **Workload labels take precedence over namespace labels.** If a namespace is labeled `autoscaling.datadoghq.com/profile=ns-profile` and a workload inside it is labeled `autoscaling.datadoghq.com/profile=workload-profile`, the workload uses `workload-profile`. - **Opt out with `excluded`.** Set `autoscaling.datadoghq.com/profile: excluded` on a workload to exempt it when its namespace is labeled. This is useful for stateful or critical workloads in an otherwise opted-in namespace. @@ -636,7 +691,7 @@ Datadog computes vertical scaling recommendations for CPU and memory by analyzin - **8-day lookback window**: All recommendations consider usage data from the past 8 days, providing enough history to capture weekly traffic patterns while remaining responsive to changes. - **Decaying weights**: For Burstable-class request recommendations (CPU or memory), older samples are weighted less heavily, so the recommendation adapts faster to recent usage shifts. - **Safety margins**: Every recommendation includes a margin above observed usage (5 to 10%) to provide a buffer against unexpected spikes. -- **OOMKill response**: When memory is Guaranteed-class (request equals limit) and an OOMKill occurs, a 20% bump is applied to reduce the likelihood of repeated out-of-memory failures. +- **OOMKill response**: When an OOMKill occurs, the memory limit is raised (by 20% by default) and re-applied on each subsequent OOMKill until the workload stabilizes, reducing the likelihood of repeated out-of-memory failures. Change the ratio with `spec.options.outOfMemory.bumpUpRatio`; see the [DatadogPodAutoscaler manifest reference][15]. - **Guaranteed-class preservation**: When a resource has request equal to limit, Datadog uses the more conservative (limit-level) computation for both, ensuring recommendations do not introduce a gap between request and limit. ## Further reading @@ -657,3 +712,7 @@ Datadog computes vertical scaling recommendations for CPU and memory by analyzin [12]: /containers/guide/manage-datadogpodautoscaler-with-argocd/ [13]: /containers/guide/manage-datdadogpodautoscaler-with-terraform/ [14]: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ +[15]: /containers/autoscaling/manifest/ +[16]: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/ +[17]: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/ +[18]: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#limitations diff --git a/hugo/content/en/containers/autoscaling/manifest.md b/hugo/content/en/containers/autoscaling/manifest.md new file mode 100644 index 00000000000..fe92603837f --- /dev/null +++ b/hugo/content/en/containers/autoscaling/manifest.md @@ -0,0 +1,579 @@ +--- +title: DatadogPodAutoscaler manifest reference +description: Configure DatadogPodAutoscaler custom resources in YAML to access options that the Datadog UI does not expose. +further_reading: +- link: "/containers/autoscaling/" + tag: "Documentation" + text: "Kubernetes Autoscaling" +- link: "/containers/guide/manage-datadogpodautoscaler-with-argocd/" + tag: "Documentation" + text: "Manage DatadogPodAutoscaler with ArgoCD" +- link: "/containers/guide/manage-datdadogpodautoscaler-with-terraform/" + tag: "Documentation" + text: "Manage DatadogPodAutoscaler with Terraform" +--- + +The `DatadogPodAutoscaler` (DPA) custom resource defines autoscaling behavior for a single Kubernetes workload. The [Autoscaling UI][1] with {{< ui >}}Export Recommendation{{< /ui >}} is a good place to start: configure a workload, then copy the generated manifest. Editing the manifest directly gives you access to every field in the custom resource definition (CRD) and makes the DPA a normal part of your GitOps workflow, where the manifest is the reviewed, versioned source of truth. + +This page covers the configuration options available in the manifest. Examples on this page use API version `datadoghq.com/v1alpha2`. + +For setup and prerequisites, see [Kubernetes Autoscaling][2]. That page covers enabling Workload Autoscaling and the Admission Controller on the Datadog Cluster Agent, required Agent versions, and enabling [in-place vertical scaling][3]. + +## Anatomy of a manifest + +The following annotated skeleton shows the structure of a `DatadogPodAutoscaler`. Every field is optional except `targetRef`. + +```yaml +apiVersion: datadoghq.com/v1alpha2 +kind: DatadogPodAutoscaler +metadata: + name: my-app # required: conventionally the workload name + namespace: my-namespace # required: must match the target workload + annotations: # optional + ad.datadoghq.com/tags: '{"team": "my-team"}' # optional: tags on this DPA's telemetry +spec: + owner: Local # optional: Local = this manifest is the source of truth (use for GitOps) + # Remote = created and managed from the Datadog UI + + targetRef: # required: the workload being autoscaled - one DPA per workload + apiVersion: apps/v1 + kind: Deployment + name: my-app + + applyPolicy: # optional + mode: Apply # Apply | Preview (Preview = compute recommendations, change nothing) + + scaleUp: # optional: horizontal, upward + strategy: Max # Max | Min | Disabled + stabilizationWindowSeconds: 600 + rules: + - type: Percent # Percent | Pods + value: 50 + periodSeconds: 120 # 1..3600 + + scaleDown: # optional: horizontal, downward + strategy: Max + stabilizationWindowSeconds: 600 + rules: + - type: Percent + value: 10 + periodSeconds: 1800 + + update: # optional: vertical + strategy: Auto # Auto | Disabled | TriggerRollout + # resizePendingPeriod: 600 # see Vertical rollout timing + # rolloutFallbackDelay: 900 # see Vertical rollout timing + + constraints: # optional + minReplicas: 3 + maxReplicas: 100 + containers: # optional: per-container vertical configuration + - name: "*" # "*" matches all containers + enabled: true + controlledResources: [cpu, memory] + controlledValues: RequestsAndLimits # RequestsAndLimits | RequestsOnly + minAllowed: + cpu: "500m" + memory: 1Gi + maxAllowed: + cpu: "4" + memory: 8Gi + + objectives: # optional: configures horizontal scaling (also used by multidimensional). Exactly one entry. + - type: ContainerResource # PodResource | ContainerResource | CustomQuery + containerResource: + container: my-app + name: cpu # cpu | memory + value: + type: Utilization # Utilization | AbsoluteValue + utilization: 65 + + fallback: # optional: in-cluster horizontal fallback if recommendations go stale + horizontal: + enabled: true + direction: ScaleUp # ScaleUp | ScaleDown | All (default ScaleUp) + triggers: + staleRecommendationThresholdSeconds: 600 # 100..3600, default 600 + + options: # optional + burstable: false # true = remove CPU limits, keep CPU request recommendations + outOfMemory: + bumpUpRatio: "1.2" # +20% memory limit after an OOMKill (default) +``` + +### Supported target workloads + +| `targetRef.kind` | `apiVersion` | Status | +|---|---|---| +| `Deployment` | `apps/v1` | Supported | +| `Rollout` (Argo Rollouts) | `argoproj.io/v1alpha1` | Supported | +| `StatefulSet` | `apps/v1` | Supported | + +For an Argo Rollout, point `targetRef` at the Rollout itself rather than at any Deployment it manages: + +```yaml + targetRef: + apiVersion: argoproj.io/v1alpha1 + kind: Rollout + name: my-app +``` + +### Choose a scaling mode + +The combination of `objectives` and `applyPolicy.update.strategy` determines whether a DPA scales horizontally, vertically, or both: + +| `objectives` set | `applyPolicy.update.strategy` | Resulting mode | +|---|---|---| +| yes | `Disabled` (or unset) | Horizontal only | +| no | `Auto` | Vertical only | +| yes | `Auto` | Multidimensional (both) | + +## Container constraints + +Most vertical options are expressed through `spec.constraints.containers[]`: + +| Field | Type | Default | Meaning | +|---|---|---|---| +| `name` | string, **required** | - | Container name, or `"*"` to match every container that has no entry of its own (see [Exclude a container](#exclude-a-container)) | +| `enabled` | bool | `true` | `false` disables resource autoscaling for this container | +| `controlledResources` | list of `cpu`, `memory` | `[cpu, memory]` | Which resources receive vertical recommendations. An empty list is equivalent to `enabled: false` | +| `controlledValues` | enum | `RequestsAndLimits` | Whether recommendations write both requests _and_ limits, or requests only | +| `minAllowed` | resource map | - | Lower bound for the container's requests | +| `maxAllowed` | resource map | - | Upper bound for the container's requests | + +If `constraints.containers` is omitted entirely, resource scaling is enabled for **all** containers, with no bounds. + +## Right-size CPU and memory + +When a DPA combines horizontal scaling (`objectives`) with vertical scaling (`update.strategy: Auto`), the default behavior is to produce vertical recommendations for **memory only**. CPU requests and limits are left untouched, and the `VerticalAbleToRecommend` condition may show as `Unknown`. + +If you have seen a DPA right-size memory while leaving CPU unchanged, this is the reason. It is not related to Quality of Service (QoS) class preservation. + +Use `controlledResources` to declare which resources receive vertical recommendations: + +| `controlledResources` | Vertical recommendations produced | +|---|---| +| unset | memory only (default) | +| `[memory]` | memory only, same as unset | +| `[cpu, memory]` | **memory and CPU** | +| `[cpu]` | CPU only | + +Full example: + +```yaml +apiVersion: datadoghq.com/v1alpha2 +kind: DatadogPodAutoscaler +metadata: + name: my-app + namespace: my-namespace +spec: + owner: Local + targetRef: + apiVersion: apps/v1 + kind: Deployment + name: my-app + applyPolicy: + mode: Apply + update: + strategy: Auto # required - without it, nothing vertical is applied + constraints: + minReplicas: 3 + maxReplicas: 60 + containers: + - name: "*" + controlledResources: + - cpu # opts CPU into vertical rightsizing + - memory + controlledValues: RequestsAndLimits + objectives: + - type: ContainerResource + containerResource: + container: my-app + name: cpu + value: + type: Utilization + utilization: 65 +``` + +This feature requires Datadog Cluster Agent 7.78.0+. On older versions, the `controlledResources` field is accepted by the CRD but has no effect. + +## Remove CPU limits with burstable mode + +CPU limit recommendations are derived from sustained usage percentiles over a multi-day window. A short warm-up spike (a JVM starting up, for example) is statistically invisible in that window, so the recommended CPU limit can land too low and the application is throttled at the wrong moment. Memory is not affected in the same way, because peak memory usage gives a reliable ceiling. + +Burstable mode **removes CPU limits entirely** while still applying CPU _request_ recommendations: + +```yaml +apiVersion: datadoghq.com/v1alpha2 +kind: DatadogPodAutoscaler +metadata: + name: my-java-app + namespace: my-namespace +spec: + owner: Local + targetRef: + apiVersion: apps/v1 + kind: Deployment + name: my-java-app + applyPolicy: + mode: Apply + update: + strategy: Auto + options: + burstable: true +``` + +If `options.burstable` is left unset, the Cluster Agent's cluster-wide default applies. Set it explicitly to `false` to opt a single workload out of that default. + +Effect on the pod: + +| | Before | After | +|---|---|---| +| CPU request | `1` | `400m` (recommendation) | +| CPU limit | `2` | **removed** | +| Memory request | `500Mi` | `450Mi` (recommendation) | +| Memory limit | `2Gi` | `2Gi`, preserved | + +Before enabling it: + +- Pods that were **Guaranteed** QoS become **Burstable** QoS. This changes their eviction priority under node pressure. +- Without a CPU limit, a container can consume available node CPU. Kernel CPU shares still apply. +- Burstable mode **takes precedence over** `controlledValues` for CPU limits. If both are set, burstable wins. + +## Tune the OOMKill memory bump + +After an OOMKill, the memory limit is raised by **20%** relative to the limit in force at the time. The bump is applied immediately and re-applied on each subsequent OOMKill until the workload stabilizes. This is a self-correcting ratchet rather than a one-time adjustment. + +To change the ratio: + +```yaml +spec: + options: + outOfMemory: + bumpUpRatio: "1.5" # 1.2 = +20% (default), 1.5 = +50% +``` + +Quote the value: it is a Kubernetes quantity, not a floating-point number. + +**When to raise it.** Raise the ratio for workloads whose memory usage can peak sharply above previous peaks. A larger bump reaches the right memory limit faster and avoids several successive bumps before the workload stabilizes. When you raise it, also set a `minAllowed` memory floor (see [Set per-container bounds](#set-per-container-bounds)) so the limit cannot fall back below a safe value between recommendation cycles. + +## Right-size requests only + +If you have deliberately tuned limits (for burst headroom, a platform requirement, or a QoS guarantee) and want Datadog to right-size only **requests**, use `controlledValues: RequestsOnly`. + +```yaml +spec: + constraints: + containers: + - name: my-app + controlledResources: [cpu, memory] + controlledValues: RequestsOnly # limits are not right-sized +``` + +| `controlledValues` | Requests | Limits | +|---|---|---| +| `RequestsAndLimits` (default) | recommended | recommended | +| `RequestsOnly` | recommended | not right-sized, **except** where a limit must move to keep the pod spec valid (see below) | + +Interactions to be aware of: + +- On a container where `request == limit`, lowering the request breaks the Guaranteed QoS class. If you need Guaranteed, keep `RequestsAndLimits`; the recommender handles `request == limit` containers explicitly. +- Burstable mode overrides this for CPU limits (see [Remove CPU limits with burstable mode](#remove-cpu-limits-with-burstable-mode)). +- **OOMKill handling still adjusts the memory limit.** `RequestsOnly` does not suppress the memory bump. After an OOMKill, the memory limit is raised, and the request is potentially raised with it (Kubernetes rejects any pod whose request exceeds its limit). Read `RequestsOnly` as "limits are not _right-sized_", not "limits are never modified". See [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump). + +Choosing a combination: + +| Goal | Configuration | +|---|---| +| Right-size everything | `controlledResources: [cpu, memory]` + `controlledValues: RequestsAndLimits` | +| Right-size requests, leave limits as written | `controlledValues: RequestsOnly` | +| Right-size memory only, leave CPU alone | `controlledResources: [memory]` | +| Right-size CPU requests, no CPU limit at all | `options.burstable: true` | +| Leave a container entirely alone | `enabled: false` | + +## Set per-container bounds + +`minAllowed` and `maxAllowed` are guard-rails on the requests the recommender may produce. They are recommended for latency-sensitive workloads, and advisable whenever you change the OOM bump ratio (see [Tune the OOMKill memory bump](#tune-the-oomkill-memory-bump)), so memory cannot drift below a safe floor between recommendation cycles. + +```yaml +spec: + constraints: + minReplicas: 2 + maxReplicas: 100 + containers: + - name: api + enabled: true + minAllowed: + cpu: "1" + memory: 1Gi + maxAllowed: + cpu: "4" + memory: 5Gi + - name: worker + enabled: true # no bounds - recommendations are unconstrained +``` + +## Exclude a container + +There are two distinct things you may want to exclude a container from. + +### Exclude a container from vertical recommendations + +```yaml +spec: + constraints: + containers: + - name: my-app + enabled: true + - name: istio-proxy + enabled: false # resources for this container are never modified +``` + +Equivalent, if you prefer to be explicit: + +```yaml + - name: istio-proxy + controlledResources: [] # empty list is equivalent to enabled: false +``` + +A common pattern is to autoscale everything except a known sidecar: + +```yaml +spec: + constraints: + containers: + - name: "*" + enabled: true + controlledResources: [cpu, memory] + - name: istio-proxy + enabled: false +``` + +**How `"*"` and named entries combine:** the `"*"` entry applies to every container that does **not** have a named entry. A container with its own named entry takes only the settings declared under that name. The two are **not merged**, so the wildcard contributes nothing to it. + +In the example above, `istio-proxy` is governed solely by `enabled: false` and does not inherit `controlledResources` from the wildcard. Every other container in the pod uses the wildcard entry. + +**Note**: If you add a named entry only to set a bound, repeat any wildcard settings you still want. In the example below, `my-app` falls back to the default `RequestsAndLimits` rather than the `RequestsOnly` set on the wildcard: + +```yaml + - name: "*" + controlledValues: RequestsOnly + - name: my-app + maxAllowed: + memory: 8Gi # controlledValues is NOT inherited - repeat it if you want it +``` + +### Exclude a container from the horizontal signal + +`enabled: false` governs _vertical_ behavior only. The horizontal objective is chosen separately, and this is where sidecars most often distort scaling decisions: + +```yaml + objectives: + # Recommended: scale on the application container's CPU + - type: ContainerResource + containerResource: + container: my-app + name: cpu + value: + type: Utilization + utilization: 65 +``` + +compared with: + +```yaml + objectives: + # Risky when sidecars are present: pod-level utilization is diluted by + # sidecar requests, so a busy application container can appear idle. + - type: PodResource + podResource: + name: cpu + value: + type: Utilization + utilization: 65 +``` + +**Rule of thumb:** if the pod has any sidecar, use `ContainerResource` scoped to the main container. Reserve `PodResource` for genuinely single-container pods. + +## Configure sidecars + +### Ordinary sidecars (`spec.containers`) + +Nothing special is required. They appear as containers and can be bounded, excluded, or targeted like any other; see [Exclude a container](#exclude-a-container). + +### Native sidecars (`spec.initContainers` with `restartPolicy: Always`) + +Kubernetes 1.29+ allows a long-running sidecar to be declared in `initContainers` with `restartPolicy: Always`, the [native sidecar][4] pattern. It runs for the pod's whole lifetime but lives in a different field of the pod spec: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app +spec: + template: + spec: + initContainers: + - name: log-shipper + image: log-shipper:1.2 + restartPolicy: Always # this is what makes it a native sidecar + resources: + requests: {cpu: 100m, memory: 128Mi} + limits: {memory: 256Mi} + containers: + - name: my-app + image: my-app:4.5 + resources: + requests: {cpu: "1", memory: 2Gi} + limits: {cpu: "2", memory: 4Gi} +``` + +**Native sidecars are fully supported.** They are treated as ordinary containers: recommendations are produced and applied for them, and they appear in the workload's container list where they can be configured or excluded. Reference them in `constraints.containers[]` by **name**, exactly like any other container. There is no separate `initContainers` block in the DPA spec, and a `"*"` entry covers them too. + +```yaml +spec: + constraints: + containers: + - name: my-app + controlledResources: [cpu, memory] + controlledValues: RequestsAndLimits + - name: log-shipper # native sidecar, referenced by name + enabled: false +``` + +Points to be aware of: + +- `restartPolicy: Always` is what distinguishes them. Ordinary init containers (those that run to completion before the application starts) are not native sidecars and are not managed by a DPA. +- **Cost and savings figures may under-count native sidecars.** Their resource requests are reported under a separate aggregation, so the cost figures shown for a workload with native sidecars can look inconsistent with its observed usage. This is a known limitation that affects the cost display only; recommendations are unaffected. +- **Injected sidecars** (such as Istio's) are added by a mutating admission webhook at pod level and never appear in the Deployment manifest. They are still picked up, because the container list is reconciled from running pods rather than from the workload manifest alone. +- **Do not exclude autoscaled containers from Agent collection.** A `DatadogPodAutoscaler` relies on the metrics the Agent collects for the containers it manages. If a container in an autoscaled workload is filtered out through the Agent's container discovery configuration, no metrics exist for it and it cannot be right-sized. Confirm that no container in an autoscaled workload is excluded from collection. For how inclusion and exclusion rules work, see [Container Discovery Management][7]. + +## Additional manifest options + +### Preview (dry-run) mode + +```yaml +spec: + applyPolicy: + mode: Preview # recommendations are computed and visible in .status, but nothing is applied +``` + +Useful as a temporary stop switch. If a horizontal configuration is invalid, vertical rightsizing continues to run; setting `mode: Preview` freezes both while you correct it. + +### Disable one scaling direction + +```yaml +spec: + applyPolicy: + scaleUp: + strategy: Max + scaleDown: + strategy: Disabled # never scale down + update: + strategy: Auto +``` + +For vertical-only scaling, omit `objectives` and set `update.strategy: Auto`, as in the [Choose a scaling mode](#choose-a-scaling-mode) table. With no `objectives`, horizontal scaling has no target to act on, so you do not also need to set `scaleUp` and `scaleDown` to `Disabled`. + +### Vertical rollout timing + +The controller applies a vertical change by the least disruptive route available and escalates if that route stalls. Two fields control how long it waits at each step: + +```yaml +spec: + applyPolicy: + update: + strategy: Auto + resizePendingPeriod: 600 # 1..3600 seconds + rolloutFallbackDelay: 900 # 1..3600 seconds +``` + +| Field | Controls | +|---|---| +| `resizePendingPeriod` | How long to wait before evicting a pod when the kubelet reports the resize as pending (accepted but not progressing, often because the node lacks headroom) | +| `rolloutFallbackDelay` | How long to wait before falling back to a full rollout when evictions are blocked, typically by a PodDisruptionBudget | + +Both are optional and accept 1 to 3600 seconds. Leaving them unset uses the controller's built-in defaults. + +- **Raise** `resizePendingPeriod` where eviction is expensive (long warm-up, large caches, slow drain). You trade a longer period at the old size for fewer restarts. +- **Raise** `rolloutFallbackDelay` on workloads with a tight PodDisruptionBudget, so a temporary budget constraint does not immediately escalate to a full rollout. +- **Lower either** where restarts are cheap and you want recommendations to take effect faster. + +These control _how_ a change is delivered, not _whether_ one happens. To stop changes entirely, use `update.strategy: Disabled` or `applyPolicy.mode: Preview`. These fields apply to [in-place vertical scaling][3]; see the overview for cluster-level enablement and Kubernetes requirements. + +### Local fallback tuning + +```yaml +spec: + fallback: + horizontal: + enabled: true + direction: ScaleUp # default; use All to allow fallback scale-in as well + triggers: + staleRecommendationThresholdSeconds: 600 # 100..3600 +``` + +Fallback recommendations are computed inside the cluster from Agent-collected metrics, so scaling continues if Datadog cannot deliver a recommendation within the threshold. + +This feature also requires cluster-side configuration on both the Cluster Agent and the node Agents. See [Kubernetes Autoscaling][2] or contact [Datadog Support][6]. + +### Absolute-value objectives + +Instead of a utilization percentage, target an absolute value: + +```yaml + objectives: + - type: ContainerResource + containerResource: + container: my-app + name: cpu + value: + type: AbsoluteValue + absoluteValue: "1.5" # target cores per pod +``` + +### Custom query objectives + +Scale on any Datadog metric rather than CPU or memory: + +```yaml + objectives: + - type: CustomQuery + customQuery: + window: 5m + request: + queries: + - name: a + source: Metrics + metrics: + query: "avg:my.queue.depth{service:my-app}" + value: + type: AbsoluteValue + absoluteValue: "100" +``` + +`source` may also be `ApmMetrics`, with fields such as `service`, `resourceName`, `operationName`, and `stat`. + +Custom queries are supported for **horizontal scaling only**. Combining a custom query with vertical scaling is **not supported**, because the autoscaler cannot infer which dimension an arbitrary query should act on. + +### Tag a DPA's telemetry + +```yaml +metadata: + annotations: + ad.datadoghq.com/tags: '{"team": "my-team", "tier": "critical"}' +``` + +This adds the tags to the autoscaling telemetry emitted for this DPA. For the list of metrics the Cluster Agent emits, see the [Datadog Cluster Agent integration][5]. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: https://app.datadoghq.com/orchestration/scaling/workload +[2]: /containers/autoscaling/ +[3]: /containers/autoscaling/#in-place-vertical-scaling +[4]: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ +[5]: /integrations/datadog-cluster-agent/#metrics +[6]: /help/ +[7]: /containers/guide/container-discovery-management/