Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions hugo/config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3317,6 +3317,11 @@ menu:
parent: containers
identifier: containers_autoscaling
weight: 2
- name: Manifest reference
url: containers/autoscaling/manifest
parent: containers_autoscaling
identifier: containers_autoscaling_manifest
weight: 200
- name: Cluster
url: containers/autoscaling/cluster
parent: containers_autoscaling
Expand Down
87 changes: 73 additions & 14 deletions hugo/content/en/containers/autoscaling/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,8 @@ helm upgrade -f datadog-values.yaml <RELEASE_NAME> datadog/datadog
{{% /tab %}}
{{< /tabs >}}

**Note**: Vertical scaling recommendations are applied to new pods through the [Admission Controller](/containers/cluster_agent/admission_controller/) mutating webhook. The Admission Controller is enabled by default. If you disable it, horizontal scaling continues to work but vertical scaling has no effect.

### Idle cost and savings estimates

{{< tabs >}}
Expand Down Expand Up @@ -210,6 +212,59 @@ _Fixed cost values are subject to refinement over time._
{{% /tab %}}
{{< /tabs >}}

### In-place vertical scaling

By default, applying a vertical recommendation requires a full pod rollout: the pod template is updated, Kubernetes recreates the pods, and the new resources take effect as those pods are admitted. For slow-starting or latency-sensitive services, that is a meaningful disruption.

In-place vertical scaling instead updates container resources on the running pods through the Kubernetes [pod resize subresource][16], so most resizes happen with no restart. In-place vertical scaling is supported on Kubernetes 1.33+, where the `InPlacePodVerticalScaling` feature gate is enabled by default. It requires Datadog Cluster Agent 7.78+.

A resize takes effect on the running container without restarting the application process. Applications that read their CPU or memory requests and limits only at startup do not see the new values until they restart, and environment variables populated from resource fields through the [downward API][17] are not refreshed on an in-place resize. Keep this in mind for workloads that size internal components (thread pools, heap, or caches) from their resource requests or limits.

With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling for a workload, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`. To tune how long the controller waits before evicting a pending resize or falling back to a rollout, see the [DatadogPodAutoscaler manifest reference][15].

#### Enable in-place vertical scaling

Enable the feature on the Cluster Agent. This also grants the Cluster Agent the permissions it needs to resize and, where necessary, evict pods.

{{< tabs >}}
{{% tab "Datadog Operator" %}}

```yaml
spec:
features:
autoscaling:
workload:
enabled: true
inPlaceVerticalScaling:
enabled: true
```

{{% /tab %}}
{{% tab "Helm" %}}

```yaml
datadog:
autoscaling:
workload:
enabled: true
inPlaceVerticalScaling:
enabled: true
```

{{% /tab %}}
{{< /tabs >}}

#### Behavior and limitations

- **`resizePolicy` stays under your control.** Datadog never sets or overrides the container-level `resizePolicy`; it is immutable after pod creation and is an application-level decision. When unset, Kubernetes defaults to `NotRequired` for CPU and memory, meaning resize without restart. Set `RestartContainer` per resource on containers that cannot absorb a live change.
Comment thread
clamoriniere marked this conversation as resolved.
- **Kubernetes limitations apply.** See the [upstream limitations][18]:
- Only CPU and memory can be resized.
- QoS class cannot change.
- Requests and limits can be changed but not removed entirely.
- Windows pods and pods under static CPU or memory manager policies are excluded.
- **Burstable mode still requires a rollout.** Burstable mode removes the CPU limit, and in-place resize can change a limit but not remove one. See the [DatadogPodAutoscaler manifest reference][15].
- If a resize is infeasible or stays pending, the pod is evicted through the Kubernetes Eviction API, which respects PodDisruptionBudgets.

## Usage

### Identify resources to rightsize
Expand Down Expand Up @@ -249,7 +304,7 @@ The Setup wizard is best for trying autoscaling on a single workload, getting ha

#### Path B: GitOps

Define a `DatadogPodAutoscaler` custom resource that targets your workload and apply it through whatever tooling you already use to ship Kubernetes manifests, whether that's `kubectl apply`, Helm, ArgoCD, Terraform, or another GitOps tool. Authoring the manifest is the same regardless of delivery mechanism. See the [example configurations](#example-datadogpodautoscaler-configurations) below for ready-to-edit starting points covering cost optimization, balanced scaling, vertical-only resizing, and custom-query horizontal scaling.
Define a `DatadogPodAutoscaler` custom resource that targets your workload and apply it through whatever tooling you already use to ship Kubernetes manifests, whether that's `kubectl apply`, Helm, ArgoCD, Terraform, or another GitOps tool. Authoring the manifest is the same regardless of delivery mechanism. See the [example configurations](#example-datadogpodautoscaler-configurations) below for ready-to-edit starting points covering cost optimization, balanced scaling, vertical-only resizing, and custom-query horizontal scaling. For a complete reference of the manifest fields and options, including options the UI does not expose, see the [DatadogPodAutoscaler manifest reference][15].

For tool-specific guides, see:

Expand All @@ -258,7 +313,7 @@ For tool-specific guides, see:

### Example DatadogPodAutoscaler configurations

The following examples demonstrate common `DatadogPodAutoscaler` configurations for different scaling strategies. Use them as starting points and adjust the values to match your workload's requirements. If you would rather pick a template in the UI, follow [Path A](#path-a-datadog-ui-setup-wizard) above.
The following examples demonstrate common `DatadogPodAutoscaler` configurations for different scaling strategies. Use them as starting points and adjust the values to match your workload's requirements. If you would rather pick a template in the UI, follow [Path A](#path-a-datadog-ui-setup-wizard) above. For the full range of manifest fields and options not shown here, see the [DatadogPodAutoscaler manifest reference][15].

{{< tabs >}}
{{% tab "Optimize Cost" %}}
Expand Down Expand Up @@ -361,15 +416,7 @@ spec:

Pick this template when a workload can't be scaled horizontally, or when you want pure rightsizing without changing replica counts. Common cases are singleton services, stateful workloads, and leader-elected components. The defining setting is `scaleDown.strategy: Disabled` and `scaleUp.strategy: Disabled`, which leaves only `update.strategy: Auto` to apply CPU and memory recommendations.

By default, the controller applies vertical recommendations by triggering a rollout (evict and recreate pods). Cluster Agent **7.78+** also supports **in-place pod resizing**, which updates a pod's CPU and memory requests and limits without restarting it. In-place resize is opt-in: set `autoscaling.workload.in_place_vertical_scaling.enabled: true` on the Cluster Agent (or set the environment variable `DD_AUTOSCALING_WORKLOAD_IN_PLACE_VERTICAL_SCALING_ENABLED=true`).

Your cluster must also expose the `pods/resize` subresource. This is the default in Kubernetes 1.33+ where the `InPlacePodVerticalScaling` feature gate is beta. On Kubernetes 1.27 to 1.32, the feature gate must be enabled on `kube-apiserver` and every `kubelet`.

When both prerequisites are met:

- **Default**: Workloads with `applyPolicy.update.strategy: Auto` (the default) resize in place.
- **Fallback**: If the kubelet reports a resize as `Infeasible`, the controller falls back to a rollout.
- **Opt-out**: To force a workload to always use rollout-based vertical scaling regardless of the cluster setting, set `applyPolicy.update.strategy: TriggerRollout` on its `DatadogPodAutoscaler`.
By default, the controller applies vertical recommendations by triggering a rollout (evict and recreate pods). Cluster Agent **7.78+** also supports **in-place pod resizing**, which updates a pod's CPU and memory requests and limits without restarting it. With `applyPolicy.update.strategy: Auto` (the default), the controller resizes in place wherever the cluster supports it and falls back to a rollout otherwise. To force rollout-based vertical scaling, set `applyPolicy.update.strategy: TriggerRollout`. For enablement, Kubernetes requirements, and behavior, see [In-place vertical scaling](#in-place-vertical-scaling).

```yaml
apiVersion: datadoghq.com/v1alpha2
Expand Down Expand Up @@ -427,7 +474,7 @@ spec:
type: Percent
value: 50
stabilizationWindowSeconds: 130
# Vertical updates disabled horizontal only
# Vertical updates disabled, horizontal only
update:
strategy: Disabled
constraints:
Expand Down Expand Up @@ -469,6 +516,14 @@ spec:
{{% /tab %}}
{{< /tabs >}}

The examples above cover the most common strategies. The manifest supports additional options that the templates don't show, including:

- **CPU rightsizing** alongside memory (`constraints.containers[].controlledResources`). When a workload combines horizontal and vertical scaling, vertical recommendations cover memory only by default.
- **Burstable mode** (`spec.options.burstable`) to remove CPU limits on spiky workloads while still rightsizing CPU requests.
- **Per-container bounds** (`minAllowed` and `maxAllowed`), request-only rightsizing (`controlledValues: RequestsOnly`), and per-container targeting for horizontal scaling (`ContainerResource` objectives).

For the full field reference, see the [DatadogPodAutoscaler manifest reference][15].

### Cluster profiles

A `DatadogPodAutoscalerClusterProfile` is a cluster-scoped resource that holds a `DatadogPodAutoscaler` template. The Cluster Agent watches `Deployment` and `StatefulSet` resources (and, on 7.79+, the namespaces that contain them) for the `autoscaling.datadoghq.com/profile` label, and creates a managed `DatadogPodAutoscaler` for every matching workload. One profile applies to many workloads; one workload still maps to one `DatadogPodAutoscaler`.
Expand Down Expand Up @@ -558,7 +613,7 @@ The template body accepts the same fields as a `DatadogPodAutoscaler` spec, minu

#### Activation precedence

Cluster Agent 7.79.0+ adds namespace-level activation, the `excluded` opt-out, and the precedence rule between them. On Cluster Agent 7.78.0, only the workload-level label is read — the rules below that involve namespaces or the `excluded` value do not apply.
Cluster Agent 7.79.0+ adds namespace-level activation, the `excluded` opt-out, and the precedence rule between them. On Cluster Agent 7.78.0, only the workload-level label is read. The rules below that involve namespaces or the `excluded` value do not apply.

- **Workload labels take precedence over namespace labels.** If a namespace is labeled `autoscaling.datadoghq.com/profile=ns-profile` and a workload inside it is labeled `autoscaling.datadoghq.com/profile=workload-profile`, the workload uses `workload-profile`.
- **Opt out with `excluded`.** Set `autoscaling.datadoghq.com/profile: excluded` on a workload to exempt it when its namespace is labeled. This is useful for stateful or critical workloads in an otherwise opted-in namespace.
Expand Down Expand Up @@ -636,7 +691,7 @@ Datadog computes vertical scaling recommendations for CPU and memory by analyzin
- **8-day lookback window**: All recommendations consider usage data from the past 8 days, providing enough history to capture weekly traffic patterns while remaining responsive to changes.
- **Decaying weights**: For Burstable-class request recommendations (CPU or memory), older samples are weighted less heavily, so the recommendation adapts faster to recent usage shifts.
- **Safety margins**: Every recommendation includes a margin above observed usage (5 to 10%) to provide a buffer against unexpected spikes.
- **OOMKill response**: When memory is Guaranteed-class (request equals limit) and an OOMKill occurs, a 20% bump is applied to reduce the likelihood of repeated out-of-memory failures.
- **OOMKill response**: When an OOMKill occurs, the memory limit is raised (by 20% by default) and re-applied on each subsequent OOMKill until the workload stabilizes, reducing the likelihood of repeated out-of-memory failures. Change the ratio with `spec.options.outOfMemory.bumpUpRatio`; see the [DatadogPodAutoscaler manifest reference][15].
- **Guaranteed-class preservation**: When a resource has request equal to limit, Datadog uses the more conservative (limit-level) computation for both, ensuring recommendations do not introduce a gap between request and limit.

## Further reading
Expand All @@ -657,3 +712,7 @@ Datadog computes vertical scaling recommendations for CPU and memory by analyzin
[12]: /containers/guide/manage-datadogpodautoscaler-with-argocd/
[13]: /containers/guide/manage-datdadogpodautoscaler-with-terraform/
[14]: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
[15]: /containers/autoscaling/manifest/
[16]: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/
[17]: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
[18]: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#limitations
Loading
Loading