Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 2 additions & 9 deletions hugo/content/en/observability_pipelines/_index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Observability Pipelines
description: Learn how Observability Pipelines lets you collect, process, and route logs, metrics, and traces within your own infrastructure to destinations such as Datadog, Amazon S3, Splunk, and Microsoft Sentinel.
description: Learn how Observability Pipelines lets you collect, process, and route logs and metrics within your own infrastructure to destinations such as Datadog, Amazon S3, Splunk, and Microsoft Sentinel.
disable_toc: false
further_reading:
- link: "/observability_pipelines/configuration/explore_templates/"
Expand Down Expand Up @@ -78,7 +78,7 @@ further_reading:

{{< img src="observability_pipelines/op_marketecture_06042025.png" alt="A graphic showing data being aggregated from a variety of sources, processed and enriched by the observability pipelines worker in your own environment, and then being routed to the security, analytics, and storage destinations of your choice" style="width:100%;" >}}

Datadog Observability Pipelines allows you to collect and process {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment.
Datadog Observability Pipelines allows you to collect and process logs and metrics within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment.

With out-of-the-box templates, you can build pipelines that redact sensitive data, enrich data, filter out noisy events, and route data to destinations like Datadog, SIEM tools, or cloud storage.

Expand Down Expand Up @@ -141,13 +141,6 @@ Observability Pipelines includes prebuilt templates for common data routing and
|----------|-------------|
| Metric Tag Governance | Manage the quality and volume of your metrics by keeping only the metrics you need, standardizing metrics tagging, and removing unwanted tags to prevent high cardinality. |

{{% /tab %}}
{{% tab "Traces" %}}

| Template | Description |
|----------|-------------|
| Trace Sampling | Ingest, process, and route traces to control costs while retaining the traces you need for troubleshooting and analysis. |

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ further_reading:

{{< img src="observability_pipelines/setup/pipeline_ui.png" alt="The pipelines page with a source going to two processors groups and two destinations" style="width:100%;" >}}

Observability Pipelines lets you collect and process {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:
Observability Pipelines lets you collect and process logs and metrics within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:

- [Source][1]: Receives data from a tool like the Datadog Agent.
- [Processors][2]: Transform, enrich, or filter data.
Expand Down Expand Up @@ -120,17 +120,6 @@ See [Metric Types][3] for more information.
[3]: /metrics/types/?tab=gauge#metric-types
[4]: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality

{{% /tab %}}

{{% tab "Traces" %}}

You can ingest, process, and send {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}} to different destinations using the [Trace Sampling][1] template.

See [Set Up Pipelines][2] for more information on setting up a source, processors, and destinations.

[1]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling
[2]: /observability_pipelines/configuration/set_up_pipelines/

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,6 @@ Metrics capture signals about your environment and offer insight into your syste

To help you manage the quality and volume of your metrics, use the Metric Tag Governance template to process them in Observability Pipelines before sending them to your destinations. You can use processors to keep only the metrics you need, standardize metrics tagging, and remove unwanted tags to prevent high cardinality.

{{% /tab %}}
{{% tab "Traces" %}}

### Trace Sampling

Use the Trace Sampling template to ingest, process, and route your {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}} telemetry data to control costs while retaining the traces you need for troubleshooting and analysis.

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ For RHEL and CentOS, the Observability Pipelines Worker supports versions 8.0 or

{% /if %}

The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your {% tooltip contents="Contact your account manager to discuss use cases and pricing." %}logs, metrics, and traces{% /tooltip %}, and then route them to different destinations.
The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs and metrics, and then route them to different destinations.

<!-- Kubernetes - Overview -->
{% if equals($platform, "kubernetes") %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,6 @@ products:
- name: Metrics
icon: metrics
url: /observability_pipelines/configuration/?tab=metrics#pipeline-types
- name: Traces
icon: apm
url: /observability_pipelines/configuration/?tab=traces#pipeline-types
---

{{< product-availability >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,31 +110,6 @@ See [Export a Pipeline Configuration to JSON or Terraform][14] if you want to pr
[6]: /observability_pipelines/configuration/pipeline_simulation/
[11]: /observability_pipelines/search_syntax/metrics/

{{% /tab %}}
{{% tab "Traces" %}}

1. Navigate to [Observability Pipelines][1].
1. Select the [Trace Sampling][2] template, or click {{< ui >}}New Pipeline{{< /ui >}} and select {{< ui >}}Traces Pipeline{{< /ui >}}.
1. Select and set up a [trace source][3].
1. Select and set up [destinations][5] for your processed traces.
1. Click {{< ui >}}Edit{{< /ui >}} on a processor group to add, update, and validate your [processors][4] using [Pipeline Simulation][6].
- **Notes**:
- For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors.
- To copy a processor, click the copy icon for that processor and then paste it (`Cmd+V` on Mac, `Ctrl+V` on Windows or Linux).

#### Add another processor group

{{< img src="observability_pipelines/setup/another_processor_group.png" alt="The Pipelines page showing two processor groups sending logs to the same destination" style="width:100%;" >}}

{{% observability_pipelines/set_up_pipelines/add_another_processor_group %}}

[1]: https://app.datadoghq.com/observability-pipelines
[2]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling
[3]: /observability_pipelines/sources/?tab=traces#sources
[4]: /observability_pipelines/processors/?tab=traces#processors
[5]: /observability_pipelines/destinations/?tab=traces#destinations
[6]: /observability_pipelines/configuration/pipeline_simulation/

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -208,14 +208,8 @@ On the Worker installation page:
{{% /tab %}}
{{% tab "OpenTelemetry" %}}

**Metrics**

{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_metrics %}}

**Traces**

{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces %}}

{{% /tab %}}
{{% tab "Prometheus" %}}

Expand Down
14 changes: 4 additions & 10 deletions hugo/content/en/observability_pipelines/destinations/_index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
---
title: Destinations
disable_toc: false
aliases:
- /observability_pipelines/destinations/datadog_apm/
- /observability_pipelines/destinations/opentelemetry/traces/
further_reading:
- link: "logs/processing/pipelines"
tag: "Documentation"
Expand All @@ -9,7 +12,7 @@ further_reading:

## Overview

Use the Observability Pipelines Worker to send your processed {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information.
Use the Observability Pipelines Worker to send your processed logs and metrics to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information.

Select a destination in the left navigation menu to see more information about it.

Expand Down Expand Up @@ -88,15 +91,6 @@ These are the available destinations:
[5]: /observability_pipelines/destinations/opentelemetry/metrics
[6]: /observability_pipelines/destinations/prometheus/

{{% /tab %}}
{{% tab "Traces" %}}

- [Datadog APM][1]
- [OpenTelemetry Traces][2]

[1]: /observability_pipelines/destinations/datadog_apm/
[2]: /observability_pipelines/destinations/opentelemetry/traces

{{% /tab %}}
{{< /tabs >}}

Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
---
title: OpenTelemetry Metrics Destination
title: OpenTelemetry Destination
disable_toc: false
code_lang: metrics
type: multi-code-lang
weight: 1
products:
- name: Metrics
icon: metrics
url: /observability_pipelines/configuration/?tab=metrics#pipeline-types
---

{{< product-availability >}}

## Overview

Use Observability Pipelines' {{< tooltip text=" OpenTelemetry destination" tooltip="Contact your account manager to request access." >}} to send metrics over HTTP/S to an OpenTelemetry (OTel) Collector or another OpenTelemetry Protocol (OTLP)-compatible endpoint.
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -133,13 +133,8 @@ Some Observability Pipelines components require setting up environment variables
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opensearch %}}

### OpenTelemetry

**Metrics**
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_metrics %}}

**Traces**
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces %}}

### Prometheus
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/prometheus %}}

Expand Down
19 changes: 3 additions & 16 deletions hugo/content/en/observability_pipelines/processors/_index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
title: Processors
disable_toc: false
aliases:
- /observability_pipelines/processors/tail_based_sampling/
further_reading:
- link: https://www.datadoghq.com/blog/rehydrate-archived-logs-with-observability-pipelines
tag: Blog
Expand All @@ -14,7 +16,7 @@ further_reading:

<div class="alert alert-info">The processors outlined in this documentation are specific to on-premises logging environments. To parse, structure, and enrich cloud-based logs, see the <a href="https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics">Log Management</a> documentation.</div>

Use Observability Pipelines' processors to parse, structure, and enrich your {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}}. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.
Use Observability Pipelines' processors to parse, structure, and enrich your logs and metrics. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.

Processor groups are executed from top to bottom. The order of the processors is important because events are checked by each processor, but only events that match the processor's filters are processed. To modify the order of the processors, use the drag handle on the top left corner of the processor you want to move.

Expand Down Expand Up @@ -84,21 +86,6 @@ These are the available processors:
[4]: /observability_pipelines/processors/tag_allow_block_list/
[5]: /observability_pipelines/processors/tag_cardinality_control/

{{% /tab %}}
{{% tab "Traces" %}}

- [Custom Processor][1]
- [Filter][2]
- [Sample][3]
- [Sensitive Data Scanner][4]
- [Tail-Based Sampling][5]

[1]: /observability_pipelines/processors/custom_processor/
[2]: /observability_pipelines/processors/filter/
[3]: /observability_pipelines/processors/sample/
[4]: /observability_pipelines/processors/sensitive_data_scanner/
[5]: /observability_pipelines/processors/tail_based_sampling/

{{% /tab %}}
{{< /tabs >}}

Expand Down
Loading
Loading