[DOCS-15714] Add Node.js experiment setup instructions by mehulsonowal · Pull Request #40594 · DataDog/documentation

mehulsonowal · 2026-10-09T16:51:32Z

What does this PR do? What is the motivation?

Fixes DOCS-15714

Adds Node.js setup and usage instructions for Agent Observability experiments and reorganizes the detailed experiment documentation under Guides. This PR supersedes #39963 and contains the finalized documentation changes with all authored commits GitHub-verified under the mehulsonowal account.

Merge readiness

Ready for merge

AI assistance

Documentation edits were prepared with Pi and validated with the repository hooks and whitespace checks.

Additional notes

The previous PR is closed in favor of this branch. All authored commits on this branch use the mehulsonowal GitHub signing identity.

github-actions · 2026-10-09T16:55:16Z

Preview links (active after the `build_preview` check completes)

New or renamed files

https://docs-staging.datadoghq.com/docs/nodejs-experiments-setup-mehulsonowal/llm_observability/guide/experiments

Removed or renamed files (these should redirect)

https://docs-staging.datadoghq.com/docs/nodejs-experiments-setup-mehulsonowal/llm_observability/improve/experiments/setup

Renamed files

Modified Files

joepeeples · 2026-10-09T18:35:44Z

Opened DOCS-15968 to assign a Docs writer and follow up with editorial review.

domalessi

Left some feedback in line. Almost there!

domalessi · 2026-10-09T19:36:29Z

+NODE_OPTIONS="--import dd-trace/initialize.mjs" node <YOUR_APP_ENTRYPOINT>
+```
+
+For more information, see the [Node.js tracer command-line setup](/llm_observability/instrument/sdk?tab=nodejs#command-line-setup).


Links to other pages should be defined reference style. Please apply this feedback to all inline links (inline anchor links to sections on the same page are fine).

domalessi · 2026-10-09T19:37:39Z

+{{% /tab %}}
+    {{< /tabs >}}
+
+### APM Trace correlation


Suggested change

### APM Trace correlation

### APM trace correlation

domalessi · 2026-10-09T19:46:30Z

+  text: "Debug and evaluate your AI app from your coding agent with Datadog Agent Observability"
+---
+
+This guide describes how to set up and use Agent Observability experiments with the Python or Node.js SDK. For complete runnable Node.js examples, see the [Node.js experiments examples](https://github.com/DataDog/llm-observability/tree/main/experiments/nodejs).


Link should also be reference style

Suggested change

This guide describes how to set up and use Agent Observability experiments with the Python or Node.js SDK. For complete runnable Node.js examples, see the [Node.js experiments examples](https://github.com/DataDog/llm-observability/tree/main/experiments/nodejs).

This guide describes how to set up and use Agent Observability experiments with the Python or Node.js SDK. See [runnable Node.js experiments examples](https://github.com/DataDog/llm-observability/tree/main/experiments/nodejs).

domalessi · 2026-10-09T19:47:46Z

I wonder if we should have a more featured/explicit call-out to the Setup docs. You can find it in Further reading, but a user might not readily spot it. The page feels fairly sparse and odd without a direct pointer to setup instructions, I think?

domalessi · 2026-10-09T20:00:26Z

+
+### 3. Define evaluators
+
+Evaluators measure how well your model or agent performs on each record. Both SDKs support function-based evaluators and reusable class-based evaluators.


Clarify the distinction between local evaluators defined as functions or classes and managed remote evaluators, so the later remote-evaluator section does not appear to introduce an unexplained third type.

Suggested change

Evaluators measure how well your model or agent performs on each record. Both SDKs support function-based evaluators and reusable class-based evaluators.

Evaluators measure how well your model or agent performs on each record. Both SDKs support local evaluators defined as functions or classes. The Node.js SDK also supports managed remote evaluators, which run evaluations configured in Datadog.

domalessi · 2026-10-09T20:00:27Z

+{{% /tab %}}
+
+{{% tab "Node.js" %}}
+The Node.js SDK supports function and class-based evaluators. Return a Boolean, number, string, or JSON-serializable object. Use an object map when you want to assign evaluator names explicitly.


Label the first example as function-based and remove the repeated SDK overview. This also makes the Node.js tab parallel to the Python tab.

Suggested change

The Node.js SDK supports function and class-based evaluators. Return a Boolean, number, string, or JSON-serializable object. Use an object map when you want to assign evaluator names explicitly.

#### Function-based evaluators

Return a Boolean, number, string, or JSON-serializable object. Use an object map to assign evaluator names explicitly.

domalessi · 2026-10-09T20:00:28Z

+{{% /tab %}}
+
+{{% tab "Node.js" %}}
+Summary evaluators are functions that receive the inputs, outputs, expected outputs, and an object containing the results from each evaluator. The optional fifth argument contains record metadata.


Add a heading for the function-based example, parallel to the Python tab and the class-based subsection that follows. Scope the explanation to functions rather than implying that all summary evaluators are functions.

Suggested change

Summary evaluators are functions that receive the inputs, outputs, expected outputs, and an object containing the results from each evaluator. The optional fifth argument contains record metadata.

#### Function-based summary evaluators

Summary evaluator functions receive the inputs, outputs, expected outputs, and an object containing the results from each evaluator. The optional fifth argument contains record metadata.

domalessi · 2026-10-09T20:00:28Z

+{{% /tab %}}
+{{< /tabs >}}
+
+### 5. Create and run the experiment.


Introduce the first examples so readers can distinguish creating and running an experiment from the execution options shown later. Also remove the period from the heading.

Suggested change

### 5. Create and run the experiment.

### 5. Create and run the experiment

Create an experiment with your dataset, task, and evaluators, then run it and inspect the results:

domalessi · 2026-10-09T20:00:29Z

+{{% /tab %}}
+{{< /tabs >}}
+
+### 6. Review your experiment results in Datadog.


Suggested change

### 6. Review your experiment results in Datadog.

### 6. Review your experiment results in Datadog

domalessi · 2026-10-09T20:02:00Z

+```python
+def num_exact_matches(inputs, outputs, expected_outputs, evaluators_results):
+    return evaluators_results["exact_match"].count(True)
+```
+
+Summary evaluator functions can take a list of any non-null type as `inputs` (string, number, Boolean, object, or array); `outputs` and `expected_outputs` can be lists of any type. `evaluators_results` is a dictionary of lists of results from evaluators, keyed by the name of the evaluator function.


Suggested change

```python

def num_exact_matches(inputs, outputs, expected_outputs, evaluators_results):

return evaluators_results["exact_match"].count(True)

```

Summary evaluator functions can take a list of any non-null type as `inputs` (string, number, Boolean, object, or array); `outputs` and `expected_outputs` can be lists of any type. `evaluators_results` is a dictionary of lists of results from evaluators, keyed by the name of the evaluator function.

Summary evaluator functions can take a list of any non-null type as `inputs` (string, number, Boolean, object, or array); `outputs` and `expected_outputs` can be lists of any type. `evaluators_results` is a dictionary of lists of results from evaluators, keyed by the name of the evaluator function.

```python

def num_exact_matches(inputs, outputs, expected_outputs, evaluators_results):

return evaluators_results["exact_match"].count(True)

```

datadog-official · 2026-10-09T21:09:45Z

Tests

🔄 Datadog auto-retried 2 jobs - 0 passed on retry

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 3bc2cfb | Docs | View more details | Give us feedback!}

[DOCS-15714] Update Agent Observability experiments guide

342aa21

mehulsonowal requested a review from a team as a code owner October 9, 2026 16:51

github-actions Bot added Architecture Everything related to the Doc backend Guide Content impacting a guide labels Oct 9, 2026

mehulsonowal mentioned this pull request Oct 9, 2026

Add Node.js experiment setup instructions #39963

Closed

1 task

[DOCS-15714] Document Node.js evaluator APIs

ec0fcd0

joepeeples added the editorial review Waiting on a more in-depth review label Oct 9, 2026

domalessi reviewed Oct 9, 2026

View reviewed changes

[DOCS-15714] Address experiments guide review feedback

3bc2cfb

	This guide describes how to set up and use Agent Observability experiments with the Python or Node.js SDK. For complete runnable Node.js examples, see the [Node.js experiments examples](https://github.com/DataDog/llm-observability/tree/main/experiments/nodejs).
	This guide describes how to set up and use Agent Observability experiments with the Python or Node.js SDK. See [runnable Node.js experiments examples](https://github.com/DataDog/llm-observability/tree/main/experiments/nodejs).


		### 3. Define evaluators

		Evaluators measure how well your model or agent performs on each record. Both SDKs support function-based evaluators and reusable class-based evaluators.

-The Node.js SDK supports function and class-based evaluators. Return a Boolean, number, string, or JSON-serializable object. Use an object map when you want to assign evaluator names explicitly.
+#### Function-based evaluators
+Return a Boolean, number, string, or JSON-serializable object. Use an object map to assign evaluator names explicitly.

-Summary evaluators are functions that receive the inputs, outputs, expected outputs, and an object containing the results from each evaluator. The optional fifth argument contains record metadata.
+#### Function-based summary evaluators
+Summary evaluator functions receive the inputs, outputs, expected outputs, and an object containing the results from each evaluator. The optional fifth argument contains record metadata.

	### 6. Review your experiment results in Datadog.
	### 6. Review your experiment results in Datadog

Conversation

mehulsonowal commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do? What is the motivation?

Merge readiness

AI assistance

Additional notes

Uh oh!

github-actions Bot commented Oct 9, 2026

Preview links (active after the build_preview check completes)

New or renamed files

Removed or renamed files (these should redirect)

Renamed files

Modified Files

Uh oh!

joepeeples commented Oct 9, 2026

Uh oh!

domalessi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

datadog-official Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mehulsonowal commented Oct 9, 2026 •

edited

Loading

Preview links (active after the `build_preview` check completes)

datadog-official Bot commented Oct 9, 2026 •

edited

Loading