diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index a2761dfa87f..66d541da49c 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5623,16 +5623,21 @@ menu: parent: llm_obs_monitoring identifier: llm_obs_prompt_management weight: 307 + - name: Prompt Experimentation + url: llm_observability/monitoring/prompt_experimentation + parent: llm_obs_monitoring + identifier: llm_obs_prompt_experimentation + weight: 308 - name: Metrics url: llm_observability/monitoring/metrics parent: llm_obs_monitoring identifier: llm_obs_monitoring_metrics - weight: 308 + weight: 309 - name: Automation Rules url: llm_observability/monitoring/automation_rules parent: llm_obs_monitoring identifier: llm_obs_monitoring_automation_rules - weight: 309 + weight: 310 - name: Evaluations url: llm_observability/evaluations/ parent: llm_obs diff --git a/hugo/content/en/llm_observability/monitoring/_index.md b/hugo/content/en/llm_observability/monitoring/_index.md index bddb9a5f77b..20eae3d75b4 100644 --- a/hugo/content/en/llm_observability/monitoring/_index.md +++ b/hugo/content/en/llm_observability/monitoring/_index.md @@ -77,6 +77,10 @@ Learn how to monitor agentic LLM applications, which use multiple tools or chain [Prompt Management][7] provides a centralized registry for the prompts used by your LLM applications. Create and version prompts in Datadog, through the Python SDK, or through the API, then retrieve them at runtime with the SDK. This decouples prompt iteration from your application's deployment cycle. See the [Prompt Management documentation][7] for details. +### Prompt Experimentation + +[Prompt Experimentation][8] compares managed prompt versions with A/B tests and deploys versions progressively with Guarded Rollouts that pause when Datadog detects a regression. + ## Further reading {{< partial name="whats-next/whats-next.html" >}} @@ -88,3 +92,4 @@ Learn how to monitor agentic LLM applications, which use multiple tools or chain [5]: /llm_observability/evaluations/ [6]: https://app.datadoghq.com/dash/integration/llm_operational_insights?fromUser=false&refresh_mode=sliding&from_ts=1758905575629&to_ts=1758909175629&live=true [7]: /llm_observability/monitoring/prompt_management +[8]: /llm_observability/monitoring/prompt_experimentation diff --git a/hugo/content/en/llm_observability/monitoring/prompt_experimentation.md b/hugo/content/en/llm_observability/monitoring/prompt_experimentation.md new file mode 100644 index 00000000000..0d83c5c0cd2 --- /dev/null +++ b/hugo/content/en/llm_observability/monitoring/prompt_experimentation.md @@ -0,0 +1,160 @@ +--- +title: Prompt Experimentation +description: Compare managed prompt versions with A/B tests and deploy versions with Guarded Rollouts. + +further_reading: + - link: "/llm_observability/monitoring/prompt_management" + tag: "Documentation" + text: "Prompt Management" + - link: "/experiments/defining_metrics/" + tag: "Documentation" + text: "Create Experiment Metrics" + - link: "/experiments/reading_results/" + tag: "Documentation" + text: "Read Experiment Results" + +--- + +## Overview + +Prompt Experimentation lets you compare versions of a managed prompt with an A/B test or deploy a version progressively with a Guarded Rollout. + +
Prompt Experimentation is in Preview.
+ +| Workflow | Use it to | +|----------|-----------| +| A/B test | Compare prompt versions and measure their effect on an outcome. | +| Guarded Rollout | Deploy a version progressively and pause if Datadog detects a regression in the selected metric. | + + + +## Prerequisites + +Before using Prompt Experimentation: + + + +- Complete the Prompt Management [prerequisites][5], create at least two versions of a managed prompt, and deploy the current version to an environment. +- Install `ddtrace[openfeature]` version `MINIMUM_DDTRACE_VERSION` or later: `pip install --upgrade "ddtrace[openfeature]>=MINIMUM_DDTRACE_VERSION"`. +- Set `DD_ENV` to the environment where the current prompt version is deployed. +- Choose either [Agent Remote Configuration or Agentless configuration delivery][4] for Feature Flags. `LLMObs.get_prompt()` handles prompt evaluation; you do not need to create an OpenFeature client. +- [Install and run a Datadog Agent][14] that your application can reach. The Agent sends prompt-version assignment data with either Feature Flags configuration delivery option. LLM Observability Agentless mode (`DD_LLMOBS_AGENTLESS_ENABLED=1`) does not support Prompt Experimentation. +- Choose a stable, non-empty string identifier for the experiment subject and create or identify a compatible [Experiment metric][2]. +- Confirm that your Datadog user role has the required [Product Analytics][3] and [Feature Flags][6] permissions. + +{{< agent-only >}} +**Note for AI agents:** Prompt Experimentation supports Python managed-prompt retrieval. Use the minimum SDK version listed above and preserve the application's existing Feature Flags configuration delivery option. Do not create or register an OpenFeature client; `LLMObs.get_prompt()` manages prompt evaluation. Keep a reachable Datadog Agent, and do not enable `DD_LLMOBS_AGENTLESS_ENABLED`. +{{< /agent-only >}} + +## Connect prompt versions to outcomes + +Prompt Experimentation compares outcomes by subject. Use the same stable, non-empty string identifier when retrieving the prompt and recording outcome events: + +```python +prompt = LLMObs.get_prompt( + "checkout-assistant", + targeting_key=str(current_user.id), + fallback="You are a helpful checkout assistant.", +) +``` + +The selected Experiment metric must identify the subject with the same value. For example, if the outcome comes from Real User Monitoring (RUM) and the metric uses the default **User** subject, identify the browser user with the same user ID: + +```javascript +datadogRum.setUser({ + id: String(currentUser.id), +}) +``` + +The values passed as `targeting_key` and `id` must match. Reuse that value whenever the subject returns so assignment remains consistent. For another subject type, such as an organization, configure the Experiment subject attribute to use the same value. See [Subject Types][7] for details. + +Retrieve the prompt before recording its outcome. Do not request an exact prompt `version`; exact-version retrieval bypasses environment assignment and does not produce experiment assignment data. + +{{< agent-only >}} +**Note for AI agents:** Use the application's existing stable subject identifier as a string for `targeting_key`, and use the same value for the Experiment subject. If the identity or outcome source is unclear, ask the user instead of inventing one. Do not install RUM unless the selected outcome uses RUM. Do not request an exact prompt version. Preserve the existing fallback, prompt variables, provider, model, and application behavior. +{{< /agent-only >}} + +### Choose an Experiment metric + +Select a compatible **Datadog Experiment metric** available in the workflow's metric selector. These are not generic Datadog metrics or monitors. + +To create or edit a metric, navigate to [{{< ui >}}Product Analytics{{< /ui >}} > {{< ui >}}Experiments{{< /ui >}} > {{< ui >}}Metrics{{< /ui >}}][8]. See [Create Experiment Metrics][2] for instructions. + +When configuring a metric: + +- Select an event that represents the intended outcome and scope it to the relevant application or service. +- Select the subject type whose attribute matches the prompt `targeting_key`. +- Set whether an increase or decrease is the desired change. Guarded Rollouts use this direction to identify regression. + +If an existing Experiment metric measures the intended outcome, reuse it. Otherwise, follow [Create Experiment Metrics][2] to configure a supported data source. For browser or mobile outcomes collected with RUM, follow the [RUM setup][9] for your platform and [enable Product Analytics][10]. + +An A/B test cannot start without a primary metric. A Guarded Rollout uses one selected metric for each prompt environment. Changing or removing that selection affects future deployments; it does not alter an active rollout. + +{{< agent-only >}} +**Note for AI agents:** Do not create or edit Experiment metrics, start or end experiments, deploy prompt versions, or control Guarded Rollouts unless the user explicitly requests those external changes. When asked only to prepare application code, implement the local integration and report the remaining Datadog UI steps. +{{< /agent-only >}} + +## Run an A/B test + +1. Open a managed prompt and click {{< ui >}}Run A/B test{{< /ui >}}. +1. Select the environment in which to run the test. +1. Continue to the Product Analytics experiment setup. Datadog carries the prompt and environment into the setup, and its versions are available as experiment variants. +1. Select the experiment subject and primary metric. +1. Configure the audience, traffic split, exposure, and optional duration. +1. Start the experiment. +1. Confirm that assignments appear on the {{< ui >}}Flag & Exposures{{< /ui >}} page before analyzing the results. + +Use Product Analytics to start, monitor, conclude, or cancel the experiment. An experiment result does not change prompt serving on its own. To make a selected version the environment default, return to the prompt and deploy that version. + +For details about configuration and analysis, see [Plan and Launch Experiments][11] and [Read Experiment Results][12]. + + + +## Use a Guarded Rollout + +1. Open the candidate prompt version and click {{< ui >}}Set Environments{{< /ui >}}. +1. For each environment that should use progressive exposure, select {{< ui >}}Guarded rollout{{< /ui >}}. +1. Select the Experiment metric to monitor for regressions. +1. Click {{< ui >}}Set Environments{{< /ui >}} to deploy the candidate version and start the rollout. +1. Follow the serving split and rollout state from the prompt page. + +Exposure grows automatically while the rollout is running. You can pause, resume, or cancel the rollout from the prompt page: + +- **Pause** freezes exposure at its current share. +- **Resume** continues the exposure schedule. +- **Cancel** stops the rollout and returns eligible traffic to the version that served before the rollout began. + +If Datadog detects a regression in the selected metric, the rollout pauses. The current traffic split remains in place until you resume, cancel, or deploy another version. A Guarded Rollout does not automatically roll back the candidate. + +If the rollout reaches its final step without pausing, the candidate reaches full exposure for traffic eligible for the rollout. + +
Guarded Rollouts can continue when outcome data is sparse, missing, or cannot be matched to prompt assignments. Reaching full exposure means Datadog did not detect a regression in the available data. It does not prove that the candidate improved or is safe.
+ +## Verify and troubleshoot + +| Symptom | Check | +|---------|-------| +| Prompt Experimentation controls are unavailable | Confirm that the prompt has at least two versions and a deployed environment, and that your user role has the required permissions. If the controls are still unavailable, contact Datadog Support. | +| The prompt always returns its fallback, or the experiment receives no assignment data | Confirm the supported `ddtrace` version and `ddtrace[openfeature]` installation, `DD_ENV`, configuration source, Agent connectivity, and that the retrieval does not specify an exact `version`. | +| The A/B test cannot start | Select a primary metric, then review the Product Analytics and Feature Flags permissions and any required approvals. | +| The experiment has no results, or the selected Guarded metric has no usable data | Confirm that prompt retrieval and outcome events use the same subject identifier, outcomes occur after retrieval, the metric has the correct scope, and enough data has been collected. Review [Experiment Diagnostics][13]. | +| Pause or resume is denied | Confirm the user's Feature Flags permissions and any applicable approval requirements. | + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /llm_observability/monitoring/prompt_management/ +[2]: /experiments/defining_metrics/ +[3]: /account_management/rbac/permissions/#product-analytics +[4]: /feature_flags/concepts/configuration_sources/ +[5]: /llm_observability/monitoring/prompt_management/#prerequisites +[6]: /account_management/rbac/permissions/#feature-flags +[7]: /experiments/concepts/subject_types/ +[8]: https://app.datadoghq.com/product-analytics/experimentation-metrics +[9]: /real_user_monitoring/#get-started +[10]: /product_analytics/#enable-product-analytics +[11]: /experiments/plan_and_launch_experiments/ +[12]: /experiments/reading_results/ +[13]: /experiments/diagnostics/ +[14]: /getting_started/agent/ diff --git a/hugo/content/en/llm_observability/monitoring/prompt_management.md b/hugo/content/en/llm_observability/monitoring/prompt_management.md index a3546e15827..2438a8edd95 100644 --- a/hugo/content/en/llm_observability/monitoring/prompt_management.md +++ b/hugo/content/en/llm_observability/monitoring/prompt_management.md @@ -21,6 +21,8 @@ Prompt Management provides a centralized registry for the prompts used by your L Runtime retrieval is supported in Python through the `ddtrace` SDK. Prompt retrieval and Prompt Tracking are separate: `LLMObs.get_prompt()` can retrieve a managed prompt without enabling Agent Observability, but Agent Observability must be enabled to create LLM spans and associate prompt metadata with them. +After creating prompt versions, use [Prompt Experimentation][10] to compare them with an A/B test or deploy one progressively with a Guarded Rollout. + Prompt Management works alongside [Prompt Tracking][1]. When Agent Observability is enabled, managed prompts passed directly to supported, automatically instrumented LLM calls are associated with the resulting spans. ## Prerequisites @@ -332,3 +334,4 @@ To retrieve an exact version regardless of any targeting rule, pass `version` as [7]: /llm_observability/instrumentation/sdk/?tab=python#manual-instrumentation [8]: /api/latest/agent-observability/ [9]: /api/latest/feature-flags/list-environments/ +[10]: /llm_observability/monitoring/prompt_experimentation/