fix(feature-flags): add safe agentless EVP fallback by leoromanovsky · Pull Request #12299 · DataDog/dd-trace-java

leoromanovsky · 2026-08-26T08:46:57Z

Motivation

Java already selects a local EVP route from Agent /info and, for agentless Feature Flags, falls back to direct intake for both exposures and flag evaluations. The remaining direct-intake transport has four gaps:

Direct requests do not honor DD_PROXY_HTTPS and DD_PROXY_NO_PROXY.
An ambiguous local transport failure leaves subsequent batches on the failed local route instead of switching them to direct intake.
Direct requests follow redirects while carrying DD-API-KEY.
The endpoint derived from DD_SITE is not restricted to the expected HTTPS Event Platform intake origin.

Changes

flowchart LR
  Signals["Exposures + flag evaluations"] --> Discover["Startup /info discovery"]
  Discover -->|"v4 advertised"| V4["/evp_proxy/v4"]
  Discover -->|"v2 only"| V2["/evp_proxy/v2"]
  Discover -->|"No compatible route"| Direct["Direct intake + DD-API-KEY"]
  V4 --> Intake["Event Platform intake"]
  V2 --> Intake
  V4 -->|"Definitive failure"| Direct
  V2 -->|"Definitive failure"| Direct
  Direct --> Intake

Select the local EVP v4 or v2 route from Agent /info during startup.
Use direct intake when no supported local route is available.
Replay definitive local-route failures through direct intake and keep the direct selection sticky.
Switch future batches after ambiguous local transport failures without replaying the current batch.
Route direct intake through configured HTTPS proxy settings with NO_PROXY support.
Reject HTTP and HTTPS redirects from direct Feature Flags intake.
Add focused route, failure-classification, proxy, exposure, flag-evaluation, and redirect tests.

Decisions

Replay the current batch only after a definitive refusal or HTTP 403, 404, or 405.
Keep reset, broken pipe, and timeout failures future-only to avoid duplicate delivery.
Keep HTTP 429 and 5xx responses on the selected local route.
Keep TLS verification enabled and use raw DD-API-KEY authentication only for direct intake.
Do not follow direct Feature Flags intake redirects.

Validation

The Java system tests are defined and enabled in DataDog/system-tests#7601.

Ran the Java spring-boot system-test stack against the branch artifact in these scenarios:

FEATURE_FLAGGING_AND_EXPERIMENTATION
FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_DIRECT
FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_SERVERLESS

In each topology, the tests exercised both /api/v2/exposures and /api/v2/flagevaluation, route-specific headers and authentication, aggregate evaluation counts, and zero events on the unused route.

Local validation:

./gradlew :communication:test --tests datadog.communication.BackendApiFactoryTest
./gradlew :products:feature-flagging:feature-flagging-lib:test --tests com.datadog.featureflag.FeatureFlagBackendApiFactoryTest --tests com.datadog.featureflag.ExposureWriterTests --tests com.datadog.featureflag.FlagEvaluationWriterImplTest
./gradlew :communication:spotlessCheck :communication:forbiddenApisMain :products:feature-flagging:feature-flagging-lib:spotlessCheck :products:feature-flagging:feature-flagging-lib:forbiddenApisMain

Route direct feature flag intake through standard HTTPS proxy settings and attach the canonical fixed-width API key fingerprint. Environment: Datadog workspace

dd-octo-sts · 2026-08-26T09:08:44Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.00 s	13.95 s	[-0.3%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent	12.91 s	13.06 s	[-1.9%; -0.4%] (maybe better)
startup:petclinic:appsec:Agent	17.55 s	17.22 s	[+1.1%; +2.7%] (significantly worse)
startup:petclinic:iast:Agent	17.46 s	17.61 s	[-1.5%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent	17.48 s	17.51 s	[-1.3%; +0.9%] (no difference)
startup:petclinic:sca:Agent	17.44 s	17.38 s	[-0.7%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.73 s	16.69 s	[-0.7%; +1.2%] (no difference)

Commit: 489706ce · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Keep this PR focused on proxy-aware direct Event Platform intake and leave fingerprinting to an independent change. Environment: Datadog workspace

Keep the direct-intake diff free of fingerprint-only formatting artifacts. Environment: Datadog workspace

Use the original two-argument direct intake factory now that the unrelated fingerprint header plumbing has moved out of this PR. Environment: Datadog workspace

linear-code · 2026-08-26T21:30:47Z

FFL-1482

FFL-1484

Reject URL authority confusion before adding DD-API-KEY. Environment: Datadog workspace

Read standard proxy environment variables without registering them as Datadog configuration aliases, and use the existing non-regex parser for no-proxy hosts. Environment: Datadog workspace

Disable HTTP and HTTPS redirects for Feature Flags direct EVP intake so DD-API-KEY remains bound to the configured origin. Environment: Datadog workspace

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 489706ce96

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-27T16:23:19Z

+    noProxyHosts =
+        configuredNoProxyHosts == null
+            ? Collections.emptySet()
+            : parseStringIntoSetOfNonEmptyStrings(configuredNoProxyHosts);


Preserve single-character no-proxy entries

When DD_PROXY_NO_PROXY, NO_PROXY, or no_proxy contains a one-character entry, this parser silently omits it because parseStringIntoSetOfNonEmptyStrings only adds tokens whose length is greater than one. In particular, NO_PROXY=* produces an empty set, so the explicit wildcard handling in IntakeProxySelector.shouldBypassProxy is unreachable and direct feature-flag traffic still uses HTTPS_PROXY; single-character internal hostnames also regress from the previous getSpacedList behavior. Parse these values with a helper that retains all nonempty tokens.

Useful? React with 👍 / 👎.

Yeah, this is a simple fix that is worth it.

chatgpt-codex-connector · 2026-08-27T16:23:19Z

+            || normalizedHost.equals(normalized)
+            || (normalized.startsWith(".")
+                && (normalizedHost.equals(normalized.substring(1))
+                    || normalizedHost.endsWith(normalized)))) {


Match bare NO_PROXY domains against subdomains

When a standard domain entry such as NO_PROXY=datadoghq.com is used with HTTPS_PROXY, this matcher bypasses only the exact host because suffix matching is restricted to entries starting with a dot. Consequently, event-platform-intake.datadoghq.com is still sent through the proxy, even though bare domain entries in no-proxy lists are expected to cover that domain and its subdomains. Apply boundary-aware suffix matching to bare domain entries as well.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-27T16:23:19Z


-    // DD_PROXY_NO_PROXY is specified as a space-separated list of hosts
-    noProxyHosts = tryMakeImmutableSet(configProvider.getSpacedList(PROXY_NO_PROXY));
+    String configuredHttpsProxy = configProvider.getString(PROXY_HTTPS);


Redact credentials from proxy configuration telemetry

When DD_PROXY_HTTPS contains supported userinfo such as http://user:password@proxy:8080, reading it through ConfigProvider.getString records the complete URL in ConfigCollector; ConfigSetting does not classify proxy.https as sensitive, so the telemetry configuration payload serializes the proxy username and password verbatim. Read this setting without collecting the raw value or add explicit redaction for the proxy URL before telemetry emission.

Useful? React with 👍 / 👎.

This might warrant a second look unless the bot is hallucinating.

datadog-datadog-us1-prod

NO_PROXY=* does not bypass the new intake proxy. The parser removes the wildcard before the proxy selector reads it.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 489706c · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-us1-prod · 2026-08-27T16:26:33Z

+    noProxyHosts =
+        configuredNoProxyHosts == null
+            ? Collections.emptySet()
+            : parseStringIntoSetOfNonEmptyStrings(configuredNoProxyHosts);


Keep the NO_PROXY wildcard

Feature Flag requests with an API key can go through a proxy that the user explicitly disables.

Assertion details

Input: Set an HTTPS proxy and set DD_PROXY_NO_PROXY=*, NO_PROXY=*, or no_proxy=*.

Expected: The wildcard must bypass the proxy for all direct intake hosts.

Actual: The parser removes the one-character * value. The proxy selector then uses the configured HTTPS proxy.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

dougqh

Looks good to me, and I very much appreciate the thoroughness of the testing.

danyal002

LGTM. Some of the bot comments might be worth looking further into.

danyal002 · 2026-08-31T16:28:15Z


-    // DD_PROXY_NO_PROXY is specified as a space-separated list of hosts
-    noProxyHosts = tryMakeImmutableSet(configProvider.getSpacedList(PROXY_NO_PROXY));
+    String configuredHttpsProxy = configProvider.getString(PROXY_HTTPS);


This might warrant a second look unless the bot is hallucinating.

danyal002 · 2026-08-31T16:29:15Z

+    noProxyHosts =
+        configuredNoProxyHosts == null
+            ? Collections.emptySet()
+            : parseStringIntoSetOfNonEmptyStrings(configuredNoProxyHosts);


Yeah, this is a simple fix that is worth it.

Preserve one-character NO_PROXY entries and prevent HTTPS proxy URLs from exposing credentials through configuration telemetry. Environment: Datadog workspace

…agentless-evp-java-hardening # Conflicts: # utils/config-utils/src/main/java/datadog/trace/api/ConfigSetting.java # utils/config-utils/src/test/java/datadog/trace/api/ConfigSettingTest.java

datadog-datadog-us1-prod

A plain NO_PROXY domain does not cover its subdomains. This can send direct intake traffic through a proxy that blocks Datadog.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 75471e9 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-us1-prod · 2026-08-31T17:09:23Z

+            || normalizedHost.equals(normalized)
+            || (normalized.startsWith(".")
+                && (normalizedHost.equals(normalized.substring(1))
+                    || normalizedHost.endsWith(normalized)))) {


Plain NO_PROXY domains do not match subdomains

Direct Feature Flags intake can fail when the proxy blocks Datadog traffic.

Assertion details

Input: Set DD_PROXY_HTTPS and set NO_PROXY=datadoghq.com. Then send direct intake traffic to event-platform-intake.datadoghq.com.

Expected: A plain NO_PROXY domain must bypass the proxy for that domain and its subdomains.

Actual: The selector uses the configured HTTPS proxy because suffix matching only applies to entries that start with a dot.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

Add proxy-aware feature flag intake authentication

8d99fa5

Route direct feature flag intake through standard HTTPS proxy settings and attach the canonical fixed-width API key fingerprint. Environment: Datadog workspace

This comment has been minimized.

Sign in to view

leoromanovsky added 3 commits August 26, 2026 15:50

Separate direct intake proxying from API key fingerprinting

bae1e35

Keep this PR focused on proxy-aware direct Event Platform intake and leave fingerprinting to an independent change. Environment: Datadog workspace

Merge live Java master after EVP split

1234612

Restore Java formatting after the fingerprint split

3f82e4a

Keep the direct-intake diff free of fingerprint-only formatting artifacts. Environment: Datadog workspace

leoromanovsky changed the title ~~Add proxy-aware feature flag intake authentication~~ Add proxy-aware Feature Flags direct intake Aug 26, 2026

leoromanovsky added 3 commits August 26, 2026 16:00

Update direct-intake tests after fingerprint removal

f9bcf76

Use the original two-argument direct intake factory now that the unrelated fingerprint header plumbing has moved out of this PR. Environment: Datadog workspace

Remove unnecessary Groovy imports

96278a2

fix(feature-flags): handle ambiguous EVP failures safely

19572f6

leoromanovsky changed the title ~~Add proxy-aware Feature Flags direct intake~~ fix(feature-flags): add safe agentless EVP fallback Aug 26, 2026

leoromanovsky mentioned this pull request Aug 26, 2026

test(ffe): validate Node and Java EVP egress DataDog/system-tests#7588

Closed

fix(feature-flags): validate direct EVP intake site

ce39375

Reject URL authority confusion before adding DD-API-KEY. Environment: Datadog workspace

leoromanovsky mentioned this pull request Aug 27, 2026

test(ffe): validate Node and Java EVP egress DataDog/system-tests#7601

Draft

fix(config): align proxy configuration validation

147145e

Read standard proxy environment variables without registering them as Datadog configuration aliases, and use the existing non-regex parser for no-proxy hosts. Environment: Datadog workspace

leoromanovsky added tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature type: feature Enhancements and improvements labels Aug 27, 2026

fix(feature-flags): reject direct intake redirects

489706c

Disable HTTP and HTTPS redirects for Feature Flags direct EVP intake so DD-API-KEY remains bound to the configured origin. Environment: Datadog workspace

leoromanovsky marked this pull request as ready for review August 27, 2026 16:14

leoromanovsky requested review from a team as code owners August 27, 2026 16:14

leoromanovsky requested review from amarziali, danyal002 and pavlokhrebto and removed request for a team August 27, 2026 16:14

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

datadog-datadog-us1-prod Bot reviewed Aug 27, 2026

View reviewed changes

dougqh approved these changes Aug 28, 2026

View reviewed changes

leoromanovsky removed the request for review from pavlokhrebto August 31, 2026 16:21

danyal002 approved these changes Aug 31, 2026

View reviewed changes

fix(feature-flags): harden proxy configuration

75471e9

Preserve one-character NO_PROXY entries and prevent HTTPS proxy URLs from exposing credentials through configuration telemetry. Environment: Datadog workspace

leoromanovsky requested a review from a team as a code owner August 31, 2026 16:53

leoromanovsky requested review from mcculls and removed request for a team August 31, 2026 16:53

Merge remote-tracking branch 'origin/master' into leo.romanovsky/ffe-…

6c0444c

…agentless-evp-java-hardening # Conflicts: # utils/config-utils/src/main/java/datadog/trace/api/ConfigSetting.java # utils/config-utils/src/test/java/datadog/trace/api/ConfigSettingTest.java

datadog-datadog-us1-prod Bot reviewed Aug 31, 2026

View reviewed changes

Uh oh!

Conversation

leoromanovsky commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Validation

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

linear-code Bot commented Aug 26, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

danyal002 Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

danyal002 Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-us1-prod Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

dougqh left a comment

Choose a reason for hiding this comment

Uh oh!

danyal002 left a comment

Choose a reason for hiding this comment

Uh oh!

danyal002 Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

danyal002 Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-us1-prod Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leoromanovsky commented Aug 26, 2026 •

edited

Loading

dd-octo-sts Bot commented Aug 26, 2026 •

edited

Loading