Add agentless Feature Flagging configuration source#11892
Add agentless Feature Flagging configuration source#11892leoromanovsky wants to merge 8 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
1858f32 to
82bb199
Compare
be3f8fc to
f9e70e6
Compare
| final StringBuilder endpoint = | ||
| new StringBuilder("https://api.") | ||
| .append(config.getSite().toLowerCase(Locale.ROOT)) | ||
| .append(DATADOG_API_SERVER_DISTRIBUTION_PATH); |
There was a problem hiding this comment.
🟡 Blocker: This is going to be a CDN-backed API. We are waiting on the DNS to be settled.
| // TODO before merge: confirm the final backend route with the server-distribution API owners. | ||
| private static final String DATADOG_API_SERVER_DISTRIBUTION_PATH = | ||
| "/api/v2/feature-flagging/config/server-distribution"; |
There was a problem hiding this comment.
🟡 Blocker: The path is not settled yet.
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
There was a problem hiding this comment.
All retry, ETag, shutdown, and idempotency paths look correct. One real gap: apply() silently returns false for 401/403 with no log above DEBUG — users with a wrong or missing DD_API_KEY have no way to diagnose why feature flags never load. Fixed by separating the auth-failure branch in apply() with a LOGGER.warn(...) call.
🤖 Datadog Autotest · Commit f9e70e6 · What is Autotest? · Any feedback? Reach out in #autotest
| static final boolean DEFAULT_INJECT_DATADOG_ATTRIBUTE = true; | ||
| static final String DEFAULT_SITE = "datadoghq.com"; | ||
|
|
||
| public static final String DEFAULT_FEATURE_FLAGGING_CONFIGURATION_SOURCE = "agentless"; |
There was a problem hiding this comment.
nit: move these up one block to preserve publics and package visibility groups
…ess-narrative # Conflicts: # dd-trace-api/src/main/java/datadog/trace/api/config/FeatureFlaggingConfig.java # products/feature-flagging/feature-flagging-agent/src/main/java/com/datadog/featureflag/FeatureFlaggingSystem.java
pavlokhrebto
left a comment
There was a problem hiding this comment.
Advisory review of the agentless configuration source, cross-checked against the Agentless-mode RFC. The core state machine (cold vs. warm handling, retry classification, readiness, atomic snapshot swap, shutdown / last-known-good) matches the RFC precisely — nice work. Findings below are precision-first and mostly non-blocking; inline comments are attached to the relevant lines.
Descoped config note (C8): The RFC lists DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_EXTRA_HEADERS (default empty object); this PR intentionally removes it. Fine if deliberate, but noting it since the mock-CDN test flow may want custom headers — worth tracking as a follow-up.
| @Override | ||
| public UfcHttpResponse fetch(final HttpUrl endpoint, final Config config, final String etag) | ||
| throws IOException { | ||
| final Map<String, String> headers = new HashMap<>(); |
There was a problem hiding this comment.
Reconcile source-mode metadata with system-tests. These requests send DD-API-KEY, If-None-Match, and the standard Datadog-Meta-* headers, but no source-mode metadata — matching this PR's decision that source selection is not sent to the backend. However, the RFC's System Tests Validation section lists sending auth/source-mode metadata as required proof, and the mock-CDN flow expects it. This passed the current draft scenario, but if the shared system-tests later assert a source-mode header, this SDK would fail. Worth reconciling with the RFC author / system-tests owner before the manifest (system-tests#7300) leaves draft.
| return false; | ||
| } | ||
| if (attempt == MAX_ATTEMPTS) { | ||
| LOGGER.debug("Feature Flagging HTTP configuration source request failed", e); |
There was a problem hiding this comment.
Sustained non-auth failures are invisible at the default log level. Auth failures (401/403) emit a rate-limited WARN (good), but timeout/5xx exhaustion and IOException only log at DEBUG. A persistently unreachable or 5xx-ing endpoint produces zero default-level signal; the only symptom is PROVIDER_NOT_READY returning defaults forever. Since a failed config source silently means all flags fall back to code defaults, consider a rate-limited WARN on sustained fetch failure too.
There was a problem hiding this comment.
Good idea; addressing.
| return parsed; | ||
| } | ||
|
|
||
| private static String endpointFromConfiguredBaseUrl(final String configuredBaseUrl) { |
There was a problem hiding this comment.
dd_env is dropped for custom base URLs. endpointFromConfiguredBaseUrl never appends ?dd_env=<env>, whereas the Datadog-managed default (datadogApiServerDistributionEndpoint) always does. So pointing ..._BASE_URL at a server-distribution-style backend silently loses env scoping. Likely intentional for opaque custom endpoints, but it is undocumented and an easy footgun — could you confirm the intent and document it (and ideally add a test pinning the behavior)?
There was a problem hiding this comment.
Good idea to add dd_env automatically; seems like a handy bit of information to have.
| executor.shutdownNow(); | ||
| } | ||
|
|
||
| private void pollOnceSafely() { |
There was a problem hiding this comment.
Test gap + error handling: scheduled-path crash safety. pollOnceSafely exists to stop a throwing poll from permanently cancelling the scheduleWithFixedDelay task, but no test exercises that on the scheduled path — failedGatewayDispatchDoesNotAdvanceEtag only calls pollOnce() directly. A test that a throwing listener on the scheduled path does not kill subsequent polls would lock this in. Also note this catches only RuntimeException, not Error — a listener throwing an Error would still kill the poller; worth a comment or widening to Throwable.
There was a problem hiding this comment.
Interesting! Ok, will improve the testing around this.
| final class AgentlessConfigurationSource implements ConfigurationSourceService { | ||
| private static final Logger LOGGER = LoggerFactory.getLogger(AgentlessConfigurationSource.class); | ||
|
|
||
| // TODO before merge: confirm the final backend route with the server-distribution API owners. |
There was a problem hiding this comment.
Unresolved backend route TODO. Flagging so it is not missed — by its own text this blocks merge: confirm the final server-distribution route with the API owners.
There was a problem hiding this comment.
Known; this is a blocker to merging until we get the DNS.
There was a problem hiding this comment.
Known; waiting on CDN DNS to be known.
| config, | ||
| millis(config.getFeatureFlaggingConfigurationSourcePollIntervalSeconds()), | ||
| new OkHttpUfcHttpClient( | ||
| OkHttpUtils.buildHttpClient( |
There was a problem hiding this comment.
Request-timeout semantics (no change requested). The 2s default matches the RFC. Just noting the mapping: it is applied as connect + read + write each = 2s (via OkHttpUtils.buildHttpClient), with no overall callTimeout — so large payloads can still stream (good), but a 2s connect could be tight on cold edges. Might be worth confirming with FFE against a real CDN edge.
There was a problem hiding this comment.
Thanks for mentioning; I'll think about how to improve this, and yes, we will validate on the real CDN edge. I believe even without that information, that 2s is sufficient.
| return parsed.toString(); | ||
| } | ||
|
|
||
| private static String datadogApiServerDistributionEndpoint(final Config config) { |
There was a problem hiding this comment.
GovCloud not guarded. The RFC scopes direct CDN mode to commercial sites (GovCloud not included), but https://api.<site> is built for any site with no guard. A defensive check or doc note would match the stated scope.
There was a problem hiding this comment.
Thanks, yes, GovCloud is not supported; I'll make it explicit.
…figuration-source
NOTE TO REVIEWERS
This PR intentionally keeps the complete feature in one PR so the final architecture remains visible. Its four commits are deliberately layered like a stacked PR, and each commit contains the final production behavior and tests for that layer.
How to Review
Commit Guide
LOC is rename-aware per commit against its parent.
b8bbd269e04baefdbf6e304, last-known-good, bounded jittered retry, no overlap, in-flight cancellation, shutdown safety, and system-test-parity tests.d378350cf1f9e70e67c4Per-Commit Validation
Each commit was checked out and validated independently before publication:
ConfigTest, feature-flagging lib/agent tests, and relevant Spotless checks.All four gates completed with
BUILD SUCCESSFUL.Stack Position
You are here: the Java source-mode foundation. It makes Datadog-managed agentless delivery the default, keeps custom HTTP under agentless, and preserves Agent Remote Configuration as explicit opt-in.
flowchart LR subgraph JAVA["dd-trace-java"] J1["JAVA-01 · #11892<br/>Agentless + RC sources"] --> J2["JAVA-02 · #11639<br/>Aggregate evaluation EVP"] end subgraph SYSTEM["system-tests"] ST1["ST-01 · #7298<br/>Mock agentless backend"] --> ST2["ST-02 · #7299<br/>Side-effect contracts"] ST2 --> STM1["ST-M01 · #7300<br/>Enable Java configuration"] ST2 --> NEXT["Next drafts<br/>Enable Java side effects"] end subgraph DOGFOOD["ffe-dogfooding"] DOG0["DOG-00 · #92<br/>Agentless evaluation baseline"] --> DOG1["DOG-01 · #93<br/>Side-effect conduit"] end J1 --> STM1 J1 --> DOG0 J2 --> NEXT J2 --> DOG1 STM1 --> GREEN["Java proof<br/>both sources × side effects"] NEXT --> GREEN DOG1 --> GREEN classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class J1 current;Motivation
Java Feature Flagging needs a tracer-side configuration-source split so SDKs can fetch UFC from an agentless HTTP backend while preserving the existing Agent Remote Configuration path.
Changes
DD_FEATURE_FLAGS_CONFIGURATION_SOURCEwithagentless,remote_config, and reservedoffline; default isagentless.DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL.DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDSandDD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS.AgentlessConfigurationSource, an HTTP UFC poller used for both the default Datadog-managed endpoint and a configured custom HTTP endpoint.remote_configon the existingRemoteConfigServiceImpl/ Agent RC path.DD-API-KEY, accepted-200-only ETags/304, malformed UFC rejection, last-known-good preservation, and no overlapping polls.clamp(P/6, 2s, 10s)thenclamp(P/3, 5s, 30s).DD_FEATURE_FLAGS_ENABLEDand extra-header env from this PR.Decisions
No new provider kill switch. This keeps the existing provider bootstrap behavior through
DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED; changing enablement is out of scope for this PR.Custom HTTP delivery remains under
agentless. The source mode describes direct HTTP UFC delivery without the Datadog Agent. With no base URL, the SDK derives the first-party Datadog endpoint ashttps://api.<site>/api/v2/feature-flagging/config/server-distribution?dd_env=<env>. SettingDD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URLchooses a system-test, dogfood, or operator-managed HTTP endpoint without introducing a fourth source mode. A bare host uses the standard server-distribution path; a URL with a path is used as the exact UFC endpoint.Remote Configuration remains a distinct mode. It is Agent-mediated and uses the RC protocol's security, signing, targeting, capability, and subscription lifecycle. Those semantics are materially different from direct HTTP polling, so
remote_configremains a peer source mode rather than another agentless endpoint choice.No offline factory yet.
offlineremains a reserved configuration-source value. This PR does not expose a startup-bytes API or offline factory because that API still needs design work.Source selection stays SDK-local.
DD_FEATURE_FLAGS_CONFIGURATION_SOURCEdecides which local source implementation starts. It is not sent to the backend.Initialization Modes
Solid connectors are implemented in this PR. The dashed offline connector previews startup-provided UFC bytes.
flowchart TD Gate{DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED} Disabled[Feature Flagging subsystem not started] System[FeatureFlaggingSystem.start] Select{DD_FEATURE_FLAGS_CONFIGURATION_SOURCE} Exposure[ExposureWriter starts for every enabled mode] subgraph AgentlessMode[agentless mode - current default] AgentlessSource[AgentlessConfigurationSource] OperationalDefaults[SDK-owned operation<br/>30s poll default; 2s request timeout default<br/>ETag; retry; last-known-good; no overlap] EndpointChoice{agentless base URL set?} DatadogManaged[Datadog-managed agentless<br/>first-party; CDN-backed] CustomHttp[Custom HTTP endpoint<br/>system tests; dogfood; operator backend] AgentlessSource --- OperationalDefaults AgentlessSource -- polls --> EndpointChoice EndpointChoice -- no --> DatadogManaged EndpointChoice -- yes --> CustomHttp end RemoteConfig[remote_config<br/>explicit opt-in; Agent-mediated] RemoteConfigSource[RemoteConfigServiceImpl] AgentRc[Datadog Agent Remote Configuration<br/>RC security; signing; targeting; subscriptions] OfflineReserved[offline today<br/>reserved; no configuration service] OfflineSource[Later: OfflineConfigurationSource<br/>customer UFC JSON bytes at startup; no network] Ufc[Shared UFC deserialize and evaluate pipeline] Gateway[FeatureFlaggingGateway] Provider[OpenFeature provider] Gate -- false --> Disabled Gate -- true --> System System --> Select System --> Exposure Select -- unset or agentless --> AgentlessSource DatadogManaged --> Ufc CustomHttp --> Ufc Select -- remote_config --> RemoteConfig RemoteConfig --> RemoteConfigSource RemoteConfigSource -- subscribes --> AgentRc RemoteConfigSource --> Ufc Select -- offline today --> OfflineReserved OfflineReserved -. later .-> OfflineSource OfflineSource -. startup UFC bytes .-> Ufc Ufc --> Gateway Gateway --> ProviderCustomer Usage Example
Default Datadog-managed agentless delivery:
Custom HTTP delivery keeps agentless mode and changes only the endpoint:
Explicit Agent Remote Configuration delivery still uses the existing Agent path:
Verification
System Test Evidence
I built
dd-java-agent,dd-trace-api, anddd-openfeaturefrom the rewrittenhead
f9e70e67c4, rebuilt the Java Spring Boot weblog image with those localartifacts, and verified the JAR inside the image matched the local agent JAR.
I then ran the Java manifest-enabled agentless scenario from the companion
system-tests draft:
This exercises the Java manifest row proposed by the companion draft against
the exact code currently published in this PR.
Next Steps
DataDog/system-tests#7300.
merge the activation only when that gate is green.