test(ffe): add agentless end-to-end harness#7298
Conversation
|
|
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9af665a8a9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eb57dc60db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| default_libs_with_dev, | ||
| 3600, | ||
| "false", | ||
| "DEFAULT,FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS", |
There was a problem hiding this comment.
When this test file changes, this new expectation makes the CI selector emit FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS, and compute-workflow-parameters schedules it as end-to-end work; however I checked .github/workflows/run-end-to-end.yml:320-322 and the workflow only has the existing FEATURE_FLAGGING_AND_EXPERIMENTATION run step, with no AGENTLESS step found. Those jobs can build weblogs and then skip every scenario run step, so PRs changing or enabling this test get green end-to-end jobs without running the selected scenario; either keep the dormant scenario out of the selector or add the matching run step.
Useful? React with 👍 / 👎.
cbeauchesne
left a comment
cbeauchesne
left a comment
There was a problem hiding this comment.
Great job, congrats. I've few request changes and question. There are also two missing bits :
- add rules in
utils/scripts/libraries_and_scenarios_rules.ymlfor this new scenario - add the scenario in the CI
|
|
||
| def configure(self, config: pytest.Config) -> None: | ||
| try: | ||
| if not self.replay: |
There was a problem hiding this comment.
I'm not sure it's usefull, we never use xdist with end-to-end scenario.
cbeauchesne
left a comment
cbeauchesne
left a comment
There was a problem hiding this comment.
A small non blocking comment.
Great piece of work, congrat !
| class FeatureFlaggingAgentlessEndToEndScenario(DdTraceEndToEndScenario): | ||
| """FFE end-to-end scenario with UFC available before the weblog starts.""" | ||
|
|
||
| _default_scenario_groups: tuple[ScenarioGroup, ...] = () |
There was a problem hiding this comment.
So, with that, the scenario is not activated in the official system-test workflow, unless you explicitly require it.
In system-tests CI if you modify a test file that runs on this scenario, or this present file, or something in utils/mocked_backend/ffe.py, it will be executed. For any other modification (many system-tests internals), it won't be executed.
Is it what you want ?
2 participants
Motivation
Server SDK Feature Flags are moving to authenticated CDN delivery as the default configuration path, with Agent Remote Configuration remaining an explicit alternative. The existing FFE end-to-end scenario can exercise only the Agent/RC lifecycle: it starts an Agent and weblog, then supplies UFC after the application is running.
That cannot prove the agentless startup contract. An agentless SDK may request UFC while the application is starting, so the test backend must already exist before the weblog starts. The scenario must also run without an Agent; otherwise a passing test cannot rule out an accidental Agent dependency.
This PR adds that missing, dormant test foundation. It starts a deterministic CDN-style UFC backend before a no-Agent weblog, injects the endpoint and credentials, and verifies authenticated configuration fetch plus evaluation. No SDK manifest is enabled here, so ordinary SDK CI behavior does not change until a later activation PR.
flowchart LR subgraph BEFORE["Existing end-to-end coverage"] B1["Start Agent and weblog"] --> B2["SDK starts"] B3["RC fixture"] --> B4["Agent Remote Configuration"] B4 --> B2 B2 --> B5["Evaluate flag"] end subgraph THIS["Agentless harness added here"] A1["1. Start mock CDN backend"] --> A2["2. Start weblog and SDK<br/>without an Agent"] A2 --> A3["3. Fetch UFC with API key"] A3 --> A4["4. Evaluate flag"] end GAP["Missing on main:<br/>pre-start UFC service<br/>and no-Agent lifecycle"] -.-> A1 classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class A1,A2,A3,A4 current;Stack Position
This is the inactive foundation of the stack. Later PRs add the telemetry topology and then activate individual Java capabilities only when their implementation is green.
flowchart TD ST1["#7298 · this PR<br/>no-Agent CDN configuration harness<br/>all manifests disabled"] ST2["#7299 · next stacked PR<br/>side-effect transport contracts<br/>sidecar preferred, direct fallback"] STM["#7300 and small follow-ups<br/>enable Java capabilities one at a time"] J1["dd-trace-java #11892<br/>agentless configuration source"] J2["dd-trace-java side-effects work<br/>exposures · evaluation EVP · OTLP"] ST1 --> ST2 --> STM J1 --> STM J2 --> STM classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class ST1 current;Architecture Boundary
Configuration source and telemetry transport are independent. The Agent belongs only to the explicit Remote Configuration branch. Default agentless configuration has no Agent dependency. Post-evaluation side effects are a stacked follow-up: prefer the Datadog serverless sidecar when available and fall back to direct authenticated intake otherwise.
flowchart LR subgraph CONFIG["Configuration delivery"] RC["RC fixture"] --> RCA["Agent<br/>RC only"] RCA --> RCS["SDK remote_config source"] CDN["Datadog CDN<br/>or controlled mock"] -->|"authenticated HTTP"| AGS["SDK agentless source<br/>default"] RCS --> EVAL["Same UFC / OpenFeature evaluator"] AGS --> EVAL end subgraph FOLLOWUP["Telemetry transport · stacked follow-up"] EVAL --> FX["Exposure EVP · evaluation EVP · OTLP metrics"] FX --> PICK{"Serverless sidecar available?"} PICK -->|"yes"| SIDECAR["Datadog serverless sidecar"] PICK -->|"no"| DIRECT["Direct authenticated intake"] SIDECAR --> INTAKE["Datadog backend"] DIRECT --> INTAKE end classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class CDN,AGS,EVAL current;Changes
ETag/If-None-Match, and metadata-only status inspection.FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESSscenario that starts the backend before the weblog.all,end_to_end, andtracer_releaseselections.Decisions
Validation
./format.sh --check— passes.java@1.65.0-SNAPSHOT+f9e70e67c4— 1 passed in 2.11s.Next Steps
interfaces.agent.