Add system-tests for agentless functionality - #7607
Draft
bwoebi wants to merge 17 commits into
Draft
Conversation
…urations Agentless remote config has no agent to relay client state via /v0.7/config: the native RC client polls the backend endpoint directly and reports its per-config apply state inline on that same request. Add the protobuf message definitions and wire them into the proxy's request deserializer so that state is visible to the test harness.
TUF derives a key's keyid from the canonical JSON of the key object, not from the raw public key bytes. libdatadog's rust-tuf recomputes the keyid this way when verifying the root, and previously failed to match our test signature (metadata root signature threshold not met). Fix the derivation and regenerate the two pinned fixtures that embed the old (wrong) keyid. Also: TUF metadata version 0 is spec-invalid, but was used as an empty-state sentinel for the mocked backend; real clients (correctly) reject it. Move the sentinel to version 1, and add an opt-in agent_refresh_interval field to targets metadata (default None/omitted, a no-op for all existing callers) so agentless scenarios can override the native RC client's 60s default poll interval down to something tests can wait on.
When context.scenario.include_agent is False, there is no agent to relay client state via /v0.7/config: watch /api/v0.1/configurations directly (via interfaces.datadog_direct) and read the client's reported state out of the request instead of a follow-up response, normalizing protobuf's stringified uint64 fields back to int along the way. Also re-sign real backend config pushes at version + 1 and set agent_refresh_interval=1, to stay strictly greater than the version-1 empty-state sentinel and keep the agentless client's poll interval fast enough for tests to wait on (see previous commit).
Replace hardcoded interfaces.agent/_LOGS_PATH/_DEBUGGER_PATH/etc. references throughout BaseDebuggerTest with overridable class attributes (_backend_interface, _snapshot_paths, _traces_path, _telemetry_path, _symbols_interface, _symbols_path), defaulting to today's agent-mode values so no existing test changes behavior. Add AgentlessBaseDebuggerTest, which points these at interfaces.datadog_direct and the single unified agentless debugger-intake path (libdatadog's agentless debugger sender collapses logs/snapshots/diagnostics/symdb onto one path, unlike the agent-relayed protocol). Span-decoration collection (_collect_span_decoration) calls get_spans_list(), which only interfaces.agent implements (it parses /v0.4/traces msgpack; agentless traces use a different wire format). Gate that call behind hasattr() so it's skipped for agentless backends rather than raising, per decision: span-decoration assertions stay agent-only for now.
AgentlessEndToEndScenario hardcoded FFE's own MockFFEAgentlessBackendServer into its base __init__/configure(). Replace that with a _create_mock_backend() hook (default None, meaning no mock backend), structurally typed via a small AgentlessBackendServer protocol, so other agentless scenarios can reuse this base without carrying FFE-specific setup. FeatureFlaggingAgentlessEndToEndScenario now overrides the hook and keeps its own env/extra_hosts wiring. Move the generic 'capture direct egress' wiring (DD_SITE=mock-intake.invalid, HTTPS_PROXY through the proxy container, mounting the mitmproxy CA bundle so the weblog trusts the intercepted TLS, watchdog/teardown registration for interfaces.datadog_direct) up from FeatureFlaggingAgentlessEndToEndScenario into the shared base, gated on a new capture_direct_egress constructor param (default True). Add pass-through rc_api_enabled/rc_backend_enabled params: when rc_backend_enabled is set, wire the same TUF test root env vars already used to make the real Datadog Agent trust the mocked RC backend (utils/_context/containers.py::AgentContainer), so the native agentless RC client trusts it too. Override _wait_for_app_readiness() to skip the interfaces.library.ready wait when capturing direct egress: that readiness signal only ever fires on agent-facing traffic, which agentless mode never sends, so it would time out on every run otherwise. Also fix a resource leak in _start_mock_backend(): the mock backend was only assigned to self._mock_backend after backend.reset() succeeded, so a failing reset() skipped cleanup in configure()'s except handler and leaked the backend.
APM_TRACING_AGENTLESS (AgentlessEndToEndScenario) covers direct-to-intake trace submission, client-side stats, and Remote Configuration under DD_AGENTLESS_ENABLED, with no Datadog Agent. DEBUGGER_AGENTLESS (new DebuggerAgentlessScenario, in its own module) covers agentless Dynamic Instrumentation (probe upload/logs/snapshots) and Symbol DB the same way, reusing the TUF-trust and mocked RC backend wiring generalized in the previous commit. Both are excluded from the tracer_release scenario group, same as the other agentless scenarios; add them to test_group_rules.py's exclusion list.
…er, SymDB) tests/test_agentless.py (APM_TRACING_AGENTLESS): trace submission, client-side stats, and Remote Configuration all land directly on the intake with no Datadog Agent, each asserted on host/path/headers, and RC additionally drives a real config push through to ACKNOWLEDGED via send_apm_tracing_command(). tests/debugger/test_debugger_agentless.py (DEBUGGER_AGENTLESS): a log probe installed via the agentless native RC client emits a snapshot to the agentless debugger intake; Symbol DB (forced via _DD_SYMBOL_DATABASE_FORCE_UPLOAD) uploads to the same unified path. Both verified end-to-end against a live weblog build of the (unmerged) dd-trace-py bob/agentless-setting branch. Endpoints/shapes are derived from reading tracer/libdatadog source, since that branch isn't released; re-confirm against real captures once it merges.
Scenario and test code is generic (no python-only guards), scoped to python for now via manifest entries only, so dropping these entries is enough once another tracer implements DD_AGENTLESS_ENABLED.
Test_Agentless_Trace_Submission and Test_Agentless_Stats only checked headers and non-empty content, not the actual shape of the trace/stats payloads. Add real assertions on captured payloads (root span for GET /: service, type, error, http.method, http.status_code; matching stats bucket entry: AgentHostname, ClientComputed, Service, Type, Hits, TopLevelHits, Errors), derived from and verified against live-captured traffic against the branch. Verified live: APM_TRACING_AGENTLESS still 3/3 passing with these assertions.
Trace submission, stats, RC, debugger snapshot, and SymDB requests each now assert their full required header set (dd-api-key, content-type, user-agent, datadog-meta-* tracer identity, dd-evp-origin, etc.), not just dd-api-key/content-type as before. Caught a real discrepancy along the way: SymDB uploads share dd-evp-origin: agent-debugger with logs/snapshots/diagnostics rather than a distinct agent-symdb value, and don't send x-datadog-additional-tags at all - fixed the assertion to match confirmed live behavior.
Trace submission now asserts content-encoding: zstd and that the wire size is genuinely smaller than the decoded body, following an audit of the agentless Rust encoder against the real Agent that found compression had been silently disabled (now fixed on the dd-trace-py branch). New Test_Agentless_Stats_Multi_Flush polls for two same-runtime stats flushes and asserts ClientStatsPayload.Sequence increments by exactly 1 per flush - a guarantee agent-mode never gives a real signal for, since the Agent's own re-aggregation always resets Sequence to 0 on relay.
Contributor
|
|
This comment has been minimized.
This comment has been minimized.
Telemetry (POST /api/v2/apmtelemetry) has no per-product transport setting of its own and simply follows the global DD_AGENTLESS_ENABLED switch - it was already flowing agentlessly and captured in every prior run, but had zero test coverage. Notably, this is the transport for generate-metrics events (internal tracer metrics like spans_created), which is distinct from DogStatsD-based runtime metrics (CPU/memory/GC) - those have no agentless transport at all on this branch and are silently dropped without an agent; out of scope for this commit. Caught and fixed a real setup-dispatch bug along the way: system-tests matches setup_<X> to test_<X> by exact name, so a single shared setup_telemetry method silently never ran for either test method - confirmed live via the pytest debug log that each test now independently triggers and waits for its own telemetry traffic.
Crashtracker submission was completely broken agentlessly: a real triggered crash (SIGSEGV) produced zero network activity toward either agentless mechanism for 90+ seconds. Root-caused on the dd-trace-py branch (ddtrace/internal/core/crashtracking.py, ddtrace/internal/settings/crashtracker.py, uncommitted there) to two compounding bugs - a too-short collection timeout that got the collector SIGKILLed mid-symbolication, and the crash receiver subprocess's environment never forwarding HTTP(S)_PROXY, so it silently couldn't reach the mock intake through the proxy this harness requires. Test_Agentless_Crashtracking covers both agentless delivery mechanisms confirmed by a live capture after the fix: the dedicated errors-intake endpoint (POST .../api/v2/errorsintake) and a telemetry 'logs' event - both share the telemetry-intake host, just on different paths. Distinguishes the full crash report from crashtracker's lightweight 'ping' log by tags, matching the convention tests/parametric/test_crashtracking.py already uses for the agent-mode equivalent. The crash-triggering request needs a much longer client timeout than this project's 5s default: DD_CRASHTRACKING_WAIT_FOR_RECEIVER defaults to true, so the crashing process blocks in its own signal handler until the report actually finishes uploading (up to the now-correct 30s collector timeout) before it can exit - the parent's response, and therefore this HTTP call, legitimately takes that long.
cbeauchesne
requested changes
Sep 2, 2026
Comment on lines
+95
to
+97
| use_proxy_for_weblog=capture_direct_egress, | ||
| rc_api_enabled=rc_api_enabled, | ||
| rc_backend_enabled=rc_backend_enabled, |
Collaborator
There was a problem hiding this comment.
I'm not really sure to understand the purpose here. The proxy is a way to observe data from libraries. But with agentless, we have a mocked backend that collect everything, so we have a very natural way to observe all of this. Do you have some input why both are needed ?
Contributor
Author
There was a problem hiding this comment.
This is only used for FFE (preserving what was there), on my agentless tests it's None. I.e. just the mocked backend.
| return | ||
| super()._wait_for_app_readiness() | ||
|
|
||
| def _create_mock_backend(self) -> AgentlessBackendServer | None: |
Collaborator
There was a problem hiding this comment.
IIUC, it's an abstractmethod ? if yes, can you mark it as so ?
Contributor
Author
There was a problem hiding this comment.
Nope, AgentlessEndToEndScenario is directly instantiated (e.g. by apm_tracing_agentless) with no backend.
| self._wait_successful = False | ||
| found_ids: set[str] = set() | ||
| interfaces.agent.wait_for( | ||
| self._backend_interface.wait_for( |
Collaborator
There was a problem hiding this comment.
Does it mean it's using the real backend? If yes, then we'll need to change that : every time we have used the real backend, it ended up by having unusable scenarios : backend does not guarantee anything about ingestion sucess and delays.
Contributor
Author
There was a problem hiding this comment.
Nope, _backend_interface is always a local ProxyBasedInterfaceValidator.
|
|
||
|
|
||
| @scenarios.apm_tracing_agentless | ||
| @features.not_reported |
Collaborator
There was a problem hiding this comment.
Can you add a feature ?
Contributor
Author
There was a problem hiding this comment.
done, dd_agentless_enabled feature now exists
8 real type errors: an int assigned into a str-typed dict literal (tuf.py), a missing dict annotation and an attribute access on the base Scenario class that only EndToEndScenario actually has (_remote_config.py, defaulted via getattr), two subclass interface-attribute overrides incompatible with the base class's narrower inferred type (debugger/utils.py, fixed by annotating both as their shared ProxyBasedInterfaceValidator base), and mock_backend_status()'s object|None return type being indexed directly in two FFE tests (narrowed with isinstance checks). Also guards a third, newly-surfaced get_spans_list() call the same way the existing hasattr-gated one already is, now that widening _backend_interface's annotation makes mypy correctly see it's not always present.
- Rename a locally-shadowing 'environment' variable to 'weblog_environment' in AgentlessEndToEndScenario.__init__, and merge two separate 'if capture_direct_egress:' blocks that were checking the exact same condition. - Default DD_API_KEY once in AgentlessEndToEndScenario itself instead of repeating the same value in every scenario declaration - safe no-op for FeatureFlaggingAgentlessEndToEndScenario, which already sets (and later re-sets, in _start_mock_backend) the identical value explicitly. Left DD_AGENTLESS_ENABLED un-defaulted: unlike DD_API_KEY, unconditionally defaulting it in the shared base would silently turn on dd-trace-py's real agentless tracer mode for FFE's scenario variants, which use a wholly separate agentless mechanism and don't want it. - Register the new utils/_context/_scenarios/debugger_agentless.py in utils/scripts/libraries_and_scenarios_rules.yml (scenario_groups: agentless), matching agentless_endtoend.py's existing entry - it was missing. - Replace @features.not_reported with real feature-parity IDs on the three test classes that had it: crashtracking (an exact existing match), trace submission and telemetry (no agentless-specific feature-parity page exists yet, so mapped to the closest existing ones - trace_data_integrity and telemetry_metrics_collected respectively; flagged this mapping for confirmation in the PR review reply since it's a judgment call, not a certainty).
…nd telemetry cbeauchesne added a new DD_AGENTLESS_ENABLED entry (feature=644, owned by language-platform) to the feature-parity dashboard specifically for this - trace_data_integrity and telemetry_metrics_collected were both approximate stand-ins, not correct matches. Test_Agentless_Crashtracking keeps @features.crashtracking (317), which was already an exact existing match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding tests to assert basic functionality of sending stuff to the intake.
Handling traces, stats, RC, debugger and SymDB.