[appsec] system tests for agentic onboarding runtime detection by christophe-papazian · Pull Request #7320 · DataDog/system-tests

christophe-papazian · 2026-07-16T09:45:39Z

APPSEC-69171

System tests for RFC-1110 (Agentic Onboarding Runtime Detection, feature #563): validate the appsec.agentic_onboarding boolean in app-started telemetry when DD_APPSEC_AGENTIC_ONBOARDING is set — true when AppSec is active at startup, false otherwise (flag is presence-only).

Reuses existing telemetry scenarios; the two tests are missing_feature until each tracer implements the signal.

🤖 Generated with Claude Code

github-actions · 2026-07-16T09:46:04Z

CODEOWNERS have been resolved as:

tests/appsec/test_agentic_onboarding.py                                 @DataDog/asm-libraries @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-07-16T10:07:04Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 18082e8 | Docs | Datadog PR Page | Give us feedback!}

datadog-system-tests-org · 2026-07-16T10:47:53Z

📊 dbt CICD (full report)

Impact Lineage

⚠️ Analysis failed. An error occurred while computing the impact lineage for this PR.

Error details

Error fetching impact lineage results.

Drift Detection

⚠️ Drift detection failed. An error occurred while computing prod-vs-CI tests for this PR.

Error details

Error fetching drift detection results.

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: c900870 | Docs | Give us feedback!}

christophe-papazian · 2026-07-16T15:19:38Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 322e797c4c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 70986e693b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

RFC-1110: validate the appsec.agentic_onboarding boolean reported in app-started telemetry when DD_APPSEC_AGENTIC_ONBOARDING is set (true when AppSec is active at startup, false otherwise). Reuses the existing telemetry scenarios; tests are missing_feature until tracers implement it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The signal targets the 7 application tracers (RFC-1110); C++ proxies are out of scope, so the test is irrelevant there (cpp_kong already blankets tests/appsec as irrelevant). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- assert origin=env_var in the AppSec-disabled (false) case so the entry is driven by the flag itself, not a default/product-state config - clarify docstrings: the signal may ride app-started or the subsequent app-client-configuration-change (RFC-1110), both intentionally accepted Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- exact-case match on the config key (intake normalization relies on it) - assert the raw DD_APPSEC_AGENTIC_ONBOARDING key is never transmitted - strict boolean checks (reject numeric 0/1 via `is`) - precise return typing; drop redundant comment - cpp: drop redundant cpp_httpd entry (blanket already covers), use bare missing_feature for cpp/cpp_nginx (RFC-1110 targets the 7 app tracers) - python: feature merged on main -> activate at v4.12.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- #4: replace entries[-1] with a loop asserting every reported entry is consistent (origin=env_var + expected bool) — catches contradictory/ duplicate reports without a count assertion that would flake on prefork/multiprocess weblogs - python: activate at v4.13.0-rc1 (feature merged on main) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…telemetry Some tracers report telemetry configuration entries under the raw dotted name (appsec.agentic_onboarding) while others (e.g. Java) normalize it to the env-var form (DD_APPSEC_AGENTIC_ONBOARDING, uppercased with dots -> underscores). Accept either, matching the convention used by other telemetry tests (e.g. ["DD_APPSEC_ENABLED", "appsec.enabled"]). Also drops the incorrect "raw key must not be reported" assertions from #7320: the two names are just the raw and normalized forms of the same config, not a forbidden separate entry. Presence-only refers to the env-var's value never being read, not to a second telemetry entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

christophe-papazian force-pushed the christophe-papazian/appsec-agentic-onboarding-runtime-detection branch from c68ce33 to 322e797 Compare July 16, 2026 13:45

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread tests/appsec/test_agentic_onboarding.py Outdated

Comment thread tests/appsec/test_agentic_onboarding.py Outdated

christophe-papazian marked this pull request as ready for review July 16, 2026 17:21

christophe-papazian requested review from a team as code owners July 16, 2026 17:21

christophe-papazian requested review from claponcet, jandro996, juanjux, mhlidd, quinna-h and zacharycmontoya and removed request for a team July 16, 2026 17:21

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread tests/appsec/test_agentic_onboarding.py

avara1986 approved these changes Jul 17, 2026

View reviewed changes

cbeauchesne approved these changes Jul 17, 2026

View reviewed changes

e-n-0 approved these changes Jul 17, 2026

View reviewed changes

christophe-papazian and others added 4 commits July 17, 2026 11:33

christophe-papazian force-pushed the christophe-papazian/appsec-agentic-onboarding-runtime-detection branch from 70986e6 to 83de56e Compare July 17, 2026 09:33

christophe-papazian merged commit 160baa9 into main Jul 17, 2026
2084 of 2085 checks passed

christophe-papazian deleted the christophe-papazian/appsec-agentic-onboarding-runtime-detection branch July 17, 2026 11:20

christophe-papazian mentioned this pull request Jul 17, 2026

[appsec] accept normalized or raw config name for agentic onboarding telemetry #7337

Open

Uh oh!

Conversation

christophe-papazian commented Jul 16, 2026

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-system-tests-org Bot commented Jul 16, 2026

Uh oh!

christophe-papazian commented Jul 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Jul 16, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 16, 2026 •

edited

Loading