debugger: snapshot correlation tests

[grantseltzer]

Motivation

The Live Debugger lets an engineer drop logpoints into running code and see snapshots of what happened. Those snapshots are supposed to tell a complete story of a request from start to finish, but today the related snapshots often arrive in pieces, with no signal that anything is missing. There is now an approved plan for how every language's tracer should behave so the story stays whole, but there was no automated way to say whether each language actually follows it.

This adds that check, so we can track, per language, who meets the standard and who does not yet.

Changes

Adds end-to-end tests that hold each language's tracer to the agreed snapshot-correlation standard:

• Related snapshots from a single request stay together: either the whole chain arrives or none of it does, never a partial chain, and they share one request id.
• A logpoint inside a loop records once per request instead of flooding.
• Each snapshot reports which process and which thread it came from.

The tests run for all six languages (Go, Java, .NET, Node, Python, Ruby). Because no tracer has built this behavior yet, every language is currently marked "not yet implemented," so the tests run without failing the build and will turn green automatically once a language implements the standard. The change also adds the test endpoints to each language's sample app, Go included.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/debugger/test_debugger_snapshot_correlation.py                    @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_snapshot_log_correlation.json         @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_snapshot_log_correlation_loop.json    @DataDog/debugger @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/debugger/utils.py                                                 @DataDog/debugger @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/build/docker/dotnet/weblog/Controllers/DebuggerController.cs      @DataDog/apm-dotnet @DataDog/asm-dotnet @DataDog/system-tests-core
utils/build/docker/golang/app/chi/debugger.go                           @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/chi/main.go                               @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/echo/debugger.go                          @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/echo/main.go                              @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/gin/debugger.go                           @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/gin/main.go                               @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http-orchestrion/debugger.go          @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http-orchestrion/main.go              @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http/debugger.go                      @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http/main.go                          @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/java/spring-boot/src/main/java/com/datadoghq/system_tests/springboot/debugger/DebuggerController.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/nodejs/express/debugger/index.js                     @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express4-typescript/debugger/index.ts         @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/fastify/debugger/index.js                     @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/python/flask/debugger/debugger_controller.py         @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/ruby/rails72/config/routes.rb                        @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails80/config/routes.rb                        @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/shared/rails/app/controllers/debugger_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core

[datadog-datadog-prod-us1]

 

⚠️ Warnings

🚦 2 Pipeline jobs failed

Testing the test | System Tests (nodejs, prod) / End-to-end #1 / fastify 1     

Testing the test | all-jobs-are-green     

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f8ce791 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af522adb7c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[datadog-datadog-prod-us1]

Go's correlationLoop weblog endpoints were missing time.Sleep(time.Second) per loop iteration, making Test_Debugger_Per_Span_Budget a false-positive gate for Go: without the sleep the 3-iteration loop completes in microseconds, so a time-based rate-limit (1/sec) and a span-scoped budget (1/span) both emit ≤1 snapshot and the test can't distinguish them. Fix applied: added time.Sleep(time.Second) to all five Go weblog variants (chi, echo, gin, net-http, net-http-orchestrion) and updated all affected Go line-number mappings in tests/debugger/utils.py (+1 on each entry due to the new "time" import line).

_{📊 Validated against 12 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit af522ad · What is Autotest? · Any feedback? Reach out in #autotest}