chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642
Open
jbachorik wants to merge 32 commits into
Open
chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642jbachorik wants to merge 32 commits into
jbachorik wants to merge 32 commits into
Conversation
…ALLOC_PERTURB_ for glibc test runs Antagonist races tracer-style context propagation against virtual-thread carrier churn to target the ContextStorageMode.THREAD stale-carrier use-after-free. Since such heap corruption manifests silently until an unrelated later allocation, also turn on glibc's own corruption checks by default for chaos runs and ddprof-test suites (skipped on musl and whenever a sanitizer/allocator already owns malloc via LD_PRELOAD). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
CI Test ResultsRun: #29432708695 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-15 16:52:30 UTC |
J9 hits a pre-existing, unrelated glibc heap-corruption bug that MALLOC_CHECK_ now surfaces as a hard abort; gate it off until that's investigated separately.
Contributor
Reliability & Chaos Results❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124747408 ❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos |
Contributor
|
isRawPointer() only checked RAW_POINTER_MASK, letting raw J9 ASGCT BCIs that never went through encode() misroute into JVMSupport::resolve(), which asserts false for non-HotSpot VMs. Require ENCODED_MASK too.
It was registered in Main.java's antagonist factory but never added to either config's ANTAGONISTS string, so it silently never ran in CI.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1dd0fb1c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Include ACTIVE_PHASE_MILLIS in STALE_RACER_SLEEP_MILLIS so racers wake as the next active phase begins instead of ~3s early. Bound stopGracefully's total wait to the caller's timeout via a shared deadline instead of giving each join/tryAcquire the full budget independently.
…ntext via @trace driverLoop() called JavaProfiler.getInstance() directly, but JavaProfiler is a chaosCompileOnly dependency never bundled into chaos.jar and not visible to the app classloader (the agent's copy is shaded/relocated). The resulting NoClassDefFoundError is an Error, not an Exception, so it escaped the catch (Exception e) block and silently killed the driver thread — leaving activePhase stuck true and letting the sibling pinningChurnLoop run full-throttle for the rest of the test instead of its intended active/quiet duty cycle, which is the likely driver behind the OOM seen in reliability-chaos-aarch64 [profiler, gmalloc, 21.0.3-tem]. Rewrite the antagonist to only touch the tracer through @Trace-annotated methods (the same proven-safe dd-trace-api compileOnly pattern already used by TraceContextAntagonist), letting the real tracer and its virtual-thread instrumentation drive setContext/clearContext instead of calling com.datadoghq.profiler.* directly. Drops the custom-attribute/baggage simulation in favor of nested @trace hops plus forceUnmount and stale-buffer racer threads that still race OtelContextStorage's thread-scoped fallback across carrier churn. Verified with a 90s live run (JDK 21.0.11, profiler+tracer, gmalloc): no NoClassDefFoundError, no exceptions, completed cleanly with RC=0. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without a RUNTIME arg, chaos_check.sh silently proceeds with an empty value, which becomes --duration s in the chaos harness invocation and fails 15+ seconds later — after the JDK/agent/jar setup — with a NumberFormatException buried in the harness output, reported generically as "FAIL:Chaos harness crashed (RC=1)". Check for a missing RUNTIME right after arg parsing and fail immediately with a clear usage message. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 3e379a7)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123961967 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
|
Not a review, but one thing that I can't quickly see from looking at the code: does it trigger actual migrations of vthreads across carriers? Because that is the fault scenario that led to the heap corruption. |
Collaborator
Author
|
@rkennke It is trying to - but it is very probabilistic. It is trying to hold back carrier threads so new ones have to be spawned, then let the old one die so the vthread will have to be remounted but DBB still point to now invalid memory. |
Contributor
J9
|
Contributor
I see. If you have the time/nerve(/AI tokens) for it, you could perhaps try using a custom FJP where you can control the thread lifecycles. Probabilistic is fine, though, better than nothing. |
Collaborator
Author
|
@rkennke That's a great idea. Thanks! |
Extract chaos_check.sh's logic into utils/run-chaos-harness.sh for local repro. Add a #ifdef DEBUG-only watchdog in ProfiledThread's free path (threadLocalData.cpp) that detects writes into freed OTel-context memory, and beef up VirtualThreadContextCascadeAntagonist's stale-carrier racing (continuous decoupled driver, randomized race window, dead-code removal). Fixes from review: Main.java antagonist-start try/finally, watcher thread signal-mask inheritance, racer-loop backoff under saturation, redundant frame_ut.cpp assertions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
watchFreedContextMemory() deliberately read freed ProfiledThread memory to detect stale writes, tripping ASan/TSan on every free (e.g. in forced_unwind_ut) unrelated to the actual race under investigation.
Contributor
Benchmark Results (commit b71ca62)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124279353 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10191 ms (21 iters) | ✅ 10399 ms (21 iters) | ≈ +2% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8940 ms (24 iters) | ✅ 8819 ms (24 iters) | ≈ -1.4% (±9.6%) | — / — |
| finagle-chirper | 21 | ✅ 5958 ms (33 iters) | ✅ 5999 ms (33 iters) | ≈ +0.7% (±24.8%) | |
| finagle-chirper | 25 | ✅ 5524 ms (36 iters) | ✅ 5491 ms (36 iters) | ≈ -0.6% (±25%) | |
| fj-kmeans | 21 | ✅ 2746 ms (68 iters) | ✅ 2742 ms (68 iters) | ≈ -0.1% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2737 ms (69 iters) | ✅ 2836 ms (66 iters) | 🔴 +3.6% | — / — |
| future-genetic | 21 | ✅ 2083 ms (89 iters) | ✅ 2065 ms (90 iters) | ≈ -0.9% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2041 ms (90 iters) | ✅ 2141 ms (87 iters) | 🔴 +4.9% | — / — |
| naive-bayes | 21 | ✅ 1291 ms (133 iters) | ✅ 1255 ms (136 iters) | ≈ -2.8% (±31.7%) | — / — |
| naive-bayes | 25 | ✅ 983 ms (174 iters) | ✅ 999 ms (171 iters) | ≈ +1.6% (±31.9%) | — / — |
| reactors | 21 | ✅ 16505 ms (15 iters) | ✅ 16271 ms (15 iters) | ≈ -1.4% (±8.3%) | — / — |
| reactors | 25 | ✅ 18606 ms (14 iters) | ✅ 18677 ms (15 iters) | ≈ +0.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2094 / 2059 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 5 | 2307 / 2237 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8584 / 8425 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 8769 / 8940 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1252 / 1221 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1320 / 1251 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2917 / 2984 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2900 / 2964 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 3507 / 3542 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3509 / 3455 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1682 / 1599 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1520 / 1920 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 2e34507)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124382794 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
profiler+tracer chaos runs OOMKilled on aarch64 (exit 137) with gmalloc/jemalloc but not tcmalloc, which already had aggressive-decommit tuning for the same thread-churn RSS-inflation issue. Apply the equivalent tuning to the other two allocators.
Contributor
Benchmark Results (commit cfe1245)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124478924 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Measured on matching-arch hosts: this workload floats at ~2.6-2.75GB RSS regardless of MALLOC_CHECK_/allocator tuning, ruling that out as the OOM cause. Trim -Xmx to 1536m on aarch64 for real headroom, and log the container's actual cgroup memory limit so future OOMs are diagnosable from the job log directly.
Contributor
Benchmark Results (commit 33a8f3d)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124498331 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10359 ms (21 iters) | ✅ 10432 ms (21 iters) | ≈ +0.7% (±11%) | — / — |
| akka-uct | 25 | ✅ 8965 ms (24 iters) | ✅ 8729 ms (24 iters) | ≈ -2.6% (±10.8%) | — / — |
| finagle-chirper | 21 | ✅ 6009 ms (33 iters) | ✅ 6032 ms (33 iters) | ≈ +0.4% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5446 ms (36 iters) | ✅ 5464 ms (36 iters) | ≈ +0.3% (±24.1%) | |
| fj-kmeans | 21 | ✅ 2702 ms (69 iters) | ✅ 2788 ms (67 iters) | 🔴 +3.2% | — / — |
| fj-kmeans | 25 | ✅ 2748 ms (68 iters) | ✅ 2757 ms (67 iters) | ≈ +0.3% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2153 ms (86 iters) | ✅ 2092 ms (89 iters) | 🟢 -2.8% | — / — |
| future-genetic | 25 | ✅ 2057 ms (91 iters) | ✅ 2038 ms (91 iters) | ≈ -0.9% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1257 ms (136 iters) | ✅ 1280 ms (134 iters) | ≈ +1.8% (±32.7%) | — / — |
| naive-bayes | 25 | ✅ 1026 ms (167 iters) | ✅ 982 ms (173 iters) | ≈ -4.3% (±30.9%) | — / — |
| reactors | 21 | ✅ 17155 ms (15 iters) | ✅ 16050 ms (15 iters) | ≈ -6.4% (±7.4%) | — / — |
| reactors | 25 | ✅ 18361 ms (15 iters) | ✅ 18840 ms (15 iters) | ≈ +2.6% (±3.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2050 / 1918 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 2273 / 2115 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8651 / 8737 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 8617 / 8305 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 1262 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1279 / 1244 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2954 / 3068 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2901 / 2857 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 3474 / 3522 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 7 / 3 | 3505 / 3513 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1780 / 1682 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1893 / 1910 | ✅ / ✅ | ✅ / ✅ |
Collaborator
Author
|
@copilot resolve the merge conflicts in this pull request |
Contributor
Resolved by merging |
Cleans /tmp/ddprof_root before and after each run so failed-upload chunks from one run can't starve the next run's JFR writer, and logs disk usage for future disk-full crash diagnosis.
Contributor
Benchmark Results (commit 17d0792)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124513178 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10378 ms (21 iters) | ✅ 10302 ms (21 iters) | ≈ -0.7% (±11.7%) | — / — |
| akka-uct | 25 | ✅ 8861 ms (24 iters) | ✅ 8870 ms (24 iters) | ≈ +0.1% (±9.7%) | — / — |
| finagle-chirper | 21 | ✅ 6046 ms (33 iters) | ✅ 6109 ms (33 iters) | ≈ +1% (±25.7%) | |
| finagle-chirper | 25 | ✅ 5515 ms (36 iters) | ✅ 5589 ms (35 iters) | ≈ +1.3% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2780 ms (67 iters) | ✅ 2656 ms (70 iters) | 🟢 -4.5% | — / — |
| fj-kmeans | 25 | ✅ 2831 ms (66 iters) | ✅ 2820 ms (66 iters) | ≈ -0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2067 ms (89 iters) | ✅ 2164 ms (86 iters) | 🔴 +4.7% | — / — |
| naive-bayes | 21 | ✅ 1283 ms (133 iters) | ✅ 1289 ms (133 iters) | ≈ +0.5% (±33.1%) | — / — |
| naive-bayes | 25 | ✅ 968 ms (175 iters) | ✅ 1008 ms (170 iters) | ≈ +4.1% (±32.5%) | — / — |
| reactors | 21 | ✅ 15934 ms (15 iters) | ✅ 16648 ms (15 iters) | ≈ +4.5% (±8.1%) | — / — |
| reactors | 25 | ✅ 18828 ms (15 iters) | ✅ 18359 ms (15 iters) | ≈ -2.5% (±4.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2003 / 2011 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2162 / 2344 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8597 / 8674 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8651 / 8404 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1284 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1283 / 1277 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2971 / 3043 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 2874 / 2900 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3512 / 3521 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 3440 / 3533 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1649 / 1644 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1911 / 1819 | ✅ / ✅ | ✅ / ✅ |
Container cgroup limit confirmed at 6GiB; the 2.6-2.75GB RSS ceiling assumed in 33a8f3d was an inferred guess, not a measured cgroup value.
Contributor
Benchmark Results (commit be3c3bf)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124562849 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10226 ms (21 iters) | ✅ 10174 ms (21 iters) | ≈ -0.5% (±10.9%) | — / — |
| akka-uct | 25 | ✅ 8753 ms (24 iters) | ✅ 8941 ms (24 iters) | ≈ +2.1% (±10.1%) | — / — |
| finagle-chirper | 21 | ✅ 5920 ms (33 iters) | ✅ 6013 ms (33 iters) | ≈ +1.6% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5449 ms (36 iters) | ✅ 5493 ms (36 iters) | ≈ +0.8% (±25.2%) | |
| fj-kmeans | 21 | ✅ 2761 ms (68 iters) | ✅ 2778 ms (67 iters) | ≈ +0.6% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2743 ms (68 iters) | ✅ 2822 ms (66 iters) | 🔴 +2.9% | — / — |
| future-genetic | 21 | ✅ 2057 ms (91 iters) | ✅ 2122 ms (87 iters) | 🔴 +3.2% | — / — |
| future-genetic | 25 | ✅ 2099 ms (88 iters) | ✅ 2105 ms (88 iters) | ≈ +0.3% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1276 ms (134 iters) | ✅ 1252 ms (137 iters) | ≈ -1.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1014 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.3% (±31.5%) | — / — |
| reactors | 21 | ✅ 15190 ms (15 iters) | ✅ 16624 ms (15 iters) | 🔴 +9.4% | — / — |
| reactors | 25 | ✅ 18330 ms (15 iters) | ✅ 18900 ms (15 iters) | ≈ +3.1% (±4.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2013 / 1963 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 7 | 2311 / 2165 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8339 / 8974 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8490 / 8255 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1255 / 1304 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1254 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2940 / 2992 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2880 / 2882 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 9 | 3513 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1457 / 1670 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1843 / 1882 | ✅ / ✅ | ✅ / ✅ |
Throttles alloc-storm/direct-memory/dump-storm via cgroup memory watermarks (85%/70%) instead of trimming heap to make off-heap headroom. Validated: the jemalloc/2560m/profiler+tracer config that OOMKilled in CI (job 1858271921) now completes cleanly under a real 6GiB-capped cgroup.
Contributor
Benchmark Results (commit ae3daa1)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124604420 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10249 ms (21 iters) | ✅ 10262 ms (21 iters) | ≈ +0.1% (±10.8%) | — / — |
| akka-uct | 25 | ✅ 8927 ms (24 iters) | ✅ 9068 ms (24 iters) | ≈ +1.6% (±10.9%) | — / — |
| finagle-chirper | 21 | ✅ 5962 ms (33 iters) | ✅ 6010 ms (33 iters) | ≈ +0.8% (±25.6%) | |
| finagle-chirper | 25 | ✅ 5482 ms (36 iters) | ✅ 5470 ms (36 iters) | ≈ -0.2% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2712 ms (69 iters) | ✅ 2721 ms (69 iters) | ≈ +0.3% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2829 ms (66 iters) | ✅ 2855 ms (66 iters) | ≈ +0.9% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2049 ms (90 iters) | ✅ 2126 ms (88 iters) | 🔴 +3.8% | — / — |
| future-genetic | 25 | ✅ 2047 ms (91 iters) | ✅ 2024 ms (92 iters) | ≈ -1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1266 ms (135 iters) | ✅ 1352 ms (127 iters) | ≈ +6.8% (±33.9%) | — / — |
| naive-bayes | 25 | ✅ 1013 ms (169 iters) | ✅ 1026 ms (166 iters) | ≈ +1.3% (±32%) | — / — |
| reactors | 21 | ✅ 16418 ms (15 iters) | ✅ 15608 ms (15 iters) | ≈ -4.9% (±8.2%) | — / — |
| reactors | 25 | ✅ 18164 ms (15 iters) | ✅ 17971 ms (15 iters) | ≈ -1.1% (±6.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1939 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 2164 / 2347 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 8291 / 8431 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 8283 / 8356 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1278 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1274 / 1286 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2934 / 3045 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2890 / 2824 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 7 | 3502 / 3552 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3464 / 3429 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1765 / 1536 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1844 / 1973 | ✅ / ✅ | ✅ / ✅ |
…s heap to 3072m Cgroup-only signal missed heap exhaustion when Xmx is a small fraction of the cgroup limit (job 1860026763). Governor now also watches heap usage; pace() wired into weakref-wave, hidden-class-churn, context-hop, bounded-pool, consumer-group alongside the existing alloc-storm/direct-memory/dump-storm. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 2a3e926)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124688617 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
…nsitions; disable Gradle daemon in inline chaos.jar build burstLoop() had its own fixed sleep but never checked MemoryGovernor, letting it allocate at full rate regardless of memory pressure. Governor now logs each throttle/release transition so CI runs show whether it engaged. A lingering Gradle daemon from the inline chaosJar build was also sharing the CI container's cgroup with the chaos JVM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit d1f2a0f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747414 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10284 ms (21 iters) | ✅ 10144 ms (21 iters) | ≈ -1.4% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8813 ms (24 iters) | ✅ 8820 ms (24 iters) | ≈ +0.1% (±9.7%) | — / — |
| finagle-chirper | 21 | ✅ 6022 ms (33 iters) | ✅ 6007 ms (33 iters) | ≈ -0.2% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5472 ms (36 iters) | ✅ 5464 ms (36 iters) | ≈ -0.1% (±24.7%) | |
| fj-kmeans | 21 | ✅ 2765 ms (68 iters) | ✅ 2829 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2804 ms (66 iters) | ✅ 2813 ms (66 iters) | ≈ +0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2040 ms (91 iters) | ✅ 2152 ms (86 iters) | 🔴 +5.5% | — / — |
| future-genetic | 25 | ✅ 2029 ms (92 iters) | ✅ 2091 ms (89 iters) | 🔴 +3.1% | — / — |
| naive-bayes | 21 | ✅ 1270 ms (136 iters) | ✅ 1270 ms (134 iters) | ≈ 0% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1019 ms (168 iters) | ✅ 1029 ms (166 iters) | ≈ +1% (±31.7%) | — / — |
| reactors | 21 | ✅ 16566 ms (15 iters) | ✅ 16611 ms (15 iters) | ≈ +0.3% (±7.3%) | — / — |
| reactors | 25 | ✅ 18343 ms (15 iters) | ✅ 18406 ms (15 iters) | ≈ +0.3% (±6.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2015 / 1901 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 2301 / 2165 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 8670 / 8927 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 8793 / 8343 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1265 / 1259 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 7 / ✅ | 1282 / 1256 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2964 / 2930 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2932 / 2896 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3525 / 3504 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3474 / 3487 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1770 / 1625 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1745 / 1812 | ✅ / ✅ | ✅ / ✅ |
pace() now brakes harder (50ms vs 5ms) and update() fires a one-shot System.gc() above 0.93 cgroup / 0.90 heap, since plain throttling wasn't enough to stop a heap OOME in job 1861234807. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 2ef0fd2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124764606 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
pace() now peeks at heap usage inline every 64th call so a fast burst escalates before the next sampler tick, since 500ms was too coarse for a heap that filled 16%->97% between samples (job 1861521304). Also add -XX:+ExitOnOutOfMemoryError so a heap OOME exits immediately instead of cascading into a SIGSEGV with no hs_err.log. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kaahos
reviewed
Jul 15, 2026
kaahos
left a comment
kaahos
left a comment
Contributor
There was a problem hiding this comment.
I've started to take a look at it and left some comments!
| # chunk) into an immediate, attributable SIGABRT instead of a crash much | ||
| # later in an unrelated allocation. Only meaningful against glibc's own | ||
| # malloc, not the LD_PRELOAD-replaced allocators below. | ||
| export MALLOC_CHECK_=3 |
Contributor
There was a problem hiding this comment.
for quite recent glibc (>= 2.34; the CI images use Ubuntu 22.04/glibc 2.35), malloc checking is disabled in the main C library and requires preloading libc_malloc_debug (using LD_PRELOAD). from glibc heap-consistency documentation:
Like in the case of mcheck, one would need to preload the libc_malloc_debug library to enable MALLOC_CHECK_ functionality. Without this preloaded library, setting MALLOC_CHECK_ will have no effect.
I think we should try to detect and preload libc_malloc_debug for the glibc 2.34+, or fail setup clearly if it cannot be found, so that the intended corruption detection is guaranteed to be active, wdyt?
Comment on lines
+126
to
+128
| DD_AGENT_JAR="${DD_AGENT_JAR}" DDPROF_JAR="${DDPROF_JAR}" OUTPUT_JAR="${PATCHED_AGENT}" \ | ||
| "${ROOT}/utils/patch-dd-java-agent.sh" | ||
| if [ ! -f "${PATCHED_AGENT}" ]; then |
Contributor
There was a problem hiding this comment.
since the script uses set +e, we might check the patcher's exit status directly instead of inferring success from the existence of the jar? patch-dd-java-agent.sh creates the JAR before completing validation, so a validation failure can leave an output file which would bypass this check.
Suggested change
| DD_AGENT_JAR="${DD_AGENT_JAR}" DDPROF_JAR="${DDPROF_JAR}" OUTPUT_JAR="${PATCHED_AGENT}" \ | |
| "${ROOT}/utils/patch-dd-java-agent.sh" | |
| if [ ! -f "${PATCHED_AGENT}" ]; then | |
| if ! DD_AGENT_JAR="${DD_AGENT_JAR}" DDPROF_JAR="${DDPROF_JAR}" OUTPUT_JAR="${PATCHED_AGENT}" \ | |
| "${ROOT}/utils/patch-dd-java-agent.sh"; then |
| // Such values occur on non-HotSpot VMs (e.g. J9), which never call encode() | ||
| // and therefore never set ENCODED_MASK; only encode(..., rawPointer=true) | ||
| // (HotSpot-only, per its own assert) is allowed to set RAW_POINTER_MASK. | ||
| int rawAsgctBciWithBit30 = 1 << 30; |
Contributor
There was a problem hiding this comment.
Hotspot is off by default and since isRawPointer() requires Hotspot to be on, the test always passes. I'd recommend forcing Hotspot to be activated.
Suggested change
| int rawAsgctBciWithBit30 = 1 << 30; | |
| int rawAsgctBciWithBit30 = 1 << 30; | |
| VMHotspotGuard hotspot(true); |
Contributor
Benchmark Results (commit 08390ce)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124799195 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10375 ms (21 iters) | ✅ 10149 ms (21 iters) | ≈ -2.2% (±10.8%) | — / — |
| akka-uct | 25 | ✅ 8799 ms (24 iters) | ✅ 8810 ms (24 iters) | ≈ +0.1% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 6004 ms (33 iters) | ✅ 5987 ms (33 iters) | ≈ -0.3% (±24.9%) | |
| finagle-chirper | 25 | ✅ 5479 ms (36 iters) | ✅ 5475 ms (36 iters) | ≈ -0.1% (±24.1%) | |
| fj-kmeans | 21 | ✅ 2749 ms (68 iters) | ✅ 2820 ms (66 iters) | ≈ +2.6% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2871 ms (66 iters) | ✅ 2849 ms (66 iters) | ≈ -0.8% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2027 ms (92 iters) | ✅ 2133 ms (88 iters) | 🔴 +5.2% | — / — |
| future-genetic | 25 | ✅ 2031 ms (91 iters) | ✅ 2110 ms (88 iters) | 🔴 +3.9% | — / — |
| naive-bayes | 21 | ✅ 1299 ms (131 iters) | ✅ 1319 ms (131 iters) | ≈ +1.5% (±32.8%) | — / — |
| naive-bayes | 25 | ✅ 1010 ms (170 iters) | ✅ 1014 ms (169 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16954 ms (15 iters) | ✅ 16040 ms (15 iters) | ≈ -5.4% (±7.8%) | — / — |
| reactors | 25 | ✅ 18876 ms (15 iters) | ✅ 18543 ms (15 iters) | ≈ -1.8% (±3.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1961 / 2029 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2142 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 8351 / 8465 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 8501 / 8600 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1273 / 1282 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1308 / 1277 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 3025 / 2980 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2846 / 2882 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3479 / 3552 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3476 / 3468 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1819 / 1741 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1852 / 1863 | ✅ / ✅ | ✅ / ✅ |
RC=3 (-XX:+ExitOnOutOfMemoryError) past the halfway point of the run means antagonists got a fair shot before hitting the heap ceiling — treat that as tolerated, not a failure. Before the halfway point it still fails, with a message distinguishing it from an actual crash. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 68ceb93)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124817306 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10255 ms (21 iters) | ≈ -0.5% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8778 ms (24 iters) | ✅ 8848 ms (24 iters) | ≈ +0.8% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6033 ms (33 iters) | ✅ 5964 ms (33 iters) | ≈ -1.1% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5465 ms (36 iters) | ✅ 5488 ms (36 iters) | ≈ +0.4% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2779 ms (67 iters) | ✅ 2736 ms (68 iters) | ≈ -1.5% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2839 ms (66 iters) | ✅ 2842 ms (66 iters) | ≈ +0.1% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2039 ms (90 iters) | ✅ 2072 ms (89 iters) | ≈ +1.6% (±2.5%) | — / — |
| future-genetic | 25 | ✅ 2069 ms (89 iters) | ✅ 2002 ms (92 iters) | 🟢 -3.2% | — / — |
| naive-bayes | 21 | ✅ 1245 ms (138 iters) | ✅ 1288 ms (133 iters) | ≈ +3.5% (±33.1%) | — / — |
| naive-bayes | 25 | ✅ 1022 ms (167 iters) | ✅ 1028 ms (166 iters) | ≈ +0.6% (±32.1%) | — / — |
| reactors | 21 | ✅ 16345 ms (15 iters) | ✅ 16388 ms (15 iters) | ≈ +0.3% (±8.8%) | — / — |
| reactors | 25 | ✅ 18285 ms (15 iters) | ✅ 18580 ms (15 iters) | ≈ +1.6% (±4.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 2006 / 2091 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2306 / 2458 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 8510 / 8550 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8164 / 8361 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1265 / 1270 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2929 / 2905 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 2873 / 2848 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 2 | 3570 / 3504 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 10 / 5 | 3485 / 3462 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1722 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1897 / 1886 | ✅ / ✅ | ✅ / ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Adds a new
vthread-context-cascadechaos antagonist that races tracer-stylecontext propagate/activate/restore semantics against virtual-thread carrier
churn, targeting the
ContextStorageMode.THREADstale-carrierDirectByteBufferuse-after-free. Also defaultsMALLOC_CHECK_/MALLOC_PERTURB_for glibc-backed test runs (chaos reliability runner andthe shared
ddprof-testGradle test-task configuration), turning silentheap corruption into an immediate, attributable
SIGABRTinstead of acrash much later in an unrelated allocation.
Motivation:
While building the
vthread-context-cascadereproducer it became clearthat this class of use-after-free bug corrupts glibc heap/tcache metadata
silently and only manifests as a crash far later, in an unrelated
allocation call — making it unlikely to be caught within a short chaos
run or CI test window. Enabling glibc's own heap-corruption checks by
default closes that gap cheaply, without needing a sanitizer build (which
can't run against a live JVM, see
reference_sanitizers_no_jvm).Additional Notes:
The
MALLOC_CHECK_/MALLOC_PERTURB_defaults are skipped on musl (noMALLOC_CHECK_support) and whenever a sanitizer/allocator alreadyreplaces malloc via
LD_PRELOAD(ASan/TSan/tcmalloc/jemalloc configs),mirroring the existing
gmallocvstcmalloc/jemallocmutualexclusivity in
chaos_check.sh.How to test the change?:
./gradlew :ddprof-stresstest:chaosJarbuilds cleanly with the new antagonist registered../gradlew :build-logic:conventions:compileKotlincompiles cleanly with theProfilerTestPluginchange.-Dddprof.debug.context.storage.mode=thread, tuned virtual-thread scheduler flags,MALLOC_CHECK_=3/MALLOC_PERTURB_set) for several minutes; no crash reproduced yet in this window — the antagonist is being landed as a reliability-suite addition to keep exercising this race over longer soak runs, not as a guaranteed-repro smoke test.For Datadog employees:
If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.This PR doesn't touch any of that.
JIRA: PROF-15360
Unsure? Have a question? Request a review!