bug: PHP-FPM worker crashes with heap corruption in _dd_curl_reset_headers (1.18.0)

[masak1yu]

Bug report

Summary

PHP-FPM worker processes crash with SIGABRT due to heap corruption in the background writer thread (`_dd_writer_loop`) with dd-trace-php 1.18.0. Each crash causes in-flight HTTP requests to be dropped with a 504 Gateway Timeout from the upstream reverse proxy.

Environment

• dd-trace-php version: 1.18.0
• PHP: 7.4 NTS, PHP-FPM (standard process manager, not FrankenPHP / not ZTS)
• OS: Linux x86_64 (glibc)
• libcurl: system libcurl (`/lib64/libcurl.so.4`)
• Rollback version that resolved the issue: 1.0.0

Symptom

PHP-FPM worker processes terminate with `SIGABRT` under production load. The crash is triggered by glibc detecting heap corruption (`malloc_printerr`) when the background writer thread calls `curl_slist_free_all` on the stored HTTP header list (`writer->headers`).

From the upstream reverse proxy (nginx), the crashing worker produces a `504 Gateway Timeout` (`upstream timed out, 110: Connection timed out`), interrupting request processing mid-flight.

Coredump stack trace

#0  0x00007f59ee354690 in raise () from /lib64/libpthread.so.0
#1  0x00007f59dfa617b0 in libdd_crashtracker::collector::signal_handler_manager::chain_signal_handler ()
    at libdatadog/libdd-crashtracker/src/collector/signal_handler_manager.rs:125
#2  libdd_crashtracker::collector::crash_handler::handle_posix_sigaction () at libdatadog/libdd-crashtracker/src/collector/crash_handler.rs:209
#3  <signal handler called>
#4  0x00007f59ee594ae0 in raise () from /lib64/libc.so.6
#5  0x00007f59ee595f88 in abort () from /lib64/libc.so.6
#6  0x00007f59ee5d4b94 in __libc_message () from /lib64/libc.so.6
#7  0x00007f59ee5da80a in malloc_printerr () from /lib64/libc.so.6
#8  0x00007f59ee5dad16 in munmap_chunk () from /lib64/libc.so.6
#9  0x00007f59eb30021d in curl_slist_free_all () from /lib64/libcurl.so.4
#10 0x00007f59df6512d8 in _dd_curl_reset_headers (writer=<optimized out>)
    at /go/src/github.com/DataDog/apm-reliability/dd-trace-php/tmp/build_extension/ext/coms.c:928
#11 0x00007f59df6521db in _dd_curl_send_stack (metrics=0x7f59d4f1e990, stack=<optimized out>, writer=0x7f59e0215960)
    at /go/src/github.com/DataDog/apm-reliability/dd-trace-php/tmp/build_extension/ext/coms.c:1057
#12 _dd_writer_loop (_=<optimized out>) at /go/src/github.com/DataDog/apm-reliability/dd-trace-php/tmp/build_extension/ext/coms.c:1268
#13 0x00007f59ee34a40b in start_thread () from /lib64/libpthread.so.0
#14 0x00007f59ee64de7f in clone () from /lib64/libc.so.6

Impact

Intermittent crashes over multiple days, with each crash dropping an in-flight request and leaving dozens of core dump files across multiple web hosts.

Workaround

Rolling back to 1.0.0 resolved the issue immediately.

Notes

• `DD_TRACE_CLI_ENABLED=false` and `DD_TRACE_SIDECAR_TRACE_SENDER=false` were already set in the environment at the time the crash occurred. These flags are documented to suppress the background writer thread for CLI, but the crash happened in PHP-FPM (web) regardless. The reason this configuration did not prevent the crash is unknown.
• Code inspection of `coms.c` shows that `writer->headers` is declared as `_Atomic(struct curl_slist *)` (line 412) but is written with a plain non-atomic assignment in `_dd_curl_set_headers` (line 955).
• `dd_agent_curl_headers` (the global slist used to seed per-request headers) is read from the writer thread without a lock, while it can be freed or reassigned from the main PHP thread (e.g., `ddtrace_coms_curl_shutdown`). A race between these two accesses is a candidate root cause for the heap corruption.

Related

• bug: malloc_consolidate heap corruption with 1.17.0 on FrankenPHP #3729 — similar heap corruption in FrankenPHP (ZTS, `malloc_consolidate`); different call site and environment, but may share the same underlying cause.

[realFlowControl]

Hello @masak1yu,

sorry for your problems and thanks for opening an bug report. In your notes section you mention:

• DD_TRACE_CLI_ENABLED=false and DD_TRACE_SIDECAR_TRACE_SENDER=false were already set in the environment at the time the crash occurred. These flags are documented to suppress the background writer thread for CLI, but the crash happened in PHP-FPM (web) regardless. The reason this configuration did not prevent the crash is unknown.

The workaround for this bug is to switch to the sidecar sender, so you'd need to set DD_TRACE_SIDECAR_TRACE_SENDER=true (true instead of false).

Could you try that and report back?

[masak1yu]

Thank you for the suggestion. Before applying the workaround, we'd like to reproduce the bug locally first to confirm the root cause. However, we've been unable to reproduce the crash after running for 20+ hours on both ARM and Intel. Do you have any hints on how to trigger it reliably?

[realFlowControl]

Sorry @masak1yu for my late response. The underlying bug is a race condition between two threads, where the write is freeing things that another thread has already freed. As such there is no reliable way to trigger it, it just races and sooner or later you'd hit it. As I mean, the way to fix this race condition is to migrate to the sidecar sender by setign DD_TRACE_SIDECAR_TRACE_SENDER=true.