perf(datadog): reuse one UDP socket per batch and coalesce metrics#13653
perf(datadog): reuse one UDP socket per batch and coalesce metrics#13653shreemaan-abhishek wants to merge 2 commits into
Conversation
The datadog plugin opened a fresh UDP socket for every entry in a flushed batch and sent 6 separate datagrams per entry (request.counter, request.latency, upstream.latency, apisix.latency, ingress.size, egress.size). For a batch of N entries this cost N socket setups and 6N sendto syscalls. Reuse a single socket across the whole batch and coalesce each entry's metrics into one newline-delimited DogStatsD datagram, cutting the per-batch work to 1 socket setup and N sends. Newline-delimited multi-metric packets are standard DogStatsD and parsed by the agent. The test mock now splits received datagrams on newlines, as a real DogStatsD server does, so existing metric assertions still hold. Signed-off-by: Abhishek Choudhary <shreemaan.abhishek@gmail.com>
dosubot
Bot
added
size:L
There was a problem hiding this comment.
Pull request overview
This PR improves the performance characteristics of the datadog plugin’s batch flush path by reusing a single UDP socket for the whole batch and by coalescing each entry’s metrics into one newline-delimited DogStatsD datagram (with the test mock updated to parse newline-delimited datagrams like a real DogStatsD server).
Changes:
- Reuse one UDP socket per batch flush rather than creating/closing a socket per entry.
- Coalesce per-entry metrics into a single newline-delimited DogStatsD payload (1 datagram per entry).
- Update the layer4 DogStatsD mock to split incoming datagrams on newlines for existing per-metric assertions.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| t/lib/mock_layer4.lua | Logs each newline-delimited metric within a received DogStatsD datagram to preserve existing per-metric test assertions. |
| apisix/plugins/datadog.lua | Builds multi-metric DogStatsD payloads per entry and reuses a single UDP socket across the batch flush. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for i = 1, #entries do | ||
| local ok, err = send_metric_over_udp(entries[i], metadata) | ||
| if not ok then | ||
| return false, err, i | ||
| -- coalesce the per-request metrics into one datagram | ||
| local send_ok, send_err = sock:send(build_metrics(entries[i], metadata)) | ||
| if not send_ok then | ||
| sock:close() | ||
| return false, "failed to send metrics to dogstatsd server: host[" .. host | ||
| .. "] port[" .. tostring(port) .. "] err: " .. send_err, i | ||
| end |
| local sock = udp() | ||
| local ok, err = sock:setpeername(host, port) | ||
| if not ok then | ||
| return false, "failed to connect to UDP server: host[" .. host | ||
| .. "] port[" .. tostring(port) .. "] err: " .. err | ||
| end |
Collapse the send loop to a single exit so there is one checked sock:close() instead of an unchecked close on the failure path plus a checked one at the end, matching the existing udp-logger idiom. Signed-off-by: Abhishek Choudhary <shreemaan.abhishek@gmail.com>
membphis
left a comment
membphis
left a comment
There was a problem hiding this comment.
[P2] Coalesced DogStatsD datagram has no size guard
This PR changes the Datadog plugin from sending each metric in its own UDP datagram to sending all metrics for one entry as a newline-delimited DogStatsD datagram. DogStatsD supports newline-delimited packets, but the implementation does not enforce any packet size limit or split oversized payloads.
APISIX allows long tag sources: constant_tags has no maxItems limit, route/service/consumer names can be long, and include_path can add a long route path. With those inputs, a single metric line may still be acceptable while the coalesced 5/6-line datagram exceeds the DogStatsD Agent buffer or practical UDP packet size limits, causing silent metric loss.
Please add packet-size-aware splitting or fall back to smaller datagrams when the coalesced payload is too large, and add a test covering long tags/path.
Description
The
datadogplugin opened a fresh UDP socket for every entry in a flushed batch and sent 6 separate datagrams per entry (request.counter,request.latency,upstream.latency,apisix.latency,ingress.size,egress.size). For a batch of N entries this cost N socket setups and 6Nsendtosyscalls in the batch-processor flush.This PR:
Per-batch work drops from
N sockets + 6N sendsto1 socket + N sends. Newline-delimited multi-metric packets are standard DogStatsD and are parsed by the Datadog agent, so this is wire-compatible with existing receivers. The plugin schema and the metrics emitted are unchanged.The test mock (
t/lib/mock_layer4.luadogstatsd()) now splits received datagrams on newlines, as a real DogStatsD server does, so the existing per-metric assertions int/plugin/datadog.tcontinue to hold without changes.Checklist
t/plugin/datadog.tassertions cover the emitted metrics; mock updated to reflect coalesced framing)