fix: drain the mailbox before stopping by Aaalibaba42 · Pull Request #2258 · DataDog/libdatadog

Aaalibaba42 · 2026-07-21T13:43:50Z

What does this PR do?

A brief description of the change being made with this pull request.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

pr-commenter · 2026-07-21T13:57:31Z

Benchmarks

Comparison

Benchmark execution time: 2026-07-22 12:16:09

Comparing candidate commit 0cc0d67 in PR branch jwiriath/drain-before-stop with baseline commit 3081603 in branch main.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 23 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:trace_buffer/4_senders/no_delay

🟥 execution_time [+161.280µs; +187.934µs] or [+6.998%; +8.154%]
🟥 throughput [-119973.273op/s; -102531.223op/s] or [-7.671%; -6.556%]

Benchmark execution time: 2026-07-22 12:14:27

Comparing candidate commit 0cc0d67 in PR branch jwiriath/drain-before-stop with baseline commit 3081603 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`0cc0d67`	1784721827	jwiriath/drain-before-stop

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
trace_buffer/1_senders/10us_delay	execution_time	56.323ms	56.592ms ± 0.158ms	56.569ms ± 0.090ms	56.659ms	56.859ms	57.111ms	57.463ms	1.58%	1.638	5.012	0.28%	0.011ms	1	200
trace_buffer/1_senders/10us_delay	throughput	15662.376op/s	15903.426op/s ± 44.145op/s	15909.728op/s ± 25.270op/s	15934.996op/s	15956.841op/s	15975.552op/s	15979.234op/s	0.44%	-1.602	4.808	0.28%	3.122op/s	1	200
trace_buffer/1_senders/1us_delay	execution_time	50.013ms	50.080ms ± 0.031ms	50.072ms ± 0.011ms	50.086ms	50.148ms	50.183ms	50.202ms	0.26%	1.637	3.551	0.06%	0.002ms	1	200
trace_buffer/1_senders/1us_delay	throughput	17927.398op/s	17971.313op/s ± 11.121op/s	17974.171op/s ± 3.894op/s	17976.879op/s	17983.492op/s	17990.184op/s	17995.165op/s	0.12%	-1.632	3.534	0.06%	0.786op/s	1	200
trace_buffer/1_senders/no_delay	execution_time	332.973µs	340.503µs ± 5.549µs	339.644µs ± 1.447µs	341.546µs	346.232µs	354.030µs	402.769µs	18.59%	7.458	78.131	1.63%	0.392µs	1	200
trace_buffer/1_senders/no_delay	throughput	2234530.176op/s	2643773.421op/s ± 38741.345op/s	2649835.756op/s ± 11336.614op/s	2660511.966op/s	2678071.606op/s	2688413.235op/s	2702920.193op/s	2.00%	-6.360	61.314	1.46%	2739.427op/s	1	200
trace_buffer/2_senders/10us_delay	execution_time	56.679ms	57.067ms ± 0.201ms	57.042ms ± 0.142ms	57.214ms	57.428ms	57.562ms	57.756ms	1.25%	0.515	0.051	0.35%	0.014ms	1	200
trace_buffer/2_senders/10us_delay	throughput	31165.420op/s	31542.047op/s ± 111.123op/s	31555.485op/s ± 78.641op/s	31628.259op/s	31707.774op/s	31750.423op/s	31757.687op/s	0.64%	-0.497	0.017	0.35%	7.858op/s	1	200
trace_buffer/2_senders/1us_delay	execution_time	50.061ms	50.114ms ± 0.033ms	50.109ms ± 0.017ms	50.126ms	50.180ms	50.217ms	50.233ms	0.25%	1.201	1.615	0.07%	0.002ms	1	200
trace_buffer/2_senders/1us_delay	throughput	35832.816op/s	35917.949op/s ± 23.508op/s	35921.709op/s ± 12.169op/s	35933.605op/s	35946.299op/s	35954.154op/s	35956.042op/s	0.10%	-1.197	1.602	0.07%	1.662op/s	1	200
trace_buffer/2_senders/no_delay	execution_time	834.556µs	886.130µs ± 21.226µs	883.970µs ± 12.397µs	899.114µs	924.168µs	939.364µs	1004.555µs	13.64%	1.302	4.946	2.39%	1.501µs	1	200
trace_buffer/2_senders/no_delay	throughput	1791838.740op/s	2032432.468op/s ± 47407.956op/s	2036269.420op/s ± 28589.314op/s	2063308.364op/s	2095917.292op/s	2127239.130op/s	2156836.443op/s	5.92%	-0.965	3.272	2.33%	3352.249op/s	1	200
trace_buffer/4_senders/10us_delay	execution_time	56.990ms	57.448ms ± 0.170ms	57.430ms ± 0.109ms	57.547ms	57.757ms	57.848ms	58.131ms	1.22%	0.496	0.977	0.29%	0.012ms	1	200
trace_buffer/4_senders/10us_delay	throughput	61929.316op/s	62666.184op/s ± 184.668op/s	62684.986op/s ± 119.410op/s	62784.702op/s	62964.926op/s	63061.582op/s	63168.783op/s	0.77%	-0.472	0.935	0.29%	13.058op/s	1	200
trace_buffer/4_senders/1us_delay	execution_time	50.094ms	50.167ms ± 0.037ms	50.160ms ± 0.020ms	50.181ms	50.245ms	50.277ms	50.286ms	0.25%	1.137	1.207	0.07%	0.003ms	1	200
trace_buffer/4_senders/1us_delay	throughput	71590.653op/s	71760.436op/s ± 52.217op/s	71769.939op/s ± 28.363op/s	71796.358op/s	71822.932op/s	71840.758op/s	71865.400op/s	0.13%	-1.133	1.197	0.07%	3.692op/s	1	200
trace_buffer/4_senders/no_delay	execution_time	2.309ms	2.479ms ± 0.052ms	2.488ms ± 0.033ms	2.515ms	2.552ms	2.573ms	2.582ms	3.78%	-0.645	0.349	2.11%	0.004ms	1	200
trace_buffer/4_senders/no_delay	throughput	1394409.117op/s	1452660.182op/s ± 31138.087op/s	1447113.599op/s ± 19225.728op/s	1469008.598op/s	1504128.498op/s	1545949.794op/s	1559446.471op/s	7.76%	0.771	0.625	2.14%	2201.795op/s	1	200
trace_buffer/8_senders/10us_delay	execution_time	57.660ms	57.972ms ± 0.248ms	57.878ms ± 0.117ms	58.130ms	58.465ms	58.569ms	58.627ms	1.29%	0.985	-0.269	0.43%	0.018ms	1	200
trace_buffer/8_senders/10us_delay	throughput	122810.041op/s	124199.761op/s ± 528.971op/s	124398.640op/s ± 251.627op/s	124571.988op/s	124745.786op/s	124861.043op/s	124869.676op/s	0.38%	-0.975	-0.289	0.42%	37.404op/s	1	200
trace_buffer/8_senders/1us_delay	execution_time	50.077ms	50.285ms ± 0.060ms	50.270ms ± 0.031ms	50.311ms	50.401ms	50.479ms	50.564ms	0.59%	1.398	3.954	0.12%	0.004ms	1	200
trace_buffer/8_senders/1us_delay	throughput	142392.829op/s	143184.696op/s ± 169.230op/s	143227.329op/s ± 87.930op/s	143292.190op/s	143364.636op/s	143385.163op/s	143779.936op/s	0.39%	-1.384	3.916	0.12%	11.966op/s	1	200
trace_buffer/8_senders/no_delay	execution_time	6.349ms	6.557ms ± 0.073ms	6.559ms ± 0.047ms	6.605ms	6.676ms	6.744ms	6.763ms	3.10%	-0.032	0.208	1.10%	0.005ms	1	200
trace_buffer/8_senders/no_delay	throughput	1064657.487op/s	1098209.467op/s ± 12162.859op/s	1097683.072op/s ± 7890.439op/s	1105856.688op/s	1119534.392op/s	1126556.575op/s	1133993.573op/s	3.31%	0.105	0.208	1.10%	860.044op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
trace_buffer/1_senders/10us_delay	execution_time	[56.570ms; 56.614ms] or [-0.039%; +0.039%]	None	None	None
trace_buffer/1_senders/10us_delay	throughput	[15897.307op/s; 15909.544op/s] or [-0.038%; +0.038%]	None	None	None
trace_buffer/1_senders/1us_delay	execution_time	[50.076ms; 50.084ms] or [-0.009%; +0.009%]	None	None	None
trace_buffer/1_senders/1us_delay	throughput	[17969.772op/s; 17972.854op/s] or [-0.009%; +0.009%]	None	None	None
trace_buffer/1_senders/no_delay	execution_time	[339.734µs; 341.272µs] or [-0.226%; +0.226%]	None	None	None
trace_buffer/1_senders/no_delay	throughput	[2638404.243op/s; 2649142.598op/s] or [-0.203%; +0.203%]	None	None	None
trace_buffer/2_senders/10us_delay	execution_time	[57.039ms; 57.095ms] or [-0.049%; +0.049%]	None	None	None
trace_buffer/2_senders/10us_delay	throughput	[31526.646op/s; 31557.447op/s] or [-0.049%; +0.049%]	None	None	None
trace_buffer/2_senders/1us_delay	execution_time	[50.110ms; 50.119ms] or [-0.009%; +0.009%]	None	None	None
trace_buffer/2_senders/1us_delay	throughput	[35914.691op/s; 35921.207op/s] or [-0.009%; +0.009%]	None	None	None
trace_buffer/2_senders/no_delay	execution_time	[883.189µs; 889.072µs] or [-0.332%; +0.332%]	None	None	None
trace_buffer/2_senders/no_delay	throughput	[2025862.181op/s; 2039002.755op/s] or [-0.323%; +0.323%]	None	None	None
trace_buffer/4_senders/10us_delay	execution_time	[57.424ms; 57.471ms] or [-0.041%; +0.041%]	None	None	None
trace_buffer/4_senders/10us_delay	throughput	[62640.591op/s; 62691.777op/s] or [-0.041%; +0.041%]	None	None	None
trace_buffer/4_senders/1us_delay	execution_time	[50.162ms; 50.172ms] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/1us_delay	throughput	[71753.200op/s; 71767.673op/s] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/no_delay	execution_time	[2.472ms; 2.487ms] or [-0.293%; +0.293%]	None	None	None
trace_buffer/4_senders/no_delay	throughput	[1448344.743op/s; 1456975.622op/s] or [-0.297%; +0.297%]	None	None	None
trace_buffer/8_senders/10us_delay	execution_time	[57.938ms; 58.007ms] or [-0.059%; +0.059%]	None	None	None
trace_buffer/8_senders/10us_delay	throughput	[124126.450op/s; 124273.071op/s] or [-0.059%; +0.059%]	None	None	None
trace_buffer/8_senders/1us_delay	execution_time	[50.277ms; 50.293ms] or [-0.016%; +0.016%]	None	None	None
trace_buffer/8_senders/1us_delay	throughput	[143161.243op/s; 143208.150op/s] or [-0.016%; +0.016%]	None	None	None
trace_buffer/8_senders/no_delay	execution_time	[6.547ms; 6.567ms] or [-0.153%; +0.153%]	None	None	None
trace_buffer/8_senders/no_delay	throughput	[1096523.812op/s; 1099895.123op/s] or [-0.153%; +0.153%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`0cc0d67`	1784721827	jwiriath/drain-before-stop

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	8.556ms	8.584ms ± 0.018ms	8.578ms ± 0.010ms	8.594ms	8.617ms	8.634ms	8.637ms	0.70%	1.035	0.644	0.20%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[8.581ms; 8.586ms] or [-0.028%; +0.028%]	None	None	None

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`0cc0d67`	1784721827	jwiriath/drain-before-stop

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.053µs	1.070µs ± 0.006µs	1.070µs ± 0.005µs	1.076µs	1.078µs	1.079µs	1.080µs	0.93%	-0.735	-0.329	0.58%	0.000µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[1.070µs; 1.071µs] or [-0.081%; +0.081%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`0cc0d67`	1784721827	jwiriath/drain-before-stop

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	14.843µs	15.262µs ± 0.257µs	15.203µs ± 0.096µs	15.328µs	15.670µs	16.018µs	17.499µs	15.10%	4.003	28.463	1.68%	0.018µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[15.226µs; 15.297µs] or [-0.233%; +0.233%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`0cc0d67`	1784721827	jwiriath/drain-before-stop

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
receiver_entry_point/report/2644	execution_time	3.753ms	3.782ms ± 0.017ms	3.779ms ± 0.009ms	3.789ms	3.817ms	3.838ms	3.858ms	2.08%	1.506	3.328	0.45%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
receiver_entry_point/report/2644	execution_time	[3.780ms; 3.784ms] or [-0.063%; +0.063%]	None	None	None

Baseline

Baseline benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3081603`	1784719733	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
trace_buffer/1_senders/10us_delay	execution_time	56.287ms	56.656ms ± 0.292ms	56.544ms ± 0.088ms	56.676ms	57.404ms	57.482ms	57.659ms	1.97%	1.690	1.964	0.51%	0.021ms	1	200
trace_buffer/1_senders/10us_delay	throughput	15608.962op/s	15885.627op/s ± 81.082op/s	15916.823op/s ± 24.847op/s	15936.490op/s	15956.388op/s	15971.269op/s	15989.476op/s	0.46%	-1.673	1.906	0.51%	5.733op/s	1	200
trace_buffer/1_senders/1us_delay	execution_time	49.913ms	50.076ms ± 0.035ms	50.075ms ± 0.014ms	50.085ms	50.139ms	50.177ms	50.197ms	0.24%	0.345	3.167	0.07%	0.002ms	1	200
trace_buffer/1_senders/1us_delay	throughput	17929.403op/s	17972.555op/s ± 12.404op/s	17972.946op/s ± 5.071op/s	17978.823op/s	17990.229op/s	17992.353op/s	18031.380op/s	0.33%	-0.334	3.185	0.07%	0.877op/s	1	200
trace_buffer/1_senders/no_delay	execution_time	334.845µs	341.829µs ± 5.075µs	341.200µs ± 1.888µs	343.105µs	346.800µs	350.747µs	399.372µs	17.05%	7.447	80.884	1.48%	0.359µs	1	200
trace_buffer/1_senders/no_delay	throughput	2253536.753op/s	2633417.551op/s ± 35490.393op/s	2637747.042op/s ± 14595.926op/s	2652006.200op/s	2666058.117op/s	2676807.501op/s	2687811.601op/s	1.90%	-6.312	63.890	1.34%	2509.550op/s	1	200
trace_buffer/2_senders/10us_delay	execution_time	56.565ms	56.991ms ± 0.214ms	56.956ms ± 0.138ms	57.117ms	57.416ms	57.571ms	57.644ms	1.21%	0.748	0.203	0.37%	0.015ms	1	200
trace_buffer/2_senders/10us_delay	throughput	31226.094op/s	31584.444op/s ± 118.070op/s	31603.415op/s ± 76.883op/s	31675.978op/s	31743.001op/s	31779.110op/s	31822.033op/s	0.69%	-0.729	0.168	0.37%	8.349op/s	1	200
trace_buffer/2_senders/1us_delay	execution_time	50.029ms	50.110ms ± 0.038ms	50.103ms ± 0.017ms	50.120ms	50.193ms	50.224ms	50.282ms	0.36%	1.563	2.980	0.08%	0.003ms	1	200
trace_buffer/2_senders/1us_delay	throughput	35798.240op/s	35921.217op/s ± 27.317op/s	35925.706op/s ± 12.334op/s	35938.173op/s	35951.266op/s	35955.287op/s	35979.151op/s	0.15%	-1.558	2.957	0.08%	1.932op/s	1	200
trace_buffer/2_senders/no_delay	execution_time	848.253µs	893.499µs ± 18.430µs	892.690µs ± 12.649µs	906.561µs	919.584µs	941.898µs	963.711µs	7.96%	0.265	0.754	2.06%	1.303µs	1	200
trace_buffer/2_senders/no_delay	throughput	1867779.863op/s	2015401.133op/s ± 41413.647op/s	2016377.418op/s ± 28979.480op/s	2043520.960op/s	2082620.627op/s	2111283.475op/s	2122008.015op/s	5.24%	-0.106	0.493	2.05%	2928.387op/s	1	200
trace_buffer/4_senders/10us_delay	execution_time	57.117ms	57.454ms ± 0.141ms	57.442ms ± 0.089ms	57.541ms	57.695ms	57.850ms	57.896ms	0.79%	0.573	0.430	0.24%	0.010ms	1	200
trace_buffer/4_senders/10us_delay	throughput	62180.723op/s	62658.934op/s ± 153.361op/s	62671.562op/s ± 97.467op/s	62762.100op/s	62883.459op/s	62936.426op/s	63028.310op/s	0.57%	-0.557	0.405	0.24%	10.844op/s	1	200
trace_buffer/4_senders/1us_delay	execution_time	50.112ms	50.171ms ± 0.036ms	50.162ms ± 0.018ms	50.186ms	50.246ms	50.277ms	50.315ms	0.30%	1.221	1.529	0.07%	0.003ms	1	200
trace_buffer/4_senders/1us_delay	throughput	71549.732op/s	71755.163op/s ± 51.730op/s	71767.193op/s ± 25.803op/s	71789.653op/s	71818.735op/s	71833.078op/s	71839.455op/s	0.10%	-1.217	1.514	0.07%	3.658op/s	1	200
trace_buffer/4_senders/no_delay	execution_time	2.072ms	2.305ms ± 0.081ms	2.302ms ± 0.052ms	2.352ms	2.428ms	2.501ms	2.570ms	11.64%	0.143	0.289	3.49%	0.006ms	1	200
trace_buffer/4_senders/no_delay	throughput	1400748.301op/s	1563912.431op/s ± 54682.536op/s	1563833.162op/s ± 35058.053op/s	1599905.311op/s	1652019.652op/s	1690696.130op/s	1737467.251op/s	11.10%	0.089	0.205	3.49%	3866.639op/s	1	200
trace_buffer/8_senders/10us_delay	execution_time	57.594ms	57.902ms ± 0.225ms	57.811ms ± 0.111ms	58.027ms	58.388ms	58.492ms	58.619ms	1.40%	1.106	0.407	0.39%	0.016ms	1	200
trace_buffer/8_senders/10us_delay	throughput	122828.019op/s	124350.828op/s ± 482.194op/s	124543.262op/s ± 240.617op/s	124700.537op/s	124854.321op/s	124938.380op/s	125013.309op/s	0.38%	-1.092	0.369	0.39%	34.096op/s	1	200
trace_buffer/8_senders/1us_delay	execution_time	50.218ms	50.289ms ± 0.052ms	50.273ms ± 0.027ms	50.317ms	50.392ms	50.461ms	50.512ms	0.47%	1.319	1.911	0.10%	0.004ms	1	200
trace_buffer/8_senders/1us_delay	throughput	142540.694op/s	143173.271op/s ± 148.890op/s	143216.883op/s ± 76.427op/s	143278.608op/s	143349.008op/s	143367.213op/s	143375.859op/s	0.11%	-1.312	1.882	0.10%	10.528op/s	1	200
trace_buffer/8_senders/no_delay	execution_time	6.159ms	6.550ms ± 0.103ms	6.552ms ± 0.069ms	6.627ms	6.695ms	6.745ms	6.844ms	4.46%	-0.573	0.835	1.57%	0.007ms	1	200
trace_buffer/8_senders/no_delay	throughput	1052057.920op/s	1099584.029op/s ± 17519.851op/s	1098951.082op/s ± 11555.934op/s	1110337.131op/s	1132127.017op/s	1145381.360op/s	1168963.120op/s	6.37%	0.694	1.098	1.59%	1238.841op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
trace_buffer/1_senders/10us_delay	execution_time	[56.616ms; 56.697ms] or [-0.071%; +0.071%]	None	None	None
trace_buffer/1_senders/10us_delay	throughput	[15874.390op/s; 15896.864op/s] or [-0.071%; +0.071%]	None	None	None
trace_buffer/1_senders/1us_delay	execution_time	[50.072ms; 50.081ms] or [-0.010%; +0.010%]	None	None	None
trace_buffer/1_senders/1us_delay	throughput	[17970.836op/s; 17974.274op/s] or [-0.010%; +0.010%]	None	None	None
trace_buffer/1_senders/no_delay	execution_time	[341.126µs; 342.532µs] or [-0.206%; +0.206%]	None	None	None
trace_buffer/1_senders/no_delay	throughput	[2628498.924op/s; 2638336.178op/s] or [-0.187%; +0.187%]	None	None	None
trace_buffer/2_senders/10us_delay	execution_time	[56.961ms; 57.020ms] or [-0.052%; +0.052%]	None	None	None
trace_buffer/2_senders/10us_delay	throughput	[31568.081op/s; 31600.808op/s] or [-0.052%; +0.052%]	None	None	None
trace_buffer/2_senders/1us_delay	execution_time	[50.104ms; 50.115ms] or [-0.011%; +0.011%]	None	None	None
trace_buffer/2_senders/1us_delay	throughput	[35917.432op/s; 35925.003op/s] or [-0.011%; +0.011%]	None	None	None
trace_buffer/2_senders/no_delay	execution_time	[890.945µs; 896.053µs] or [-0.286%; +0.286%]	None	None	None
trace_buffer/2_senders/no_delay	throughput	[2009661.600op/s; 2021140.667op/s] or [-0.285%; +0.285%]	None	None	None
trace_buffer/4_senders/10us_delay	execution_time	[57.435ms; 57.474ms] or [-0.034%; +0.034%]	None	None	None
trace_buffer/4_senders/10us_delay	throughput	[62637.679op/s; 62680.188op/s] or [-0.034%; +0.034%]	None	None	None
trace_buffer/4_senders/1us_delay	execution_time	[50.166ms; 50.176ms] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/1us_delay	throughput	[71747.993op/s; 71762.332op/s] or [-0.010%; +0.010%]	None	None	None
trace_buffer/4_senders/no_delay	execution_time	[2.294ms; 2.316ms] or [-0.485%; +0.485%]	None	None	None
trace_buffer/4_senders/no_delay	throughput	[1556333.957op/s; 1571490.904op/s] or [-0.485%; +0.485%]	None	None	None
trace_buffer/8_senders/10us_delay	execution_time	[57.870ms; 57.933ms] or [-0.054%; +0.054%]	None	None	None
trace_buffer/8_senders/10us_delay	throughput	[124284.000op/s; 124417.655op/s] or [-0.054%; +0.054%]	None	None	None
trace_buffer/8_senders/1us_delay	execution_time	[50.282ms; 50.296ms] or [-0.014%; +0.014%]	None	None	None
trace_buffer/8_senders/1us_delay	throughput	[143152.636op/s; 143193.906op/s] or [-0.014%; +0.014%]	None	None	None
trace_buffer/8_senders/no_delay	execution_time	[6.535ms; 6.564ms] or [-0.219%; +0.219%]	None	None	None
trace_buffer/8_senders/no_delay	throughput	[1097155.946op/s; 1102012.112op/s] or [-0.221%; +0.221%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3081603`	1784719733	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	8.541ms	8.572ms ± 0.017ms	8.571ms ± 0.012ms	8.583ms	8.602ms	8.616ms	8.622ms	0.59%	0.593	-0.054	0.19%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[8.570ms; 8.574ms] or [-0.027%; +0.027%]	None	None	None

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3081603`	1784719733	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.061µs	1.076µs ± 0.006µs	1.076µs ± 0.003µs	1.080µs	1.083µs	1.085µs	1.098µs	2.04%	-0.094	0.299	0.54%	0.000µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[1.075µs; 1.077µs] or [-0.075%; +0.075%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3081603`	1784719733	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	14.893µs	15.272µs ± 0.217µs	15.251µs ± 0.094µs	15.343µs	15.534µs	15.720µs	17.076µs	11.96%	4.195	29.991	1.42%	0.015µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[15.242µs; 15.302µs] or [-0.197%; +0.197%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`3081603`	1784719733	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
receiver_entry_point/report/2644	execution_time	3.750ms	3.788ms ± 0.032ms	3.779ms ± 0.011ms	3.791ms	3.867ms	3.900ms	3.908ms	3.41%	1.965	3.411	0.85%	0.002ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
receiver_entry_point/report/2644	execution_time	[3.783ms; 3.792ms] or [-0.118%; +0.118%]	None	None	None

datadog-prod-us1-5 · 2026-07-21T13:58:17Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 74.64% (-0.01%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0cc0d67 | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-07-21T14:21:46Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	88.18 MB	88.18 MB	+0% (+4.10 KB) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.13 MB	8.13 MB	0% (0 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	99.46 MB	99.46 MB	+0% (+4.12 KB) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.93 MB	10.93 MB	+0% (+256 B) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	26.59 MB	26.59 MB	+0% (+2.00 KB) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	191.30 MB	191.30 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	1.04 GB	1.04 GB	+0% (+24.96 KB) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.66 MB	8.66 MB	+.01% (+1.00 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	25.36 MB	25.36 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	50.55 MB	50.56 MB	+0% (+4.81 KB) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	23.16 MB	23.16 MB	+0% (+2.00 KB) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	195.94 MB	195.94 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	1.03 GB	1.03 GB	+0% (+23.64 KB) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.68 MB	6.68 MB	+.01% (+1.00 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	27.26 MB	27.26 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	48.13 MB	48.13 MB	+.01% (+5.22 KB) 🔍

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	78.64 MB	78.64 MB	+0% (+4.16 KB) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	9.05 MB	9.05 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	94.14 MB	94.14 MB	+0% (+4.22 KB) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.02 MB	11.02 MB	+.03% (+4.17 KB) 🔍

This reverts commit 986a45a.

Aaalibaba42 · 2026-07-22T12:04:03Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-22T12:04:07Z

View all feedbacks in Devflow UI.

2026-07-22 12:04:07 UTC ℹ️ Start processing command /merge

2026-07-22 12:04:14 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-07-22 12:49:10 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 1h (p90).

2026-07-22 13:02:04 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for b15dc27:

build_rust_on_alpine_amd

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

Aaalibaba42 · 2026-07-22T13:24:07Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-22T13:24:11Z

View all feedbacks in Devflow UI.

2026-07-22 13:24:11 UTC ℹ️ Start processing command /merge

2026-07-22 13:24:16 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).

2026-07-22 14:10:47 UTC ℹ️ MergeQueue: This merge request was merged

github-actions Bot added telemetry data-pipeline labels Jul 21, 2026

Aaalibaba42 force-pushed the jwiriath/drain-before-stop branch 2 times, most recently from 65854b2 to c0a8553 Compare July 21, 2026 15:55

github-actions Bot added the ci-build label Jul 21, 2026

Aaalibaba42 force-pushed the jwiriath/drain-before-stop branch from aed69f3 to b2aedb5 Compare July 22, 2026 09:54

fix: drain the mailbox before stopping

55dfdda

github-actions Bot removed the ci-build label Jul 22, 2026

Aaalibaba42 added 2 commits July 22, 2026 11:54

tmp: run the test 100 times

eebc4d5

Revert "tmp: run the test 100 times"

0c7a7f0

This reverts commit 986a45a.

Aaalibaba42 force-pushed the jwiriath/drain-before-stop branch from b2aedb5 to 0c7a7f0 Compare July 22, 2026 09:54

feat: drain only the currently sent messages.

c26f414

Aaalibaba42 marked this pull request as ready for review July 22, 2026 10:42

Aaalibaba42 requested review from a team as code owners July 22, 2026 10:42

Aaalibaba42 requested a review from VianneyRuhlmann July 22, 2026 10:48

VianneyRuhlmann approved these changes Jul 22, 2026

View reviewed changes

Comment thread libdd-telemetry/src/worker/mod.rs Outdated

fix: don't iterate twice

0cc0d67

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: rejected and removed mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress labels Jul 22, 2026

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: rejected mergequeue-status: queued labels Jul 22, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 0c78ffc into main Jul 22, 2026
175 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the jwiriath/drain-before-stop branch July 22, 2026 14:10

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Jul 22, 2026

Uh oh!

Conversation

Aaalibaba42 commented Jul 21, 2026

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

pr-commenter Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Explanation

More details about the CI and significant changes

scenario:trace_buffer/4_senders/no_delay

Explanation

More details about the CI and significant changes

Candidate

Group 1

Group 2

Group 1

Group 2

Group 3

Baseline

Group 1

Group 2

Group 1

Group 2

Group 3

Uh oh!

datadog-prod-us1-5 Bot commented Jul 21, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

Uh oh!

Aaalibaba42 commented Jul 22, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

Aaalibaba42 commented Jul 22, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pr-commenter Bot commented Jul 21, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented Jul 21, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

dd-octo-sts Bot commented Jul 21, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 22, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 22, 2026 •

edited

Loading