Skip to content

[BUG] Do not queue a curl session closed by the Retry-After cap - #4632

Merged
marcalff merged 5 commits into
open-telemetry:mainfrom
ThomsonTan:fix/curl-retry-after-cap-requeue
Sep 24, 2026
Merged

marcalff merged 5 commits into
open-telemetry:mainfrom
ThomsonTan:fix/curl-retry-after-cap-requeue

Conversation

@ThomsonTan

@ThomsonTan ThomsonTan commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #4631

Changes

When a retryable response carries a Retry-After beyond max_backoff,
PerformCurlMessage() closes the session with Cleanup(). The IO loop then
asked IsRetryable() again, still got true, and queued the closed session
at the server's retry time. doRetrySessions() stops at the first entry
that is not due, so that entry held back every later retry on the client
and kept the background thread alive. WaitBackgroundThreadExit() and
~HttpClient() could not join until the server's time, and the loop
busy-spun meanwhile.

For significant contributions please make sure you have completed the following items:

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@ThomsonTan
ThomsonTan requested a review from a team as a code owner September 23, 2026 15:50
@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.64%. Comparing base (4679325) to head (e053836).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4632      +/-   ##
==========================================
+ Coverage   86.53%   86.64%   +0.12%     
==========================================
  Files         525      525              
  Lines       20482    20482              
==========================================
+ Hits        17722    17745      +23     
+ Misses       2760     2737      -23     
Files with missing lines Coverage Δ
...lemetry/ext/http/client/curl/http_operation_curl.h 90.91% <ø> (ø)
ext/src/http/client/curl/http_client_curl.cc 93.19% <100.00%> (-0.01%) ⬇️
ext/src/http/client/curl/http_operation_curl.cc 63.20% <100.00%> (+1.99%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


started_at = std::chrono::steady_clock::now();
http_client.WaitBackgroundThreadExit();
const auto joined_in = std::chrono::steady_clock::now() - started_at;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: could we add a separate case that calls WaitBackgroundThreadExit() immediately after the capped request finishes, without sending the second request?

Here, the original bug spends the 30 seconds waiting for the second request to retry, so the later join can still return promptly. The retry assertion catches the bug, but a separate case would directly cover the shutdown scenario from #4631.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added RetryAfterBeyondMaxBackoffDoesNotDelayShutdown which calls WaitBackgroundThreadExit() right after the capped request, fails after about 30 s against the pre-fix IO loop, and passes with this fix.

@lalitb lalitb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the quick fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Retry-After beyond max_backoff stalls the curl retry queue and HttpClient shutdown

4 participants