Skip to content

fix(gax): stop retrying a call after the caller cancels it - #9359

Draft
shivanee-p wants to merge 1 commit into
mainfrom
shivaneep-retries-bug
Draft

shivanee-p wants to merge 1 commit into
mainfrom
shivaneep-retries-bug

Conversation

@shivanee-p

Copy link
Copy Markdown
Contributor

retryable decides whether to retry purely on the error code returned by the in-flight call. The abort that cancel() itself triggers is indistinguishable from a transport failure, and a transport that reports it with a retryable code gets retried.

By the time the error arrives, cancel() has already run: it nulled canceller and cleared the retry timer, and the timer for the next attempt does not exist yet. So the call hangs rather than reporting cancellation.

Nothing on the gRPC path reports an abort with a retryable code today, so this is currently latent. The REST fallback reaches it as soon as transport failures are mapped to UNAVAILABLE, which is what #9346 does: cancelling a fallback call then dispatches a second request and never settles.

Record the cancellation and check it before the retry decision. An error that already carries CANCELLED is passed through untouched, since the transport has already said everything worth saying; anything else is reported as CANCELLED with the original kept as cause.

`retryable` decides whether to retry purely on the error code returned by the
in-flight call. Nothing records that the caller asked to cancel, so the abort
that `cancel()` itself triggers is indistinguishable from a transport failure,
and a transport that reports it with a retryable code gets retried.

The consequence is worse than a redundant request. By the time the error
arrives, `cancel()` has already run: it nulled `canceller` and cleared the
retry timer, and the timer for the next attempt does not exist yet. So the
retry is dispatched with nothing left that can stop it. It runs unbounded and
the caller's callback is never invoked — the call hangs rather than reporting
cancellation.

Nothing on the gRPC path reports an abort with a retryable code today, so this
is currently latent. The REST fallback reaches it as soon as transport failures
are mapped to UNAVAILABLE, which is what #9346 does: cancelling a fallback call
then dispatches a second request and never settles.

Record the cancellation and check it before the retry decision. An error that
already carries CANCELLED is passed through untouched, since the transport has
already said everything worth saying; anything else is reported as CANCELLED
with the original kept as `cause`.

Fixing it here rather than in the transport keeps the guarantee where the
retry decision is made, so it holds for any transport and any future error
mapping, instead of depending on every transport resolving its own aborts
correctly.

The regression test hangs to the mocha timeout without the fix rather than
failing an assertion, which is the symptom itself.
</CodeContent>
<parameter name="Description">Commit message for the retries.ts cancellation fix.
@shivanee-p
shivanee-p marked this pull request as ready for review September 17, 2026 00:15
@shivanee-p
shivanee-p requested a review from a team as a code owner September 17, 2026 00:15
@github-actions
github-actions Bot requested a review from bshaffer September 17, 2026 00:15

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a cancelled flag in retryable calls to prevent automatic retries when a request is explicitly cancelled by the caller, even if the transport reports a retryable error code. It also adds corresponding unit tests to verify this behavior. Feedback on the changes suggests using optional chaining (err?.code) when checking the error code to prevent potential null pointer exceptions or TypeScript compilation errors under strict null checks.

if (cancelled) {
// A transport that already reported the abort as CANCELLED has said
// everything worth saying; re-wrapping would only bury it.
if (err.code === Status.CANCELLED) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since err can be null or undefined (as indicated by the use of non-null assertions like err!.code! on line 187), accessing err.code directly here will cause a TypeScript compilation error under strict null checks, or a runtime TypeError if err is indeed nullish.

Using optional chaining (err?.code) is safer and adheres to defensive programming practices.

Suggested change
if (err.code === Status.CANCELLED) {
if (err?.code === Status.CANCELLED) {

@shivanee-p
shivanee-p marked this pull request as draft September 17, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant