Repository navigation
fix(spanner): trigger best-effort rollback on indeterminate commit failures - #14831
sakthivelmanii wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces automatic rollback behavior on commit failures for specific error codes (such as DEADLINE_EXCEEDED, CANCELLED, UNAVAILABLE, and RESOURCE_EXHAUSTED with a retry delay) across AsyncTransactionManagerImpl, TransactionManagerImpl, and TransactionRunnerImpl. It also adds comprehensive unit tests to verify these rollback scenarios. The review feedback suggests two improvements: adding null checks for txn and commitError in rollbackBestEffort to prevent potential NullPointerExceptions, and increasing the assertion timeout in OrphanedTransactionTest from 200ms to 2000ms to prevent flaky test failures in busy CI/CD environments.
| static void rollbackBestEffort(TransactionContextImpl txn, Throwable commitError) { | ||
| try { | ||
| ApiFuture<Empty> unused = txn.rollbackAsync(); | ||
| } catch (RuntimeException rollbackError) { | ||
| commitError.addSuppressed(rollbackError); | ||
| } | ||
| } |
There was a problem hiding this comment.
To prevent potential NullPointerExceptions, it is safer to add a null check for txn before calling txn.rollbackAsync(). Additionally, checking if commitError is non-null before calling addSuppressed ensures robust exception handling.
static void rollbackBestEffort(TransactionContextImpl txn, Throwable commitError) {
if (txn == null) {
return;
}
try {
ApiFuture<Empty> unused = txn.rollbackAsync();
} catch (RuntimeException rollbackError) {
if (commitError != null) {
commitError.addSuppressed(rollbackError);
}
}
}| } | ||
|
|
||
| private void assertRollbackRequestReceived() throws InterruptedException { | ||
| long deadline = System.currentTimeMillis() + 200L; |
There was a problem hiding this comment.
Using a short timeout of 200ms for checking if a rollback request was received can lead to flaky test failures in busy CI/CD environments where thread scheduling and gRPC mock server processing might be delayed. Increasing this timeout to 2000ms provides a safer margin while still keeping the test fast due to the short sleep interval.
| long deadline = System.currentTimeMillis() + 200L; | |
| long deadline = System.currentTimeMillis() + 2000L; |
Problem
TransactionRunnerImpl,TransactionManagerImpl, andAsyncTransactionManagerImpldo not roll back ifCommitfails with an indeterminate error (DEADLINE_EXCEEDED,CANCELLED,UNAVAILABLE, orRESOURCE_EXHAUSTEDwithRetryInfo).CommitRPC does not complete on Cloud Spanner, the server-side transaction remainsACTIVE(orPRECOMMITTED) holding row locks for ~15 seconds until server-side idle cleanup aborts it.Solution
Commitfails with an indeterminate error (DEADLINE_EXCEEDED,CANCELLED,UNAVAILABLE, orRESOURCE_EXHAUSTEDwithRetryInfo),TransactionRunnerImpl,TransactionManagerImpl, andAsyncTransactionManagerImplnow trigger a best-effort fire-and-forgetrollbackAsync().CommitRPC did not complete on Cloud Spanner, this background rollback promptly releases server-side row locks rather than leaving the transaction inACTIVEorPRECOMMITTEDstate until server-side idle timeout (~15s).ALREADY_EXISTSorINVALID_ARGUMENT) andRESOURCE_EXHAUSTEDwithoutRetryInfodo not trigger a rollback, as the transaction has already failed definitively on Cloud Spanner.addSuppressed(...).