Skip to content

fix(spanner): trigger best-effort rollback on indeterminate commit failures - #14831

Open
sakthivelmanii wants to merge 2 commits into
mainfrom
fix_bug_577654501
Open

sakthivelmanii wants to merge 2 commits into
mainfrom
fix_bug_577654501

Conversation

@sakthivelmanii

Copy link
Copy Markdown
Contributor

Problem

  • When a read-write transaction executes statements that acquire locks and the user callback returns normally, TransactionRunnerImpl, TransactionManagerImpl, and AsyncTransactionManagerImpl do not roll back if Commit fails with an indeterminate error (DEADLINE_EXCEEDED, CANCELLED, UNAVAILABLE, or RESOURCE_EXHAUSTED with RetryInfo).
  • If the Commit RPC does not complete on Cloud Spanner, the server-side transaction remains ACTIVE (or PRECOMMITTED) holding row locks for ~15 seconds until server-side idle cleanup aborts it.

Solution

  • When a read-write transaction callback returns normally and Commit fails with an indeterminate error (DEADLINE_EXCEEDED, CANCELLED, UNAVAILABLE, or RESOURCE_EXHAUSTED with RetryInfo), TransactionRunnerImpl, TransactionManagerImpl, and AsyncTransactionManagerImpl now trigger a best-effort fire-and-forget rollbackAsync().
  • If the Commit RPC did not complete on Cloud Spanner, this background rollback promptly releases server-side row locks rather than leaving the transaction in ACTIVE or PRECOMMITTED state until server-side idle timeout (~15s).
  • Deterministic commit failures (such as ALREADY_EXISTS or INVALID_ARGUMENT) and RESOURCE_EXHAUSTED without RetryInfo do not trigger a rollback, as the transaction has already failed definitively on Cloud Spanner.
  • The original commit exception is always rethrown with any synchronous rollback failure attached via addSuppressed(...).

@sakthivelmanii sakthivelmanii self-assigned this Oct 29, 2026
@sakthivelmanii
sakthivelmanii requested review from a team as code owners October 29, 2026 06:00

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces automatic rollback behavior on commit failures for specific error codes (such as DEADLINE_EXCEEDED, CANCELLED, UNAVAILABLE, and RESOURCE_EXHAUSTED with a retry delay) across AsyncTransactionManagerImpl, TransactionManagerImpl, and TransactionRunnerImpl. It also adds comprehensive unit tests to verify these rollback scenarios. The review feedback suggests two improvements: adding null checks for txn and commitError in rollbackBestEffort to prevent potential NullPointerExceptions, and increasing the assertion timeout in OrphanedTransactionTest from 200ms to 2000ms to prevent flaky test failures in busy CI/CD environments.

Comment on lines +1449 to +1455
static void rollbackBestEffort(TransactionContextImpl txn, Throwable commitError) {
try {
ApiFuture<Empty> unused = txn.rollbackAsync();
} catch (RuntimeException rollbackError) {
commitError.addSuppressed(rollbackError);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent potential NullPointerExceptions, it is safer to add a null check for txn before calling txn.rollbackAsync(). Additionally, checking if commitError is non-null before calling addSuppressed ensures robust exception handling.

  static void rollbackBestEffort(TransactionContextImpl txn, Throwable commitError) {
    if (txn == null) {
      return;
    }
    try {
      ApiFuture<Empty> unused = txn.rollbackAsync();
    } catch (RuntimeException rollbackError) {
      if (commitError != null) {
        commitError.addSuppressed(rollbackError);
      }
    }
  }

}

private void assertRollbackRequestReceived() throws InterruptedException {
long deadline = System.currentTimeMillis() + 200L;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a short timeout of 200ms for checking if a rollback request was received can lead to flaky test failures in busy CI/CD environments where thread scheduling and gRPC mock server processing might be delayed. Increasing this timeout to 2000ms provides a safer margin while still keeping the test fast due to the short sleep interval.

Suggested change
long deadline = System.currentTimeMillis() + 200L;
long deadline = System.currentTimeMillis() + 2000L;

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant