feat(spanner): implement x-goog-spanner-request-id header propagation and retry tracking

[thecodewreck]

Description

Implements the end-to-end x-goog-spanner-request-id header specification across all Google Cloud Spanner Node.js client operations, enabling end-to-end gRPC RPC tracing, attempt tracking, and deterministic error correlation.

Key Changes

1. Header Format Compliance:

   • Conforms to the standard structure <version>.<process>.<client>.<channel>.<request>.<attempt>.
   • <version> is fixed to 1.
   • <process> generates a 64-bit random integer formatted as a 16-character hexadecimal string (%016x), matching the Go and Java implementations. Supports custom overrides via SPANNER_PROCESS_ID and GOOGLE_CLOUD_SPANNER_PROCESS_ID environment variables.
   • <client> sequentially increments per Database / Client instance in the process.
   • <channel> defaults to 0 (Node.js gRPC channel pool).
   • <request> monotonically increments per distinct user request / RPC initiated by the client.
   • <attempt> tracks individual call attempts across retries (starting at 1).

2. Full API & Workload Coverage:

   • Standalone Operations: Query (database.run), Read (table.read), direct table mutations (table.upsert, table.insert, etc.), and partitioned operations (runPartitionedUpdate, partitionRead, partitionQuery).
   • Transactions & Lifecycle: Batch DML (executeBatchDml), Batch writes (batchWriteAtLeastOnce), transaction rollbacks (transaction.rollback), and transaction commits (transaction.commit) including retry tracking across aborted transaction replays.
   • Error Correlation: Appends (x-goog-spanner-request-id: <reqId>) to gRPC error messages upon failure.
   • Tracing: Attaches the request ID to OpenTelemetry span attributes (gcp.spanner.request_id).

3. Concurrency & Thread Safety:

   • Atomic counters ensure uniqueness of <request> identifiers during high-concurrency parallel queries without duplicate IDs or race conditions.

---

Verification & Testing

• Unit Tests: test/request_id_header.ts (12/12 passing)
• Integration Tests: test/spanner.ts (15/15 passing for XGoogRequestId suite)
• Workload Verification: 28/28 scenarios passing across sync & async workloads (WL-1 through WL-14)
• Code Style: Verified clean with Prettier across all modified files.

---

Modified Files

• handwritten/spanner/src/batch-transaction.ts
• handwritten/spanner/src/database.ts
• handwritten/spanner/src/index.ts
• handwritten/spanner/src/request_id_header.ts
• handwritten/spanner/src/session.ts
• handwritten/spanner/src/transaction.ts
• handwritten/spanner/test/request_id_header.ts
• handwritten/spanner/test/spanner.ts

[gemini-code-assist]

Code Review

This pull request implements robust request ID tracking and injection across Spanner operations, including support for 64-bit process IDs, environment variable overrides, and a gRPC interceptor to automatically increment attempt numbers on retries. Feedback on these changes includes resolving the parent database from session instances to correctly extract client and channel IDs, using isNaN to avoid coercing a valid attempt number of 0 to 1, preferring the imported Status enum from @grpc/grpc-js over grpc.status, and utilizing a deep copy of configuration objects to prevent accidental mutation of user-provided headers.

[thecodewreck]

/gemini review

[gemini-code-assist]

Code Review

This pull request enhances the Spanner client's request ID tracking by introducing a gRPC interceptor (createRequestIdInterceptor) to automatically increment attempt numbers on retries, appending request IDs to error messages, and updating request ID injection across transactions, databases, and sessions. It also includes comprehensive test coverage for these changes. The review feedback highlights several improvement opportunities: returning a shallow copy of headers in injectRequestIDIntoHeaders when session is falsy to prevent accidental mutations of shared headers, robustly resolving actualDatabase to handle cases where session is already a Database instance, fixing a potential TypeScript compilation issue in nextNthRequest, and removing the redundant _nthClientId property in favor of the existing _clientId property.

[olavloite]

RequestId is something that is only used internally, so it is not really something that should be modified by an external contributor. I think it would be better if you could open an issue with the problem that you are experiencing, so we can discuss it there first.

[thecodewreck]

/gemini review

[gemini-code-assist]

Code Review

This pull request enhances request ID tracking in the Spanner client by introducing a gRPC interceptor (createRequestIdInterceptor) to increment attempt numbers on retries, supporting 64-bit hex process IDs with environment variable overrides, and appending request IDs to error messages. It also updates various operations (like batch writes, commits, and rollbacks) to properly inject and propagate these headers. The review feedback identifies a critical bug in the newly added createRequestIdInterceptor where mutating the shared metadata object in-place during retries causes incorrect attempt numbers, and suggests caching the base request ID and initial attempt on the first call to resolve the issue.

[gemini-code-assist]

There is a bug in createRequestIdInterceptor when google-gax retries a request. Since google-gax reuses the same metadata instance across retries, mutating the metadata in-place on the first attempt means that subsequent attempts will read the already-mutated request ID (e.g., ending in .2 instead of .1). This causes parsedAttempt to increase on each retry, leading to incorrect/exponentially growing attempt numbers (e.g., .1, .2, .4, .7 instead of .1, .2, .3, .4).

To fix this, we should parse and cache the original base request ID and initial attempt number on the very first call, and then use those cached values to compute the attempt number for all subsequent retries.

function createRequestIdInterceptor(config: any) {
  let attemptCount = 0;
  let baseRequestId: string | null = null;
  let initialAttempt = 1;
  return (options: any, nextCall: any) => {
    return new grpc.InterceptingCall(nextCall(options), {
      start: function (metadata: grpc.Metadata, listener: any, next: any) {
        attemptCount++;
        if (baseRequestId === null) {
          const currentReqIds = metadata.get(X_GOOG_SPANNER_REQUEST_ID_HEADER);
          if (currentReqIds && currentReqIds.length > 0) {
            const currentReqId = String(currentReqIds[0]);
            const lastDot = currentReqId.lastIndexOf('.');
            if (lastDot !== -1) {
              baseRequestId = currentReqId.substring(0, lastDot);
              const parsedAttempt = parseInt(
                currentReqId.substring(lastDot + 1),
                10
              );
              initialAttempt = isNaN(parsedAttempt) ? 1 : parsedAttempt;
            }
          }
        }
        if (baseRequestId !== null) {
          const newAttempt = initialAttempt + (attemptCount - 1);
          const newReqId = baseRequestId + '.' + newAttempt;
          metadata.set(X_GOOG_SPANNER_REQUEST_ID_HEADER, newReqId);
          if (config && config.headers) {
            config.headers[X_GOOG_SPANNER_REQUEST_ID_HEADER] = newReqId;
          }
        }
        next(metadata, listener);
      },
    });
  };
}