Repository navigation
fix(storage): retry retryable HTTP status codes (408/429/5xx) for XML multipart upload operations - #14802
fix(storage): retry retryable HTTP status codes (408/429/5xx) for XML multipart upload operations#14802chandra-siri wants to merge 1 commit into
Conversation
… multipart upload operations
There was a problem hiding this comment.
Code Review
This pull request enhances Google Cloud Storage retry handling by integrating HttpResponseException and Retryable exceptions into the retry strategy, adjusting boundary checks in RewindableContent to allow rewinding to the end of the content, and adding comprehensive tests for multipart upload client retries. The review feedback correctly identifies copy-paste typos in DefaultRetryHandlingBehaviorTest.java where HTTP 408 test cases incorrectly wrapped C.HTTP_409 instead of C.HTTP_408.
| STORAGE_EXCEPTION_0_HTTP_RESPONSE_EXCEPTION_408( | ||
| new StorageException(0, "code 0 wrapping http 408", C.HTTP_409)), |
There was a problem hiding this comment.
There is a discrepancy here where STORAGE_EXCEPTION_0_HTTP_RESPONSE_EXCEPTION_408 is wrapping C.HTTP_409 instead of C.HTTP_408. This appears to be a copy-paste typo. Please use C.HTTP_408 to ensure the test correctly validates behavior for HTTP 408 (Request Timeout).
| STORAGE_EXCEPTION_0_HTTP_RESPONSE_EXCEPTION_408( | |
| new StorageException(0, "code 0 wrapping http 408", C.HTTP_409)), | |
| STORAGE_EXCEPTION_0_HTTP_RESPONSE_EXCEPTION_408( | |
| new StorageException(0, "code 0 wrapping http 408", C.HTTP_408)), |
References
- Keep duplicated or similar text/exception messages in sync across the codebase. If a change is made to one instance, ensure it is applied to all other locations to maintain consistency.
| COALESCED_HTTP_RESPONSE_EXCEPTION_401(StorageException.coalesce(C.HTTP_401)), | ||
| COALESCED_HTTP_RESPONSE_EXCEPTION_403(StorageException.coalesce(C.HTTP_403)), | ||
| COALESCED_HTTP_RESPONSE_EXCEPTION_404(StorageException.coalesce(C.HTTP_404)), | ||
| COALESCED_HTTP_RESPONSE_EXCEPTION_408(StorageException.coalesce(C.HTTP_409)), |
There was a problem hiding this comment.
Similarly, COALESCED_HTTP_RESPONSE_EXCEPTION_408 is using C.HTTP_409 instead of C.HTTP_408. Please update this to use C.HTTP_408.
| COALESCED_HTTP_RESPONSE_EXCEPTION_408(StorageException.coalesce(C.HTTP_409)), | |
| COALESCED_HTTP_RESPONSE_EXCEPTION_408(StorageException.coalesce(C.HTTP_408)), |
References
- Keep duplicated or similar text/exception messages in sync across the codebase. If a change is made to one instance, ensure it is applied to all other locations to maintain consistency.
Summary
XML Multipart Upload operations in
MultipartUploadClient(such asuploadPart,completeMultipartUpload,abortMultipartUpload,listParts, andlistMultipartUploads) execute rawcom.google.api.client.http.HttpRequestcalls that throw a plaincom.google.api.client.http.HttpResponseExceptionon non-2xx responses.When
Retrying.DefaultRetrier.run(...)caught those exceptions,StorageException.coalesce(Throwable)fell through togetStorageException(t)->new StorageException(UNKNOWN_CODE, t.getMessage(), t)(code = 0,reason = null,retryable = false), andDefaultStorageRetryStrategy.InterceptorImpl.deepShouldRetry(BaseServiceException)routed theHttpResponseExceptioncause (sinceHttpResponseException extends IOException) intoshouldRetryIOException(IOException), classifying retryable HTTP status codes (408,429,500,502,503,504) as unretryable on Attempt #1.Changes
StorageException.coalesce(Throwable):tort.getCause()is anHttpResponseException, delegate tonew StorageException((IOException) e)soBaseHttpServiceExceptionpreserves the HTTP status code (getCode()), message (including XML<Error>bodies), cause, and retryability.DefaultStorageRetryStrategy.InterceptorImpl.deepShouldRetry(BaseServiceException):code == UNKNOWN_CODE && reason == null, evaluatecause instanceof HttpResponseException(shouldRetryCodeReason(statusCode, null)) andcause instanceof Retryable((idempotent && isRetryable()) ? RETRY : NO_RETRY) beforecause instanceof IOException, matchingbeforeEval(Exception).RewindableContent.PathRewindableContent.rewindTo(long):offset < sizeto0 <= offset && offset <= sizeso retrying a 0-bytePathpayload inuploadPartcan rewind to offset0without throwingIllegalArgumentException.StorageExceptionHttpResponseExceptionTest: Unit tests verifyingStorageException.coalescepreserves HTTP status codes, messages, causes, and retryability forHttpResponseException.DefaultRetryHandlingBehaviorTest: Permutation matrix cases forSTORAGE_EXCEPTION_0_HTTP_RESPONSE_EXCEPTION_*,COALESCED_HTTP_RESPONSE_EXCEPTION_*, andSTORAGE_EXCEPTION_0_AUTH_RETRYABLE_*.RetryingTest: End-to-endDefaultRetrier+DefaultStorageRetryStrategyretry tests with rawHttpResponseException.MultipartUploadClientImplRetryTest: HermeticMockHttpTransportretry tests across allMultipartUploadClientImploperations, status codes (408,429,500,502,503,504,400,403,404), andRequestBodypayload rewinding (ByteBuffer, multi-ByteBuffer,Path, and 0-bytePath).RewindablePathContentTest&ITMultipartUploadClientRetryTest: Unit tests forPathRewindableContent.rewindTobounds andFakeHttpServerintegration tests viaMultipartUploadClient.create(...).