Repository navigation
Optimize S3 server-side compose capability check using encoded part sizes - #149
Merged
Merged
Conversation
…izes - Add S3Utils.getPartSize to resolve part sizes in O(1) time from encoded part key name suffixes with statObject fallback - Add canCompose and canComposePartialUploads to S3ServerSideComposeHelper to eliminate N+1 statObject HTTP calls - Manage compose capability flag in S3ServerSideComposeHelper with automatic fallback disablement in S3ConcatenationService - Streamline prepareStreamWithExistingIncompletePart in S3StorageService to inspect parts in a single pass
|
Coverage is 95.049% — feature/s3-compose-part-size-optimization into master. No base build found for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR optimizes the S3 server-side compose capability check and tail rollback inspection by resolving part sizes directly from the part key name suffix, eliminating N+1 remote
statObjectHTTP roundtrips.Changes
S3Utils.getPartSize: Extracts the_<size>suffix from part keys in O(1) time without network calls, with automatic fallback tostatObjectfor legacy or un-suffixed keys.S3ServerSideComposeHelper:composeSupportedtoggle guarded byReentrantReadWriteLock(isComposeSupported(),setComposeSupported(boolean)).canCompose(bucket, partKeys, minPartSize)evaluating S3 compose requirements (all non-final parts >= 5 MiB) using O(1) size lookups.canComposePartialUploads(partialUploads, minPartSize)centralizing multi-part size validation logic.MinioClientin the explicit endpoint constructor viaS3Utils.buildMinioClient.S3StorageService:statObjectcalls infinalizeCompletedUploadIfFinishedwith delegated call tos3ComposeHelper.canCompose(...).prepareStreamWithExistingIncompletePartto calculate part sizes in a single pass.s3ComposeHelper.S3ConcatenationService:s3ComposeHelper.canComposePartialUploads(...)inmerge.s3ComposeHelper.setComposeSupported(false)on compose failure before falling back to streaming re-upload so subsequent requests immediately bypass failing compose calls.S3UtilsTest,S3ServerSideComposeHelperTest,S3StorageServiceTest, andS3ConcatenationServiceTest.fmt-maven-plugin).