[SPARK-57498][CORE] Fix ResourceProfile ID collision when custom profile is created before default - #57566
Open
ybapat wants to merge 1 commit into
Open
[SPARK-57498][CORE] Fix ResourceProfile ID collision when custom profile is created before default#57566ybapat wants to merge 1 commit into
ybapat wants to merge 1 commit into
Conversation
…ile is created before default Start nextProfileId at 1 instead of 0 to avoid colliding with DEFAULT_RESOURCE_PROFILE_ID (0). Previously a custom ResourceProfile created before getOrCreateDefaultProfile() was called would receive ID 0, indistinguishable from the default profile. Also reset nextProfileId in clearDefaultProfile() for clean test isolation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
ResourceProfile.nextProfileIdwas initialized tonew AtomicInteger(0), so the first call togetNextProfileId()returned0— which is identical toDEFAULT_RESOURCE_PROFILE_ID. Any customResourceProfilecreated before the default profile was initialized would receive ID0, colliding with the default profile.This PR fixes the counter to start at
1, so ID0remains exclusively reserved for the default profile.Changes:
nextProfileIdat1instead of0inResourceProfilecompanion object.resetNextProfileIdForTesting()(private[spark]) to enable test isolation without exposing the reset in production code paths.afterEachreset call inResourceProfileSuiteto prevent test-order-dependent failures.Why are the changes needed?
Without this fix,
new ResourceProfileBuilder().build()called beforeResourceProfile.getOrCreateDefaultProfile()produces a profile withid == 0 == DEFAULT_RESOURCE_PROFILE_ID, which can lead to incorrect resource allocation or scheduling decisions.Does this PR introduce any user-facing change?
No. The fix only affects internal ID assignment. The default profile always receives ID
0and custom profiles now always start at ID1.How was this patch tested?
ResourceProfileSuitepasses."custom ResourceProfile id must not collide with DEFAULT_RESOURCE_PROFILE_ID"covers the regression case.Closes #57498
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com