fix: Un-skipping tests in sagemaker-serve#5807
Open
mujtaba1747 wants to merge 5 commits intoaws:masterfrom
Open
fix: Un-skipping tests in sagemaker-serve#5807mujtaba1747 wants to merge 5 commits intoaws:masterfrom
mujtaba1747 wants to merge 5 commits intoaws:masterfrom
Conversation
2ce2819 to
fb43456
Compare
mollyheamazon
previously approved these changes
Apr 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: #5802
Description of changes: Fix skipped tests in sagemaker-serve
test_model_customization_deployment.py
Problem:
cleanup_e2e_endpoints is a session-scoped fixture, but under pytest-xdist each worker gets its own session. This means multiple workers independently call Endpoint.get_all() and delete every e2e-* endpoint — including endpoints that another worker just created and is actively using.
Observed failure: Test running on worker gw2 deploys e2e-1777420521-4224, its base inference component goes InService, then a different worker's cleanup deletes it. The test on gw2 then fails when it tries to Endpoint.get() the endpoint it just created.
The teardown path had the same problem — it ran the same Endpoint.get_all() + delete loop after tests, risking a second round of collisions.
Changes:
Use filelock to coordinate cleanup across xdist workers. The first worker to acquire the lock performs the e2e-* endpoint cleanup and writes a sentinel file; remaining workers see the sentinel and skip.
Remove the post-yield (after-test) cleanup entirely — it was the main source of the race condition, deleting endpoints that other workers were still using.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.