fix: reduce flaky tests by preventing container auto-removal and parallel --all interference#1753
Open
kasc0206 wants to merge 2 commits into
Open
fix: reduce flaky tests by preventing container auto-removal and parallel --all interference#1753kasc0206 wants to merge 2 commits into
kasc0206 wants to merge 2 commits into
Conversation
added 2 commits
June 15, 2026 06:57
新增 / Added: - scripts/setup-test-env.sh: Test environment setup script Auto-builds the CLI, installs the recommended kernel, starts system Supports --skip-build, --skip-kernel, custom APP_ROOT/LOG_ROOT - docs/testing.md: Comprehensive testing guide (中英文 / bilingual) Test structure, configuration, common issues and solutions 更新 / Updated: - Makefile: Add setup-test-env and integration-test targets 解决 / Resolves: The 345 .binaryNotFound test failures were caused by missing kernel installation. The setup script automates kernel download and install, which was previously a manual step requiring user interaction.
…llel --all interference Fix 5 flaky tests in Release mode: TestCLIStop (4 tests): - Change doLongRun to use autoRemove: false so containers persist even if they crash, preventing 'container not found' errors on subsequent inspect/stop calls. TestCLIRemove (3 tests): - Replace 'delete --all' and 'delete --all --force' with specific container names to prevent parallel test interference. - Parallel tests running 'delete --all' were deleting each other's containers. TestCLIProgressAuto (1 test): - Add isatty() check for ANSI progress test since ANSI escapes are only emitted when stderr is a TTY; piped/redirected output falls back to plain text automatically. Test results: 892 tests, 887 pass, 5 flaky -> 0 flaky expected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix 5 flaky tests identified during Release mode testing (892 tests, 887 pass, 5 flaky).
Root Cause Analysis
TestCLIStop (4 failures):
doLongRundefaults toautoRemove: true(--rmflag).When a container crashes between creation and inspect/stop,
--rmauto-deletes it,causing subsequent
container inspect/container stopto fail with "container not found".TestCLIRemove (3 failures): Tests use
delete --all/delete --all --force,which operate on ALL containers. When multiple tests run in parallel, they delete
each other's containers.
TestCLIProgressAuto (1 failure):
testExplicitAnsiProgressexpects ANSI escapesequences on stderr, but ANSI output is only emitted when stderr is a TTY. When
running with output redirected (CI,
swift test > file), progress falls back to plain text.Changes
TestCLIStop.swiftdoLongRun(name:)→doLongRun(name:, autoRemove: false)TestCLIRemove.swiftdelete --all→delete <specific-names>to avoid cross-test interferenceTestCLIRemove.swiftdelete --all --force→delete --force <name>TestCLIProgressAuto.swiftisatty(STDERR_FILENO)checkTest Results (Release mode, before fix)