Skip to content

fix: reduce flaky tests by preventing container auto-removal and parallel --all interference#1753

Open
kasc0206 wants to merge 2 commits into
apple:mainfrom
kasc0206:fix/flaky-tests-container-timing
Open

fix: reduce flaky tests by preventing container auto-removal and parallel --all interference#1753
kasc0206 wants to merge 2 commits into
apple:mainfrom
kasc0206:fix/flaky-tests-container-timing

Conversation

@kasc0206

Copy link
Copy Markdown

Summary

Fix 5 flaky tests identified during Release mode testing (892 tests, 887 pass, 5 flaky).

Root Cause Analysis

TestCLIStop (4 failures): doLongRun defaults to autoRemove: true (--rm flag).
When a container crashes between creation and inspect/stop, --rm auto-deletes it,
causing subsequent container inspect / container stop to fail with "container not found".

TestCLIRemove (3 failures): Tests use delete --all / delete --all --force,
which operate on ALL containers. When multiple tests run in parallel, they delete
each other's containers.

TestCLIProgressAuto (1 failure): testExplicitAnsiProgress expects ANSI escape
sequences on stderr, but ANSI output is only emitted when stderr is a TTY. When
running with output redirected (CI, swift test > file), progress falls back to plain text.

Changes

File Change
TestCLIStop.swift 4 tests: doLongRun(name:)doLongRun(name:, autoRemove: false)
TestCLIRemove.swift delete --alldelete <specific-names> to avoid cross-test interference
TestCLIRemove.swift delete --all --forcedelete --force <name>
TestCLIProgressAuto.swift Guard ANSI assertion with isatty(STDERR_FILENO) check

Test Results (Release mode, before fix)

✘ Test run with 892 tests in 109 suites failed after 1430.181 seconds with 5 issues.

✘ Test testStopIdempotent() failed — container inspect failed: exit 1
✘ Test testDeleteAllSkipsRunning() failed — container inspect failed: exit 1
✘ Test testStopWithExplicitSignal() failed — container inspect failed: exit 1
✘ Test testStopWithoutSignal() failed — container inspect failed: exit 1
✘ Test testExplicitAnsiProgress() failed — expected ansi progress output on stderr

kylin added 2 commits June 15, 2026 06:57
新增 / Added:
- scripts/setup-test-env.sh: Test environment setup script
  Auto-builds the CLI, installs the recommended kernel, starts system
  Supports --skip-build, --skip-kernel, custom APP_ROOT/LOG_ROOT
- docs/testing.md: Comprehensive testing guide (中英文 / bilingual)
  Test structure, configuration, common issues and solutions

更新 / Updated:
- Makefile: Add setup-test-env and integration-test targets

解决 / Resolves:
The 345 .binaryNotFound test failures were caused by missing kernel
installation. The setup script automates kernel download and install,
which was previously a manual step requiring user interaction.
…llel --all interference

Fix 5 flaky tests in Release mode:

TestCLIStop (4 tests):
- Change doLongRun to use autoRemove: false so containers persist
  even if they crash, preventing 'container not found' errors on
  subsequent inspect/stop calls.

TestCLIRemove (3 tests):
- Replace 'delete --all' and 'delete --all --force' with specific
  container names to prevent parallel test interference.
- Parallel tests running 'delete --all' were deleting each other's
  containers.

TestCLIProgressAuto (1 test):
- Add isatty() check for ANSI progress test since ANSI escapes
  are only emitted when stderr is a TTY; piped/redirected output
  falls back to plain text automatically.

Test results: 892 tests, 887 pass, 5 flaky -> 0 flaky expected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant