Skip to content

test: harden the flakiest unit test and the contract test harness download - #386

Draft
kinyoklion wants to merge 1 commit into
mainfrom
devin/1786577527-harden-flaky-ci
Draft

test: harden the flakiest unit test and the contract test harness download#386
kinyoklion wants to merge 1 commit into
mainfrom
devin/1786577527-harden-flaky-ci

Conversation

@kinyoklion

Copy link
Copy Markdown
Member

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions

Related issues

Comes out of a review of every Build and Test run (PR + main) over the last three months, including all attempts of re-run builds. The FDv2 OutOfMemoryError flakes found by that review are being fixed separately in #384 and are deliberately untouched here.

Describe the solution you've provided

Two independent flake sources, both outside the SDK's production code.

StateDebounceManagerTest.closeCancelsPendingTimer scheduled a debounce timer, slept for a fraction of the window, and expected close() to win the race. On a loaded runner the sleep overshoots the window, the timer fires first, and the test fails. It now drives the timer through the existing ManualTaskExecutor, so cancellation is observed directly rather than inferred from wall-clock ordering, and additionally asserts that the scheduled task really was cancelled. resetCancelsPendingTimer had the identical shape and gets the same treatment. Two sibling tests were already converted this way in earlier fixes; these were missed.

The contract test step fetched the harness downloader with curl -s ... | sh. Beyond having no retry, that pipeline is silently fatal in the wrong direction: a failed fetch feeds sh an empty script, which exits 0, so the contract tests would be skipped and the build would go green. The downloader is now fetched to a file with --fail and retries, and run from there, so a fetch failure fails the build. Both suites share that one copy, taken from the branch whose downloader retries the release download — the script is version-agnostic, VERSION still selects which harness binary each suite fetches, which is what fixes the observed Download failed failures on the v3 run.

Describe alternatives you've considered

Retrying the whole contract-test step would also mask genuine failures, and re-running the suite after the emulator has booted is expensive. Pinning VERSION=v2 to an exact release (as the v3 line already does) is a separate recommendation: the floating tag makes contract results non-reproducible and has already produced a red-then-green build with no code change, but changing it is a process decision rather than a flake fix, so it is not part of this PR.

Additional context

Verified with :launchdarkly-android-client-sdk:test (full unit test suite) plus repeated --rerun-tasks runs of the changed class, and :launchdarkly-android-client-sdk:lint. The Makefile change was exercised both ways: a successful download and run of the harness, and a deliberately bad downloader URL, which now aborts the target instead of quietly skipping the tests.

Link to Devin session: https://app.devin.ai/sessions/fb9541c6fa994866a832460742c89682
Requested by: @kinyoklion

…on harness download errors

Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
@kinyoklion kinyoklion self-assigned this Aug 12, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot added the devin-pr PRs created by Devin label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devin-pr PRs created by Devin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant