ci(java): retry a failed merge-queue IT once, report it to #ci-cleanup, retry dependency downloads

[sonika-shah]

Summary

Since #34892 the Java ITs run only in the merge queue, so one flaky test or one network blip ejects a PR and restarts everything queued behind it. In the 12 hours after #35469, 6 PRs were ejected and none of them by their own change: 4 by flaky tests, 2 by Maven Central dropping the connection before any test ran.

This adds a safety net for the merge-queue IT lanes. It does not replace fixing the flaky tests.

• Retry a failed test once. After the lane finishes, failsafe re-runs only the failed test methods, once, against the same server. If it passes it's reported as a flake (<flakyFailure>, Flakes: N) and the lane stays green. If it fails twice the lane still fails. Green runs cost nothing extra. CI-only via -DintegrationTests.rerunFailingTestsCount=1; the pom default stays 0, so local runs are unchanged.
• Keep the server up for that retry. Failsafe closes the JUnit launcher session between the two passes, and TestSuiteBootstrap tore the whole stack down on session close, so the retry hit a dead server. Teardown now runs in a JVM shutdown hook.
• Give the retry fresh names. TestNamespace.prefix is deterministic per JVM/class/method, so the retry reused the failed attempt's names and hit 409 Entity already exists on whatever that attempt created. TestNamespaceExtension now adds __retry to the method id the second time it sees a test's unique id; first attempts are unchanged.
• Retry a failed dependency download. A lane re-runs mvn verify (up to 3 attempts, 30s apart) only when the attempt failed on a dependency download within 10 minutes and wrote no test report. Before each retry it deletes ~/.m2/**/*.lastUpdated: Maven caches a "not found" there and would otherwise never ask Central again, which is exactly what the Fixes #30361: Show failed rows sample when a test passes within its threshold #35042 ejection's log shows. Test failures and hangs are never retried this way, and every attempt is kept in maven.log.
• Post the flakes to #ci-cleanup. Each lane annotates the tests that passed only on retry; after a green merge-queue run the workflow posts one message to #ci-cleanup with those tests, the PRs in the group and the run link (same mechanism as the Playwright flaky post to #pw-health). The retry hides nothing: every flake lands in the channel so it gets an owner.

Decision record: ADR:2026-10-15-merge-queue-its-retry-a-failed-test-once.

Verified locally (postgres-opensearch, parallel lane)

Run	Result
Throwaway IT that fails its first attempt, then calls /v1/system/version; this PR	Attempt 1 fails, attempt 2 reaches the same server (same start timestamp) and passes → Flakes: 1, <flakyFailure> in the XML, BUILD SUCCESS
Throwaway IT that creates an untracked classification via ns.prefix, then fails its first attempt	Without the __retry suffix: retry fails 409 Entity already exists → BUILD FAILURE. With it: retry creates …__retry, passes → Flakes: 1, BUILD SUCCESS
Same IT, bootstrap reverted to teardown-on-session-close	Retry gets Connection refused → BUILD FAILURE, so the harness change is required
Flaky annotation step on a failsafe XML with one flaky, one failed-twice and one passing test	Annotates only the flaky test ✅; Slack text rendered with jq ✅
Download retry loop, simulated under bash with a stub mvn that models Maven's not-found cache (attempt 1 writes *.lastUpdated; a later attempt that finds one fails without touching the network)	Before this change: 3 attempts, all fail, only the last attempt logged. After: markers cleared, attempt 2 downloads and passes, both attempts in maven.log ✅ · download error then compile error → no second retry ✅ · attempt over 10 min with no report (hang) → not retried ✅. (The first version of this test used a stub that cached nothing, which is why it missed the not-found case.)

Details

Why the retry needs the harness change. Surefire 3.1.2's JUnitPlatformProvider.invokeAllTests runs all tests, calls closeLauncher() (which closes the LauncherSession), then opens a new session and re-executes only the failed test IDs. TestSuiteBootstrap.launcherSessionClosed() called cleanup(), which stops the app and every container. On the second session STARTED is already true, so it skipped boot and the retried test ran against nothing. The bootstrap now registers cleanup as a shutdown hook once boot succeeds, and no longer overrides launcherSessionClosed. forkedProcessExitTimeoutInSeconds is 180 so the fork can finish that teardown before Surefire halts it. External mode (JPW_MODE=external) is unchanged: it never boots, so it registers no hook.

Download retry condition. Retry only if exit != 0, there are no failsafe-reports/TEST-*.xml files, and the maven log matches Could not (resolve dependencies|transfer artifact|find artifact). This retries the whole mvn verify rather than a separate pre-fetch step, because some of the artifacts that failed (e.g. junit-platform-launcher) are resolved by Surefire at test start-up, which a pre-fetch wouldn't cover.

Ejections this would have prevented (all after #35469):

• chore: remove unused isPayloadTooLargeError helper from OpenSearchBulkSink #35391 MetricResourceIT: approval workflow ran 6s after the 5-minute wait
• Fixes #35317: check hybrid search with a query, not the stored pipeline #35321 TableCertificationPropagationIT: async cascade update dropped on a version conflict
• feat(domain): apply the global navbar domain filter server-side from a persisted defaultDomain #33908 OntologyStructureIT: MySQL deadlock in edit-lock acquire
• feat(governance): hold approval gated changes off the entity until approved (#4673) #32038 TaskResourceIT: count vs list over shared fixtures
• feat(websocket): cross-pod delivery via a generic relay behind sendToOne #34620 and Fixes #30361: Show failed rows sample when a test passes within its threshold #35042: Could not transfer artifact … Connection reset / Could not find artifact … in central

The flaky tests are being fixed separately.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[gitar-bot]

Code Review ✅ Approved

🟡 Medium risk · The merge-queue retry changes could let intermittent integration-test failures pass, allowing a real regression to merge despite flake reporting.

Adds a CI safety net for merge-queue IT lanes: failed tests get one re-run against the same server and flakes are reported to #ci-cleanup, while dependency-download failures get up to three retries after clearing Maven's cached not-found markers. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

_{For more verification details, comment gitar display:verbose.}

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}