HDDS-16398. Fix intermittent AlreadyClosedException in TestCommitWatcher

[yandrey321]

What changes were proposed in this pull request?

TestCommitWatcher#testReleaseBuffersOnException (and testReleaseBuffers, which shares the same setup) intermittently fail because the test starts writing to a freshly-allocated RATIS THREE pipeline before that pipeline's Ratis group has elected a leader.

@beforeeach calls cluster.waitForClusterToBeReady(), which waits only for datanode registration and SCM readiness — not for the pipeline's leader election. When the first putBlock/watch lands before a leader is settled, Ratis returns NotLeaderException; under the test's deliberately short (3s) request/watch timeouts the retries are exhausted and the client is closed, surfacing as:

NotLeaderException
-> RaftRetryFailureException
-> AlreadyClosedException: SlidingWindow$Client:client-...->RAFT is closed.
The failure occurs at future1.get() during setup, before the test reaches the datanode-shutdown scenario it actually exercises.

The fix adds one call in @beforeeach, after the cluster is ready, to wait for the RATIS THREE pipeline to reach OPEN before any writes:

cluster.waitForClusterToBeReady();
// Wait for the RATIS THREE pipeline to reach OPEN state before any writes.
// A pipeline only opens once it is healthy, which requires an elected Ratis
// leader; otherwise the first write can race leader election and fail with
// NotLeaderException -> RaftRetryFailureException -> AlreadyClosedException.
cluster.waitForPipelineTobeReady(HddsProtos.ReplicationFactor.THREE, 60000);
This closes the race deterministically: SCM transitions a RATIS pipeline to OPEN only when Pipeline.isHealthy() is true, and for a RATIS pipeline isHealthy() requires all datanodes to have reported and leaderId != null. So once the pipeline is OPEN, a leader has been elected and reported — exactly the precondition the writes depend on. The change reuses the existing MiniOzoneCluster.waitForPipelineTobeReady(...) helper (no new abstraction) and, being in @beforeeach, protects both tests in the class.

This is a test-only change; no production code is affected.

Generated-by: Claude Code (Claude Opus 4.8)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16398

How was this patch tested?

CI: https://github.com/yandrey321/ozone/actions/runs/34411454199/job/102906678997

[chihsuan]

Thanks for looking into this flaky test! @yandrey321 Could you share the CI log of the failure you investigated? The Jira stack trace is truncated.

Would it also be worth running the intermittent-test-check workflow on your branch to confirm it's stable?

Small nit: the code snippets in the description could use ``` fences so they render properly. Thanks!

[yandrey321]

Thanks for looking into this flaky test! @yandrey321 Could you share the CI log of the failure you investigated? The Jira stack trace is truncated.

Would it also be worth running the intermittent-test-check workflow on your branch to confirm it's stable?

Small nit: the code snippets in the description could use ``` fences so they render properly. Thanks!

here is the link to the failed CI: https://github.com/yandrey321/ozone/actions/runs/34395742534/job/102619847831

I saw multiple CI runs with the same signature.

[chihsuan]

here is the link to the failed CI: https://github.com/yandrey321/ozone/actions/runs/34395742534/job/102619847831

Thanks for the link! @yandrey321 I ran the intermittent-test-check on your branch and it still failed 3 times: https://github.com/chihsuan/ozone/actions/runs/34612123189

The failures have different signatures, so I suspect the root cause is different and there may be multiple issues. Could you take a look?

[yandrey321]

The failures have different signatures, so I suspect the root cause is different and there may be multiple issues. Could you take a look?
it should be fixed now.

[chihsuan]

Thanks! @yandrey321 I checked the datanode logs from my earlier intermittent run, and I'm not sure the current fix addresses the actual failure. Please see inline comments. Would it be worth running intermittent-test-check on the new head as well?

[chihsuan]

I wonder if leader election is really the cause here. In the DN logs from my run, the second WriteChunk failed with CHUNK_FILE_INCONSISTENCY after the first PutBlock closed the block file. The container was then marked UNHEALTHY, followed by pipeline closure. Would using a different blockID per iteration avoid this?

[chihsuan]

Do we still need this wait? I noticed allocateContainer only picks pipelines already in OPEN state, and OPEN already implies a reported leader, so this may not add any guarantee.

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerImpl.java

Lines 182 to 186 in f9f6184

	pipelines = pipelineManager
	.getPipelines(replicationConfig, Pipeline.PipelineState.OPEN);
	if (!pipelines.isEmpty()) {
	pipeline = pipelines.get(random.nextInt(pipelines.size()));
	containerInfo = createContainer(pipeline, owner);

[chihsuan]

nit: Could we drop createPipelineOnDatanode here? The groups should already exist for an OPEN pipeline, so these calls normally just hit duplicate-group errors that the helper swallows. Same for line 246.

[chihsuan]

nit: Could we trim this comment and keep the explanation in one place? The same explanation appears here, at line 241, and in init(), and the exception chain is already in the PR description.