Skip to content

HDDS-16398. Fix intermittent AlreadyClosedException in TestCommitWatcher #11227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yandrey321 wants to merge 4 commits into
base: master
Choose a base branch
from
Open

HDDS-16398. Fix intermittent AlreadyClosedException in TestCommitWatcher #11227

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
766fc18
fixed transient CI failure
yandrey321 Sep 9, 2026
a557792
addressed review comments
yandrey321 Sep 14, 2026
3465e0c
addressed review comments
yandrey321 Sep 16, 2026
010c5ea
removed unneeded changes
yandrey321 Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions .../integration-test/src/test/java/org/apache/hadoop/hdds/scm/storage/TestCommitWatcher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,13 +157,22 @@ public void testReleaseBuffers() throws Exception {
try (XceiverClientSpi xceiverClient = mgr.acquireClient(pipeline)) {
assertEquals(1, xceiverClient.getRefcount());
XceiverClientRatis ratisClient = assertInstanceOf(XceiverClientRatis.class, xceiverClient);
// Commit one request synchronously before the async write burst so the
// Ratis client is fully established and the container already exists.
// Issuing the burst directly on a freshly-acquired client was seen to
// fail intermittently with AlreadyClosedException (HDDS-16398).
ratisClient.sendCommandAsync(
ContainerTestHelper.getCreateContainerRequest(containerId, pipeline))
.getResponse().get();
CommitWatcher watcher = new CommitWatcher(bufferPool, ratisClient);
BlockID blockID = ContainerTestHelper.getTestBlockID(containerId);
List<XceiverClientReply> replies = new ArrayList<>();
long length = 0;
List<CompletableFuture<ContainerCommandResponseProto>>
futures = new ArrayList<>();
for (int i = 0; i < capacity; i++) {
// Use a distinct block per iteration; a real client never rewrites the
// same blockID and offset with different data.
BlockID blockID = ContainerTestHelper.getTestBlockID(containerId);
ContainerCommandRequestProto writeChunkRequest =
ContainerTestHelper
.getWriteChunkRequest(pipeline, blockID, CHUNK_SIZE);
Expand Down Expand Up @@ -223,13 +232,22 @@ public void testReleaseBuffersOnException() throws Exception {
try (XceiverClientSpi xceiverClient = mgr.acquireClient(pipeline)) {
assertEquals(1, xceiverClient.getRefcount());
XceiverClientRatis ratisClient = assertInstanceOf(XceiverClientRatis.class, xceiverClient);
// Commit one request synchronously before the async write burst so the
// Ratis client is fully established and the container already exists.
// Issuing the burst directly on a freshly-acquired client was seen to
// fail intermittently with AlreadyClosedException (HDDS-16398).
ratisClient.sendCommandAsync(
ContainerTestHelper.getCreateContainerRequest(containerId, pipeline))
.getResponse().get();

@chihsuan chihsuan Sep 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if leader election is really the cause here. In the DN logs from my run, the second WriteChunk failed with CHUNK_FILE_INCONSISTENCY after the first PutBlock closed the block file. The container was then marked UNHEALTHY, followed by pipeline closure. Would using a different blockID per iteration avoid this?

CommitWatcher watcher = new CommitWatcher(bufferPool, ratisClient);
BlockID blockID = ContainerTestHelper.getTestBlockID(containerId);
List<XceiverClientReply> replies = new ArrayList<>();
long length = 0;
List<CompletableFuture<ContainerCommandResponseProto>>
futures = new ArrayList<>();
for (int i = 0; i < capacity; i++) {
// Use a distinct block per iteration; a real client never rewrites the
// same blockID and offset with different data.
BlockID blockID = ContainerTestHelper.getTestBlockID(containerId);
ContainerCommandRequestProto writeChunkRequest =
ContainerTestHelper
.getWriteChunkRequest(pipeline, blockID, CHUNK_SIZE);
Expand Down