[fix][client] Remove the dead letter candidate entry using the entry-level message id - #26477
Open
SongOf wants to merge 1 commit into
Open
[fix][client] Remove the dead letter candidate entry using the entry-level message id#26477SongOf wants to merge 1 commit into
SongOf wants to merge 1 commit into
Conversation
Contributor
Author
|
@nodece could you review this pr |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ConsumerImpl#possibleSendToDeadLetterTopicMessagesis always keyed by the entry-level messageid:
receiveIndividualMessagesFromBatch()registers a batch underbatchMessage(a
MessageIdImpl), andmessageReceived()registers a non-batched message under its ownMessageIdImpl.processPossibleToDLQ()respects that when it looks the entry up:but not when it deletes it, once the messages have reached the DLQ:
MessageIdAdvUtils#equals/#hashCodeinclude the batch index, so as soon asmessageIdstillcarries one the removal silently matches nothing.
That is reachable on a plain multi-topic / partitioned consumer, with no tracing or custom code
involved:
MultiTopicsConsumerImpladdstopicMessage.getMessageId()— aTopicMessageIdImplthat keeps the batch index — to its unacked-message tracker, unlike
ConsumerImpl#trackMessage()which discards it first. When the ack timeout fires, those ids are routed back to
ConsumerImpl#redeliverUnacknowledgedMessages(Set)and reachprocessPossibleToDLQ()with thebatch index intact.
The consequences for a consumer using batching + a dead letter policy on a partitioned topic:
grows for the lifetime of the consumer;
and writes the whole batch to the dead letter topic a second time, so the DLQ accumulates
duplicates.
Modifications
ConsumerImpl#processPossibleToDLQ(): compute the entry-level key once(
MessageIdAdvUtils.discardBatch(messageId)) and use it for both the lookup and the removal.acknowledgeAsync(messageId)deliberately keeps using the original id — acknowledging isper-batch-index and must not have the batch index stripped.
Verifying this change
This change added tests and can be verified as follows:
DeadLetterTopicTest#testPossibleToDeadLetterEntryIsRemovedWhenRedeliveringABatchMessageId—publishes one batch, lets it reach
maxRedeliverCountso the batch is registered as a DLQcandidate, then redelivers a single message id that still carries its batch index (the shape
MultiTopicsConsumerImplpasses down), and asserts the entry is gone once the batch has beenwritten to the DLQ. The test first asserts that all messages of the batch really did arrive on
the dead letter topic, so the final assertion cannot pass vacuously. On the unpatched code it
fails for the real reason:
Local runs:
The
:pulsar-client-original:testmodule passes as well.Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
This is an internal bug fix in the consumer's dead letter bookkeeping. It changes no public API,
schema, configuration default, wire protocol, REST endpoint, CLI option or metric.