Add SQS shared-stream producer and trigger#68782
Draft
FrankYang0529 wants to merge 1 commit into
Draft
Conversation
Many event-driven triggers and asset watchers can watch the same SQS queue. Without sharing, each opens its own consumer, multiplying ReceiveMessage calls and in-flight load against a single queue. The producer-side ack channel (apache#67523) lets sibling triggers share one consumer in the triggerer, with message deletion gated on trigger-event persistence -- so a triggerer crash leaves an unconfirmed message to be redelivered rather than dropped (at-least-once). This adds the SQS backend for that ack channel. Signed-off-by: PoAn Yang <payang@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
SqsSharedStreamProducerandSqsSharedStreamTriggerto the Amazon provider, backing the producer-side ack channel for shared-stream triggers (#67523). Many event-driven triggers watching the same SQS queue can now share a single consumer.How it works
SqsSharedStreamProducerowns one SQS consumer for a shared-stream group:open_streamreceives messages without deleting them and yields(body, SqsBrokerPayload); deletion happens only inadvance.advance:ackedandrejectedmessages are deleted (rejected is dropped);failed(ack timeout / overflow) resets visibility to0for immediate redelivery; a broadcast no subscriber was online for is left to lapse and be redelivered when a subscriber returns.MessageGroupIdwhen present, falling back to the receipt handle, so FIFO groups advance in order while standard queues get maximum per-message parallelism.related: #67523
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.