Skip to content

Add SQS shared-stream producer and trigger#68782

Draft
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:add-sqs-SharedStreamProducer
Draft

Add SQS shared-stream producer and trigger#68782
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:add-sqs-SharedStreamProducer

Conversation

@FrankYang0529

Copy link
Copy Markdown
Member

Adds SqsSharedStreamProducer and SqsSharedStreamTrigger to the Amazon provider, backing the producer-side ack channel for shared-stream triggers (#67523). Many event-driven triggers watching the same SQS queue can now share a single consumer.

How it works

  • SqsSharedStreamProducer owns one SQS consumer for a shared-stream group: open_stream receives messages without deleting them and yields (body, SqsBrokerPayload); deletion happens only in advance.
  • A background task extends each outstanding message's visibility timeout while it is being processed, so a slow subscriber does not cause SQS to redeliver a message still in flight.
  • On advance: acked and rejected messages are deleted (rejected is dropped); failed (ack timeout / overflow) resets visibility to 0 for immediate redelivery; a broadcast no subscriber was online for is left to lapse and be redelivered when a subscriber returns.
  • The advance lane is the FIFO MessageGroupId when present, falling back to the receipt handle, so FIFO groups advance in order while standard queues get maximum per-message parallelism.

related: #67523


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Many event-driven triggers and asset watchers can watch the same SQS
queue. Without sharing, each opens its own consumer, multiplying
ReceiveMessage calls and in-flight load against a single queue. The
producer-side ack channel (apache#67523) lets sibling triggers share one
consumer in the triggerer, with message deletion gated on trigger-event
persistence -- so a triggerer crash leaves an unconfirmed message to be
redelivered rather than dropped (at-least-once). This adds the SQS
backend for that ack channel.

Signed-off-by: PoAn Yang <payang@apache.org>
@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Jun 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant