Skip to content

[Kafka Streams runner] Use a single Impulse bootstrap topic with fan-out #39382

Description

@junaiddshaukat

Summary

Currently every Impulse transform gets its own bootstrap topic (__beam_impulse_<applicationId>_<transformId>), because Kafka Streams rejects registering one topic on two source nodes and a pipeline can contain several Impulses (e.g. an empty Create plus the dummy branch PAssert adds).

Per review of #39380 this can be optimized: keep a single per-application bootstrap topic with one source node, and fan the created element out to all impulse processors from there, instead of one topic per transform.

Part of #18479.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions