Skip to content

feat(connectors): implement Avro payload support with separate encoder, decoder and transform crates#3141

Open
Tyooughtul wants to merge 7 commits intoapache:masterfrom
Tyooughtul:feat/1846-avro-support
Open

feat(connectors): implement Avro payload support with separate encoder, decoder and transform crates#3141
Tyooughtul wants to merge 7 commits intoapache:masterfrom
Tyooughtul:feat/1846-avro-support

Conversation

@Tyooughtul
Copy link
Copy Markdown
Contributor

@Tyooughtul Tyooughtul commented Apr 20, 2026

Which issue does this PR close?

Closes #1846

Rationale

Issue #1846 requested support for Apache Avro as a first-class payload format in connectors, including dedicated StreamEncoder / StreamDecoder implementations and cross-format transformations. This PR delivers that support with schema-driven encoding/decoding and runtime integration.

What changed?

What changed?

Connectors only supported JSON, Raw, Text, Proto and FlatBuffer payloads. Avro data had to be handled as opaque raw bytes without schema validation or format conversion.
So, I added full Avro support across the connector stack, SDK-level AvroStreamEncoder/Decoder with inline/file-based schema loading, a new AvroConvert transform for cross-format conversion, runtime config fields for producers/consumers, and HTTP sink base64 encoding for Avro payloads.

Local Execution

  • Passed / not passed
    Passed
  • Pre-commit hooks ran / not ran
    Ran

AI Usage

I used codex to check the code I wrote, and it helped me discover some bugs and wrote the pr draft.

- Add apache-avro dependency to workspace and SDK crate
- Extend Payload and Schema enums with Avro variant
- Implement AvroStreamDecoder with JSON extraction and raw passthrough
- Implement AvroStreamEncoder with JSON-to-Avro conversion and field mappings
- Update FlatBuffer and Proto encoders to handle Payload::Avro
- Update ProtoConvert transform to support Avro format conversion
- Implement AvroConvert supporting Avro <-> JSON/Text/Raw conversions
- Register AvroConvert in TransformType and from_config dispatch
- Add avro_schema_json and avro_schema_path to stream configs
- Use AvroStreamDecoder in sink setup for Schema::Avro
- Use AvroStreamEncoder in source setup for Schema::Avro
- Base64-encode Payload::Avro with metadata marker for HTTP transport
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 20, 2026

Codecov Report

❌ Patch coverage is 45.84585% with 541 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.66%. Comparing base (868ae62) to head (bfe9d51).

Files with missing lines Patch % Lines
core/connectors/sdk/src/encoders/avro.rs 45.23% 215 Missing and 9 partials ⚠️
core/connectors/sdk/src/decoders/avro.rs 46.52% 145 Missing and 9 partials ⚠️
core/connectors/sdk/src/transforms/avro_convert.rs 55.64% 102 Missing and 4 partials ⚠️
core/connectors/runtime/src/sink.rs 21.42% 11 Missing ⚠️
core/connectors/runtime/src/source.rs 21.42% 11 Missing ⚠️
core/connectors/sdk/src/encoders/proto.rs 0.00% 11 Missing ⚠️
...ore/connectors/sdk/src/transforms/proto_convert.rs 0.00% 7 Missing ⚠️
core/connectors/sinks/http_sink/src/lib.rs 0.00% 7 Missing ⚠️
core/connectors/sdk/src/lib.rs 0.00% 6 Missing ⚠️
core/connectors/sdk/src/transforms/mod.rs 0.00% 3 Missing ⚠️
... and 1 more
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3141       +/-   ##
=============================================
- Coverage     73.64%   52.66%   -20.98%     
  Complexity      943      943               
=============================================
  Files          1141     1143        +2     
  Lines        100869    91816     -9053     
  Branches      78044    69009     -9035     
=============================================
- Hits          74285    48359    -25926     
- Misses        23928    40983    +17055     
+ Partials       2656     2474      -182     
Components Coverage Δ
Rust Core 46.47% <45.84%> (-28.14%) ⬇️
Java SDK 62.30% <ø> (ø)
C# SDK 69.11% <ø> (-0.29%) ⬇️
Python SDK 81.43% <ø> (ø)
Node SDK 91.53% <ø> (ø)
Go SDK 39.41% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/runtime/src/configs/connectors.rs 28.18% <ø> (ø)
core/connectors/sdk/src/encoders/flatbuffer.rs 57.61% <0.00%> (-23.38%) ⬇️
core/connectors/sdk/src/transforms/mod.rs 29.62% <0.00%> (-3.71%) ⬇️
core/connectors/sdk/src/lib.rs 55.05% <0.00%> (-5.19%) ⬇️
...ore/connectors/sdk/src/transforms/proto_convert.rs 58.37% <0.00%> (-14.65%) ⬇️
core/connectors/sinks/http_sink/src/lib.rs 57.35% <0.00%> (-28.93%) ⬇️
core/connectors/runtime/src/sink.rs 72.40% <21.42%> (-2.12%) ⬇️
core/connectors/runtime/src/source.rs 66.90% <21.42%> (-2.84%) ⬇️
core/connectors/sdk/src/encoders/proto.rs 43.42% <0.00%> (-28.21%) ⬇️
core/connectors/sdk/src/transforms/avro_convert.rs 55.64% <55.64%> (ø)
... and 2 more

... and 362 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Tyooughtul Tyooughtul changed the title Feat/1846 avro support feat/1846 avro support Apr 20, 2026
@Tyooughtul Tyooughtul changed the title feat/1846 avro support feat(connectors): implement Avro payload support with separate encoder, decoder and transform crates Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for Avro payload to connectors runtime

1 participant