Bug: Kafka Avro deserialization fails with Avro 1.12.2 class allowlist #2687

Expected Behaviour

A Lambda function that uses @Deserialization(type = DeserializationType.KAFKA_AVRO) with Apache Avro 1.12.2 deserializes records into the Avro class declared in the handler signature. This is how it works with Avro 1.12.1.

Current Behaviour

Avro 1.12.2 adds a JVM-wide class allowlist (org.apache.avro.util.ClassSecurityValidator). By default the allowlist trusts 16 JDK types and no application classes. KafkaAvroDeserializer creates a SpecificDatumReader for the handler's record type, Avro rejects that type, and every invocation fails:

java.lang.SecurityException: Forbidden com.example.Order! This class is not trusted to be included in Avro schemas.
    at org.apache.avro.util.ClassSecurityValidator.validate(ClassSecurityValidator.java:60)
    at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:99)
    at org.apache.avro.specific.SpecificData.getClass(SpecificData.java:391)
    at org.apache.avro.specific.SpecificDatumReader.<init>(SpecificDatumReader.java:62)
    at software.amazon.lambda.powertools.kafka.serializers.KafkaAvroDeserializer.deserializeObject(KafkaAvroDeserializer.java:34)

This affects every KAFKA_AVRO user who upgrades Avro to 1.12.2. It also blocks our Dependabot bump in #2673, where 3 powertools-kafka tests fail with this error.

The Avro 1.12.2 release notes tell users to list trusted classes in the org.apache.avro.SERIALIZABLE_CLASSES or org.apache.avro.SERIALIZABLE_PACKAGES system properties. For environments with trusted schemas and data, they also offer SERIALIZABLE_PACKAGES=*, which turns the protection off for the whole JVM.

Code snippet

public class OrderHandler implements RequestHandler<ConsumerRecords<String, Order>, String> {

    @Override
    @Deserialization(type = DeserializationType.KAFKA_AVRO)
    public String handleRequest(ConsumerRecords<String, Order> records, Context context) {
        records.forEach(r -> context.getLogger().log(r.value().toString()));
        return "OK";
    }
}

Possible Solution

Powertools trusts the exact classes the developer declared, and nothing more.

powertools-kafka never reads a schema from the Kafka record. We use the schema-registry ID only to strip the wire-format header, and we decode with the schema compiled into the generic type T of ConsumerRecords<K, T>. The only classes Avro resolves are T and the records, enums and fixed types nested in T's schema. A producer that controls the record bytes cannot make Avro load any other class.

The fix:

1. On first use of a type, walk its compiled schema and collect the full names of all record, enum and fixed types.
2. Add those exact class names to an allowlist owned by Powertools.
3. Install ClassSecurityValidator.composite(previousGlobal, powertoolsAllowlist) once, so user configuration from system properties or the API keeps working.

Out of scope: package-level trust, *, and java-class or java-key-class targets. Customers who need extra classes use Avro's API. We document this in the Advanced section of the Kafka docs:

static {
    ClassSecurityValidator.setGlobal(ClassSecurityValidator.composite(
            ClassSecurityValidator.getGlobal(),
            ClassSecurityValidator.builder().add(MyExtraClass.class).build()));
}

Apache Camel took the same approach in apache/camel#26026.

Steps to Reproduce

1. Add org.apache.avro:avro:1.12.2 to a function that uses powertools-kafka.
2. Use the handler above with an Avro-generated Order class.
3. Invoke the function with an MSK or self-managed Kafka event that has an Avro value.
4. The invocation fails with java.lang.SecurityException: Forbidden ....

Powertools for AWS Lambda (Java) version

2.11.0

AWS Lambda function runtime

Java 21