feat(core): add Substrait dialect support

[nielspardon]

What

Adds a typed model in the core Java SDK for creating and consuming Substrait dialect YAML files, faithful to substrait/text/dialect_schema.yaml (introduced in substrait v0.76.0). Until now the only producer of a dialect in this repo was the Scala DialectGenerator in the spark module, whose ad-hoc model isn't reusable.

Usage

Build a dialect, serialize it to YAML, and parse it back:

import io.substrait.dialect.Dialect;
import io.substrait.dialect.Dialect.*;

DialectDocument dialect =
    DialectDocument.builder()
        .name("Example Dialect")
        .putDependencies("arithmetic", "extension:io.substrait:functions_arithmetic")
        // Types: a bare entry and a configured one.
        .addSupportedTypes(SupportedType.of(TypeKind.BOOL))
        .addSupportedTypes(
            SupportedType.builder().type(TypeKind.PRECISION_TIMESTAMP).maxPrecision(9).build())
        // Relations: a bare entry and one carrying configuration.
        .addSupportedRelations(SupportedRelation.of(RelationKind.FILTER))
        .addSupportedRelations(
            SupportedRelation.builder()
                .relation(RelationKind.JOIN)
                .addJoinTypes(JoinType.INNER, JoinType.LEFT)
                .build())
        // Functions reference a dependency alias.
        .addSupportedScalarFunctions(
            DialectFunction.builder()
                .source("arithmetic")
                .name("add")
                .systemMetadata(
                    SystemFunctionMetadata.builder().name(\"+\").notation(Notation.INFIX).build())
                .addSupportedImpls("i32_i32", "i64_i64")
                .build())
        .build();

String yaml = Dialect.toYaml(dialect);          // create
DialectDocument parsed = Dialect.load(yaml);     // consume

The configuration-free entries serialize as bare enum strings and the configured ones as mappings:

---
name: "Example Dialect"
dependencies:
  arithmetic: "extension:io.substrait:functions_arithmetic"
supported_types:
- "BOOL"
- type: "PRECISION_TIMESTAMP"
  max_precision: 9
supported_relations:
- "FILTER"
- relation: "JOIN"
  join_types:
  - "INNER"
  - "LEFT"
supported_scalar_functions:
- source: "arithmetic"
  name: "add"
  system_metadata:
    name: "+"
    notation: "INFIX"
  supported_impls:
  - "i32_i32"
  - "i64_i64"

A fuller example covering every union and configuration option lives in DialectRoundTripTest.

Design

• New io.substrait.dialect package with a @Value.Enclosing Dialect holder and nested Immutables types, mirroring the existing SimpleExtension pattern (Jackson + @Value.Immutable, static load(...)/toYaml(...) helpers).
• The three polymorphic unions (supported_types / supported_relations / supported_expressions) are oneOf [bare-enum-string | config-object] in the schema. Each is modeled as one enum-tag class per category (SupportedType/SupportedRelation/SupportedExpression) carrying a dialect-local kind enum plus typed config fields. Custom Jackson (de)serializers collapse config-free entries to a bare enum string and expand configured ones to objects.
• Config sub-enums (JoinType, SetOperation, ...) are dialect-local with exactly the schema's constants, keeping the dialect vocabulary decoupled from the relational-algebra model (whose Join.JoinType/Set.SetOp carry extra UNKNOWN/deprecated values).
• The existing Type/Rel/Expression hierarchies model full algebra instances — the wrong abstraction level for capability tags — so they are intentionally not reused for the kind enums.

Validation & tests

• Schema validation is test-scope only (networknt json-schema-validator); the published core jar gains no new runtime dependency.
• processTestResources copies the dialect schema, the published spark_dialect.yaml, and the spec's per-section dialect fixtures onto the test classpath.
• Tests: a schema-validated build -> serialize -> validate -> parse -> assert-equal round-trip; bare-string collapse behavior; parsing/re-validating the real Spark dialect; and a parameterized round-trip over all five spec fixtures (types, relations, expressions, functions, execution_behavior).

The spark module is left unchanged; migrating its DialectGenerator onto this model is a natural follow-up.

🤖 Generated with AI