diff --git a/docs/content.zh/docs/sql/reference/data-types.md b/docs/content.zh/docs/sql/reference/data-types.md index 6ad9cd02c7a782..c1d0082ca70b24 100644 --- a/docs/content.zh/docs/sql/reference/data-types.md +++ b/docs/content.zh/docs/sql/reference/data-types.md @@ -1086,7 +1086,7 @@ The type can be declared using the above combinations where `p1` is the number o and `9` (both inclusive). If no `p1` is specified, it is equal to `2` by default. If no `p2` is specified, it is equal to `6` by default. -### Constructured Data Types +### Constructed Data Types #### `ARRAY` @@ -1507,7 +1507,7 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN flexibility to support highly nested and evolving schema. `VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps, -or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity +or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity and nesting are significant. `VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas @@ -1515,11 +1515,78 @@ without requiring upfront schema definition. For example, if a new field is adde can be directly incorporated into the `VARIANT` data without modifying the table schema. This is particularly useful in dynamic environments where schemas may evolve over time. -A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric -targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer -such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of -the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns -`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`. +A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`, +`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`, +`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ` +are stored with microsecond precision. + +The `PARSE_JSON` function produces only the kinds that JSON syntax can express: + +| JSON input | Stored `VARIANT` kind | +|-----------------------------------------------------|-------------------------------------------------| +| `null` | `NULL` | +| `true` / `false` | `BOOLEAN` | +| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` | +| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` | +| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` | +| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` | +| Number exceeding 38 digits of precision or scale | `DOUBLE` | +| String | `STRING` | +| Array | array (elements encoded by the same rules) | +| Object | object (values encoded by the same rules) | + +Because JSON has no literal for them, `PARSE_JSON` never produces `FLOAT`, `DATE`, `TIMESTAMP`, +`TIMESTAMP_LTZ`, or `BYTES`. + +The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`, +`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail. `PARSE_JSON('1e400')` fails as well +because a value outside the `DOUBLE` range cannot be stored as a finite number. In all of these +cases `TRY_PARSE_JSON` returns `NULL`. +A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast +it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`. + +A `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. A cast succeeds only when +the target holds the stored value without altering it, so a value is never wrapped, rounded, +truncated, or padded to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`. + +| Stored kind | Succeeds for | +|-----------------|------------------------------------------------------| +| numeric kinds | any numeric target that holds the value | +| `BOOLEAN` | `BOOLEAN` | +| `DATE` | `DATE` | +| `TIMESTAMP` | `TIMESTAMP(p)` that keeps the fractional seconds | +| `TIMESTAMP_LTZ` | `TIMESTAMP_LTZ(p)` that keeps the fractional seconds | +| `BYTES` | `BINARY(n)`, `VARBINARY(n)` | +| any scalar | `STRING`, `CHAR(n)`, `VARCHAR(n)` | +| `NULL` | SQL `NULL` for any nullable target | + +The conditions above mean: + +- An **integer target** needs the value in range and without a fractional part, so + `PARSE_JSON('7.0')` reaches `INT` as `7` while `PARSE_JSON('7.2')` does not, and + `CAST(PARSE_JSON('1000') AS TINYINT)` fails instead of wrapping. +- A **`DECIMAL`** target has to fit the precision and the scale. Trailing zeros may be appended, so + `42` reaches `DECIMAL(5, 2)` as `42.00`, but a scale that would have to round is rejected. +- **`FLOAT`** and **`DOUBLE`** are approximate by definition, so they take any numeric kind and drop + decimal digits, rejecting only a magnitude out of range such as `1e40` to a `FLOAT`. +- **fractional seconds** have to survive the target precision. A variant keeps microseconds, so + `.123` reaches `TIMESTAMP(3)` while `.123456` does not. +- **`CHAR(n)`** and **`BINARY(n)`** require the exact length, **`VARCHAR(n)`** and **`VARBINARY(n)`** + at most `n`. Nothing is padded or truncated. + +To reach a type the table does not list, wrap the cast in a regular cast. Only the inner cast is a +`VARIANT` cast, so the outer one applies the usual rules and may round, truncate, or overflow: + +```sql +CAST(CAST(PARSE_JSON('1000') AS SMALLINT) AS TINYINT) -- returns -24 (after overflow) +CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3 (truncated) +``` + +A cast to a character string renders the value exactly as a regular SQL cast of the stored kind +would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, and a `TIMESTAMP_LTZ` is shifted +into the session time zone. Use `JSON_STRING` for the JSON representation instead, where a string +stays quoted as `"foo"` and an object or array is serialized. A variant that stores a JSON `null` +casts to SQL `NULL`. **Declaration** diff --git a/docs/content/docs/sql/reference/data-types.md b/docs/content/docs/sql/reference/data-types.md index 45ace31e365083..d9c056d071a4c9 100644 --- a/docs/content/docs/sql/reference/data-types.md +++ b/docs/content/docs/sql/reference/data-types.md @@ -1094,7 +1094,7 @@ The type can be declared using the above combinations where `p1` is the number o and `9` (both inclusive). If no `p1` is specified, it is equal to `2` by default. If no `p2` is specified, it is equal to `6` by default. -### Constructured Data Types +### Constructed Data Types #### `ARRAY` @@ -1515,7 +1515,7 @@ close to the semantics of JSON. Compared to `ROW` and `STRUCTURED` type, `VARIAN flexibility to support highly nested and evolving schema. `VARIANT` allows for deeply nested data structures, such as arrays within arrays, maps within maps, -or combinations of both.This capability makes `VARIANT` ideal for scenarios where data complexity +or combinations of both. This capability makes `VARIANT` ideal for scenarios where data complexity and nesting are significant. `VARIANT` allows schema evolution, enabling the storage of data with changing or unknown schemas @@ -1523,11 +1523,78 @@ without requiring upfront schema definition. For example, if a new field is adde can be directly incorporated into the `VARIANT` data without modifying the table schema. This is particularly useful in dynamic environments where schemas may evolve over time. -A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. Numeric -targets are lenient: a variant holding any numeric value casts to any numeric type, so a JSON integer -such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require the stored value to be of -the matching kind. When the value cannot be converted, `CAST` fails the job and `TRY_CAST` returns -`NULL`. Use the `JSON_STRING` function to obtain the JSON string representation of a `VARIANT`. +A `VARIANT` stores a single value of one of the following kinds: `NULL`, `BOOLEAN`, `TINYINT`, +`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision 38), `STRING`, `DATE`, +`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object. `TIMESTAMP` and `TIMESTAMP_LTZ` +are stored with microsecond precision. + +The `PARSE_JSON` function produces only the kinds that JSON syntax can express: + +| JSON input | Stored `VARIANT` kind | +|-----------------------------------------------------|-------------------------------------------------| +| `null` | `NULL` | +| `true` / `false` | `BOOLEAN` | +| Integer within the 64-bit signed range | smallest of `TINYINT`/`SMALLINT`/`INT`/`BIGINT` | +| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL` | +| Decimal in plain notation, up to precision/scale 38 | `DECIMAL` | +| Number in scientific notation, e.g. `1.5e3` | `DOUBLE` | +| Number exceeding 38 digits of precision or scale | `DOUBLE` | +| String | `STRING` | +| Array | array (elements encoded by the same rules) | +| Object | object (values encoded by the same rules) | + +Because JSON has no literal for them, `PARSE_JSON` never produces `FLOAT`, `DATE`, `TIMESTAMP`, +`TIMESTAMP_LTZ`, or `BYTES`. + +The JSON specification has no `NaN` or infinity literals, so `PARSE_JSON('NaN')`, +`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail. `PARSE_JSON('1e400')` fails as well +because a value outside the `DOUBLE` range cannot be stored as a finite number. In all of these +cases `TRY_PARSE_JSON` returns `NULL`. +A `VARIANT` has no dedicated kind for these values. To keep one, store it as a JSON string and cast +it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS FLOAT)`. + +A `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. A cast succeeds only when +the target holds the stored value without altering it, so a value is never wrapped, rounded, +truncated, or padded to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`. + +| Stored kind | Succeeds for | +|-----------------|------------------------------------------------------| +| numeric kinds | any numeric target that holds the value | +| `BOOLEAN` | `BOOLEAN` | +| `DATE` | `DATE` | +| `TIMESTAMP` | `TIMESTAMP(p)` that keeps the fractional seconds | +| `TIMESTAMP_LTZ` | `TIMESTAMP_LTZ(p)` that keeps the fractional seconds | +| `BYTES` | `BINARY(n)`, `VARBINARY(n)` | +| any scalar | `STRING`, `CHAR(n)`, `VARCHAR(n)` | +| `NULL` | SQL `NULL` for any nullable target | + +The conditions above mean: + +- An **integer target** needs the value in range and without a fractional part, so + `PARSE_JSON('7.0')` reaches `INT` as `7` while `PARSE_JSON('7.2')` does not, and + `CAST(PARSE_JSON('1000') AS TINYINT)` fails instead of wrapping. +- A **`DECIMAL`** target has to fit the precision and the scale. Trailing zeros may be appended, so + `42` reaches `DECIMAL(5, 2)` as `42.00`, but a scale that would have to round is rejected. +- **`FLOAT`** and **`DOUBLE`** are approximate by definition, so they take any numeric kind and drop + decimal digits, rejecting only a magnitude out of range such as `1e40` to a `FLOAT`. +- **fractional seconds** have to survive the target precision. A variant keeps microseconds, so + `.123` reaches `TIMESTAMP(3)` while `.123456` does not. +- **`CHAR(n)`** and **`BINARY(n)`** require the exact length, **`VARCHAR(n)`** and **`VARBINARY(n)`** + at most `n`. Nothing is padded or truncated. + +To reach a type the table does not list, wrap the cast in a regular cast. Only the inner cast is a +`VARIANT` cast, so the outer one applies the usual rules and may round, truncate, or overflow: + +```sql +CAST(CAST(PARSE_JSON('1000') AS SMALLINT) AS TINYINT) -- returns -24 (after overflow) +CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3 (truncated) +``` + +A cast to a character string renders the value exactly as a regular SQL cast of the stored kind +would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, and a `TIMESTAMP_LTZ` is shifted +into the session time zone. Use `JSON_STRING` for the JSON representation instead, where a string +stays quoted as `"foo"` and an object or array is serialized. A variant that stores a JSON `null` +casts to SQL `NULL`. **Declaration** diff --git a/docs/data/sql_functions.yml b/docs/data/sql_functions.yml index c6b683c70c9e3a..31d746a1aca66e 100644 --- a/docs/data/sql_functions.yml +++ b/docs/data/sql_functions.yml @@ -1234,6 +1234,8 @@ variant: parser will keep the last occurrence of all fields with the same key, otherwise when `allowDuplicateKeys` is false it will throw an error. The default value of `allowDuplicateKeys` is false. + + See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds. - sql: TRY_PARSE_JSON(json_string[, allow_duplicate_keys]) description: | Try to parse a JSON string into a Variant if possible. If the JSON string is invalid, return @@ -1243,6 +1245,8 @@ variant: parser will keep the last occurrence of all fields with the same key, otherwise when `allowDuplicateKeys` is false it will throw an error. The default value of `allowDuplicateKeys` is false. + + See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds. - sql: JSON_STRING(variant) description: | Generate a json string from a Variant object. diff --git a/docs/data/sql_functions_zh.yml b/docs/data/sql_functions_zh.yml index 998f46d4134870..eb88c3897836d6 100644 --- a/docs/data/sql_functions_zh.yml +++ b/docs/data/sql_functions_zh.yml @@ -1320,6 +1320,8 @@ variant: 同键的字段，否则当 allowDuplicateKeys 为 false 时，它会抛出一个错误。默认情况下， allowDuplicateKeys 的值为 false。 + See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds. + - sql: TRY_PARSE_JSON(json_string[, allow_duplicate_keys]) description: | 尽可能将 JSON 字符串解析为 Variant。如果 JSON 字符串无效，则返回 NULL。如果希望抛出错误而不是返回 NULL， @@ -1329,6 +1331,8 @@ variant: 同键的字段，否则当 allowDuplicateKeys 为 false 时，它会抛出一个错误。默认情况下， allowDuplicateKeys 的值为 false。 + See the `VARIANT` data type documentation for how JSON values are mapped to variant kinds. + - sql: JSON_STRING(variant) description: | Generate a json string from a Variant object. diff --git a/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java b/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java index c0f15788f3f754..dd7ac96b6fc9f7 100644 --- a/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java +++ b/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java @@ -94,7 +94,8 @@ public interface Variant { float getFloat() throws VariantTypeException; /** - * Get the scalar value of variant as BigDecimal, if the variant type is {@link Type#DECIMAL}. + * Get the scalar value of variant as {@link BigDecimal}, if the variant type is {@link + * Type#DECIMAL}. * * @throws VariantTypeException If this variant is not a scalar value or is not {@link * Type#DECIMAL}. @@ -118,7 +119,8 @@ public interface Variant { String getString() throws VariantTypeException; /** - * Get the scalar value of variant as LocalDate, if the variant type is {@link Type#DATE}. + * Get the scalar value of variant as {@link LocalDate}, if the variant type is {@link + * Type#DATE}. * * @throws VariantTypeException If this variant is not a scalar value or is not {@link * Type#DATE}. @@ -126,8 +128,8 @@ public interface Variant { LocalDate getDate() throws VariantTypeException; /** - * Get the scalar value of variant as LocalDateTime, if the variant type is {@link - * Type#TIMESTAMP}. + * Get the scalar value of variant as {@link LocalDateTime}, if the variant type is {@link + * Type#TIMESTAMP}. The returned value has microsecond precision. * * @throws VariantTypeException If this variant is not a scalar value or is not {@link * Type#TIMESTAMP}. @@ -135,10 +137,11 @@ public interface Variant { LocalDateTime getDateTime() throws VariantTypeException; /** - * Get the scalar value of variant as Instant, if the variant type is {@link Type#TIMESTAMP}. + * Get the scalar value of variant as {@link Instant}, if the variant type is {@link + * Type#TIMESTAMP_LTZ}. The returned value has microsecond precision. * * @throws VariantTypeException If this variant is not a scalar value or is not {@link - * Type#TIMESTAMP}. + * Type#TIMESTAMP_LTZ}. */ Instant getInstant() throws VariantTypeException; diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java index 3bd66e01cd5ecc..fbd4bf188d7519 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java @@ -35,7 +35,6 @@ import java.util.List; import java.util.Optional; -import static org.apache.flink.table.types.logical.utils.LogicalTypeCasts.getUnsupportedCastHint; import static org.apache.flink.table.types.logical.utils.LogicalTypeCasts.supportsExplicitCast; /** @@ -70,15 +69,6 @@ public Optional> inferInputTypes( return Optional.of(argumentDataTypes); } if (!supportsExplicitCast(fromType, toType)) { - final Optional hint = getUnsupportedCastHint(fromType, toType); - if (hint.isPresent()) { - return callContext.fail( - throwOnFailure, - "Unsupported cast from '%s' to '%s'. %s", - fromType, - toType, - hint.get()); - } return callContext.fail( throwOnFailure, "Unsupported cast from '%s' to '%s'.", fromType, toType); } diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java index 58f474ec0eb70b..d22b16399c372f 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java @@ -37,7 +37,6 @@ import java.util.HashSet; import java.util.List; import java.util.Map; -import java.util.Optional; import java.util.Set; import java.util.function.BiFunction; import java.util.function.BiPredicate; @@ -210,7 +209,7 @@ public final class LogicalTypeCasts { castTo(CHAR) .implicitFrom(CHAR) .explicitFromFamily(PREDEFINED, CONSTRUCTED) - .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP) + .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP, VARIANT) .injectiveFrom(WHEN_LENGTH_FITS, CHAR) .injectiveFrom(WHEN_MAX_CHAR_LENGTH_FITS, STRING_INJECTIVE_SOURCES) .injectiveFrom(WHEN_CHAR_LENGTH_FITS_UTF8, BINARY, VARBINARY) @@ -219,7 +218,7 @@ public final class LogicalTypeCasts { castTo(VARCHAR) .implicitFromFamily(CHARACTER_STRING) .explicitFromFamily(PREDEFINED, CONSTRUCTED) - .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP) + .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP, VARIANT) .injectiveFrom(WHEN_LENGTH_FITS, CHAR, VARCHAR) .injectiveFrom(WHEN_MAX_CHAR_LENGTH_FITS, STRING_INJECTIVE_SOURCES) .injectiveFrom(WHEN_CHAR_LENGTH_FITS_UTF8, BINARY, VARBINARY) @@ -232,7 +231,7 @@ public final class LogicalTypeCasts { castTo(BINARY) .implicitFrom(BINARY) .explicitFromFamily(CHARACTER_STRING) - .explicitFrom(VARBINARY, RAW) + .explicitFrom(VARBINARY, RAW, VARIANT) .injectiveFrom(WHEN_LENGTH_FITS, BINARY) .injectiveFrom(WHEN_BINARY_LENGTH_FITS_UTF8, CHAR, VARCHAR) .build(); @@ -240,7 +239,7 @@ public final class LogicalTypeCasts { castTo(VARBINARY) .implicitFromFamily(BINARY_STRING) .explicitFromFamily(CHARACTER_STRING) - .explicitFrom(BINARY, RAW) + .explicitFrom(BINARY, RAW, VARIANT) .injectiveFrom(WHEN_LENGTH_FITS, BINARY, VARBINARY) .injectiveFrom(WHEN_BINARY_LENGTH_FITS_UTF8, CHAR, VARCHAR) .build(); @@ -252,35 +251,55 @@ public final class LogicalTypeCasts { castTo(TINYINT) .implicitFrom(TINYINT) .explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL) - .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) + .explicitFrom( + BOOLEAN, + TIMESTAMP_WITHOUT_TIME_ZONE, + TIMESTAMP_WITH_LOCAL_TIME_ZONE, + VARIANT) .injectiveFrom(TINYINT) .build(); castTo(SMALLINT) .implicitFrom(TINYINT, SMALLINT) .explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL) - .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) + .explicitFrom( + BOOLEAN, + TIMESTAMP_WITHOUT_TIME_ZONE, + TIMESTAMP_WITH_LOCAL_TIME_ZONE, + VARIANT) .injectiveFrom(TINYINT, SMALLINT) .build(); castTo(INTEGER) .implicitFrom(TINYINT, SMALLINT, INTEGER) .explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL) - .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) + .explicitFrom( + BOOLEAN, + TIMESTAMP_WITHOUT_TIME_ZONE, + TIMESTAMP_WITH_LOCAL_TIME_ZONE, + VARIANT) .injectiveFrom(TINYINT, SMALLINT, INTEGER) .build(); castTo(BIGINT) .implicitFrom(TINYINT, SMALLINT, INTEGER, BIGINT) .explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL) - .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) + .explicitFrom( + BOOLEAN, + TIMESTAMP_WITHOUT_TIME_ZONE, + TIMESTAMP_WITH_LOCAL_TIME_ZONE, + VARIANT) .injectiveFrom(TINYINT, SMALLINT, INTEGER, BIGINT) .build(); castTo(DECIMAL) .implicitFromFamily(NUMERIC) .explicitFromFamily(CHARACTER_STRING, INTERVAL) - .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) + .explicitFrom( + BOOLEAN, + TIMESTAMP_WITHOUT_TIME_ZONE, + TIMESTAMP_WITH_LOCAL_TIME_ZONE, + VARIANT) .injectiveFrom(WHEN_PRECISION_AND_SCALE_MATCH, DECIMAL) .build(); @@ -291,14 +310,22 @@ public final class LogicalTypeCasts { castTo(FLOAT) .implicitFrom(TINYINT, SMALLINT, INTEGER, BIGINT, FLOAT, DECIMAL) .explicitFromFamily(NUMERIC, CHARACTER_STRING) - .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) + .explicitFrom( + BOOLEAN, + TIMESTAMP_WITHOUT_TIME_ZONE, + TIMESTAMP_WITH_LOCAL_TIME_ZONE, + VARIANT) .injectiveFrom(FLOAT) .build(); castTo(DOUBLE) .implicitFromFamily(NUMERIC) .explicitFromFamily(CHARACTER_STRING) - .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) + .explicitFrom( + BOOLEAN, + TIMESTAMP_WITHOUT_TIME_ZONE, + TIMESTAMP_WITH_LOCAL_TIME_ZONE, + VARIANT) .injectiveFrom(DOUBLE) .build(); @@ -309,6 +336,7 @@ public final class LogicalTypeCasts { castTo(BOOLEAN) .implicitFrom(BOOLEAN) .explicitFromFamily(CHARACTER_STRING, INTEGER_NUMERIC) + .explicitFrom(VARIANT) .injectiveFrom(BOOLEAN) .build(); @@ -319,6 +347,7 @@ public final class LogicalTypeCasts { castTo(DATE) .implicitFrom(DATE, TIMESTAMP_WITHOUT_TIME_ZONE) .explicitFromFamily(TIMESTAMP, CHARACTER_STRING) + .explicitFrom(VARIANT) .injectiveFrom(DATE) .build(); @@ -331,6 +360,7 @@ public final class LogicalTypeCasts { castTo(TIMESTAMP_WITHOUT_TIME_ZONE) .implicitFrom(TIMESTAMP_WITHOUT_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE) .explicitFromFamily(DATETIME, CHARACTER_STRING, NUMERIC) + .explicitFrom(VARIANT) .injectiveFrom( WHEN_PRECISION_MATCHES, TIMESTAMP_WITHOUT_TIME_ZONE, @@ -346,6 +376,7 @@ public final class LogicalTypeCasts { castTo(TIMESTAMP_WITH_LOCAL_TIME_ZONE) .implicitFrom(TIMESTAMP_WITH_LOCAL_TIME_ZONE, TIMESTAMP_WITHOUT_TIME_ZONE) .explicitFromFamily(DATETIME, CHARACTER_STRING, NUMERIC) + .explicitFrom(VARIANT) .injectiveFrom( WHEN_PRECISION_MATCHES, TIMESTAMP_WITH_LOCAL_TIME_ZONE, @@ -648,9 +679,6 @@ private static boolean supportsCasting( // BITMAP can only be cast to BYTES (unbounded VARBINARY), because trimming or padding // would corrupt the serialized bitmap data. return allowExplicit && getLength(targetType) == VarBinaryType.MAX_LENGTH; - } else if (sourceRoot == VARIANT) { - // a VARIANT can only be explicitly cast to a supported scalar type - return allowExplicit && supportsVariantToScalarCast(targetType); } if (implicitCastingRules.get(targetRoot).contains(sourceRoot)) { @@ -731,45 +759,6 @@ private static boolean supportsConstructedCasting( return false; } - private static boolean supportsVariantToScalarCast(LogicalType targetType) { - switch (targetType.getTypeRoot()) { - case BOOLEAN: - case TINYINT: - case SMALLINT: - case INTEGER: - case BIGINT: - case FLOAT: - case DOUBLE: - case DECIMAL: - case BINARY: - case VARBINARY: - case DATE: - case TIMESTAMP_WITHOUT_TIME_ZONE: - case TIMESTAMP_WITH_LOCAL_TIME_ZONE: - return true; - default: - // TIME has no counterpart in the Variant type model. CHARACTER_STRING is handled by - // the display-oriented VariantToStringCastRule and is intentionally not offered as - // a - // user-facing cast here. - return false; - } - } - - /** - * Returns a hint pointing to the function that performs a conceptually related operation when - * an explicit cast is unsupported, or empty when no specific hint applies. - */ - public static Optional getUnsupportedCastHint( - LogicalType sourceType, LogicalType targetType) { - if (sourceType.is(VARIANT) && targetType.is(CHARACTER_STRING)) { - return Optional.of( - "Use the JSON_STRING function to convert a VARIANT to its JSON string " - + "representation."); - } - return Optional.empty(); - } - private static CastingRuleBuilder castTo(LogicalTypeRoot targetType) { return new CastingRuleBuilder(targetType); } diff --git a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java index 6fb3574b1f59f6..c51408edd81fcb 100644 --- a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java +++ b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java @@ -270,6 +270,7 @@ private static Stream testData() { // variant to scalar is explicit only Arguments.of(new VariantType(), new BooleanType(), false, true), Arguments.of(new VariantType(), new TinyIntType(), false, true), + Arguments.of(new VariantType(), new SmallIntType(), false, true), Arguments.of(new VariantType(), new IntType(), false, true), Arguments.of(new VariantType(), new BigIntType(), false, true), Arguments.of(new VariantType(), new DoubleType(), false, true), @@ -284,11 +285,12 @@ private static Stream testData() { new VarBinaryType(VarBinaryType.MAX_LENGTH), false, true), + Arguments.of(new VariantType(), new CharType(), false, true), + Arguments.of(new VariantType(), VarCharType.STRING_TYPE, false, true), // variant identity cast is implicit Arguments.of(new VariantType(), new VariantType(), true, true), - // TIME, character strings and constructed targets are not castable from variant + // TIME and constructed targets are not castable from variant Arguments.of(new VariantType(), new TimeType(), false, false), - Arguments.of(new VariantType(), VarCharType.STRING_TYPE, false, false), Arguments.of(new VariantType(), new ArrayType(new IntType()), false, false), Arguments.of(new VariantType(), new RowType(List.of()), false, false), Arguments.of( diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java index 8a278f41498b57..9b42628498f9dd 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java @@ -18,33 +18,30 @@ package org.apache.flink.table.planner.functions.casting; -import org.apache.flink.table.data.DecimalData; -import org.apache.flink.table.data.TimestampData; import org.apache.flink.table.planner.functions.casting.CastRuleUtils.CodeWriter; +import org.apache.flink.table.runtime.functions.VariantCastUtils; import org.apache.flink.table.types.logical.DecimalType; import org.apache.flink.table.types.logical.LogicalType; import org.apache.flink.table.types.logical.LogicalTypeRoot; import org.apache.flink.table.types.logical.utils.LogicalTypeChecks; import org.apache.flink.types.variant.Variant; -import java.math.BigDecimal; -import java.util.Arrays; - -import static org.apache.flink.table.planner.codegen.CodeGenUtils.className; -import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName; -import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.arrayLength; +import static org.apache.flink.table.planner.codegen.CodeGenUtils.primitiveTypeTermForType; import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.cast; -import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall; import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall; import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall; -import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.ternaryOperator; +import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral; /** * {@link LogicalTypeRoot#VARIANT} to primitive type cast rule. * - *

Numeric targets are lenient and follow regular numeric cast semantics; other targets require - * the stored value to match the target kind. On a mismatch {@code CAST} fails and {@code TRY_CAST} - * returns {@code null}. + *

A cast succeeds only when the target holds the stored value without altering it, otherwise it + * fails {@code CAST} and yields {@code null} for {@code TRY_CAST}. An integer widens or narrows as + * long as it stays in range, a {@code DECIMAL} has to fit the precision and scale without rounding, + * and a timestamp has to fit the target precision. {@code FLOAT} and {@code DOUBLE} are approximate + * by definition, so they take any numeric kind and reject only a magnitude out of range. Changing + * the kind itself is not implicit, so a decimal is not read as an integer and a {@code TIMESTAMP} + * is not read as a {@code TIMESTAMP_LTZ}. * *

{@code CHARACTER_STRING} is handled by {@link VariantToStringCastRule}; {@code TIME} has no * variant counterpart and is unsupported. @@ -128,14 +125,49 @@ protected String generateCodeBlockInternal( case SMALLINT: case INTEGER: case BIGINT: + writer.assignStmt( + returnVariable, + cast( + primitiveTypeTermForType(targetLogicalType), + staticCall( + VariantCastUtils.class, + "toIntegral", + inputTerm, + // A long literal needs the suffix to compile, since + // Long.MIN_VALUE does not fit an int literal. + integralMin(targetLogicalType) + "L", + integralMax(targetLogicalType) + "L", + strLiteral(targetLogicalType.getTypeRoot().name())))); + break; case FLOAT: + writer.assignStmt( + returnVariable, staticCall(VariantCastUtils.class, "toFloat", inputTerm)); + break; case DOUBLE: + writer.assignStmt( + returnVariable, staticCall(VariantCastUtils.class, "toDouble", inputTerm)); + break; case DECIMAL: - writer.assignStmt(returnVariable, numericExpression(inputTerm, targetLogicalType)); + final DecimalType decimalType = (DecimalType) targetLogicalType; + writer.assignStmt( + returnVariable, + staticCall( + VariantCastUtils.class, + "toDecimal", + inputTerm, + decimalType.getPrecision(), + decimalType.getScale())); break; case BINARY: case VARBINARY: - generateToBytes(context, inputTerm, returnVariable, targetLogicalType, writer); + writer.assignStmt( + returnVariable, + staticCall( + VariantCastUtils.class, + "toBytes", + inputTerm, + LogicalTypeChecks.getLength(targetLogicalType), + targetLogicalType.is(LogicalTypeRoot.BINARY))); break; case DATE: writer.assignStmt( @@ -146,17 +178,19 @@ protected String generateCodeBlockInternal( writer.assignStmt( returnVariable, staticCall( - TimestampData.class, - "fromLocalDateTime", - methodCall(inputTerm, "getDateTime"))); + VariantCastUtils.class, + "toTimestamp", + inputTerm, + LogicalTypeChecks.getPrecision(targetLogicalType))); break; case TIMESTAMP_WITH_LOCAL_TIME_ZONE: writer.assignStmt( returnVariable, staticCall( - TimestampData.class, - "fromInstant", - methodCall(inputTerm, "getInstant"))); + VariantCastUtils.class, + "toTimestampLtz", + inputTerm, + LogicalTypeChecks.getPrecision(targetLogicalType))); break; default: throw new IllegalArgumentException( @@ -165,73 +199,29 @@ protected String generateCodeBlockInternal( return writer.toString(); } - /** - * Converts a numeric variant to the numeric {@code target} via the matching {@link Number} - * accessor, mirroring regular numeric cast semantics. A non-numeric variant raises {@link - * ClassCastException}, failing {@code CAST} and yielding {@code null} for {@code TRY_CAST}. - */ - private static String numericExpression(String inputTerm, LogicalType target) { - final String number = cast(className(Number.class), methodCall(inputTerm, "get")); - if (!target.is(LogicalTypeRoot.DECIMAL)) { - return methodCall(number, numberAccessor(target)); - } - final DecimalType decimalType = (DecimalType) target; - return staticCall( - DecimalData.class, - "fromBigDecimal", - constructorCall(BigDecimal.class, methodCall(number, "toString")), - decimalType.getPrecision(), - decimalType.getScale()); - } - - private static String numberAccessor(LogicalType target) { + private static long integralMin(LogicalType target) { switch (target.getTypeRoot()) { case TINYINT: - return "byteValue"; + return Byte.MIN_VALUE; case SMALLINT: - return "shortValue"; + return Short.MIN_VALUE; case INTEGER: - return "intValue"; - case BIGINT: - return "longValue"; - case FLOAT: - return "floatValue"; - case DOUBLE: - return "doubleValue"; + return Integer.MIN_VALUE; default: - throw new IllegalArgumentException( - "Unsupported numeric target for casting from VARIANT: " + target); + return Long.MIN_VALUE; } } - private static void generateToBytes( - CodeGeneratorCastRule.Context context, - String inputTerm, - String returnVariable, - LogicalType targetLogicalType, - CodeWriter writer) { - final int targetLength = LogicalTypeChecks.getLength(targetLogicalType); - // Read the bytes once to avoid decoding the variant twice. - final String bytesTerm = newName(context.getCodeGeneratorContext(), "variantBytes"); - writer.declStmt("byte[]", bytesTerm, methodCall(inputTerm, "getBytes")); - if (BinaryToBinaryCastRule.couldPad(targetLogicalType, targetLength)) { - // BINARY(n): pad or trim to the exact target length. - writer.assignStmt( - returnVariable, - ternaryOperator( - arrayLength(bytesTerm) + " == " + targetLength, - bytesTerm, - staticCall(Arrays.class, "copyOf", bytesTerm, targetLength))); - } else if (BinaryToBinaryCastRule.couldTrim(targetLength)) { - // VARBINARY(n): trim only when longer than the target length. - writer.assignStmt( - returnVariable, - ternaryOperator( - arrayLength(bytesTerm) + " <= " + targetLength, - bytesTerm, - staticCall(Arrays.class, "copyOf", bytesTerm, targetLength))); - } else { - writer.assignStmt(returnVariable, bytesTerm); + private static long integralMax(LogicalType target) { + switch (target.getTypeRoot()) { + case TINYINT: + return Byte.MAX_VALUE; + case SMALLINT: + return Short.MAX_VALUE; + case INTEGER: + return Integer.MAX_VALUE; + default: + return Long.MAX_VALUE; } } } diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java index 29e7821bc3a281..83179ccde60005 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java @@ -18,15 +18,27 @@ package org.apache.flink.table.planner.functions.casting; +import org.apache.flink.table.runtime.functions.VariantCastUtils; import org.apache.flink.table.types.logical.LogicalType; import org.apache.flink.table.types.logical.LogicalTypeFamily; import org.apache.flink.table.types.logical.LogicalTypeRoot; +import org.apache.flink.table.types.logical.utils.LogicalTypeChecks; import org.apache.flink.types.variant.Variant; import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall; -import static org.apache.flink.table.types.logical.VarCharType.STRING_TYPE; +import static org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall; -/** {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeFamily#CHARACTER_STRING} cast rule. */ +/** + * {@link LogicalTypeRoot#VARIANT} to {@link LogicalTypeFamily#CHARACTER_STRING} cast rule. + * + *

Renders the scalar value the way a regular SQL cast of the stored kind would, so a boolean + * becomes {@code TRUE}, a timestamp uses the SQL format, and a {@code TIMESTAMP_LTZ} is shifted + * into the session time zone. A variant holding an object, array, or binary value has no scalar + * rendering and fails; use {@code JSON_STRING} for its JSON representation. + * + *

The target {@code CHAR}/{@code VARCHAR} length is enforced strictly: a value that does not fit + * fails {@code CAST} and yields {@code null} for {@code TRY_CAST}, with no padding or truncation. + */ class VariantToStringCastRule extends AbstractCharacterFamilyTargetRule { static final VariantToStringCastRule INSTANCE = new VariantToStringCastRule(); @@ -35,16 +47,50 @@ private VariantToStringCastRule() { super( CastRulePredicate.builder() .input(LogicalTypeRoot.VARIANT) - .target(STRING_TYPE) + .target(LogicalTypeRoot.CHAR) + .target(LogicalTypeRoot.VARCHAR) .build()); } + @Override + public boolean canFail(LogicalType inputLogicalType, LogicalType targetLogicalType) { + return true; + } + + /** + * Treats a variant that stores a JSON {@code null} as a {@code NULL} input, so it casts to SQL + * {@code NULL} instead of the text {@code null}. Only applied for a nullable target: a {@code + * NOT NULL} result cannot carry {@code NULL}, so a null-valued variant then fails. + */ + @Override + public CastCodeBlock generateCodeBlock( + CodeGeneratorCastRule.Context context, + String inputTerm, + String inputIsNullTerm, + LogicalType inputLogicalType, + LogicalType targetLogicalType) { + if (!targetLogicalType.isNullable()) { + return super.generateCodeBlock( + context, inputTerm, inputIsNullTerm, inputLogicalType, targetLogicalType); + } + final String isNullTerm = + "(" + inputIsNullTerm + " || " + methodCall(inputTerm, "isNull") + ")"; + return super.generateCodeBlock( + context, inputTerm, isNullTerm, inputLogicalType, targetLogicalType); + } + @Override public String generateStringExpression( CodeGeneratorCastRule.Context context, String inputTerm, LogicalType inputLogicalType, LogicalType targetLogicalType) { - return methodCall(inputTerm, "toString"); + return staticCall( + VariantCastUtils.class, + "toStringValue", + inputTerm, + context.getSessionTimeZoneTerm(), + LogicalTypeChecks.getLength(targetLogicalType), + targetLogicalType.is(LogicalTypeRoot.CHAR)); } } diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java index 6ad3ab7cb1192c..c40fd6944b08b6 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java @@ -138,75 +138,211 @@ Stream getTestSetSpecs() { } private static List variantCasts() { - // A variant is produced with PARSE_JSON so that the source column stays a STRING literal; - // there is no VARIANT literal to feed a source column directly. A JSON integer is stored in - // the smallest integer type that fits (42 -> TINYINT), and numeric casts are lenient, so it - // still widens to INT. + // A variant is produced with PARSE_JSON since there is no VARIANT literal. Numeric casts + // succeed only when the value is preserved exactly, otherwise CAST fails and TRY_CAST + // returns NULL. return List.of( TestSetSpec.forExpression("Cast a VARIANT produced by PARSE_JSON to a primitive") .onFieldsWithData("unused") .andDataTypes(STRING()) + // An integer converts to any integer target while the value stays in + // range, and to FLOAT or DOUBLE which are approximate by definition. + .testResult( + call("PARSE_JSON", "42").cast(TINYINT()), + "CAST(PARSE_JSON('42') AS TINYINT)", + (byte) 42, + TINYINT().notNull()) + .testResult( + call("PARSE_JSON", "42").cast(SMALLINT()), + "CAST(PARSE_JSON('42') AS SMALLINT)", + (short) 42, + SMALLINT().notNull()) .testResult( call("PARSE_JSON", "42").cast(INT()), "CAST(PARSE_JSON('42') AS INT)", 42, INT().notNull()) - // Integer overflow wraps around (Java narrowing), like a regular numeric - // cast. .testResult( - call("PARSE_JSON", "40000").cast(SMALLINT()), - "CAST(PARSE_JSON('40000') AS SMALLINT)", - (short) -25536, + call("PARSE_JSON", "42").cast(BIGINT()), + "CAST(PARSE_JSON('42') AS BIGINT)", + 42L, + BIGINT().notNull()) + .testResult( + call("PARSE_JSON", "42").cast(FLOAT()), + "CAST(PARSE_JSON('42') AS FLOAT)", + 42.0f, + FLOAT().notNull()) + .testResult( + call("PARSE_JSON", "42").cast(DOUBLE()), + "CAST(PARSE_JSON('42') AS DOUBLE)", + 42.0d, + DOUBLE().notNull()) + // An out-of-range value is rejected rather than wrapped. + .testResult( + call("PARSE_JSON", "1000").cast(SMALLINT()), + "CAST(PARSE_JSON('1000') AS SMALLINT)", + (short) 1000, SMALLINT().notNull()) + .testTableApiRuntimeError( + call("PARSE_JSON", "1000").cast(TINYINT()), "overflowed") + .testSqlRuntimeError("CAST(PARSE_JSON('1000') AS TINYINT)", "overflowed") .testResult( - call("PARSE_JSON", "128").cast(TINYINT()), - "CAST(PARSE_JSON('128') AS TINYINT)", - (byte) -128, - TINYINT().notNull()) + call("PARSE_JSON", "1000").tryCast(TINYINT()), + "TRY_CAST(PARSE_JSON('1000') AS TINYINT)", + null, + TINYINT()) + // A decimal reaches an integer target when it is already integral. .testResult( - call("PARSE_JSON", "2147483648").cast(INT()), - "CAST(PARSE_JSON('2147483648') AS INT)", - -2147483648, + call("PARSE_JSON", "7.0").cast(INT()), + "CAST(PARSE_JSON('7.0') AS INT)", + 7, INT().notNull()) + // A fractional value is rejected, since converting would drop digits. + .testTableApiRuntimeError( + call("PARSE_JSON", "123.456").cast(INT()), "lose precision") .testResult( - call("PARSE_JSON", "9223372036854775808").cast(BIGINT()), - "CAST(PARSE_JSON('9223372036854775808') AS BIGINT)", - -9223372036854775808L, - BIGINT().notNull()) - // An out-of-range floating point value saturates when cast to an integer, - // and a fractional value is truncated toward zero. + call("PARSE_JSON", "123.456").tryCast(INT()), + "TRY_CAST(PARSE_JSON('123.456') AS INT)", + null, + INT()) + // A DECIMAL target has to hold the value exactly. .testResult( - call("PARSE_JSON", "1e20").cast(INT()), - "CAST(PARSE_JSON('1e20') AS INT)", - 2147483647, - INT().notNull()) + call("PARSE_JSON", "123.456").cast(DECIMAL(6, 3)), + "CAST(PARSE_JSON('123.456') AS DECIMAL(6, 3))", + new BigDecimal("123.456"), + DECIMAL(6, 3).notNull()) + .testTableApiRuntimeError( + call("PARSE_JSON", "123.456").cast(DECIMAL(6, 2)), "lose precision") .testResult( - call("PARSE_JSON", "3.9").cast(INT()), - "CAST(PARSE_JSON('3.9') AS INT)", - 3, - INT().notNull()) - // A value beyond the FLOAT range becomes infinity. + call("PARSE_JSON", "123.456").tryCast(DECIMAL(6, 2)), + "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(6, 2))", + null, + DECIMAL(6, 2)) + .testTableApiRuntimeError( + call("PARSE_JSON", "123.456").cast(DECIMAL(5, 3)), "overflowed") .testResult( - call("PARSE_JSON", "1e40").cast(FLOAT()), - "CAST(PARSE_JSON('1e40') AS FLOAT)", - Float.POSITIVE_INFINITY, + call("PARSE_JSON", "123.456").tryCast(DECIMAL(5, 3)), + "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(5, 3))", + null, + DECIMAL(5, 3)) + // An integer is exact, so it reaches a DECIMAL that has room for it. + .testResult( + call("PARSE_JSON", "42").cast(DECIMAL(5, 2)), + "CAST(PARSE_JSON('42') AS DECIMAL(5, 2))", + new BigDecimal("42.00"), + DECIMAL(5, 2).notNull()) + // A decimal reaches an approximate target, where losing digits is expected. + .testResult( + call("PARSE_JSON", "123.456").cast(FLOAT()), + "CAST(PARSE_JSON('123.456') AS FLOAT)", + 123.456f, FLOAT().notNull()) - // DECIMAL overflow yields NULL instead of wrapping; TRY_CAST surfaces it. .testResult( - call("PARSE_JSON", "123.456").tryCast(DECIMAL(4, 2)), - "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(4, 2))", + call("PARSE_JSON", "123.456").cast(DOUBLE()), + "CAST(PARSE_JSON('123.456') AS DOUBLE)", + 123.456d, + DOUBLE().notNull()) + // A magnitude the target cannot represent is still rejected. + .testTableApiRuntimeError( + call("PARSE_JSON", "1e40").cast(FLOAT()), "overflowed") + .testResult( + call("PARSE_JSON", "1e40").tryCast(FLOAT()), + "TRY_CAST(PARSE_JSON('1e40') AS FLOAT)", null, - DECIMAL(4, 2)) + FLOAT()) .testResult( - call("PARSE_JSON", "42").cast(BIGINT()), - "CAST(PARSE_JSON('42') AS BIGINT)", - 42L, - BIGINT().notNull()) + call("PARSE_JSON", "1e20").cast(DOUBLE()), + "CAST(PARSE_JSON('1e20') AS DOUBLE)", + 1e20, + DOUBLE().notNull()) .testResult( call("PARSE_JSON", "true").cast(BOOLEAN()), "CAST(PARSE_JSON('true') AS BOOLEAN)", true, BOOLEAN().notNull()) + // CAST returns the raw scalar value (string unquoted) + .testResult( + call("PARSE_JSON", "\"foo\"").cast(STRING()), + "CAST(PARSE_JSON('\"foo\"') AS STRING)", + "foo", + STRING().notNull()) + .testResult( + call("PARSE_JSON", "123.456").cast(STRING()), + "CAST(PARSE_JSON('123.456') AS STRING)", + "123.456", + STRING().notNull()) + // The rendering matches a regular cast of the stored kind, so a boolean + // becomes TRUE rather than the JSON true. + .testResult( + call("PARSE_JSON", "true").cast(STRING()), + "CAST(PARSE_JSON('true') AS STRING)", + "TRUE", + STRING().notNull()) + // An object or array has no scalar value, so the error points to + // JSON_STRING. + .testTableApiRuntimeError( + call("PARSE_JSON", "[\"a\", \"b\"]").cast(STRING()), "JSON_STRING") + .testResult( + call("PARSE_JSON", "[\"a\", \"b\"]").tryCast(STRING()), + "TRY_CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)", + null, + STRING()) + .testTableApiRuntimeError( + call("PARSE_JSON", "{\"a\": 1}").cast(STRING()), "JSON_STRING") + .testResult( + call("PARSE_JSON", "{\"a\": 1}").tryCast(STRING()), + "TRY_CAST(PARSE_JSON('{\"a\": 1}') AS STRING)", + null, + STRING()) + // Bounded CHAR/VARCHAR is strict on length. + // VARCHAR(n) allows any length up to n; + // CHAR(n) requires the exact length. + .testResult( + call("PARSE_JSON", "\"ab\"").cast(VARCHAR(3)), + "CAST(PARSE_JSON('\"ab\"') AS VARCHAR(3))", + "ab", + VARCHAR(3).notNull()) + .testTableApiRuntimeError( + call("PARSE_JSON", "\"foobar\"").cast(VARCHAR(3)), "does not fit") + .testResult( + call("PARSE_JSON", "\"foobar\"").tryCast(VARCHAR(3)), + "TRY_CAST(PARSE_JSON('\"foobar\"') AS VARCHAR(3))", + null, + VARCHAR(3)) + .testResult( + call("PARSE_JSON", "\"abc\"").cast(CHAR(3)), + "CAST(PARSE_JSON('\"abc\"') AS CHAR(3))", + "abc", + CHAR(3).notNull()) + .testTableApiRuntimeError( + call("PARSE_JSON", "\"ab\"").cast(CHAR(5)), "does not fit") + .testResult( + call("PARSE_JSON", "\"ab\"").tryCast(CHAR(5)), + "TRY_CAST(PARSE_JSON('\"ab\"') AS CHAR(5))", + null, + CHAR(5)) + // A variant holding a JSON null casts to SQL NULL, not to the text 'null'. + // The length of that text must not be checked against the target either. + .testResult( + call("TRY_PARSE_JSON", "null").cast(STRING()), + "CAST(TRY_PARSE_JSON('null') AS STRING)", + null, + STRING()) + .testResult( + call("PARSE_JSON", "null").tryCast(STRING()), + "TRY_CAST(PARSE_JSON('null') AS STRING)", + null, + STRING()) + .testResult( + call("TRY_PARSE_JSON", "null").cast(CHAR(2)), + "CAST(TRY_PARSE_JSON('null') AS CHAR(2))", + null, + CHAR(2)) + .testResult( + call("PARSE_JSON", "null").tryCast(VARCHAR(2)), + "TRY_CAST(PARSE_JSON('null') AS VARCHAR(2))", + null, + VARCHAR(2)) // TRY_CAST of a value whose kind does not match the target returns NULL .testResult( call("PARSE_JSON", "\"foo\"").tryCast(INT()), @@ -227,15 +363,10 @@ private static List variantCasts() { BOOLEAN()) // A nullable variant with a concrete value still casts normally. .testResult( - call("TRY_PARSE_JSON", "42").cast(INT()), - "CAST(TRY_PARSE_JSON('42') AS INT)", - 42, - INT()) - // Casting a VARIANT to a string points the user to JSON_STRING. - .testTableApiValidationError( - call("PARSE_JSON", "42").cast(STRING()), - "Use the JSON_STRING function to convert a VARIANT to its JSON " - + "string representation.")); + call("TRY_PARSE_JSON", "42").cast(TINYINT()), + "CAST(TRY_PARSE_JSON('42') AS TINYINT)", + (byte) 42, + TINYINT())); } private static List allTypesBasic() { diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java index aca973fa3ca6c7..187d63c68968d5 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java @@ -1544,35 +1544,79 @@ Stream testCases() { .fromCase(BITMAP(), DEFAULT_BITMAP, DEFAULT_BITMAP.toBytes()) .fromCase(BITMAP(), Bitmap.empty(), Bitmap.empty().toBytes()) .fromCase(BITMAP(), null, null), - // From VARIANT to primitive types. Numeric targets are lenient: a variant holding - // any numeric kind converts to the requested numeric type (widening and narrowing). - // Non-numeric targets are strict: the stored kind must match, otherwise the cast - // fails and TRY_CAST returns null. + // From VARIANT to primitive types. A cast succeeds only when the target holds the + // stored value unaltered, except for the approximate FLOAT and DOUBLE. + // A character string renders like a regular cast of the stored kind, so these + // expectations reuse the constants of the native cases above. + CastTestSpecBuilder.testCastTo(STRING()) + .fromCase(VARIANT(), Variant.newBuilder().of(true), fromString("TRUE")) + .fromCase(VARIANT(), Variant.newBuilder().of(false), fromString("FALSE")) + .fromCase(VARIANT(), Variant.newBuilder().of("foo"), fromString("foo")) + .fromCase(VARIANT(), Variant.newBuilder().of(42), fromString("42")) + .fromCase( + VARIANT(), + Variant.newBuilder().of(new BigDecimal("123.456")), + fromString("123.456")) + .fromCase( + VARIANT(), + Variant.newBuilder().of(LocalDate.parse("2021-09-24")), + DATE_STRING) + .fromCase( + VARIANT(), + Variant.newBuilder().of(TIMESTAMP.toLocalDateTime()), + TIMESTAMP_STRING) + .fromCase( + VARIANT(), + CET_CONTEXT, + Variant.newBuilder().of(TIMESTAMP.toInstant()), + TIMESTAMP_STRING_CET) + // a binary value has no scalar rendering + .fail( + VARIANT(), + Variant.newBuilder().of(new byte[] {1, 2, 3}), + TableRuntimeException.class), CastTestSpecBuilder.testCastTo(BOOLEAN()) .fromCase(VARIANT(), Variant.newBuilder().of(true), true) .fromCase(VARIANT(), Variant.newBuilder().of(false), false) .fail(VARIANT(), Variant.newBuilder().of(1), TableRuntimeException.class), CastTestSpecBuilder.testCastTo(TINYINT()) .fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), (byte) 42) + // a wider integer kind narrows while the value is in range .fromCase(VARIANT(), Variant.newBuilder().of(42), (byte) 42) + // out of range is rejected instead of wrapping + .fail(VARIANT(), Variant.newBuilder().of(1000), TableRuntimeException.class) .fail(VARIANT(), Variant.newBuilder().of("x"), TableRuntimeException.class), CastTestSpecBuilder.testCastTo(SMALLINT()) .fromCase(VARIANT(), Variant.newBuilder().of((short) 42), (short) 42) .fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), (short) 42) + .fromCase(VARIANT(), Variant.newBuilder().of(1000), (short) 1000) + .fail( + VARIANT(), + Variant.newBuilder().of(40000), + TableRuntimeException.class) .fail( VARIANT(), Variant.newBuilder().of(true), TableRuntimeException.class), CastTestSpecBuilder.testCastTo(INT()) - // widening: a JSON integer is stored in the smallest type but still casts - // up + .fromCase(VARIANT(), Variant.newBuilder().of(42), 42) + // every integer kind converts as long as the value fits .fromCase(VARIANT(), Variant.newBuilder().of((byte) 42), 42) .fromCase(VARIANT(), Variant.newBuilder().of((short) 42), 42) - .fromCase(VARIANT(), Variant.newBuilder().of(42), 42) .fromCase(VARIANT(), Variant.newBuilder().of(42L), 42) - // narrowing from a floating point or decimal value truncates - .fromCase(VARIANT(), Variant.newBuilder().of(3.9d), 3) - .fromCase(VARIANT(), Variant.newBuilder().of(new BigDecimal("7.2")), 7) + .fail( + VARIANT(), + Variant.newBuilder().of(2147483648L), + TableRuntimeException.class) + // an approximate or decimal kind converts when the value is integral + .fromCase(VARIANT(), Variant.newBuilder().of(7.0d), 7) + .fromCase(VARIANT(), Variant.newBuilder().of(new BigDecimal("7.0")), 7) + // a fractional value would have to be rounded away, so it is rejected + .fail(VARIANT(), Variant.newBuilder().of(7.2d), TableRuntimeException.class) + .fail( + VARIANT(), + Variant.newBuilder().of(new BigDecimal("7.2")), + TableRuntimeException.class) // a non-numeric variant cannot be cast to a number .fail( VARIANT(), @@ -1587,14 +1631,28 @@ Stream testCases() { .fromCase(VARIANT(), Variant.newBuilder().of(42), 42L) .fail(VARIANT(), Variant.newBuilder().of("x"), TableRuntimeException.class), CastTestSpecBuilder.testCastTo(FLOAT()) + // every numeric kind reaches an approximate target .fromCase(VARIANT(), Variant.newBuilder().of(1.5f), 1.5f) .fromCase(VARIANT(), Variant.newBuilder().of(1.5d), 1.5f) .fromCase(VARIANT(), Variant.newBuilder().of(3), 3.0f) + .fromCase( + VARIANT(), + Variant.newBuilder().of(new BigDecimal("123.456")), + 123.456f) + // a magnitude a FLOAT cannot represent is still rejected + .fail( + VARIANT(), + Variant.newBuilder().of(1e40d), + TableRuntimeException.class) .fail(VARIANT(), Variant.newBuilder().of("x"), TableRuntimeException.class), CastTestSpecBuilder.testCastTo(DOUBLE()) .fromCase(VARIANT(), Variant.newBuilder().of(1.5d), 1.5d) .fromCase(VARIANT(), Variant.newBuilder().of(1.5f), 1.5d) .fromCase(VARIANT(), Variant.newBuilder().of(3), 3.0d) + .fromCase( + VARIANT(), + Variant.newBuilder().of(new BigDecimal("123.456")), + 123.456d) .fail(VARIANT(), Variant.newBuilder().of("x"), TableRuntimeException.class), CastTestSpecBuilder.testCastTo(DECIMAL(5, 2)) .fromCase(VARIANT(), null, null) @@ -1602,10 +1660,23 @@ Stream testCases() { VARIANT(), Variant.newBuilder().of(new BigDecimal("123.45")), DecimalData.fromBigDecimal(new BigDecimal("123.45"), 5, 2)) + // trailing zeros may be appended to reach the target scale + .fromCase( + VARIANT(), + Variant.newBuilder().of(new BigDecimal("123.4")), + DecimalData.fromBigDecimal(new BigDecimal("123.40"), 5, 2)) + // an integer is exact, so it converts when it fits .fromCase( VARIANT(), Variant.newBuilder().of(42), - DecimalData.fromBigDecimal(new BigDecimal("42"), 5, 2)) + DecimalData.fromBigDecimal(new BigDecimal("42.00"), 5, 2)) + // a scale that would have to round is rejected + .fail( + VARIANT(), + Variant.newBuilder().of(new BigDecimal("123.456")), + TableRuntimeException.class) + // an approximate kind is not read as a decimal + .fail(VARIANT(), Variant.newBuilder().of(1.5d), TableRuntimeException.class) .fail(VARIANT(), Variant.newBuilder().of("x"), TableRuntimeException.class), CastTestSpecBuilder.testCastTo(BYTES()) .fromCase(VARIANT(), null, null) @@ -1630,13 +1701,55 @@ Stream testCases() { Variant.newBuilder().of(LocalDateTime.of(2020, 1, 1, 12, 0, 0)), TimestampData.fromLocalDateTime( LocalDateTime.of(2020, 1, 1, 12, 0, 0))) - .fail(VARIANT(), Variant.newBuilder().of(1), TableRuntimeException.class), + .fail(VARIANT(), Variant.newBuilder().of(1), TableRuntimeException.class) + // a TIMESTAMP_LTZ is a different kind and is not read as a TIMESTAMP + .fail( + VARIANT(), + Variant.newBuilder().of(Instant.ofEpochSecond(1_600_000_000L)), + TableRuntimeException.class), + // A variant keeps microseconds, so a lower target precision is only allowed when + // the + // fractional seconds fit it. Truncating them away would change the value. + CastTestSpecBuilder.testCastTo(TIMESTAMP(3)) + .fromCase( + VARIANT(), + Variant.newBuilder().of(LocalDateTime.of(2020, 1, 1, 12, 0, 0)), + TimestampData.fromLocalDateTime( + LocalDateTime.of(2020, 1, 1, 12, 0, 0))) + .fromCase( + VARIANT(), + Variant.newBuilder() + .of(LocalDateTime.of(2020, 1, 1, 12, 0, 0, 123000000)), + TimestampData.fromLocalDateTime( + LocalDateTime.of(2020, 1, 1, 12, 0, 0, 123000000))) + .fail( + VARIANT(), + Variant.newBuilder() + .of(LocalDateTime.of(2020, 1, 1, 12, 0, 0, 123456000)), + TableRuntimeException.class), + CastTestSpecBuilder.testCastTo(TIMESTAMP(0)) + .fail( + VARIANT(), + Variant.newBuilder() + .of(LocalDateTime.of(2020, 1, 1, 12, 0, 0, 123000000)), + TableRuntimeException.class), CastTestSpecBuilder.testCastTo(TIMESTAMP_LTZ()) .fromCase( VARIANT(), Variant.newBuilder().of(Instant.ofEpochSecond(1_600_000_000L)), TimestampData.fromInstant(Instant.ofEpochSecond(1_600_000_000L))) - .fail(VARIANT(), Variant.newBuilder().of(1), TableRuntimeException.class)); + .fail(VARIANT(), Variant.newBuilder().of(1), TableRuntimeException.class) + // a TIMESTAMP is not read as a TIMESTAMP_LTZ either + .fail( + VARIANT(), + Variant.newBuilder().of(LocalDateTime.of(2020, 1, 1, 12, 0, 0)), + TableRuntimeException.class), + CastTestSpecBuilder.testCastTo(TIMESTAMP_LTZ(3)) + .fail( + VARIANT(), + Variant.newBuilder() + .of(Instant.ofEpochSecond(1_600_000_000L, 123456000)), + TableRuntimeException.class)); } @TestFactory diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java new file mode 100644 index 00000000000000..6f80a9536965e2 --- /dev/null +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java @@ -0,0 +1,340 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.runtime.functions; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.table.api.TableRuntimeException; +import org.apache.flink.table.data.DecimalData; +import org.apache.flink.table.data.TimestampData; +import org.apache.flink.table.utils.DateTimeUtils; +import org.apache.flink.types.variant.Variant; + +import java.math.BigDecimal; +import java.math.RoundingMode; +import java.time.Instant; +import java.time.LocalDateTime; +import java.util.TimeZone; + +/** + * Runtime helpers for casting a {@code VARIANT} value to a SQL type. + * + *

A cast succeeds only when the target holds the stored value without altering it, so a value is + * never wrapped, rounded, truncated, or padded to make it fit. Any numeric kind therefore reaches + * an integer target as long as the value is integral and in range. {@code FLOAT} and {@code DOUBLE} + * are the exception to exactness: they are approximate by definition, so they accept any numeric + * kind and reject only a magnitude they cannot represent at all. + */ +@Internal +public final class VariantCastUtils { + + /** + * The magnitude 2^63, the exclusive bound for a {@code double} that still fits a {@code long}. + * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63, whereas widening {@link + * Long#MAX_VALUE} would reach the same number only by rounding up. + */ + private static final double LONG_MAGNITUDE_LIMIT = -(double) Long.MIN_VALUE; + + /** A variant stores a timestamp with microsecond precision. */ + private static final int TIMESTAMP_PRECISION = 6; + + private VariantCastUtils() {} + + /** + * Reads a numeric variant as a {@code long} and checks it against the target range. An + * approximate or decimal value is accepted only when it is already integral, so nothing is + * rounded away. + */ + public static long toIntegral(Variant variant, long min, long max, String targetType) { + final long value; + switch (variant.getType()) { + case TINYINT: + case SMALLINT: + case INT: + case BIGINT: + value = ((Number) variant.get()).longValue(); + break; + case FLOAT: + case DOUBLE: + final double approximate = ((Number) variant.get()).doubleValue(); + // Below 2^63 the narrowing conversion stays exact. The comparison is negated so + // that NaN fails it too. + if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) { + throw overflow(approximate, targetType); + } + value = (long) approximate; + if (value != approximate) { + throw lossyCast(approximate, targetType); + } + break; + case DECIMAL: + final BigDecimal decimal = variant.getDecimal(); + final BigDecimal integral; + try { + // UNNECESSARY throws unless the value is already integral. + integral = decimal.setScale(0, RoundingMode.UNNECESSARY); + } catch (ArithmeticException e) { + throw lossyCast(decimal, targetType); + } + try { + // longValueExact rejects a value that does not fit a long instead of returning + // its low-order bits. + value = integral.longValueExact(); + } catch (ArithmeticException e) { + throw overflow(decimal, targetType); + } + break; + default: + throw unsupportedKind(variant, targetType); + } + if (value < min || value > max) { + throw overflow(value, targetType); + } + return value; + } + + /** + * Reads any numeric variant as a {@code float}. Dropping decimal digits is expected of an + * approximate type, but a magnitude outside the {@code FLOAT} range is rejected. + */ + public static float toFloat(Variant variant) { + final float value = numeric(variant, "FLOAT").floatValue(); + if (!Float.isFinite(value)) { + throw overflow(variant.get(), "FLOAT"); + } + return value; + } + + /** Reads any numeric variant as a {@code double}. See {@link #toFloat(Variant)}. */ + public static double toDouble(Variant variant) { + final double value = numeric(variant, "DOUBLE").doubleValue(); + if (!Double.isFinite(value)) { + throw overflow(variant.get(), "DOUBLE"); + } + return value; + } + + /** + * Reads an integer or decimal variant as the target {@code DECIMAL}. The value has to fit the + * precision and scale without rounding, although trailing zeros may be appended to reach the + * scale. + */ + public static DecimalData toDecimal(Variant variant, int precision, int scale) { + final BigDecimal value; + switch (variant.getType()) { + case TINYINT: + case SMALLINT: + case INT: + case BIGINT: + value = BigDecimal.valueOf(((Number) variant.get()).longValue()); + break; + case DECIMAL: + value = variant.getDecimal(); + break; + default: + throw unsupportedKind(variant, decimalTarget(precision, scale)); + } + // The integral part must fit the digits the target reserves for it. + if (value.precision() - value.scale() > precision - scale) { + throw overflow(value, decimalTarget(precision, scale)); + } + final BigDecimal rescaled; + try { + // UNNECESSARY throws unless the value fits the target scale exactly. + rescaled = value.setScale(scale, RoundingMode.UNNECESSARY); + } catch (ArithmeticException e) { + throw lossyCast(value, decimalTarget(precision, scale)); + } + final DecimalData decimal = DecimalData.fromBigDecimal(rescaled, precision, scale); + if (decimal == null) { + throw overflow(value, decimalTarget(precision, scale)); + } + return decimal; + } + + private static String decimalTarget(int precision, int scale) { + return String.format("DECIMAL(%d, %d)", precision, scale); + } + + /** + * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant keeps microseconds, so + * the value is accepted only when its fractional seconds fit the target precision. + */ + public static TimestampData toTimestamp(Variant variant, int precision) { + if (variant.getType() != Variant.Type.TIMESTAMP) { + throw unsupportedKind(variant, String.format("TIMESTAMP(%d)", precision)); + } + final LocalDateTime value = variant.getDateTime(); + checkFractionFits(value.getNano(), precision, value, "TIMESTAMP"); + return TimestampData.fromLocalDateTime(value); + } + + /** Reads a timestamp with local time zone variant. See {@link #toTimestamp(Variant, int)}. */ + public static TimestampData toTimestampLtz(Variant variant, int precision) { + if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) { + throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)", precision)); + } + final Instant value = variant.getInstant(); + checkFractionFits(value.getNano(), precision, value, "TIMESTAMP_LTZ"); + return TimestampData.fromInstant(value); + } + + /** + * Reads a binary variant, enforcing {@code targetLength} strictly with no padding or truncation + * ({@code BINARY} requires an exact length, {@code VARBINARY} an upper bound). + */ + public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLength) { + final byte[] value = variant.getBytes(); + final boolean fits = + fixedLength ? value.length == targetLength : value.length <= targetLength; + if (!fits) { + throw new TableRuntimeException( + String.format( + "The VARIANT binary value of length %d does not fit %s(%d); VARIANT " + + "casts do not pad or truncate.", + value.length, fixedLength ? "BINARY" : "VARBINARY", targetLength)); + } + return value; + } + + /** + * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular + * SQL cast of the stored kind would. {@code targetLength} is enforced strictly with no padding + * or truncation ({@code CHAR} requires an exact length, {@code VARCHAR} an upper bound). + * + * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value + */ + public static String toStringValue( + Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) { + final String value; + switch (variant.getType()) { + case BOOLEAN: + value = variant.getBoolean() ? "TRUE" : "FALSE"; + break; + case TINYINT: + case SMALLINT: + case INT: + case BIGINT: + case FLOAT: + case DOUBLE: + case DECIMAL: + value = variant.get().toString(); + break; + case STRING: + value = variant.getString(); + break; + case DATE: + value = DateTimeUtils.formatDate((int) variant.getDate().toEpochDay()); + break; + case TIMESTAMP: + // A wall-clock value needs no zone shift, which is what UTC_ZONE achieves here. A + // variant keeps microseconds, so the precision is always 6. + value = + DateTimeUtils.formatTimestamp( + TimestampData.fromLocalDateTime(variant.getDateTime()), + DateTimeUtils.UTC_ZONE, + TIMESTAMP_PRECISION); + break; + case TIMESTAMP_LTZ: + value = + DateTimeUtils.formatTimestamp( + TimestampData.fromInstant(variant.getInstant()), + sessionZone, + TIMESTAMP_PRECISION); + break; + case NULL: + // Only reachable for a NOT NULL target. A nullable target maps a null-valued + // variant to SQL NULL before this method is called. + throw new TableRuntimeException( + String.format( + "Cannot cast a VARIANT null value to %s because the target does not " + + "accept NULL.", + characterTarget(targetLength, charTarget))); + default: + // An object, array, or binary value has no scalar rendering. + throw new TableRuntimeException( + String.format( + "Cannot cast a VARIANT %s value to a character string. Use the " + + "JSON_STRING function to obtain its JSON representation.", + variant.getType())); + } + final boolean fits = + charTarget ? value.length() == targetLength : value.length() <= targetLength; + if (!fits) { + throw new TableRuntimeException( + String.format( + "The VARIANT string value of length %d does not fit %s; VARIANT string " + + "casts do not pad or truncate.", + value.length(), characterTarget(targetLength, charTarget))); + } + return value; + } + + private static String characterTarget(int targetLength, boolean charTarget) { + return String.format("%s(%d)", charTarget ? "CHAR" : "VARCHAR", targetLength); + } + + private static Number numeric(Variant variant, String targetType) { + switch (variant.getType()) { + case TINYINT: + case SMALLINT: + case INT: + case BIGINT: + case FLOAT: + case DOUBLE: + case DECIMAL: + return (Number) variant.get(); + default: + throw unsupportedKind(variant, targetType); + } + } + + private static void checkFractionFits( + int nanos, int precision, Object value, String targetType) { + // A precision of p keeps the first p fractional digits, so the remaining ones must be zero. + long unit = 1; + for (int i = precision; i < 9; i++) { + unit *= 10; + } + if (nanos % unit != 0) { + throw lossyCast(value, String.format("%s(%d)", targetType, precision)); + } + } + + private static TableRuntimeException unsupportedKind(Variant variant, String targetType) { + return new TableRuntimeException( + String.format( + "Cannot cast a VARIANT %s value to %s. A VARIANT cast does not change the " + + "type of the stored value, so cast it to its own type first and " + + "then convert with a regular cast.", + variant.getType(), targetType)); + } + + private static TableRuntimeException overflow(Object value, String targetType) { + return new TableRuntimeException( + String.format("Casting the VARIANT value %s to %s overflowed.", value, targetType)); + } + + private static TableRuntimeException lossyCast(Object value, String targetType) { + return new TableRuntimeException( + String.format( + "Casting the VARIANT value %s to %s would lose precision. Cast it to a type " + + "that holds the value exactly first, then narrow if needed.", + value, targetType)); + } +}