diff --git a/converters/salesforce/README.md b/converters/salesforce/README.md index 3b11559c..77e81464 100644 --- a/converters/salesforce/README.md +++ b/converters/salesforce/README.md @@ -42,7 +42,8 @@ This produces a self-contained executable jar at `target/ossie-salesforce-conver ## Setup -Both schemas must be obtained and placed under `src/main/resources/schemas/` before building, so they get bundled into the jar. +Obtain the Salesforce schema before building so it is bundled into the jar. +Maven copies the canonical Ossie schema from `../../core-spec/ossie-schema.json`. ### Salesforce Semantic Model Schema @@ -50,11 +51,15 @@ Both schemas must be obtained and placed under `src/main/resources/schemas/` bef 2. Copy the JSON schema content from the page 3. Save it to `src/main/resources/schemas/salesforce-semantic-model-schema.json` -### Apache Ossie Schema +Run the complete suite, including Salesforce schema checks, with: -1. Visit the [Ossie schema on GitHub](https://github.com/apache/ossie/blob/main/core-spec/ossie-schema.json) -2. Copy the raw JSON contents -3. Save it to `src/main/resources/schemas/ossie-schema.json` +```bash +mvn -DrequireSalesforceSchema=true clean verify +``` + +The property makes a missing Salesforce schema fail the test run. Without it, +schema-dependent tests retain their existing skip behavior. `verify` also checks +Apache license headers. Do not commit downloaded schemas. ## Usage @@ -168,7 +173,7 @@ ossieToSf.convert(Paths.get("input/model.yaml"), Paths.get("output/")); | Field `datatype` | Field `dataType` when a safe mapping exists | | `relationships[]` | `semanticRelationships[]` | | `from_columns` + `to_columns` | `criteria[]` | -| `metrics[]` | Not currently exported | +| `metrics[]` | Validated Tua expressions in `semanticCalculatedMeasurements[]` | | `ai_context` | `businessPreferences` | | `custom_extensions` (vendor: `SALESFORCE`) | Restored properties | @@ -229,6 +234,127 @@ dimensions. **Unsupported relationships** (containing Formula or SemanticField types) are stored in `custom_extensions` at the model level rather than being converted to Ossie relationships. +### Metric expressions + +Metrics select `TABLEAU`, then `SNOWFLAKE`, then `ANSI_SQL`, independent of entry +order. The selected expression is parsed and validated; an invalid preferred +expression fails rather than falling back to another dialect. Duplicate selected +dialect entries are errors. Successful conversion exports every declared metric. + +The target is the Salesforce/Tableau Next semantic model's +[Tua calculation language](https://developer.salesforce.com/docs/data/semantic-layer/guide/query-api-in-depth-functions.html). +Calculated measurements emit `syntax: Tua`, `dataType: Number`, and +`aggregationType: UserAgg`, so an already aggregated formula is not aggregated +again. See [calculated fields](https://developer.salesforce.com/docs/data/semantic-layer/guide/query-api-in-depth-calculated-fields.html) +and [aggregation rules](https://developer.salesforce.com/docs/data/semantic-layer/guide/query-api-in-depth-aggregation.html). + +| SQL input | Tua output | +|-----------|------------| +| `SUM`, `AVG`, `MIN`, `MAX`, `COUNT(field)` | Same aggregate | +| `COUNT(DISTINCT field)` | `COUNTD(field)` | +| `+`, `-`, `*`, `/`, parentheses, numeric constants | Explicitly grouped arithmetic | +| Searched `CASE WHEN` | `IF … THEN … ELSEIF … ELSE … END` | +| Comparisons, `AND`, `OR`, `NOT` | Equivalent grouped operators | +| `COALESCE(a, b, …)` | Nested `IFNULL` | +| `NULLIF(a, b)` | `IF a = b THEN NULL ELSE a END` | +| `IS NULL`, `IS NOT NULL` | `ISNULL`, `NOT ISNULL` | +| `ABS`, `ROUND`, `CEIL`, `FLOOR` | `ABS`, `ROUND`, `CEILING`, `FLOOR` | + +These constructs compose. For example, with declared numeric fields `profit` and +`revenue` in dataset `orders`: + +```yaml +metrics: + - name: margin + datatype: Decimal + expression: + dialects: + - dialect: SNOWFLAKE + expression: SUM(orders.profit) / NULLIF(SUM(orders.revenue), 0) +``` + +The resulting expression is: + +```text +(SUM([orders].[profit]) / (IF (SUM([orders].[revenue]) = 0) THEN NULL ELSE SUM([orders].[revenue]) END)) +``` + +**Binding and types.** References resolve against declared dataset and field +names, then against fields actually emitted to Salesforce. Physical column names +and source paths are not aliases. Unqualified SQL fields must be unique. Regular +SQL names normalize to uppercase; double-quoted names match the normalized +declaration exactly, following the [expression specification](../../core-spec/expression_language.md). +Thus `"ORDERS"."AMOUNT"` matches regular declarations `orders.amount`, while +`"orders"."amount"` requires explicitly quoted lowercase declarations. Target +API names are preserved; conversion does not rename fields or discover columns. +Names containing brackets or control characters fail because their Tua escaping +is not established. Referenced fields must also have a single, unqualified physical +column binding in the exported model. This rejects derived expressions such as +`profit+tax` that the existing field mapper can misclassify as physical columns. +Derived-field compilation and normalization of qualified/quoted physical bindings +belong to separate converter work. + +`TABLEAU` references use exact `[dataset].[field]` API names. Its supported formula +subset is the Tua equivalents above, including `IF`, `IFNULL`, `ISNULL`, `COUNTD` +and `CEILING`. Existing `ANSI_SQL` expressions using complete bracket notation +retain that spelling as a compatibility case and receive the same validation. +Bracket notation is not accepted as Snowflake SQL. + +Fields need a known compatible datatype, either declared in OSI or restored from +an existing Salesforce field type. Arithmetic and `SUM`/`AVG` require numbers; +`MIN`/`MAX` also permit text and temporal values inside numeric calculations. +Comparisons and conditional/null-handling branches must have compatible types. +Metrics must return numbers; a declared `Integer` result cannot conceal a +fractional expression. Missing metric types are inferred. A formula must be +aggregated or constant: mixed row/aggregate expressions, nested aggregates, and +aggregates without a dataset field fail. A single aggregate cannot combine fields +from multiple datasets. Separate aggregates can use datasets connected through +explicitly enabled exported relationships. Each usable edge must match one source +relationship by name, endpoints and ordered join-key pairs; join fields must resolve +as direct fields. Missing or corrupted edges cannot establish connectivity. These +checks protect metrics without changing the relationship mapper or proving join grain. + +**Limits and compatibility.** `COUNT`/`COUNTD` take a field. `ROUND` supports one +argument or a second integer-literal precision; rounding-mode overloads are not +supported. `CEIL`/`FLOOR` take one argument. They and `ROUND` at zero or negative +precision infer integral values, so compatible `Integer` metrics are accepted. Simple `CASE`, date/time and string +functions, casts, metric/calculated-field references, windows, LOD, `COUNT(*)`, +SQL comments and backslash string escapes are outside this subset. String and +Boolean literals are supported in predicates. No null-to-zero setting or implicit +cast is added. Errors name the metric and explain the rejected construct or +reference. Literal zero divisors fail; use `NULLIF` to make a zero denominator +nullable. Expressions have bounded size, nesting and generated output. + +This replaces the unvalidated SQL fallback introduced in +[#402](https://github.com/apache/ossie/pull/402). Previously accepted invalid, +unsupported or untyped formulas now fail, including invalid `TABLEAU` input. +Metric/model extension preservation remains owned by #402. CLI conversion errors +are printed to stderr with exit code 3; this small change overlaps the conversion +error handling in [#286](https://github.com/apache/ossie/pull/286). + +**Implementation choice.** The existing mapping pipeline calls a focused Java +metric compiler. JSqlParser 5.3 parses SQL; a bounded Tua frontend handles native +formulas. Both produce a small immutable metric AST. Type/aggregation checks finish +before the private emitter writes Tua. No Python runtime is needed. JSqlParser is +used under its Apache-2.0 option; its unused JMH benchmark dependency is excluded. + +The implementation has five components: `SqlMetricExpressionParser`, +`TuaMetricExpressionParser`, `MetricExpression`, `MetricExpressionTranslator` and +`MetricFieldResolver`. The resolver caches successful bindings and verified graph +reachability within one model. To add supported syntax, update the relevant +frontend and metric rule with composition and semantic tests. There is no generic +model-planning or deployment framework in this change. + +**Validation boundary.** Unit tests cover parsing, binding and failure behavior. +Independent local Tua evaluation tests use synthetic rows with nulls, duplicates, +empty inputs and zero denominators. Those tests model the intended semantics; +they are not native Tableau Next execution. The published Salesforce **output** +schema checks structure, not formula syntax, catalog bindings or authoring API +acceptance. Before deployment, validate authoring and native queries in a test +org, including numeric precision, rounding ties, empty groups, null behavior and +unguarded dynamic division and multi-dataset grain. This converter does not provision Data 360 bindings or enrich +missing fields. + ## Architecture ``` diff --git a/converters/salesforce/pom.xml b/converters/salesforce/pom.xml index 396c9bdc..2bb0d20b 100644 --- a/converters/salesforce/pom.xml +++ b/converters/salesforce/pom.xml @@ -52,9 +52,23 @@ 1.5.9 3.6.2 3.5.0 + 5.3 + + com.github.jsqlparser + jsqlparser + ${jsqlparser.version} + + + + org.openjdk.jmh + jmh-core + + + com.fasterxml.jackson.core jackson-databind diff --git a/converters/salesforce/src/main/java/org/apache/ossie/app/OssieSalesforceConverter.java b/converters/salesforce/src/main/java/org/apache/ossie/app/OssieSalesforceConverter.java index ee44b27f..c6e59090 100644 --- a/converters/salesforce/src/main/java/org/apache/ossie/app/OssieSalesforceConverter.java +++ b/converters/salesforce/src/main/java/org/apache/ossie/app/OssieSalesforceConverter.java @@ -52,6 +52,7 @@ public static void main(String[] args) { } catch (InvalidInputException e) { System.exit(2); } catch (ConversionException e) { + System.err.println("Error: " + e.getMessage()); System.exit(3); } } diff --git a/converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterConstants.java b/converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterConstants.java index 044dfe4c..02c0bb26 100644 --- a/converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterConstants.java +++ b/converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterConstants.java @@ -89,6 +89,7 @@ public enum Level { public static final String DIALECTS = "dialects"; public static final String DIALECT = "dialect"; public static final String DIALECT_TABLEAU = "TABLEAU"; + public static final String DIALECT_ANSI_SQL = "ANSI_SQL"; // Relationship properties public static final String CRITERIA = "criteria"; diff --git a/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricExpression.java b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricExpression.java new file mode 100644 index 00000000..9d84ca9b --- /dev/null +++ b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricExpression.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import java.math.BigDecimal; +import java.util.List; +import java.util.Set; + +/** Immutable compiler nodes. Neither source parsing nor type checking emits target text. */ +final class MetricExpression { + private MetricExpression() {} + sealed interface Node permits Literal, Field, Unary, Binary, Call, Conditional {} + record Literal(Object value) implements Node {} + record Field(List parts, boolean tableau) implements Node { + Field { parts = List.copyOf(parts); } + } + record Unary(String operator, Node operand) implements Node {} + record Binary(String operator, Node left, Node right) implements Node {} + record Call(String name, List arguments, boolean distinct) implements Node { + Call { arguments = List.copyOf(arguments); } + } + /** Alternating predicate/result pairs, followed by a separate ELSE expression. */ + record Conditional(List branches, Node otherwise) implements Node { + Conditional { branches = List.copyOf(branches); } + } + enum Level { CONSTANT, ROW, AGGREGATE } + enum Type { + INTEGER("Integer"), DECIMAL("Decimal"), FLOAT("Float"), STRING("String"), + BOOLEAN("Boolean"), DATE("Date"), DATETIME("DateTime"), DATETIME_TZ("DateTimeTz"), + NULL(null), UNKNOWN(null); + final String datatype; + Type(String datatype) { this.datatype = datatype; } + boolean numeric() { return this == INTEGER || this == DECIMAL || this == FLOAT; } + static Type of(String datatype) { + if (datatype == null) return UNKNOWN; + for (Type type : values()) if (datatype.equals(type.datatype)) return type; + return UNKNOWN; + } + } + record Typed(Node node, Type type, Level level, Set datasets, + List children, MetricFieldResolver.ResolvedField binding, BigDecimal number) { + Typed { datasets = Set.copyOf(datasets); children = List.copyOf(children); } + } +} diff --git a/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricExpressionTranslator.java b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricExpressionTranslator.java new file mode 100644 index 00000000..33160177 --- /dev/null +++ b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricExpressionTranslator.java @@ -0,0 +1,329 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import static org.apache.ossie.converter.MetricExpression.*; +import static org.apache.ossie.util.DataStructureUtils.*; + +import java.math.BigDecimal; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.ossie.exception.ConversionException; + +/** Compiles the documented metric subset: parse, bind/check, then emit bounded Tua text. */ +final class MetricExpressionTranslator { + private static final List DIALECTS = List.of("TABLEAU", "SNOWFLAKE", "ANSI_SQL"); + private static final Set AGGREGATES = Set.of("SUM", "AVG", "MIN", "MAX", "COUNT", "COUNTD"); + record Result(String expression, String dataType) {} + private MetricExpressionTranslator() {} + + static Result translate(Map metric, Map sourceModel, + Map targetModel) { + return translate(metric, new MetricFieldResolver(sourceModel, targetModel)); + } + + static Result translate(Map metric, MetricFieldResolver resolver) { + String name = getString(metric, "name"); + try { + Map expression = getMap(metric, "expression"); + List dialects = expression == null ? null : getList(expression, "dialects"); + if (dialects == null) { + throw new IllegalArgumentException("missing expression.dialects; provide TABLEAU, SNOWFLAKE or ANSI_SQL"); + } + Map candidates = new java.util.LinkedHashMap<>(); + for (Object entry : dialects) { + Map value = asMap(entry); + String dialect = getString(value, "dialect"); + if (DIALECTS.contains(dialect)) { + if (candidates.containsKey(dialect)) { + throw new IllegalArgumentException("ambiguous expression: multiple " + dialect + " entries"); + } + candidates.put(dialect, getString(value, "expression")); + } + } + String dialect = DIALECTS.stream().filter(candidates::containsKey).findFirst() + .orElseThrow(() -> new IllegalArgumentException( + "no supported dialect; provide TABLEAU, SNOWFLAKE or ANSI_SQL")); + String text = candidates.get(dialect); + if (text == null || text.isBlank()) { + throw new IllegalArgumentException(dialect + " expression is empty"); + } + Node parsed = dialect.equals("TABLEAU") + ? new TuaMetricExpressionParser(TuaMetricExpressionParser.tokenize(text, dialect)).parse() : SqlMetricExpressionParser.parse(text, dialect); + Typed result = new Analyzer(dialect, resolver).analyze(parsed); + resolver.validateDatasets(result.datasets()); + Type declared = Type.of(getString(metric, "datatype")); + if (metric.containsKey("datatype") && declared == Type.UNKNOWN) { + throw new IllegalArgumentException("unsupported metric datatype " + getString(metric, "datatype")); + } + if (!result.type().numeric() && result.type() != Type.NULL) { + throw new IllegalArgumentException("calculated measurements must be numeric, found " + result.type()); + } + if (declared != Type.UNKNOWN && (!declared.numeric() + || (declared == Type.INTEGER && result.type() != Type.INTEGER && result.type() != Type.NULL))) { + throw new IllegalArgumentException("datatype " + getString(metric, "datatype") + + " is incompatible with expression result " + result.type()); + } + if (result.type() == Type.NULL && !declared.numeric()) { + throw new IllegalArgumentException("all-null result needs an explicit numeric datatype"); + } + if (result.level() == Level.ROW) { + throw new IllegalArgumentException("unaggregated field in metric; use an explicit aggregate"); + } + return new Result(new Emitter().emit(result), "Number"); + } catch (IllegalArgumentException e) { + throw new ConversionException("Metric '" + name + "': " + e.getMessage(), e); + } + } + + /** Checks the complete tree before any target text is emitted. */ + private static final class Analyzer { + private final String dialect; + private final MetricFieldResolver resolver; + private int depth; + Analyzer(String dialect, MetricFieldResolver resolver) { + this.dialect = dialect; this.resolver = resolver; + } + Typed analyze(Node node) { + if (++depth > 128) throw new IllegalArgumentException("expression nesting exceeds 128 levels"); + try { return analyzeNode(node); } finally { depth--; } + } + private Typed analyzeNode(Node node) { + if (node instanceof Literal literal) { + Object value = literal.value(); + Type type = value == null ? Type.NULL : value instanceof Boolean ? Type.BOOLEAN + : value instanceof String ? Type.STRING + : ((BigDecimal) value).stripTrailingZeros().scale() <= 0 ? Type.INTEGER : Type.DECIMAL; + return new Typed(node, type, Level.CONSTANT, Set.of(), List.of(), null, + value instanceof BigDecimal number ? number : null); + } + if (node instanceof Field field) { + MetricFieldResolver.ResolvedField binding = resolver.resolve(field.parts(), field.tableau()); + if (binding == null || binding.expression() == null || binding.expression().isBlank()) { + throw new IllegalArgumentException("field resolver returned no binding"); + } + Type type = Type.of(binding.datatype()); + if (type == Type.UNKNOWN) throw new IllegalArgumentException("field reference needs known field datatypes"); + return new Typed(node, type, Level.ROW, Set.of(binding.dataset()), List.of(), binding, null); + } + if (node instanceof Unary unary) { + Typed child = analyze(unary.operand()); + String operator = unary.operator(); + Type result = child.type(); + BigDecimal number = child.number(); + if (operator.equals("ISNULL")) { result = Type.BOOLEAN; number = null; } + else if (operator.equals("NOT")) { require(child, Type.BOOLEAN, "NOT"); result = Type.BOOLEAN; number = null; } + else { numeric(child, "unary " + operator); if (operator.equals("-") && number != null) number = number.negate(); } + return new Typed(node, result, child.level(), child.datasets(), List.of(child), null, number); + } + if (node instanceof Binary binary) { + Typed left = analyze(binary.left()); Typed right = analyze(binary.right()); + String op = binary.operator(); + Type result; + if (op.equals("AND") || op.equals("OR")) { + require(left, Type.BOOLEAN, op); require(right, Type.BOOLEAN, op); result = Type.BOOLEAN; + } else if (Set.of("=", "!=", "<", "<=", ">", ">=").contains(op)) { + compatible(left.type(), right.type(), "comparison"); + if (!Set.of("=", "!=").contains(op) && (left.type() == Type.BOOLEAN || right.type() == Type.BOOLEAN)) { + throw new IllegalArgumentException("ordered comparison requires numeric, text or temporal operands"); + } + result = Type.BOOLEAN; + } else { + numeric(left, op); numeric(right, op); + result = compatible(left.type(), right.type(), op); + if (op.equals("/")) { + if (right.number() != null && right.number().signum() == 0) { + throw new IllegalArgumentException("division by literal zero; use NULLIF(denominator, 0) for a nullable denominator"); + } + result = Type.DECIMAL; + } + } + return compose(node, result, List.of(left, right)); + } + if (node instanceof Conditional conditional) { + List children = new ArrayList<>(); + Type result = Type.NULL; + for (int i = 0; i < conditional.branches().size(); i += 2) { + Typed predicate = analyze(conditional.branches().get(i)); + require(predicate, Type.BOOLEAN, "conditional predicate"); + Typed branch = analyze(conditional.branches().get(i + 1)); + result = compatible(result, branch.type(), "conditional branches"); + children.add(predicate); children.add(branch); + } + Typed otherwise = analyze(conditional.otherwise()); + result = compatible(result, otherwise.type(), "conditional branches"); + children.add(otherwise); + return compose(node, result, children); + } + Call call = (Call) node; + String name = call.name(); + if (call.distinct() && (!name.equals("COUNT") || dialect.equals("TABLEAU"))) { + throw new IllegalArgumentException("DISTINCT is supported only by SQL COUNT(DISTINCT field)"); + } + if (Set.of("COALESCE", "NULLIF", "CEIL").contains(name) && dialect.equals("TABLEAU") + || Set.of("IFNULL", "ISNULL", "CEILING", "COUNTD").contains(name) && !dialect.equals("TABLEAU")) { + throw new IllegalArgumentException(name + " is outside the supported " + dialect + " subset"); + } + int maximum = switch (name) { + case "COALESCE" -> Integer.MAX_VALUE; + case "IFNULL", "NULLIF", "ROUND" -> 2; + case "SUM", "AVG", "MIN", "MAX", "COUNT", "COUNTD", "ISNULL", "ABS", "CEIL", "CEILING", "FLOOR" -> 1; + default -> throw new IllegalArgumentException("unsupported function " + name); + }; + int minimum = Set.of("COALESCE", "IFNULL", "NULLIF").contains(name) ? 2 : 1; + if (call.arguments().size() < minimum || call.arguments().size() > maximum) { + throw new IllegalArgumentException(name + " expects " + + (minimum == maximum ? minimum : minimum + " to " + maximum) + " arguments"); + } + List arguments = call.arguments().stream().map(this::analyze).toList(); + if (AGGREGATES.contains(name)) return aggregate(call, arguments); + return switch (name) { + case "COALESCE", "IFNULL" -> { + Type result = Type.NULL; + for (Typed argument : arguments) result = compatible(result, argument.type(), name + " arguments"); + yield compose(node, result, arguments); + } + case "NULLIF" -> { + compatible(arguments.get(0).type(), arguments.get(1).type(), "NULLIF arguments"); + yield compose(node, arguments.get(0).type(), arguments); + } + case "ISNULL" -> compose(node, Type.BOOLEAN, arguments); + default -> numericFunction(call, arguments); + }; + } + + private Typed aggregate(Call call, List arguments) { + String name = call.name(); Typed argument = arguments.get(0); + if (argument.level() == Level.AGGREGATE) throw new IllegalArgumentException("nested aggregate " + name + " is unsupported"); + if (argument.datasets().isEmpty()) throw new IllegalArgumentException(name + " needs a declared field to establish its dataset"); + if (argument.datasets().size() > 1) throw new IllegalArgumentException("one aggregate cannot combine fields from multiple datasets"); + boolean count = name.equals("COUNT") || name.equals("COUNTD"); + if (count && !(argument.node() instanceof Field)) { + throw new IllegalArgumentException(name + " requires a declared field; counting expressions is unsupported"); + } + if (name.equals("MIN") || name.equals("MAX")) { + if (argument.type() == Type.BOOLEAN || argument.type() == Type.UNKNOWN) { + throw new IllegalArgumentException(name + " requires numeric, text or temporal operands"); + } + } else if (!count) numeric(argument, name); + Type result = count ? Type.INTEGER : name.equals("AVG") ? Type.DECIMAL : argument.type(); + return new Typed(call, result, Level.AGGREGATE, argument.datasets(), arguments, null, null); + } + private Typed numericFunction(Call call, List arguments) { + Typed value = arguments.get(0); numeric(value, call.name()); + BigDecimal places = BigDecimal.ZERO; + if (arguments.size() == 2) { + places = arguments.get(1).number(); + if (places == null || places.stripTrailingZeros().scale() > 0 + || places.compareTo(BigDecimal.valueOf(Integer.MIN_VALUE)) < 0 + || places.compareTo(BigDecimal.valueOf(Integer.MAX_VALUE)) > 0) { + throw new IllegalArgumentException("ROUND precision must be a 32-bit integer literal"); + } + } + Type result = value.type(); + if (result != Type.NULL && (Set.of("CEIL", "CEILING", "FLOOR").contains(call.name()) + || call.name().equals("ROUND") && places.signum() <= 0)) result = Type.INTEGER; + return compose(call, result, arguments); + } + private Typed compose(Node node, Type type, List arguments) { + Level level = Level.CONSTANT; Set datasets = new HashSet<>(); + for (Typed argument : arguments) { + if (level != Level.CONSTANT && argument.level() != Level.CONSTANT && level != argument.level()) { + throw new IllegalArgumentException("cannot mix aggregate and unaggregated field expressions"); + } + if (argument.level() != Level.CONSTANT) level = argument.level(); + datasets.addAll(argument.datasets()); + } + return new Typed(node, type, level, datasets, arguments, null, null); + } + private static void numeric(Typed value, String context) { + if (!value.type().numeric() && value.type() != Type.NULL) { + throw new IllegalArgumentException(context + " requires numeric operands, found " + value.type() + "; declare a compatible field datatype"); + } + } + private static void require(Typed value, Type expected, String context) { + if (value.type() != expected && value.type() != Type.NULL) throw new IllegalArgumentException(context + " requires " + expected + ", found " + value.type()); + } + private static Type compatible(Type left, Type right, String context) { + if (left == Type.UNKNOWN || right == Type.UNKNOWN) throw new IllegalArgumentException(context + " needs known field datatypes"); + if (left == Type.NULL) return right; + if (right == Type.NULL || left == right) return left; + if (left.numeric() && right.numeric()) return left == Type.FLOAT || right == Type.FLOAT ? Type.FLOAT : Type.DECIMAL; + throw new IllegalArgumentException(context + " has incompatible types " + left + " and " + right); + } + } + + /** Streams checked nodes so NULLIF expansion cannot allocate an unbounded string. */ + private static final class Emitter { + private static final int MAX_OUTPUT = 131072; + private final StringBuilder output = new StringBuilder(); + String emit(Typed expression) { append(expression); return output.toString(); } + private void text(String text) { + if ((long) output.length() + text.length() > MAX_OUTPUT) { + throw new IllegalArgumentException("translated expression exceeds 131072 characters"); + } + output.append(text); + } + private void append(Typed value) { + Node node = value.node(); List children = value.children(); + if (node instanceof Literal literal) { + Object content = literal.value(); + text(content == null ? "NULL" : content instanceof String string ? "'" + string.replace("'", "''") + "'" + : content instanceof Boolean bool ? bool ? "TRUE" : "FALSE" : ((BigDecimal) content).toPlainString()); + } else if (node instanceof Field) { + text(value.binding().expression()); + } else if (node instanceof Unary unary) { + switch (unary.operator()) { + case "+" -> append(children.get(0)); + case "-" -> { text("(-"); append(children.get(0)); text(")"); } + case "NOT" -> { text("(NOT "); append(children.get(0)); text(")"); } + case "ISNULL" -> { text("ISNULL("); append(children.get(0)); text(")"); } + default -> throw new IllegalStateException("unvalidated unary operator"); + } + } else if (node instanceof Binary binary) { + text("("); append(children.get(0)); text(" " + binary.operator() + " "); append(children.get(1)); text(")"); + } else if (node instanceof Conditional) { + text("(IF "); + for (int i = 0; i < children.size() - 1; i += 2) { + if (i > 0) text(" ELSEIF "); + append(children.get(i)); text(" THEN "); append(children.get(i + 1)); + } + text(" ELSE "); append(children.get(children.size() - 1)); text(" END)"); + } else { + Call call = (Call) node; + if (call.name().equals("COALESCE") || call.name().equals("IFNULL")) { + for (int i = 0; i < children.size() - 1; i++) { text("IFNULL("); append(children.get(i)); text(", "); } + append(children.get(children.size() - 1)); + for (int i = 0; i < children.size() - 1; i++) text(")"); + } else if (call.name().equals("NULLIF")) { + text("(IF ("); append(children.get(0)); text(" = "); append(children.get(1)); + text(") THEN NULL ELSE "); append(children.get(0)); text(" END)"); + } else { + text((call.distinct() ? "COUNTD" : (call.name().equals("CEIL") ? "CEILING" : call.name())) + "("); + for (int i = 0; i < children.size(); i++) { if (i > 0) text(", "); append(children.get(i)); } + text(")"); + } + } + } + } +} diff --git a/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricFieldResolver.java b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricFieldResolver.java new file mode 100644 index 00000000..46faf463 --- /dev/null +++ b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricFieldResolver.java @@ -0,0 +1,305 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import static org.apache.ossie.util.DataStructureUtils.getList; +import static org.apache.ossie.util.DataStructureUtils.getString; +import static org.apache.ossie.util.DataStructureUtils.streamMaps; + +import java.util.ArrayList; +import java.util.ArrayDeque; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; + +/** Binds metric references to declared fields that the Salesforce converter actually exported. */ +final class MetricFieldResolver { + + record Identifier(String text, boolean quoted) {} + + record ResolvedField(String expression, String datatype, String dataset) {} + + private record Field(Map dataset, Map field) {} + + private final List> datasets; + private final List> targetDatasets; + private final List> relationships; + private final List> declaredRelationships; + private record Reference(List parts, boolean tableau) { + Reference { parts = List.copyOf(parts); } + } + private final Map resolved = new HashMap<>(); + private final Map> reachable = new HashMap<>(); + private final List invalidRelationships = new ArrayList<>(); + private Map> graph; + + MetricFieldResolver(Map sourceModel, Map targetModel) { + datasets = items(sourceModel, "datasets"); + targetDatasets = items(targetModel, "semanticDataObjects"); + relationships = items(targetModel, "semanticRelationships"); + declaredRelationships = items(sourceModel, "relationships"); + } + + /** Only unchanged, enabled edges can establish a metric's dataset connectivity. */ + void validateDatasets(Set referenced) { + if (referenced.size() < 2) return; + if (graph == null) graph = validatedGraph(); + String first = referenced.stream().sorted().findFirst().orElseThrow(); + Set visited = reachable.computeIfAbsent(first, this::reachableFrom); + if (!visited.containsAll(referenced)) { + throw new IllegalArgumentException("Metric references disconnected datasets " + + referenced.stream().sorted().collect(Collectors.joining(", ")) + + "; declare supported relationships connecting them before exporting the metric" + + (invalidRelationships.isEmpty() ? "" : "; unusable relationships: " + String.join("; ", invalidRelationships))); + } + } + + private Map> validatedGraph() { + Map> result = new HashMap<>(); + for (Map dataset : targetDatasets) { + String name = getString(dataset, "apiName"); + if (name != null) result.put(name, new HashSet<>()); + } + for (Map relationship : relationships) { + if (!Boolean.TRUE.equals(relationship.get("isEnabled"))) continue; + String name = getString(relationship, "apiName"); + String left = getString(relationship, "leftSemanticDefinitionApiName"); + String right = getString(relationship, "rightSemanticDefinitionApiName"); + try { + if (!result.containsKey(left) || !result.containsKey(right)) { + throw new IllegalArgumentException("missing exported endpoint"); + } + validateRelationship(relationship, name, left, right); + result.get(left).add(right); + result.get(right).add(left); + } catch (IllegalArgumentException e) { + invalidRelationships.add("'" + name + "': " + e.getMessage()); + } + } + return result; + } + + private void validateRelationship(Map target, String name, String left, String right) { + List> matches = declaredRelationships.stream() + .filter(source -> name != null && name.equals(getString(source, "name"))).toList(); + if (matches.size() != 1) throw new IllegalArgumentException("expected one source relationship"); + Map source = matches.get(0); + if (!left.equals(getString(source, "from")) || !right.equals(getString(source, "to"))) { + throw new IllegalArgumentException("changed endpoints"); + } + List from = getList(source, "from_columns"); + List to = getList(source, "to_columns"); + List> criteria = items(target, "criteria"); + if (from == null || to == null || from.isEmpty() || from.size() != to.size() || from.size() != criteria.size()) { + throw new IllegalArgumentException("changed or missing composite join keys"); + } + for (int i = 0; i < from.size(); i++) { + Map pair = criteria.get(i); + if (!(from.get(i) instanceof String leftKey) || !(to.get(i) instanceof String rightKey) + || !leftKey.equals(pair.get("leftSemanticFieldApiName")) + || !rightKey.equals(pair.get("rightSemanticFieldApiName"))) { + throw new IllegalArgumentException("changed join key correspondence"); + } + for (String side : List.of("leftFieldType", "rightFieldType")) { + if (pair.containsKey(side) && !"TableField".equals(pair.get(side))) { + throw new IllegalArgumentException("calculated join keys are unsupported"); + } + } + resolve(List.of(new Identifier(left, true), new Identifier(leftKey, true)), true); + resolve(List.of(new Identifier(right, true), new Identifier(rightKey, true)), true); + } + } + + private Set reachableFrom(String start) { + Set visited = new HashSet<>(); + ArrayDeque pending = new ArrayDeque<>(); + pending.add(start); + while (!pending.isEmpty()) { + String dataset = pending.removeFirst(); + if (visited.add(dataset)) pending.addAll(graph.getOrDefault(dataset, Set.of())); + } + return Set.copyOf(visited); + } + + ResolvedField resolve(List parts, boolean tableau) { + return resolved.computeIfAbsent(new Reference(parts, tableau), key -> resolveUncached(key.parts(), key.tableau())); + } + + private ResolvedField resolveUncached(List parts, boolean tableau) { + String reference = parts.stream().map(Identifier::text).collect(Collectors.joining(".")); + if (parts.isEmpty() || parts.size() > 2) { + throw new IllegalArgumentException("Reference '" + reference + + "' must name a declared field or dataset.field; physical source paths are unsupported"); + } + if (tableau && parts.size() != 2) { + throw new IllegalArgumentException("TABLEAU field reference '" + reference + + "' must use [dataset].[field]"); + } + + List> candidates = datasets; + if (parts.size() == 2) { + candidates = datasets.stream() + .filter(dataset -> matches(parts.get(0), getString(dataset, "name"), tableau)) + .toList(); + if (candidates.isEmpty()) { + throw new IllegalArgumentException("Unknown dataset in reference '" + reference + + "'; use a declared dataset name, not its physical source"); + } + if (candidates.size() > 1) { + throw new IllegalArgumentException("Ambiguous dataset in reference '" + reference + + "'; dataset declarations must have distinct names"); + } + } + + Identifier fieldName = parts.get(parts.size() - 1); + List fields = new ArrayList<>(); + for (Map dataset : candidates) { + for (Map field : items(dataset, "fields")) { + if (matches(fieldName, getString(field, "name"), tableau)) { + fields.add(new Field(dataset, field)); + } + } + } + if (fields.isEmpty()) { + throw new IllegalArgumentException("Unknown field reference '" + reference + + "'; declare the field under datasets[].fields before exporting the metric"); + } + if (fields.size() > 1) { + throw new IllegalArgumentException("Ambiguous field reference '" + reference + + "'; qualify the dataset and remove duplicate field declarations"); + } + + Field match = fields.get(0); + String datasetName = getString(match.dataset(), "name"); + // An unqualified field must not accidentally select one of two equivalent datasets. + if (datasets.stream().filter(dataset -> equivalentDeclaration( + datasetName, getString(dataset, "name"), tableau)).count() > 1) { + throw new IllegalArgumentException("Ambiguous dataset for reference '" + reference + + "'; dataset declarations must have distinct names"); + } + String sourceFieldName = getString(match.field(), "name"); + Map targetDataset = exportedItem(targetDatasets, datasetName, + "dataset", reference); + List> targetFields = new ArrayList<>(items(targetDataset, "semanticDimensions")); + targetFields.addAll(items(targetDataset, "semanticMeasurements")); + Map targetField = exportedItem(targetFields, sourceFieldName, "field", reference); + + String datatype = getString(match.field(), "datatype"); + String targetType = getString(targetField, "dataType"); + if (datatype == null || datatype.isBlank()) { + datatype = SalesforceDataTypeMapper.toOssie(targetType); + } + if (SalesforceDataTypeMapper.toSalesforce(datatype) == null) { + throw new IllegalArgumentException("Field reference '" + reference + + "' has no supported datatype; declare a portable field datatype"); + } + if (targetType == null || targetType.isBlank() + || !SalesforceDataTypeMapper.areCompatible(datatype, targetType)) { + throw new IllegalArgumentException("Field reference '" + reference + "' has datatype '" + + datatype + "' but exported Salesforce dataType '" + targetType + + "'; use compatible field types"); + } + + validateDirectBinding(targetField, reference); + String targetDatasetName = getString(targetDataset, "apiName"); + String targetFieldName = getString(targetField, "apiName"); + return new ResolvedField(bracket(targetDatasetName) + "." + bracket(targetFieldName), + datatype, targetDatasetName); + } + + private static void validateDirectBinding(Map targetField, String reference) { + String column = getString(targetField, "dataObjectFieldName"); + try { + if (column == null || column.isBlank()) throw new IllegalArgumentException("missing physical column"); + MetricExpression.Node parsed = SqlMetricExpressionParser.parse(column, "ANSI_SQL"); + if (!(parsed instanceof MetricExpression.Field field) || field.parts().size() != 1 + || field.parts().get(0).quoted() || !column.equals(field.parts().get(0).text())) { + throw new IllegalArgumentException("expected one unquoted physical column"); + } + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException("Field reference '" + reference + + "' was not exported with a supported direct physical binding; derived, qualified or quoted " + + "field bindings need separate conversion support: " + e.getMessage(), e); + } + } + + private static Map exportedItem(List> items, + String name, String kind, String reference) { + List> matches = items.stream() + .filter(item -> name.equals(getString(item, "apiName"))).toList(); + if (matches.isEmpty()) { + throw new IllegalArgumentException("Declared " + kind + " in reference '" + reference + + "' was not exported as a direct Salesforce semantic " + kind + + "; calculated or omitted fields are unsupported in metric references"); + } + if (matches.size() > 1) { + throw new IllegalArgumentException("Ambiguous exported Salesforce " + kind + " for reference '" + + reference + "'; apiName values must be unique"); + } + return matches.get(0); + } + + private static boolean matches(Identifier reference, String declaration, boolean tableau) { + if (declaration == null) { + return false; + } + return tableau ? reference.text().equals(declaration) + : normalize(reference).equals(normalizeDeclaration(declaration)); + } + + private static boolean equivalentDeclaration(String first, String second, boolean tableau) { + return second != null && (tableau ? first.equals(second) + : normalizeDeclaration(first).equals(normalizeDeclaration(second))); + } + + private static String normalize(Identifier identifier) { + return identifier.quoted() ? identifier.text() : identifier.text().toUpperCase(Locale.ROOT); + } + + private static String normalizeDeclaration(String name) { + if (name.startsWith("\"") && name.endsWith("\"") && name.length() >= 2) { + String text = name.substring(1, name.length() - 1); + String unescaped = text.replace("\"\"", ""); + if (text.isEmpty() || unescaped.contains("\"")) { + throw new IllegalArgumentException("Invalid quoted declaration name '" + name + "'"); + } + return text.replace("\"\"", "\""); + } + return name.toUpperCase(Locale.ROOT); + } + + private static String bracket(String name) { + if (name == null || name.isBlank() || name.indexOf('[') >= 0 || name.indexOf(']') >= 0 + || name.chars().anyMatch(Character::isISOControl)) { + throw new IllegalArgumentException("Exported apiName '" + name + + "' cannot be represented safely in a TABLEAU field reference; rename it"); + } + return "[" + name + "]"; + } + + private static List> items(Map map, String key) { + List values = getList(map, key); + return values == null ? List.of() : streamMaps(values).toList(); + } +} diff --git a/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricMappingHandler.java b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricMappingHandler.java index 6add2b9b..5278abaf 100644 --- a/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricMappingHandler.java +++ b/converters/salesforce/src/main/java/org/apache/ossie/converter/MetricMappingHandler.java @@ -24,6 +24,7 @@ import org.apache.ossie.converter.ConverterConstants.Level; import org.apache.ossie.converter.pipeline.PipelineStep; +import org.apache.ossie.exception.ConversionException; import java.util.*; import org.apache.ossie.util.MappingUtils; @@ -76,11 +77,29 @@ private void mapOssieToSalesforce( return; } + Set names = new HashSet<>(); + for (Object metric : ossieMetrics) { + String name = getString(asMap(metric), NAME); + if (!names.add(name)) { + throw new ConversionException("Metric '" + name + "': duplicate metric name"); + } + } + // Filter mappings to get only metric-related entries Map metricMappings = MappingUtils.filterMappingsByPrefix(mappings, METRICS); + + Map mappedData = GenericMappingEngine.applyMappings(sourceData, metricMappings); metricMappings.keySet().forEach(mappings::remove); - logger.debug("Metrics are not mapped in Ossie to Salesforce direction"); + outputData.putAll(mappedData); + + List sfMetrics = getList(outputData, SEMANTIC_CALCULATED_MEASUREMENTS); + if (sfMetrics != null) { + unwrapExpressions(ossieMetrics, sfMetrics, sourceData, outputData); + } else if (!ossieMetrics.isEmpty()) { + throw new ConversionException("Metric '" + getString(asMap(ossieMetrics.get(0)), NAME) + + "': metric mappings produced no calculated measurements"); + } } /** @@ -118,6 +137,29 @@ private void mapSalesforceToOssie( } + /** + * Compiles each metric to Tua after fields have been mapped. Binding checks both + * the OSI declarations and the actual emitted fields, including their types. + */ + private void unwrapExpressions(List ossieMetrics, List sfMetrics, + Map sourceData, Map outputData) { + if (ossieMetrics.size() != sfMetrics.size()) { + throw new ConversionException("Metric export count differs from declared metrics: " + + streamMaps(ossieMetrics).map(metric -> getString(metric, NAME)).toList()); + } + MetricFieldResolver resolver = new MetricFieldResolver(sourceData, outputData); + for (int i = 0; i < ossieMetrics.size(); i++) { + Map ossieMetric = asMap(ossieMetrics.get(i)); + Map sfMetric = asMap(sfMetrics.get(i)); + MetricExpressionTranslator.Result translated = + MetricExpressionTranslator.translate(ossieMetric, resolver); + sfMetric.put(EXPRESSION, translated.expression()); + sfMetric.put(DATA_TYPE, translated.dataType()); + sfMetric.put("syntax", "Tua"); + sfMetric.put("aggregationType", "UserAgg"); + } + } + /** * Wraps expressions for SF→Ossie conversion. */ diff --git a/converters/salesforce/src/main/java/org/apache/ossie/converter/SqlMetricExpressionParser.java b/converters/salesforce/src/main/java/org/apache/ossie/converter/SqlMetricExpressionParser.java new file mode 100644 index 00000000..3464ae33 --- /dev/null +++ b/converters/salesforce/src/main/java/org/apache/ossie/converter/SqlMetricExpressionParser.java @@ -0,0 +1,250 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import static org.apache.ossie.converter.MetricExpression.*; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; +import java.util.Set; +import net.sf.jsqlparser.expression.*; +import net.sf.jsqlparser.expression.operators.relational.IsNullExpression; +import net.sf.jsqlparser.expression.operators.relational.ParenthesedExpressionList; +import net.sf.jsqlparser.parser.CCJSqlParserUtil; +import net.sf.jsqlparser.schema.Column; +import net.sf.jsqlparser.statement.select.AllColumns; +import org.apache.ossie.converter.TuaMetricExpressionParser.Kind; +import org.apache.ossie.converter.TuaMetricExpressionParser.Token; + +/** Adapts a completely consumed JSqlParser expression into the explicitly supported compiler AST. */ +final class SqlMetricExpressionParser { + private final String dialect; + private int depth; + private SqlMetricExpressionParser(String dialect) { this.dialect = dialect; } + + static Node parse(String text, String dialect) { + text = normalize(text, dialect); + try { + Expression expression = CCJSqlParserUtil.parseCondExpression(text, false, + parser -> parser.withSquareBracketQuotation(dialect.equals("ANSI_SQL"))); + if (expression == null) throw new IllegalArgumentException("could not parse a complete SQL expression"); + return new SqlMetricExpressionParser(dialect).adapt(expression); + } catch (net.sf.jsqlparser.JSQLParserException e) { + throw new IllegalArgumentException(dialect + " expression has unsupported or unexpected token: " + + e.getMessage(), e); + } catch (StackOverflowError e) { + throw new IllegalArgumentException("expression nesting exceeds parser limits", e); + } + } + + /** Simplifies redundant syntax within the original input bounds, without reparsing SQL. */ + private static String normalize(String text, String dialect) { + List tokens = TuaMetricExpressionParser.tokenize(text, dialect); + int[] closing = new int[tokens.size()]; + int[] openings = new int[128]; + int nesting = 0; + for (int i = 0; i < tokens.size() - 1; i++) { + if (symbol(tokens.get(i), "(")) openings[nesting++] = i; + if (symbol(tokens.get(i), ")")) { + if (nesting == 0) throw new IllegalArgumentException("unexpected closing parenthesis"); + closing[openings[--nesting]] = i; + } + } + boolean[] redundant = new boolean[tokens.size()]; + for (int i = 1; i < tokens.size() - 2; i++) { + // Only remove a group inside another opening parenthesis. Keep the + // function argument list and innermost group, including tuple/modifier syntax. + if (symbol(tokens.get(i - 1), "(") && symbol(tokens.get(i), "(") + && symbol(tokens.get(i + 1), "(") && closing[i] == closing[i + 1] + 1) { + redundant[i] = redundant[closing[i]] = true; + } + } + StringBuilder result = new StringBuilder(text.length()); + for (int i = 0; i < tokens.size() - 1;) { + if (redundant[i]) { i++; continue; } + Token token = tokens.get(i); + boolean sign = symbol(token, "+") || symbol(token, "-"); + boolean not = token.kind() == Kind.WORD && token.text().equalsIgnoreCase("NOT"); + // Preserve binary +/- and predicate modifiers such as IS NOT. Only + // prefix operators can be simplified; malformed modifiers must still fail. + if ((not || sign) && (i == 0 || startsOperandAfter(tokens.get(i - 1)))) { + int end = i; + int negatives = 0; + while (end < tokens.size() - 1) { + Token next = tokens.get(end); + if (not ? next.kind() != Kind.WORD || !next.text().equalsIgnoreCase("NOT") + : !symbol(next, "+") && !symbol(next, "-")) break; + if (symbol(next, "-")) negatives++; + if (++end - i > 128) throw new IllegalArgumentException("too many unary operators"); + } + // Retain a unary operation even when parity is even: dropping all + // operators would bypass numeric/Boolean checks and COUNT(field) rules. + result.append(not ? (end - i) % 2 == 0 ? "NOT NOT " : "NOT " + : negatives % 2 == 0 ? "+ " : "- "); + i = end; + } else { + // Raw slices preserve escaped strings and quoted identifier spelling. + result.append(text, token.offset(), tokens.get(i + 1).offset()).append(' '); + i++; + } + } + return result.toString(); + } + + private static boolean symbol(Token token, String value) { + return token.kind() == Kind.SYMBOL && token.text().equals(value); + } + + private static boolean startsOperandAfter(Token token) { + return token.kind() == Kind.SYMBOL + && Set.of("(", ",", "+", "-", "*", "/", "=", "!=", "<>", "<", "<=", ">", ">=").contains(token.text()) + || token.kind() == Kind.WORD + && Set.of("WHEN", "THEN", "ELSE", "AND", "OR", "NOT", "DISTINCT").contains(token.text().toUpperCase(Locale.ROOT)); + } + + private Node adapt(Expression expression) { + if (++depth > 128) throw new IllegalArgumentException("expression nesting exceeds 128 levels"); + try { return adaptNode(expression); } + finally { depth--; } + } + + private Node adaptNode(Expression expression) { + if (expression instanceof ParenthesedExpressionList list && list.size() == 1) { + return adapt(list.get(0)); + } + if (expression instanceof LongValue || expression instanceof DoubleValue) { + return new Literal(TuaMetricExpressionParser.number(expression.toString())); + } + if (expression instanceof NullValue) return new Literal(null); + if (expression instanceof BooleanValue value) return new Literal(value.getValue()); + if (expression instanceof StringValue value) { + if (value.getPrefix() != null) throw unsupported(expression); + return new Literal(value.getValue().replace("''", "'")); + } + if (expression instanceof Column column) return column(column); + if (expression instanceof SignedExpression signed) { + if (signed.getSign() != '+' && signed.getSign() != '-') throw unsupported(expression); + return new Unary(String.valueOf(signed.getSign()), adapt(signed.getExpression())); + } + if (expression instanceof NotExpression not) { + if (not.isExclamationMark()) throw unsupported(expression); + return new Unary("NOT", adapt(not.getExpression())); + } + if (expression instanceof IsNullExpression test) { + if (test.isUseIsNull() || test.isUseNotNull()) throw unsupported(expression); + Node result = new Unary("ISNULL", adapt(test.getLeftExpression())); + return test.isNot() ? new Unary("NOT", result) : result; + } + if (expression instanceof BinaryExpression binary) { + if (binary instanceof net.sf.jsqlparser.expression.operators.relational.SupportsOldOracleJoinSyntax oracle + && (oracle.getOldOracleJoinSyntax() != 0 || oracle.getOraclePriorPosition() != 0)) { + throw unsupported(expression); + } + String operator = binary.getStringExpression().toUpperCase(Locale.ROOT); + if (!Set.of("+", "-", "*", "/", "AND", "OR", "=", "!=", "<>", "<", ">", "<=", ">=").contains(operator)) { + throw unsupported(expression); + } + return new Binary(operator.equals("<>") ? "!=" : operator, + adapt(binary.getLeftExpression()), adapt(binary.getRightExpression())); + } + if (expression instanceof Function function) return function(function); + if (expression instanceof CaseExpression conditional) { + List branches = new ArrayList<>(); + // The original bounded contract supports searched CASE. Simple CASE can be + // added with explicit type checking and evaluation-count guarantees later. + if (conditional.getSwitchExpression() != null) throw unsupported(expression); + for (WhenClause branch : conditional.getWhenClauses()) { + branches.add(adapt(branch.getWhenExpression())); + branches.add(adapt(branch.getThenExpression())); + } + return new Conditional(branches, conditional.getElseExpression() == null + ? new Literal(null) : adapt(conditional.getElseExpression())); + } + if (expression instanceof AllColumns) { + throw new IllegalArgumentException("COUNT(*) is unsupported; name a declared field to count"); + } + throw unsupported(expression); + } + + private Node function(Function function) { + String name = function.getName(); + if (name == null || !name.matches("[A-Za-z_][A-Za-z_0-9]*")) { + throw new IllegalArgumentException("quoted or qualified function names are unsupported"); + } + if (function.isUnique() || function.isEscaped() || function.getNamedParameters() != null + || function.getAttribute() != null || function.getKeep() != null + || function.getNullHandling() != null || function.isIgnoreNullsOutside() + || function.isIgnoreNulls() || function.getLimit() != null + || function.getHavingClause() != null || function.getExtraKeyword() != null + || function.getOnOverflowTruncate() != null + || function.getOrderByElements() != null && !function.getOrderByElements().isEmpty()) { + throw new IllegalArgumentException("unsupported function modifiers for " + name); + } + if (function.isAllColumns()) { + throw new IllegalArgumentException("explicit ALL function modifier is outside the supported SQL subset"); + } + if (function.getParameters() instanceof ParenthesedExpressionList grouped && grouped.size() != 1) { + throw new IllegalArgumentException("tuple-valued function arguments are unsupported"); + } + List arguments = new ArrayList<>(); + if (function.getParameters() != null) { + for (Expression argument : function.getParameters()) arguments.add(adapt(argument)); + } + return new Call(name.toUpperCase(Locale.ROOT), arguments, function.isDistinct()); + } + + private Node column(Column column) { + if (column.getArrayConstructor() != null) throw unsupported(column); + // JSqlParser 5.3's Table accessors split a quoted name containing a dot + // ("Order.Items") into schema/table parts. Retain the original token + // boundaries instead of binding that expression to a different object. + List raw = new ArrayList<>(); + var source = column.getASTNode(); + if (source == null) throw new IllegalArgumentException("field reference has no source identifier tokens"); + boolean identifier = true; + for (var token = source.jjtGetFirstToken(); token != null; token = token.next) { + if (identifier) raw.add(token.image); + else if (!token.image.equals(".")) throw unsupported(column); + identifier = !identifier; + if (token == source.jjtGetLastToken()) break; + } + if (raw.isEmpty() || identifier) throw unsupported(column); + boolean bracket = raw.get(0).startsWith("["); + List parts = new ArrayList<>(); + for (String part : raw) { + if (bracket != part.startsWith("[")) { + throw new IllegalArgumentException("do not mix bracketed and SQL field identifiers"); + } + boolean quoted = part.startsWith("\"") || part.startsWith("["); + if (quoted) { + String end = bracket ? "]" : "\""; + part = part.substring(1, part.length() - 1).replace(end + end, end); + } + parts.add(new MetricFieldResolver.Identifier(part, quoted)); + } + return new Field(parts, bracket); + } + + private IllegalArgumentException unsupported(Expression expression) { + return new IllegalArgumentException(dialect + " unsupported SQL expression " + + expression.getClass().getSimpleName() + + "; only documented expression capabilities can be converted"); + } +} diff --git a/converters/salesforce/src/main/java/org/apache/ossie/converter/TuaMetricExpressionParser.java b/converters/salesforce/src/main/java/org/apache/ossie/converter/TuaMetricExpressionParser.java new file mode 100644 index 00000000..1472c592 --- /dev/null +++ b/converters/salesforce/src/main/java/org/apache/ossie/converter/TuaMetricExpressionParser.java @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import static org.apache.ossie.converter.MetricExpression.*; +import java.math.BigDecimal; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; +import java.util.Set; + +/** Native Tua frontend for the bounded supported grammar; produces the same AST as SQL. */ +final class TuaMetricExpressionParser { + private final List tokens; + private int position; + private int depth; + TuaMetricExpressionParser(List tokens) { this.tokens = tokens; } + + Node parse() { + Node node = expression(); + if (peek().kind() != Kind.END) throw error("unsupported or unexpected token '" + peek().text() + "'"); + return node; + } + private Node expression() { + if (++depth > 128) throw error("expression nesting exceeds 128 levels"); + try { return or(); } finally { depth--; } + } + private Node or() { + Node value = and(); + while (take("OR")) value = new Binary("OR", value, and()); + return value; + } + private Node and() { + Node value = not(); + while (take("AND")) value = new Binary("AND", value, not()); + return value; + } + private Node not() { + int count = 0; + while (take("NOT")) if (++count > 128) throw error("too many unary operators"); + Node value = comparison(); + while (count-- > 0) value = new Unary("NOT", value); + return value; + } + private Node comparison() { + Node value = additive(); + if (at("IS")) throw error("use ISNULL in TABLEAU expressions"); + if (Set.of("=", "!=", "<>", "<", "<=", ">", ">=").contains(peek().text())) { + String op = next().text(); + return new Binary(op.equals("<>") ? "!=" : op, value, additive()); + } + return value; + } + private Node additive() { + Node value = multiplicative(); + while (at("+") || at("-")) value = new Binary(next().text(), value, multiplicative()); + return value; + } + private Node multiplicative() { + Node value = unary(); + while (at("*") || at("/")) value = new Binary(next().text(), value, unary()); + return value; + } + private Node unary() { + List signs = new ArrayList<>(); + while (at("+") || at("-")) { + if (signs.size() >= 128) throw error("too many unary operators"); + signs.add(next().text()); + } + Node value = primary(); + for (int i = signs.size() - 1; i >= 0; i--) value = new Unary(signs.get(i), value); + return value; + } + private Node primary() { + if (take("(")) { Node value = expression(); expect(")"); return value; } + if (at("CASE")) throw error("searched CASE is SQL; use IF in TABLEAU"); + if (take("IF")) return conditional(); + if (take("NULL")) return new Literal(null); + if (at("TRUE") || at("FALSE")) return new Literal(Boolean.valueOf(next().text())); + Token token = next(); + if (token.kind() == Kind.NUMBER) return new Literal(number(token.text())); + if (token.kind() == Kind.STRING) return new Literal(token.text()); + if (token.kind() != Kind.WORD && token.kind() != Kind.IDENTIFIER) { + throw error("expected a value, found '" + token.text() + "'"); + } + if (take("(")) { + if (token.kind() != Kind.WORD) throw error("quoted function names are unsupported"); + return function(token.text().toUpperCase(Locale.ROOT)); + } + List parts = new ArrayList<>(); + addIdentifier(parts, token); + while (take(".")) addIdentifier(parts, next()); + if (parts.size() != 2) throw error("TABLEAU fields must use [dataset].[field] notation"); + return new Field(parts, true); + } + private void addIdentifier(List parts, Token token) { + if (token.kind() != Kind.IDENTIFIER || !token.bracket()) { + throw error("TABLEAU fields must use [dataset].[field] notation"); + } + parts.add(new MetricFieldResolver.Identifier(token.text(), true)); + } + private Node conditional() { + List branches = new ArrayList<>(); + do { + branches.add(expression()); expect("THEN"); branches.add(expression()); + } while (take("ELSEIF")); + Node otherwise = take("ELSE") ? expression() : new Literal(null); + expect("END"); + return new Conditional(branches, otherwise); + } + private Node function(String name) { + boolean distinct = take("DISTINCT"); + if (at("*")) throw error("COUNT(*) is unsupported; name a declared field to count"); + List arguments = new ArrayList<>(); + if (!at(")")) do { arguments.add(expression()); } while (take(",")); + expect(")"); + return new Call(name, arguments, distinct); + } + private Token peek() { return tokens.get(position); } + private Token next() { Token token = peek(); if (token.kind() != Kind.END) position++; return token; } + private boolean at(String text) { + return (peek().kind() == Kind.WORD || peek().kind() == Kind.SYMBOL) && peek().text().equalsIgnoreCase(text); + } + private boolean take(String text) { if (!at(text)) return false; next(); return true; } + private void expect(String text) { if (!take(text)) throw error("expected " + text + ", found '" + peek().text() + "'"); } + private IllegalArgumentException error(String message) { + return new IllegalArgumentException("TABLEAU at character " + (peek().offset() + 1) + ": " + message); + } + + // Preflight bounds SQL parsing too; native parsing also consumes these tokens. + enum Kind { WORD, IDENTIFIER, STRING, NUMBER, SYMBOL, END } + record Token(Kind kind, String text, int offset, boolean bracket) {} + static List tokenize(String text, String dialect) { + if (text.length() > 32768) throw new IllegalArgumentException("expression exceeds 32768 characters"); + List tokens = new ArrayList<>(); + int nesting = 0; + for (int i = 0; i < text.length();) { + char c = text.charAt(i); + if (Character.isWhitespace(c)) { i++; continue; } + int start = i; + if (c == '\'' || c == '"' || c == '[') { + if (c == '[' && dialect.equals("SNOWFLAKE")) throw lexical(dialect, i, "use double-quoted SQL identifiers"); + boolean string = c == '\'' || (c == '"' && dialect.equals("TABLEAU")); + char end = c == '[' ? ']' : c; + StringBuilder value = new StringBuilder(); + boolean closed = false; + i++; + while (i < text.length()) { + char part = text.charAt(i++); + if (part == end) { + if (i < text.length() && text.charAt(i) == end) { value.append(end); i++; } + else { closed = true; break; } + } else { + if (Character.isISOControl(part) || (string && part == '\\')) { + throw lexical(dialect, i - 1, "control characters and backslash string escapes are unsupported"); + } + value.append(part); + } + } + if (!closed) throw lexical(dialect, start, "unterminated quoted value"); + tokens.add(new Token(string ? Kind.STRING : Kind.IDENTIFIER, value.toString(), start, c == '[')); + } else if (Character.isDigit(c) || (c == '.' && i + 1 < text.length() && Character.isDigit(text.charAt(i + 1)))) { + i++; + while (i < text.length() && (Character.isDigit(text.charAt(i)) || text.charAt(i) == '.')) i++; + if (i < text.length() && (text.charAt(i) == 'e' || text.charAt(i) == 'E')) { + i++; + if (i < text.length() && (text.charAt(i) == '+' || text.charAt(i) == '-')) i++; + while (i < text.length() && Character.isDigit(text.charAt(i))) i++; + } + tokens.add(new Token(Kind.NUMBER, text.substring(start, i), start, false)); + } else if (Character.isLetter(c) || c == '_') { + i++; + while (i < text.length() && (Character.isLetterOrDigit(text.charAt(i)) || text.charAt(i) == '_' || text.charAt(i) == '$')) i++; + tokens.add(new Token(Kind.WORD, text.substring(start, i), start, false)); + } else { + if (i + 1 < text.length() && (text.startsWith("--", i) || text.startsWith("/*", i))) { + throw lexical(dialect, i, "comments are unsupported in metric expressions"); + } + String symbol = String.valueOf(c); + if (i + 1 < text.length() && Set.of("<=", ">=", "<>", "!=").contains(text.substring(i, i + 2))) { + symbol = text.substring(i, i + 2); + i++; + } + if (!"()+-*/.,=<>!".contains(String.valueOf(c))) throw lexical(dialect, start, "unsupported character '" + c + "'"); + tokens.add(new Token(Kind.SYMBOL, symbol, start, false)); + i++; + } + Token added = tokens.get(tokens.size() - 1); + if (added.kind() == Kind.NUMBER) number(added.text()); + if (added.kind() == Kind.SYMBOL && added.text().equals("(") && ++nesting > 128) { + throw lexical(dialect, start, "expression nesting exceeds 128 levels"); + } + if (added.kind() == Kind.SYMBOL && added.text().equals(")")) nesting--; + if (tokens.size() > 8192) throw lexical(dialect, start, "too many expression tokens"); + } + if (nesting > 0) throw lexical(dialect, text.length(), "expected )"); + tokens.add(new Token(Kind.END, "end of expression", text.length(), false)); + return tokens; + } + + private static IllegalArgumentException lexical(String dialect, int offset, String message) { + return new IllegalArgumentException(dialect + " at character " + (offset + 1) + ": " + message); + } + static BigDecimal number(String text) { + BigDecimal value; + try { value = new BigDecimal(text); } + catch (NumberFormatException e) { throw new IllegalArgumentException("invalid numeric literal '" + text + "'"); } + if (Math.abs((long) value.scale()) > 1000 || value.precision() > 1000) { + throw new IllegalArgumentException("numeric literal is too large"); + } + return value; + } +} diff --git a/converters/salesforce/src/test/java/org/apache/ossie/MetricCliTest.java b/converters/salesforce/src/test/java/org/apache/ossie/MetricCliTest.java new file mode 100644 index 00000000..93528956 --- /dev/null +++ b/converters/salesforce/src/test/java/org/apache/ossie/MetricCliTest.java @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie; + +import static org.junit.jupiter.api.Assertions.*; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.TimeUnit; +import org.apache.ossie.app.OssieSalesforceConverter; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class MetricCliTest { + @TempDir + Path directory; + + @Test + void reportsMetricFailureToStderrWithoutWritingAModel() throws Exception { + Path input = directory.resolve("input.yaml"); + Files.writeString(input, Files.readString(Path.of("src/test/resources/examples/ossieToSalesforce.yaml")) + .replace("SUM([Orders].[amount])", "SUM([Orders].[missing])")); + Path stderr = directory.resolve("stderr.txt"); + Process process = new ProcessBuilder( + Path.of(System.getProperty("java.home"), "bin", "java").toString(), + "-cp", System.getProperty("java.class.path"), OssieSalesforceConverter.class.getName(), + "toSF", input.toString()) + .redirectError(stderr.toFile()) + .redirectOutput(directory.resolve("stdout.txt").toFile()) + .start(); + try { + assertTrue(process.waitFor(30, TimeUnit.SECONDS), "CLI did not terminate"); + assertEquals(3, process.exitValue()); + String error = Files.readString(stderr); + assertTrue(error.contains("Metric 'total_revenue'"), error); + assertTrue(error.contains("Unknown field reference"), error); + assertTrue(error.contains("missing"), error); + assertFalse(Files.exists(directory.resolve("Customer_Orders_Model.json"))); + } finally { + process.destroyForcibly(); + } + } +} diff --git a/converters/salesforce/src/test/java/org/apache/ossie/MetricExportIntegrationTest.java b/converters/salesforce/src/test/java/org/apache/ossie/MetricExportIntegrationTest.java new file mode 100644 index 00000000..b382efc3 --- /dev/null +++ b/converters/salesforce/src/test/java/org/apache/ossie/MetricExportIntegrationTest.java @@ -0,0 +1,419 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +import org.apache.ossie.converter.ConversionDirection; +import org.apache.ossie.converter.Converter; +import org.apache.ossie.converter.ConverterFactory; +import org.apache.ossie.exception.ConversionException; +import org.apache.ossie.exception.ValidationException; +import org.apache.ossie.validator.SchemaValidator; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.*; +import static org.junit.jupiter.api.Assumptions.assumeTrue; + +/** Exercises metric compilation through the public converter and target schema. */ +class MetricExportIntegrationTest { + private static final ObjectMapper JSON = new ObjectMapper(); + private static final ObjectMapper YAML = new ObjectMapper(new YAMLFactory()); + private static boolean salesforceSchemaExists; + private static boolean ossieSchemaExists; + + private Converter converter; + + @TempDir + Path temporaryDirectory; + + @BeforeAll + static void checkSchemaAvailability() { + salesforceSchemaExists = MetricExportIntegrationTest.class + .getResource(SchemaValidator.SALESFORCE_SCHEMA_PATH) != null; + ossieSchemaExists = MetricExportIntegrationTest.class + .getResource(SchemaValidator.OSSIE_SCHEMA_PATH) != null; + if (Boolean.getBoolean("requireSalesforceSchema")) { + assertTrue(salesforceSchemaExists, + "-DrequireSalesforceSchema=true requires the Salesforce schema; see README setup instructions"); + } + } + + @BeforeEach + void setUp() { + assumeTrue(ossieSchemaExists, "Ossie schema is required; see README setup instructions"); + converter = ConverterFactory.getConverter(ConversionDirection.OSSIE_TO_SALESFORCE); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void translatesComposedMetricsThroughStringApi(String dialect) throws Exception { + Map output = convertOne(model("sales", List.of( + metric("margin", dialect, "SUM(orders.profit) / NULLIF(SUM(orders.revenue), 0)"), + metric("adjusted_average", dialect, + "ROUND(AVG(CASE WHEN orders.revenue IS NOT NULL AND NOT orders.profit < 0 " + + "THEN COALESCE(orders.profit, 0) ELSE 0 END), 2)"), + metric("customers", dialect, "COUNT(DISTINCT orders.customer_id)")))); + + List> measurements = measurements(output); + assertEquals(List.of("margin", "adjusted_average", "customers"), + measurements.stream().map(item -> item.get("apiName")).toList()); + for (Map measurement : measurements) { + assertMeasurementMetadata(measurement); + } + assertEquals("(SUM([orders].[profit]) / (IF (SUM([orders].[revenue]) = 0) " + + "THEN NULL ELSE SUM([orders].[revenue]) END))", + measurements.get(0).get("expression")); + String conditional = (String) measurements.get(1).get("expression"); + assertTrue(conditional.startsWith("ROUND(AVG((IF "), conditional); + assertTrue(conditional.contains("ISNULL([orders].[revenue])"), conditional); + assertTrue(conditional.contains("IFNULL([orders].[profit], 0)"), conditional); + assertFalse(conditional.contains("CASE"), conditional); + assertEquals("COUNTD([orders].[customer_id])", measurements.get(2).get("expression")); + + Map dataset = items(output, "semanticDataObjects").get(0); + Map profit = items(dataset, "semanticMeasurements").stream() + .filter(item -> item.get("apiName").equals("profit")).findFirst().orElseThrow(); + assertEquals("profit__c", profit.get("dataObjectFieldName")); + assertFalse(measurements.get(0).get("expression").toString().contains("profit__c"), + "A metric binds semantic field names, not physical source columns"); + } + + @Test + void fileApiWritesTheSameCompleteModelAsStringApi() throws Exception { + String input = document(List.of(model("sales", List.of( + metric("revenue", "SNOWFLAKE", "SUM(orders.revenue)"))))); + Path source = temporaryDirectory.resolve("input.yaml"); + Path outputDirectory = Files.createDirectory(temporaryDirectory.resolve("output")); + Files.writeString(source, input); + + converter.convert(source, outputDirectory); + + assertEquals(JSON.readTree(converter.convert(input).get(0)), + JSON.readTree(Files.readString(outputDirectory.resolve("sales.json")))); + try (var files = Files.list(outputDirectory)) { + assertEquals(List.of("sales.json"), files.map(path -> path.getFileName().toString()).toList()); + } + } + + @Test + void missingFieldFailsWithMetricNameAndDoesNotWriteAnyModelFiles() throws Exception { + String input = document(List.of( + model("valid", List.of(metric("revenue", "ANSI_SQL", "SUM(orders.revenue)"))), + model("invalid", List.of(metric("broken_margin", "SNOWFLAKE", "SUM(orders.missing)"))))); + Path source = temporaryDirectory.resolve("input.yaml"); + Path outputDirectory = Files.createDirectory(temporaryDirectory.resolve("output")); + Path existing = outputDirectory.resolve("existing.json"); + Files.writeString(source, input); + Files.writeString(existing, "preserve this file"); + + ConversionException error = assertThrows(ConversionException.class, + () -> converter.convert(source, outputDirectory)); + + assertTrue(error.getMessage().contains("broken_margin"), error.getMessage()); + assertTrue(error.getMessage().contains("orders.missing"), error.getMessage()); + assertTrue(error.getMessage().contains("declare"), error.getMessage()); + assertEquals("preserve this file", Files.readString(existing)); + try (var files = Files.list(outputDirectory)) { + assertEquals(List.of("existing.json"), files.map(path -> path.getFileName().toString()).toList(), + "A failure in a later model must not leave an earlier model's output behind"); + } + } + + @Test + void selectedUnsupportedTableauExpressionDoesNotFallBackToSql() throws Exception { + Map metric = metric("chosen_tableau", "TABLEAU", "BOGUS([orders].[profit])"); + metric.put("expression", Map.of("dialects", List.of( + dialect("ANSI_SQL", "SUM(orders.profit)"), + dialect("TABLEAU", "BOGUS([orders].[profit])"), + dialect("SNOWFLAKE", "SUM(orders.profit)")))); + String input = document(List.of(model("sales", List.of(metric)))); + + ConversionException error = assertThrows(ConversionException.class, () -> converter.convert(input)); + + assertTrue(error.getMessage().contains("chosen_tableau"), error.getMessage()); + assertTrue(error.getMessage().contains("BOGUS"), error.getMessage()); + } + + @Test + void preservesSupportedTableauExpressionMeaningAndValidatesReferences() throws Exception { + Map output = convertOne(model("sales", List.of( + metric("revenue", "TABLEAU", "IFNULL(SUM([orders].[revenue]), 0)")))); + + Map measurement = measurements(output).get(0); + assertEquals("IFNULL(SUM([orders].[revenue]), 0)", measurement.get("expression")); + assertMeasurementMetadata(measurement); + + String invalid = document(List.of(model("sales", List.of( + metric("unknown_tableau", "TABLEAU", "SUM([orders].[missing])"))))); + ConversionException error = assertThrows(ConversionException.class, () -> converter.convert(invalid)); + assertTrue(error.getMessage().contains("unknown_tableau"), error.getMessage()); + assertTrue(error.getMessage().contains("orders.missing"), error.getMessage()); + } + + @Test + void modelWithoutMetricsRetainsDatasetExport() throws Exception { + Map source = model("sales", List.of()); + source.remove("metrics"); + + Map output = convertOne(source); + + assertTrue(measurements(output).isEmpty()); + assertEquals(3, items(items(output, "semanticDataObjects").get(0), "semanticMeasurements").size()); + } + + @Test + void emptyMetricsRemainEmpty() throws Exception { + Map output = convertOne(model("sales", List.of())); + + assertTrue(measurements(output).isEmpty()); + assertEquals("sales", output.get("apiName")); + } + + @Test + void metricNamesAndFieldTypesAreIsolatedAcrossSemanticModels() throws Exception { + Map first = model("first", List.of( + metric("value", "SNOWFLAKE", "SUM(orders.profit)"))); + Map second = model("second", List.of( + metric("value", "ANSI_SQL", "COUNT(orders.profit)"))); + items(items(second, "datasets").get(0), "fields").get(0).put("datatype", "String"); + items(items(second, "datasets").get(0), "fields").get(0).put("dimension", Map.of("is_time", false)); + + List outputs = converter.convert(document(List.of(first, second))); + + assertEquals(2, outputs.size()); + assertEquals("SUM([orders].[profit])", measurements(parse(outputs.get(0))).get(0).get("expression")); + assertEquals("COUNT([orders].[profit])", measurements(parse(outputs.get(1))).get(0).get("expression")); + assertEquals("first", parse(outputs.get(0)).get("apiName")); + assertEquals("second", parse(outputs.get(1)).get("apiName")); + } + + @Test + void fieldsFromAnotherSemanticModelCannotSatisfyAMetricReference() throws Exception { + Map first = model("first", List.of( + metric("valid", "ANSI_SQL", "SUM(orders.profit)"))); + Map second = model("second", List.of( + metric("must_not_leak", "ANSI_SQL", "SUM(orders.profit)"))); + items(second, "datasets").get(0).put("fields", List.of(field("revenue", "Decimal"))); + String input = document(List.of(first, second)); + + ConversionException error = assertThrows(ConversionException.class, () -> converter.convert(input)); + + assertTrue(error.getMessage().contains("must_not_leak"), error.getMessage()); + assertTrue(error.getMessage().contains("orders.profit"), error.getMessage()); + } + + @Test + void declaredButOmittedCalculatedSqlFieldCannotSatisfyAMetricReference() throws Exception { + Map source = model("sales", List.of()); + Map calculated = field("adjusted", "Decimal"); + calculated.put("expression", Map.of("dialects", List.of(dialect("ANSI_SQL", "profit__c + 1")))); + items(source, "datasets").get(0).put("fields", List.of(field("profit", "Decimal"), calculated)); + Map output = convertOne(source); + assertEquals(List.of("profit"), items(items(output, "semanticDataObjects").get(0), + "semanticMeasurements").stream().map(item -> item.get("apiName")).toList()); + + source.put("metrics", List.of(metric("adjusted_total", "ANSI_SQL", "SUM(orders.adjusted)"))); + String input = document(List.of(source)); + ConversionException error = assertThrows(ConversionException.class, () -> converter.convert(input)); + + assertTrue(error.getMessage().contains("adjusted_total"), error.getMessage()); + assertTrue(error.getMessage().contains("orders.adjusted"), error.getMessage()); + assertTrue(error.getMessage().contains("not exported"), error.getMessage()); + } + + @Test + void duplicateMetricNamesAreRejectedBeforeExport() throws Exception { + String input = document(List.of(model("sales", List.of( + metric("duplicated", "ANSI_SQL", "SUM(orders.profit)"), + metric("duplicated", "ANSI_SQL", "SUM(orders.revenue)"))))); + + ConversionException error = assertThrows(ConversionException.class, () -> converter.convert(input)); + + assertTrue(error.getMessage().contains("duplicated"), error.getMessage()); + assertTrue(error.getMessage().contains("duplicate metric"), error.getMessage()); + } + + @ParameterizedTest + @ValueSource(strings = {"AVG(orders.quantity)", "SUM(orders.quantity) / 2", "1.5"}) + void integerMetricDeclarationRejectsFractionalResult(String expression) throws Exception { + Map metric = metric("integer_result", "ANSI_SQL", expression); + metric.put("datatype", "Integer"); + String input = document(List.of(model("sales", List.of(metric)))); + + ConversionException error = assertThrows(ConversionException.class, () -> converter.convert(input)); + + assertTrue(error.getMessage().contains("integer_result"), error.getMessage()); + assertTrue(error.getMessage().contains("Integer"), error.getMessage()); + assertTrue(error.getMessage().contains("incompatible"), error.getMessage()); + } + + @Test + void integerCountIsExportedAsSalesforceNumber() throws Exception { + Map metric = metric("customer_count", "ANSI_SQL", "COUNT(orders.customer_id)"); + metric.put("datatype", "Integer"); + + Map output = convertOne(model("sales", List.of(metric))); + + assertMeasurementMetadata(measurements(output).get(0)); + assertEquals("COUNT([orders].[customer_id])", measurements(output).get(0).get("expression")); + } + + @Test + void translatedComposedMetricsValidateAgainstTheSalesforceSchema() throws Exception { + assumeTrue(salesforceSchemaExists, "Salesforce schema is required; see README setup instructions"); + Map output = convertOne(model("sales", List.of( + metric("margin", "SNOWFLAKE", "SUM(orders.profit) / NULLIF(SUM(orders.revenue), 0)"), + metric("customers", "ANSI_SQL", "COUNT(DISTINCT orders.customer_id)"), + metric("rounding", "ANSI_SQL", "ROUND(AVG(ABS(orders.profit)), 2) + CEIL(1.2) - FLOOR(1.2)")))); + SchemaValidator validator = new SchemaValidator(JSON, SchemaValidator.SALESFORCE_SCHEMA_PATH); + + for (Map metric : measurements(output)) { + assertMeasurementMetadata(metric); + } + assertDoesNotThrow(() -> validator.validate(output)); + + // OSI expression objects cannot be emitted where Salesforce requires a scalar formula. + Map measurement = measurements(output).get(0); + Object expression = measurement.put("expression", Map.of("dialects", List.of( + dialect("ANSI_SQL", "SUM(orders.profit)")))); + ValidationException error = assertThrows(ValidationException.class, () -> validator.validate(output)); + assertTrue(error.getMessage().contains("expression"), error.getMessage()); + measurement.put("expression", expression); + measurement.put("dataType", "Integer"); + assertThrows(ValidationException.class, () -> validator.validate(output)); + } + + @ParameterizedTest + @ValueSource(strings = {"profit__c+1", "profit__c-1", "profit__c=1", "1"}) + void rejectsDerivedExpressionsMisclassifiedAsPhysicalColumns(String expression) throws Exception { + Map calculated = field("adjusted", "Decimal"); + calculated.put("expression", Map.of("dialects", List.of(dialect("ANSI_SQL", expression)))); + Map source = model("sales", List.of(metric("adjusted_total", "ANSI_SQL", "SUM(orders.adjusted)"))); + items(source, "datasets").get(0).put("fields", List.of(field("profit", "Decimal"), calculated)); + var error = assertThrows(ConversionException.class, () -> convertOne(source)); + assertTrue(error.getMessage().contains("adjusted_total"), error.getMessage()); + assertTrue(error.getMessage().contains("direct physical binding"), error.getMessage()); + } + + @Test + void metricCannotUseJoinCriteriaCorruptedByInheritedRelationshipFiltering() throws Exception { + Map source = model("sales", List.of(metric("combined", "ANSI_SQL", + "SUM(orders.profit) + SUM(returns.profit)"))); + Map returns = new LinkedHashMap<>(items(source, "datasets").get(0)); + returns.put("name", "returns"); + returns.put("source", "returns__dll"); + source.put("datasets", List.of(items(source, "datasets").get(0), returns)); + Map valid = Map.of("name", "orders_returns", "from", "orders", "to", "returns", + "from_columns", List.of("customer_id"), "to_columns", List.of("customer_id")); + source.put("relationships", List.of(valid)); + assertTrue(measurements(convertOne(source)).get(0).get("expression").toString().contains("[returns].[profit]")); + + source.put("relationships", List.of(Map.of("name", "removed_first", "from", "orders", "to", "returns", + "from_columns", List.of("missing"), "to_columns", List.of("customer_id")), valid)); + var error = assertThrows(ConversionException.class, () -> convertOne(source)); + assertTrue(error.getMessage().contains("combined"), error.getMessage()); + assertTrue(error.getMessage().contains("orders_returns"), error.getMessage()); + assertTrue(error.getMessage().contains("join key correspondence"), error.getMessage()); + } + + private Map convertOne(Map model) throws IOException { + List outputs = converter.convert(document(List.of(model))); + assertEquals(1, outputs.size()); + return parse(outputs.get(0)); + } + + private static String document(List> models) throws IOException { + return YAML.writeValueAsString(Map.of("version", "0.2.0.dev0", "semantic_model", models)); + } + + private static Map parse(String json) throws IOException { + return JSON.readValue(json, new TypeReference<>() {}); + } + + private static Map model(String name, List> metrics) { + Map model = new LinkedHashMap<>(); + model.put("name", name); + model.put("custom_extensions", List.of(Map.of("vendor_name", "SALESFORCE", + "data", "{\"dataspace\":\"default\"}"))); + Map dataset = new LinkedHashMap<>(); + dataset.put("name", "orders"); + dataset.put("source", "orders__dll"); + dataset.put("custom_extensions", List.of(Map.of("vendor_name", "SALESFORCE", + "data", "{\"dataObjectType\":\"Dlo\"}"))); + dataset.put("fields", List.of(field("profit", "Decimal"), field("revenue", "Decimal"), + field("quantity", "Integer"), field("customer_id", "String"))); + model.put("datasets", List.of(dataset)); + model.put("metrics", metrics); + return model; + } + + private static Map field(String name, String datatype) { + Map field = new LinkedHashMap<>(); + field.put("name", name); + field.put("datatype", datatype); + field.put("expression", Map.of("dialects", List.of(dialect("ANSI_SQL", name + "__c")))); + if (datatype.equals("String")) { + field.put("dimension", Map.of("is_time", false)); + } + return field; + } + + private static Map metric(String name, String dialect, String expression) { + Map metric = new LinkedHashMap<>(); + metric.put("name", name); + metric.put("datatype", "Decimal"); + metric.put("expression", Map.of("dialects", List.of(dialect(dialect, expression)))); + return metric; + } + + private static Map dialect(String dialect, String expression) { + return Map.of("dialect", dialect, "expression", expression); + } + + private static void assertMeasurementMetadata(Map measurement) { + assertInstanceOf(String.class, measurement.get("expression")); + assertEquals("Tua", measurement.get("syntax")); + assertEquals("UserAgg", measurement.get("aggregationType")); + assertEquals("Number", measurement.get("dataType")); + } + + private static List> measurements(Map output) { + return items(output, "semanticCalculatedMeasurements"); + } + + @SuppressWarnings("unchecked") + private static List> items(Map object, String key) { + return (List>) object.getOrDefault(key, List.of()); + } +} diff --git a/converters/salesforce/src/test/java/org/apache/ossie/OssieToSalesforceConverterTest.java b/converters/salesforce/src/test/java/org/apache/ossie/OssieToSalesforceConverterTest.java index 6d5c1237..99cd754f 100644 --- a/converters/salesforce/src/test/java/org/apache/ossie/OssieToSalesforceConverterTest.java +++ b/converters/salesforce/src/test/java/org/apache/ossie/OssieToSalesforceConverterTest.java @@ -287,12 +287,91 @@ void testCustomExtensionsRestoration() throws Exception { } @Test - void testMetricsNotConvertedInOssieToSalesforce() throws Exception { + void testMetricsConvertedToSemanticCalculatedMeasurements() throws Exception { List results = converter.convert(ossieYaml); Map sfModel = jsonMapper.readValue(results.get(0), new TypeReference>() {}); List> calcMeasurements = (List>) sfModel.get("semanticCalculatedMeasurements"); - assertNull(calcMeasurements, "Metrics from Ossie are not converted to semanticCalculatedMeasurements in Ossie->SF direction"); + assertNotNull(calcMeasurements, "Metrics from Ossie should convert to semanticCalculatedMeasurements"); + assertEquals(2, calcMeasurements.size()); + + Map totalRevenue = calcMeasurements.stream() + .filter(m -> "total_revenue".equals(m.get("apiName"))) + .findFirst() + .orElse(null); + assertNotNull(totalRevenue); + assertEquals("Sum of all order amounts", totalRevenue.get("description")); + assertEquals("Number", totalRevenue.get("dataType")); + // Legacy ANSI_SQL bracket references are validated and emitted as Tua. + assertEquals("SUM([Orders].[amount])", totalRevenue.get("expression")); + assertEquals("Tua", totalRevenue.get("syntax")); + assertEquals("UserAgg", totalRevenue.get("aggregationType")); + + Map avgOrderValue = calcMeasurements.stream() + .filter(m -> "avg_order_value".equals(m.get("apiName"))) + .findFirst() + .orElse(null); + assertNotNull(avgOrderValue); + assertEquals("AVG([Orders].[amount])", avgOrderValue.get("expression")); + } + + @Test + void testMetricExpressionPrefersTableauDialectOverAnsiSql() throws Exception { + // Normalize line endings first: the fixture file may check out with CRLF depending on + // the platform's autocrlf setting, but the substitution below is written with LF. + String yamlWithTableauMetric = ossieYaml.replace("\r\n", "\n").replace( + " metrics:\n" + + " - description: Sum of all order amounts\n" + + " name: total_revenue\n" + + " datatype: Decimal\n" + + " expression:\n" + + " dialects:\n" + + " - dialect: ANSI_SQL\n" + + " expression: SUM([Orders].[amount])\n", + " metrics:\n" + + " - description: Sum of all order amounts\n" + + " name: total_revenue\n" + + " datatype: Decimal\n" + + " expression:\n" + + " dialects:\n" + + " - dialect: ANSI_SQL\n" + + " expression: SUM([Orders].[amount])\n" + + " - dialect: TABLEAU\n" + + " expression: MAX([Orders].[amount])\n"); + assertTrue(yamlWithTableauMetric.contains("dialect: TABLEAU"), "fixture text substitution did not match"); + + List results = converter.convert(yamlWithTableauMetric); + Map sfModel = jsonMapper.readValue(results.get(0), new TypeReference>() {}); + List> calcMeasurements = (List>) sfModel.get("semanticCalculatedMeasurements"); + + Map totalRevenue = calcMeasurements.stream() + .filter(m -> "total_revenue".equals(m.get("apiName"))) + .findFirst() + .orElse(null); + assertNotNull(totalRevenue); + assertEquals("MAX([Orders].[amount])", totalRevenue.get("expression"), + "TABLEAU dialect should be preferred over ANSI_SQL when both are present"); + } + + @Test + void testMetricWithNoConvertibleDialectFailsConversion() throws Exception { + // Normalize line endings first: the fixture file may check out with CRLF depending on + // the platform's autocrlf setting, but the substitution below is written with LF. + // The Ossie schema requires every metric to have an expression and restricts `dialect` + // to its own enum, so this uses BIGQUERY (a valid dialect, but neither TABLEAU nor + // ANSI_SQL) rather than omitting the expression or inventing an unrecognized dialect. + String yamlWithUnconvertibleDialect = ossieYaml.replace("\r\n", "\n").replace( + " - dialect: ANSI_SQL\n" + + " expression: SUM([Orders].[amount])\n", + " - dialect: BIGQUERY\n" + + " expression: SUM(Orders.amount)\n"); + assertTrue(yamlWithUnconvertibleDialect.contains("dialect: BIGQUERY"), + "fixture text substitution did not match"); + + Exception exception = + assertThrows(Exception.class, () -> converter.convert(yamlWithUnconvertibleDialect)); + String message = exception.getMessage() != null ? exception.getMessage() : exception.getCause().getMessage(); + assertTrue(message.contains("total_revenue"), "error should name the unconvertible metric: " + message); } @Test diff --git a/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricExpressionSemanticsTest.java b/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricExpressionSemanticsTest.java new file mode 100644 index 00000000..4e3fc8e4 --- /dev/null +++ b/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricExpressionSemanticsTest.java @@ -0,0 +1,543 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import static org.junit.jupiter.api.Assertions.*; + +import java.math.BigDecimal; +import java.math.RoundingMode; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +/** + * Hand-computed examples evaluated independently from the production parser/AST. + * + *

This is a local regression oracle for the emitted Tua subset, not a Tableau Next execution + * test. In particular, the evaluator models three-valued Boolean logic, null-ignoring aggregates, + * and half-away-from-zero rounding; it does not establish native engine behavior or precision. + */ +class MetricExpressionSemanticsTest { + + private static final List> ORDERS = List.of( + row(10.0, 2.0, true), + row(10.0, 0.0, false), + row(-4.0, 4.0, null), + row(null, null, null)); + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void aggregatesIgnoreNullsButPreserveDuplicates(String dialect) { + assertValue(dialect, "SUM(orders.amount)", ORDERS, 16.0); + assertValue(dialect, "AVG(orders.amount)", ORDERS, 16.0 / 3); + assertValue(dialect, "MIN(orders.amount)", ORDERS, -4.0); + assertValue(dialect, "MAX(orders.amount)", ORDERS, 10.0); + assertValue(dialect, "COUNT(orders.amount)", ORDERS, 3.0); + assertValue(dialect, "COUNT(DISTINCT orders.amount)", ORDERS, 2.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void emptyAndAllNullInputsHaveDifferentCountAndSumResults(String dialect) { + for (List> rows : List.of( + List.>of(), List.of(row(null, null, null)))) { + for (String aggregate : List.of("SUM", "AVG", "MIN", "MAX")) { + assertValue(dialect, aggregate + "(orders.amount)", rows, null); + } + assertValue(dialect, "COUNT(orders.amount)", rows, 0.0); + assertValue(dialect, "COUNT(DISTINCT orders.amount)", rows, 0.0); + assertValue(dialect, "COALESCE(SUM(orders.amount), 0)", rows, 0.0); + } + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void composedConditionalArithmeticPreservesNullsAndGrouping(String dialect) { + // Per-row numerator contributions are 20, 10, -4, 0; COUNT ignores the null amount. + assertValue(dialect, + "SUM(CASE WHEN orders.flag AND orders.amount > 0 " + + "THEN orders.amount * 2 ELSE COALESCE(orders.amount, 0) END) " + + "/ NULLIF(COUNT(orders.amount), 0)", + ORDERS, 26.0 / 3); + assertValue(dialect, + "(SUM(orders.amount) + 2) * (MAX(orders.cost) - MIN(orders.cost))", + ORDERS, 72.0); + assertValue(dialect, + "SUM(CASE WHEN orders.amount < 0 THEN -orders.amount END)", + ORDERS, 4.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void guardedRatiosReturnNullForZeroAndEmptyDenominators(String dialect) { + String ratio = "SUM(orders.amount) / NULLIF(SUM(orders.cost), 0)"; + assertValue(dialect, ratio, ORDERS, 16.0 / 6); + assertValue(dialect, ratio, List.of(row(8.0, 1.0, true), row(4.0, -1.0, false)), null); + assertValue(dialect, ratio, List.of(row(8.0, null, true)), null); + assertValue(dialect, ratio, List.of(), null); + assertValue(dialect, "COALESCE(" + ratio + ", 0)", List.of(), 0.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void nestedCoalesceSelectsTheFirstNonNullValueIncludingZero(String dialect) { + String expression = "COALESCE(SUM(orders.amount), NULLIF(SUM(orders.cost), 0), 7)"; + assertValue(dialect, expression, ORDERS, 16.0); + assertValue(dialect, expression, List.of(row(0.0, 9.0, true)), 0.0); + assertValue(dialect, expression, List.of(row(null, 9.0, true)), 9.0); + assertValue(dialect, expression, List.of(row(null, 0.0, true)), 7.0); + assertValue(dialect, expression, List.of(), 7.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void nullablePredicatesUseThreeValuedLogic(String dialect) { + assertValue(dialect, + "SUM(CASE WHEN NOT (orders.flag = TRUE) THEN orders.amount ELSE 0 END)", + ORDERS, 10.0); + assertValue(dialect, + "SUM(CASE WHEN orders.amount IS NOT NULL AND " + + "(orders.flag = FALSE OR orders.flag IS NULL) " + + "THEN orders.amount ELSE 0 END)", + ORDERS, 6.0); + assertValue(dialect, + "SUM(CASE WHEN NOT (orders.flag OR orders.amount < 0) THEN 1 ELSE 0 END)", + ORDERS, 1.0); + assertValue(dialect, + "SUM(CASE WHEN orders.flag IS NULL THEN " + + "CASE WHEN orders.amount IS NULL THEN 3 ELSE 2 END ELSE 0 END)", + ORDERS, 5.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void numericFunctionsComposeAroundNegativeAggregates(String dialect) { + List> negative = List.of(row(-2.55, 0.0, true), row(-2.55, 0.0, false)); + assertValue(dialect, "ROUND(AVG(orders.amount), 1)", negative, -2.6); + assertValue(dialect, "ABS(ROUND(AVG(orders.amount), 1))", negative, 2.6); + assertValue(dialect, "CEIL(AVG(orders.amount)) + FLOOR(AVG(orders.amount))", negative, -5.0); + assertValue(dialect, "ABS(ROUND(AVG(orders.amount), 1))", List.of(), null); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void textPredicatesPreserveDuplicatesNullsAndEscapedApostrophes(String dialect) { + List> orders = List.of( + row(10.0, 0.0, true, "paid"), + row(7.0, 0.0, true, "paid"), + row(4.0, 0.0, false, "pending"), + row(9.0, 0.0, null, null), + row(2.0, 0.0, true, "O'Brien")); + assertValue(dialect, + "SUM(CASE WHEN orders.status = 'paid' THEN orders.amount ELSE 0 END)", + orders, 17.0); + assertValue(dialect, "COUNT(DISTINCT orders.status)", orders, 3.0); + assertValue(dialect, "COUNT(orders.status)", orders, 4.0); + assertValue(dialect, + "SUM(CASE WHEN COALESCE(NULLIF(orders.status, 'pending'), 'missing') = 'missing' " + + "THEN orders.amount ELSE 0 END)", + orders, 13.0); + assertValue(dialect, + "SUM(CASE WHEN COALESCE(orders.status, 'paid') = 'paid' THEN orders.amount ELSE 0 END)", + orders, 26.0); + assertValue(dialect, + "SUM(CASE WHEN orders.status = 'O''Brien' THEN orders.amount ELSE 0 END)", + orders, 2.0); + assertValue(dialect, "COUNT(DISTINCT orders.status)", List.of(), 0.0); + assertValue(dialect, "COUNT(DISTINCT orders.status)", List.of(row(null, null, null, null)), 0.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void conditionalBranchesPreserveOrderAndImplicitNull(String dialect) { + List> orders = List.of( + row(10.0, 0.0, true), row(0.0, 0.0, false), + row(-4.0, 0.0, null), row(null, null, null)); + // Positive amounts match both of the first two conditions; the first must win. + assertValue(dialect, + "SUM(CASE WHEN orders.amount > 0 THEN 1 WHEN orders.amount >= 0 THEN 2 " + + "WHEN orders.amount < 0 THEN 3 ELSE 4 END)", + orders, 10.0); + assertValue(dialect, + "SUM(CASE WHEN orders.amount > 0 THEN 1 WHEN orders.amount >= 0 THEN 2 END)", + orders, 3.0); + assertValue(dialect, + "SUM(CASE WHEN orders.amount > 0 THEN 1 WHEN orders.amount = 0 THEN 2 END)", + List.of(row(-4.0, null, null), row(null, null, null)), null); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void nestedConditionalsSkipUnusedConditionsAndResults(String dialect) { + assertValue(dialect, + "SUM(CASE WHEN orders.amount > 0 THEN " + + "CASE WHEN orders.flag THEN 1 WHEN 1 / orders.cost > 0 THEN 99 ELSE 99 END " + + "WHEN orders.amount = 0 THEN 2 WHEN orders.amount < 0 THEN " + + "CASE WHEN orders.flag IS NULL THEN 3 ELSE 1 / orders.cost END ELSE 4 END)", + List.of(row(10.0, 0.0, true), row(0.0, 0.0, false), + row(-4.0, 0.0, null), row(null, 0.0, null)), 10.0); + assertValue(dialect, + "SUM(CASE WHEN orders.amount >= 0 THEN 1 " + + "WHEN 1 / orders.cost > 0 THEN 2 ELSE 1 / orders.cost END)", + List.of(row(10.0, 0.0, true), row(0.0, 0.0, false)), 2.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void numericComparisonsTreatSignedZerosAsEqual(String dialect) { + List> zeros = List.of(row(-0.0, null, null), row(0.0, null, null)); + assertValue(dialect, + "SUM(CASE WHEN orders.amount = 0 THEN 1 ELSE 0 END)", zeros, 2.0); + for (String operator : List.of("!=", "<>")) { + assertValue(dialect, + "SUM(CASE WHEN orders.amount " + operator + " 0 THEN 1 ELSE 0 END)", zeros, 0.0); + } + assertValue(dialect, + "SUM(CASE WHEN orders.amount <= 0 AND orders.amount >= 0 THEN 1 ELSE 0 END)", + zeros, 2.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void distinctCountTreatsSignedZerosAsOneValue(String dialect) { + List> orders = List.of( + row(-0.0, null, null), row(0.0, null, null), + row(1.0, null, null), row(null, null, null)); + assertValue(dialect, "COUNT(DISTINCT orders.amount)", orders, 2.0); + assertValue(dialect, "COUNT(orders.amount)", orders, 3.0); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void nullifGuardsEitherSignOfZero(String dialect) { + assertValue(dialect, "SUM(1 / NULLIF(orders.cost, 0))", + List.of(row(null, -0.0, null), row(null, 0.0, null)), null); + } + + @ParameterizedTest + @ValueSource(strings = {"0", "-0"}) + void evaluatorRejectsEitherSignOfUnguardedZero(String zero) { + AssertionError error = assertThrows(AssertionError.class, + () -> new TuaSubsetEvaluator("1 / " + zero).evaluate(List.of())); + assertTrue(error.getMessage().contains("unguarded zero divisor")); + } + + @ParameterizedTest + @ValueSource(strings = {"SNOWFLAKE", "ANSI_SQL"}) + void repeatedUnaryOperatorsPreserveValuesAndUnknownPredicates(String dialect) { + List> orders = List.of( + row(3.0, null, true), row(7.0, null, false), + row(11.0, null, null), row(null, null, null)); + assertValue(dialect, "SUM(- -orders.amount)", orders, 21.0); + assertValue(dialect, + "SUM(CASE WHEN NOT NOT NOT orders.flag THEN orders.amount ELSE 0 END)", orders, 7.0); + assertValue(dialect, + "SUM(CASE WHEN NOT NOT NOT NOT orders.flag THEN orders.amount ELSE 0 END)", orders, 3.0); + } + + private static void assertValue( + String dialect, String sql, List> rows, Double expected) { + Map metric = Map.of( + "name", "fixture_metric", + "datatype", "Decimal", + "expression", Map.of("dialects", List.of(Map.of("dialect", dialect, "expression", sql)))); + Map source = Map.of("datasets", List.of(Map.of( + "name", "orders", + "fields", List.of( + Map.of("name", "amount", "datatype", "Decimal"), + Map.of("name", "cost", "datatype", "Decimal"), + Map.of("name", "flag", "datatype", "Boolean"), + Map.of("name", "status", "datatype", "String"))))); + Map target = Map.of("semanticDataObjects", List.of(Map.of( + "apiName", "orders", + "semanticMeasurements", List.of( + Map.of("apiName", "amount", "dataObjectFieldName", "amount__c", "dataType", "Number"), + Map.of("apiName", "cost", "dataObjectFieldName", "cost__c", "dataType", "Number")), + "semanticDimensions", List.of( + Map.of("apiName", "flag", "dataObjectFieldName", "flag__c", "dataType", "Boolean"), + Map.of("apiName", "status", "dataObjectFieldName", "status__c", "dataType", "Text"))))); + MetricExpressionTranslator.Result translated = MetricExpressionTranslator.translate(metric, source, target); + assertEquals("Number", translated.dataType(), sql); + Object actual = new TuaSubsetEvaluator(translated.expression()).evaluate(rows); + String message = dialect + ": " + sql + " -> " + translated.expression(); + if (expected == null) { + assertNull(actual, message); + } else { + assertInstanceOf(Number.class, actual, message); + assertEquals(expected, ((Number) actual).doubleValue(), 1e-12, message); + } + } + + private static Map row(Double amount, Double cost, Boolean flag) { + return row(amount, cost, flag, null); + } + + private static Map row(Double amount, Double cost, Boolean flag, String status) { + Map result = new LinkedHashMap<>(); + result.put("amount", amount); + result.put("cost", cost); + result.put("flag", flag); + result.put("status", status); + return result; + } + + /** Reads only generated Tua; it neither accepts SQL CASE/NULLIF nor uses production AST nodes. */ + private static final class TuaSubsetEvaluator { + private static final Pattern TOKEN = Pattern.compile( + "\\s*(\\[[^\\]]+\\]|[0-9]+(?:\\.[0-9]+)?(?:[eE][+-]?[0-9]+)?" + + "|'(?:[^']|'')*'|[A-Za-z_][A-Za-z_0-9]*|<>|!=|<=|>=|[().,+*/=<>-])"); + private final List tokens = new ArrayList<>(); + private int position; + + TuaSubsetEvaluator(String expression) { + Matcher matcher = TOKEN.matcher(expression); + int offset = 0; + while (offset < expression.length()) { + if (expression.substring(offset).isBlank()) { + break; + } + if (!matcher.find(offset) || matcher.start() != offset) { + throw new AssertionError("Unexpected Tua token: " + expression.substring(offset)); + } + tokens.add(matcher.group(1)); + offset = matcher.end(); + } + } + + Object evaluate(List> rows) { + Calculation calculation = expression(0); + assertEquals(tokens.size(), position, "Unconsumed generated Tua tokens"); + return calculation.value(rows, Map.of()); + } + + private Calculation expression(int minimum) { + Calculation left = prefix(); + while (position < tokens.size() && precedence(tokens.get(position)) >= minimum) { + String operator = next().toUpperCase(java.util.Locale.ROOT); + Calculation right = expression(precedence(operator) + 1); + Calculation previous = left; + left = (rows, row) -> binary(operator, previous.value(rows, row), right.value(rows, row)); + } + return left; + } + + private Calculation prefix() { + String token = next(); + if (token.equals("(")) { + Calculation result = expression(0); + expect(")"); + return result; + } + if (token.equalsIgnoreCase("IF")) { + List conditions = new ArrayList<>(); + List results = new ArrayList<>(); + do { + conditions.add(expression(0)); + expect("THEN"); + results.add(expression(0)); + } while (take("ELSEIF")); + expect("ELSE"); + Calculation otherwise = expression(0); + expect("END"); + return (rows, row) -> { + for (int i = 0; i < conditions.size(); i++) { + if (Boolean.TRUE.equals(conditions.get(i).value(rows, row))) { + return results.get(i).value(rows, row); + } + } + return otherwise.value(rows, row); + }; + } + if (token.equalsIgnoreCase("NOT") || token.equals("-")) { + Calculation child = expression(token.equals("-") ? 7 : 3); + return (rows, row) -> { + Object value = child.value(rows, row); + return value == null ? null : token.equals("-") ? -number(value) : !(Boolean) value; + }; + } + if (token.equalsIgnoreCase("NULL")) { + return (rows, row) -> null; + } + if (token.equalsIgnoreCase("TRUE") || token.equalsIgnoreCase("FALSE")) { + return (rows, row) -> Boolean.valueOf(token); + } + if (token.startsWith("'")) { + String literal = token.substring(1, token.length() - 1).replace("''", "'"); + return (rows, row) -> literal; + } + if (token.startsWith("[")) { + assertEquals("[orders]", token); + expect("."); + String field = next(); + assertTrue(field.startsWith("[") && field.endsWith("]")); + String name = field.substring(1, field.length() - 1); + return (rows, row) -> { + assertTrue(row.containsKey(name), "Unknown generated field: " + name); + return row.get(name); + }; + } + if (Character.isDigit(token.charAt(0))) { + return (rows, row) -> Double.valueOf(token); + } + expect("("); + List arguments = new ArrayList<>(); + do { + arguments.add(expression(0)); + } while (take(",")); + expect(")"); + return function(token.toUpperCase(java.util.Locale.ROOT), arguments); + } + + private static Calculation function(String name, List arguments) { + if (List.of("SUM", "AVG", "MIN", "MAX", "COUNT", "COUNTD").contains(name)) { + assertEquals(1, arguments.size()); + return (rows, row) -> { + List values = rows.stream() + .map(input -> arguments.getFirst().value(rows, input)) + .filter(java.util.Objects::nonNull).toList(); + if (name.equals("COUNT")) { + return (double) values.size(); + } + if (name.equals("COUNTD")) { + return (double) values.stream() + .map(value -> value instanceof Number && number(value) == 0.0 ? 0.0 : value) + .distinct().count(); + } + if (values.isEmpty()) { + return null; + } + return switch (name) { + case "SUM" -> values.stream().mapToDouble(TuaSubsetEvaluator::number).sum(); + case "AVG" -> values.stream().mapToDouble(TuaSubsetEvaluator::number).average().orElseThrow(); + case "MIN" -> values.stream().mapToDouble(TuaSubsetEvaluator::number).min().orElseThrow(); + case "MAX" -> values.stream().mapToDouble(TuaSubsetEvaluator::number).max().orElseThrow(); + default -> throw new AssertionError(name); + }; + }; + } + assertTrue(List.of("IFNULL", "ISNULL", "ABS", "CEILING", "FLOOR", "ROUND").contains(name), + "Unsupported generated function: " + name); + return (rows, row) -> { + Object value = arguments.getFirst().value(rows, row); + if (name.equals("IFNULL")) { + assertEquals(2, arguments.size()); + return value != null ? value : arguments.get(1).value(rows, row); + } + if (name.equals("ISNULL")) { + assertEquals(1, arguments.size()); + return value == null; + } + if (value == null) { + return null; + } + return switch (name) { + case "ABS" -> Math.abs(number(value)); + case "CEILING" -> Math.ceil(number(value)); + case "FLOOR" -> Math.floor(number(value)); + case "ROUND" -> BigDecimal.valueOf(number(value)).setScale( + arguments.size() == 1 ? 0 : (int) number(arguments.get(1).value(rows, row)), + RoundingMode.HALF_UP).doubleValue(); + default -> throw new AssertionError("Unsupported generated function: " + name); + }; + }; + } + + private static Object binary(String operator, Object left, Object right) { + if (operator.equals("AND")) { + if (Boolean.FALSE.equals(left) || Boolean.FALSE.equals(right)) { + return false; + } + return left == null || right == null ? null : Boolean.TRUE; + } + if (operator.equals("OR")) { + if (Boolean.TRUE.equals(left) || Boolean.TRUE.equals(right)) { + return true; + } + return left == null || right == null ? null : Boolean.FALSE; + } + if (left == null || right == null) { + return null; + } + return switch (operator) { + case "+" -> number(left) + number(right); + case "-" -> number(left) - number(right); + case "*" -> number(left) * number(right); + case "/" -> { + assertTrue(number(right) != 0.0, "Generated expression evaluated an unguarded zero divisor"); + yield number(left) / number(right); + } + case "=" -> equal(left, right); + case "!=", "<>" -> !equal(left, right); + case "<" -> number(left) < number(right); + case "<=" -> number(left) <= number(right); + case ">" -> number(left) > number(right); + case ">=" -> number(left) >= number(right); + default -> throw new AssertionError(operator); + }; + } + + private static boolean equal(Object left, Object right) { + return left instanceof Number && right instanceof Number + ? number(left) == number(right) : left.equals(right); + } + + private static double number(Object value) { + return ((Number) value).doubleValue(); + } + + private static int precedence(String token) { + return switch (token.toUpperCase(java.util.Locale.ROOT)) { + case "OR" -> 1; + case "AND" -> 2; + case "=", "!=", "<>", "<", ">", "<=", ">=" -> 4; + case "+", "-" -> 5; + case "*", "/" -> 6; + default -> -1; + }; + } + + private boolean take(String token) { + if (position < tokens.size() && tokens.get(position).equalsIgnoreCase(token)) { + position++; + return true; + } + return false; + } + + private String next() { + assertTrue(position < tokens.size(), "Unexpected end of generated Tua expression"); + return tokens.get(position++); + } + + private void expect(String token) { + assertEquals(token, next().toUpperCase(java.util.Locale.ROOT)); + } + } + + @FunctionalInterface + private interface Calculation { + Object value(List> rows, Map row); + } +} diff --git a/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricExpressionTranslatorTest.java b/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricExpressionTranslatorTest.java new file mode 100644 index 00000000..0692e89e --- /dev/null +++ b/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricExpressionTranslatorTest.java @@ -0,0 +1,399 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import static org.junit.jupiter.api.Assertions.*; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; +import org.apache.ossie.exception.ConversionException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.ValueSource; + +class MetricExpressionTranslatorTest { + private static final Map TYPES = Map.of( + "amount", "Decimal", "profit", "Decimal", "revenue", "Decimal", "discount", "Decimal", + "quantity", "Integer", "status", "String", "active", "Boolean", "ordered", "Date"); + + private static Map source() { + return Map.of("datasets", List.of(Map.of("name", "orders", "fields", TYPES.entrySet().stream() + .map(entry -> Map.of("name", entry.getKey(), "datatype", entry.getValue())).toList()))); + } + + private static Map target() { + return Map.of("semanticDataObjects", List.of(Map.of("apiName", "orders", "semanticMeasurements", + TYPES.entrySet().stream().map(entry -> Map.of("apiName", entry.getKey(), "dataObjectFieldName", entry.getKey() + "__c", "dataType", + SalesforceDataTypeMapper.toSalesforce(entry.getValue()))).toList()))); + } + + private static Map metric(String dialect, String expression) { + return new LinkedHashMap<>(Map.of("name", "net_value", "expression", Map.of("dialects", + List.of(Map.of("dialect", dialect, "expression", expression))))); + } + + private static String translate(String dialect, String expression) { + var result = MetricExpressionTranslator.translate(metric(dialect, expression), source(), target()); + assertEquals("Number", result.dataType()); + return result.expression(); + } + + static Stream supported() { + return Stream.of( + Arguments.of("SUM(orders.amount)", "SUM([orders].[amount])"), + Arguments.of("AVG(amount)", "AVG([orders].[amount])"), + Arguments.of("MIN(ORDERS.amount)", "MIN([orders].[amount])"), + Arguments.of("MAX(orders.amount)", "MAX([orders].[amount])"), + Arguments.of("COUNT(orders.status)", "COUNT([orders].[status])"), + Arguments.of("COUNT(DISTINCT orders.status)", "COUNTD([orders].[status])"), + Arguments.of("SUM(orders.amount * (1 - orders.discount))", + "SUM(([orders].[amount] * (1 - [orders].[discount])))"), + Arguments.of("SUM(orders.profit) / NULLIF(SUM(orders.revenue), 0)", + "(SUM([orders].[profit]) / (IF (SUM([orders].[revenue]) = 0) THEN NULL ELSE SUM([orders].[revenue]) END))"), + Arguments.of("SUM(CASE WHEN orders.status = 'paid' THEN orders.amount ELSE 0 END)", + "SUM((IF ([orders].[status] = 'paid') THEN [orders].[amount] ELSE 0 END))"), + Arguments.of("COALESCE(SUM(orders.amount), AVG(orders.revenue), 0)", + "IFNULL(SUM([orders].[amount]), IFNULL(AVG([orders].[revenue]), 0))"), + Arguments.of("SUM(CASE WHEN orders.amount IS NOT NULL THEN orders.amount END)", + "SUM((IF (NOT ISNULL([orders].[amount])) THEN [orders].[amount] ELSE NULL END))"), + Arguments.of("ROUND(AVG(ABS(orders.amount)), 2)", "ROUND(AVG(ABS([orders].[amount])), 2)"), + Arguments.of("SUM(CEIL(orders.amount) - FLOOR(orders.amount))", + "SUM((CEILING([orders].[amount]) - FLOOR([orders].[amount])))"), + Arguments.of("1 + 2 * 3 - 4 / 2", "((1 + (2 * 3)) - (4 / 2))"), + Arguments.of(".25 + 1e2", "(0.25 + 100)"), + Arguments.of("SUM(orders.quantity) + -2", "(SUM([orders].[quantity]) + (-2))"), + Arguments.of("SUM(- -orders.amount)", "SUM([orders].[amount])"), + Arguments.of("SUM(+ - - +orders.amount)", "SUM([orders].[amount])"), + Arguments.of("SUM(- + - -orders.amount)", "SUM((-[orders].[amount]))"), + Arguments.of("SUM(orders.amount - -orders.discount)", + "SUM(([orders].[amount] - (-[orders].[discount])))"), + Arguments.of("SUM(orders.amount) / - -SUM(orders.quantity)", + "(SUM([orders].[amount]) / SUM([orders].[quantity]))"), + Arguments.of("SUM((((orders.amount + orders.discount))) * 2)", + "SUM((([orders].[amount] + [orders].[discount]) * 2))"), + Arguments.of("SUM(CASE WHEN NOT NOT NOT orders.active THEN orders.amount ELSE 0 END)", + "SUM((IF (NOT [orders].[active]) THEN [orders].[amount] ELSE 0 END))"), + Arguments.of("SUM(CASE WHEN NOT NOT NOT NOT orders.active THEN orders.amount ELSE 0 END)", + "SUM((IF (NOT (NOT [orders].[active])) THEN [orders].[amount] ELSE 0 END))"), + Arguments.of("SUM(CASE WHEN orders.status = '- - NOT NOT NOT ((x))' THEN - -orders.amount ELSE 0 END)", + "SUM((IF ([orders].[status] = '- - NOT NOT NOT ((x))') THEN [orders].[amount] ELSE 0 END))"), + Arguments.of("CASE WHEN MAX(orders.status) = 'z' THEN 1 ELSE 0 END", + "(IF (MAX([orders].[status]) = 'z') THEN 1 ELSE 0 END)") + ); + } + + @ParameterizedTest + @MethodSource("supported") + void compilesComposedSqlInBothDialects(String expression, String expected) { + assertEquals(expected, translate("SNOWFLAKE", expression)); + assertEquals(expected, translate("ANSI_SQL", expression)); + // Every generated expression can be read through the bounded TABLEAU path. + assertEquals(expected, translate("TABLEAU", expected)); + } + + @Test + void notUsesSqlPrecedenceAndPreservesThreeValuedPredicates() { + assertEquals("SUM((IF ((NOT ([orders].[amount] = 0)) OR ([orders].[active] AND ISNULL([orders].[discount])))" + + " THEN 1 ELSE 0 END))", + translate("SNOWFLAKE", "SUM(CASE WHEN NOT orders.amount = 0 OR orders.active AND orders.discount IS NULL THEN 1 ELSE 0 END)")); + } + + @Test + void nestedCasesAndElseifRetainBranchOrder() { + String sql = "SUM(CASE WHEN orders.active THEN CASE WHEN orders.amount > 0 THEN 2 ELSE 3 END WHEN orders.status = 'paid' THEN 4 END)"; + String expected = "SUM((IF [orders].[active] THEN (IF ([orders].[amount] > 0) THEN 2 ELSE 3 END)" + + " ELSEIF ([orders].[status] = 'paid') THEN 4 ELSE NULL END))"; + assertEquals(expected, translate("SNOWFLAKE", sql)); + assertEquals(expected, translate("TABLEAU", expected)); + } + + @Test + void identifiersAreBoundToSemanticNamesAndStringContentsAreNotReferences() { + assertEquals("SUM([orders].[amount])", translate("SNOWFLAKE", "SUM(\"ORDERS\".\"AMOUNT\")")); + assertEquals("SUM([orders].[amount])", translate("ANSI_SQL", "SUM([orders].[amount])")); + assertEquals("SUM((IF ([orders].[status] = 'missing.field O''Brien') THEN 1 ELSE 0 END))", + translate("SNOWFLAKE", "SUM(CASE WHEN orders.status = 'missing.field O''Brien' THEN 1 ELSE 0 END)")); + assertEquals("SUM((IF ([orders].[status] = 'paid') THEN 1 ELSE 0 END))", + translate("TABLEAU", "SUM(IF [orders].[status] = \"paid\" THEN 1 ELSE 0 END)")); + } + + static Stream unsupported() { + return Stream.of( + Arguments.of("SUM(orders.missing)", "Unknown field"), + Arguments.of("SUM(missing.amount)", "Unknown dataset"), + Arguments.of("SUM(db.orders.amount)", "physical source paths"), + Arguments.of("SUM(orders.amount__c)", "Unknown field"), + Arguments.of("SUM(orders.amount) + orders.amount", "mix aggregate"), + Arguments.of("CASE WHEN orders.active THEN SUM(orders.amount) ELSE 0 END", "mix aggregate"), + Arguments.of("SUM(AVG(orders.amount))", "nested aggregate"), + Arguments.of("SUM(orders.status)", "numeric"), + Arguments.of("AVG(orders.active)", "numeric"), + Arguments.of("SUM(CASE WHEN orders.amount THEN 1 ELSE 0 END)", "BOOLEAN"), + Arguments.of("SUM(CASE WHEN orders.active THEN orders.amount ELSE 'zero' END)", "incompatible types"), + Arguments.of("COALESCE(SUM(orders.amount), '0')", "incompatible types"), + Arguments.of("NULLIF(SUM(orders.amount), '0')", "incompatible types"), + Arguments.of("SUM(orders.amount) AND TRUE", "BOOLEAN"), + Arguments.of("SUM(CASE WHEN orders.active > TRUE THEN 1 ELSE 0 END)", "ordered comparison"), + Arguments.of("orders.amount + 1", "unaggregated"), + Arguments.of("MAX(orders.status)", "must be numeric"), + Arguments.of("TRUE", "must be numeric"), + Arguments.of("COUNT(*)", "COUNT(*)"), + Arguments.of("COUNT(1)", "declared field"), + Arguments.of("COUNT(orders.quantity + 1)", "counting expressions"), + Arguments.of("SUM(1)", "establish its dataset"), + Arguments.of("SUM(DISTINCT orders.amount)", "DISTINCT"), + Arguments.of("COUNT(DISTINCT orders.amount, orders.quantity)", "expects 1"), + Arguments.of("ROUND(SUM(orders.amount), 2, 'HALF_TO_EVEN')", "expects 1 to 2"), + Arguments.of("ROUND(SUM(orders.amount), .5)", "integer literal"), + Arguments.of("ROUND(SUM(orders.amount), orders.quantity)", "integer literal"), + Arguments.of("CEIL(SUM(orders.amount), 2)", "expects 1"), + Arguments.of("FLOOR(SUM(orders.amount), 2)", "expects 1"), + Arguments.of("ABS()", "expects 1"), + Arguments.of("COALESCE(SUM(orders.amount))", "expects 2"), + Arguments.of("NULLIF(SUM(orders.amount))", "expects 2"), + Arguments.of("MEDIAN(orders.amount)", "unsupported function"), + Arguments.of("CAST(orders.amount AS DECIMAL)", "unsupported SQL expression CastExpression"), + Arguments.of("SUM(YEAR(orders.ordered))", "unsupported function"), + Arguments.of("SUM(LENGTH(orders.status))", "unsupported function"), + Arguments.of("SUM(orders.amount) OVER ()", "unsupported SQL expression AnalyticExpression"), + Arguments.of("SUM(orders.amount) FILTER (WHERE orders.active)", "unsupported SQL expression AnalyticExpression"), + Arguments.of("{ FIXED : SUM(orders.amount) }", "unsupported character"), + Arguments.of("SELECT SUM(orders.amount)", "unexpected token"), + Arguments.of("SUM(orders.amount);", "unsupported character"), + Arguments.of("SUM(orders.amount) -- comment", "comments"), + Arguments.of("SUM(orders.amount) /* comment */", "comments"), + Arguments.of("SUM(orders.amount", "expected )"), + Arguments.of("SUM(orders.amount) trailing", "unexpected token"), + Arguments.of("1.2.3", "invalid numeric"), + Arguments.of("1e", "invalid numeric"), + Arguments.of("1e1000000000", "too large"), + Arguments.of("SUM(CASE WHEN orders.status = 'unterminated THEN 1 END)", "unterminated"), + Arguments.of("SUM(orders.amount) % 2", "unsupported character"), + Arguments.of("SUM(orders.amount) ^ 2", "unsupported character"), + Arguments.of("SUM(orders.amount) || 'x'", "unsupported character"), + Arguments.of("SUM(orders.amount) / 0", "division by literal zero"), + Arguments.of("net_value + 1", "Unknown field") + ); + } + + @ParameterizedTest + @MethodSource("unsupported") + void rejectsWithMetricNameAndActionableReason(String expression, String reason) { + ConversionException exception = assertThrows(ConversionException.class, + () -> translate("SNOWFLAKE", expression)); + assertTrue(exception.getMessage().contains("Metric 'net_value'"), exception.getMessage()); + assertTrue(exception.getMessage().contains(reason), exception.getMessage()); + } + + @ParameterizedTest + @ValueSource(strings = {"SUM(orders.amount)", "SUM([orders].[missing])", "COUNT(DISTINCT [orders].[amount])", + "COALESCE(SUM([orders].[amount]), 0)", "CASE WHEN TRUE THEN 1 ELSE 0 END", "not a formula", + "SUM([orders].[amount]) OVER ()", "SUM([orders].[amount] + [orders].[status])"}) + void tableauCannotBypassParsingOrReferenceValidation(String expression) { + assertThrows(ConversionException.class, () -> translate("TABLEAU", expression)); + } + + @Test + void selectedDialectFailureNeverFallsBack() { + Map metric = metric("TABLEAU", "SUM([orders].[missing])"); + metric.put("expression", Map.of("dialects", List.of( + Map.of("dialect", "ANSI_SQL", "expression", "SUM(orders.amount)"), + Map.of("dialect", "TABLEAU", "expression", "SUM([orders].[missing])")))); + assertThrows(ConversionException.class, () -> MetricExpressionTranslator.translate(metric, source(), target())); + } + + @Test + void rejectsDuplicateDialectAndMissingOrEmptyExpressions() { + Map metric = metric("SNOWFLAKE", "SUM(orders.amount)"); + Map entry = Map.of("dialect", "SNOWFLAKE", "expression", "SUM(orders.amount)"); + metric.put("expression", Map.of("dialects", List.of(entry, entry))); + assertThrows(ConversionException.class, () -> MetricExpressionTranslator.translate(metric, source(), target())); + for (Object expression : List.of(Map.of(), Map.of("dialects", List.of()), + Map.of("dialects", List.of(Map.of("dialect", "BIGQUERY", "expression", "1"))))) { + metric.put("expression", expression); + assertThrows(ConversionException.class, () -> MetricExpressionTranslator.translate(metric, source(), target())); + } + assertThrows(ConversionException.class, () -> translate("SNOWFLAKE", " ")); + } + + @Test + void outputDatatypeMustAgreeWithTheFormula() { + Map metric = metric("SNOWFLAKE", "AVG(orders.quantity)"); + for (String datatype : List.of("Integer", "String", "Boolean", "Date", "Opaque", "Time")) { + metric.put("datatype", datatype); + assertThrows(ConversionException.class, () -> MetricExpressionTranslator.translate(metric, source(), target())); + } + metric.put("datatype", "Decimal"); + assertEquals("Number", MetricExpressionTranslator.translate(metric, source(), target()).dataType()); + Map countMetric = metric("SNOWFLAKE", "COUNT(orders.quantity)"); + countMetric.put("datatype", "Integer"); + assertEquals("Number", MetricExpressionTranslator.translate(countMetric, source(), target()).dataType()); + Map nullMetric = metric("SNOWFLAKE", "NULL"); + nullMetric.put("datatype", "Decimal"); + assertEquals("NULL", MetricExpressionTranslator.translate(nullMetric, source(), target()).expression()); + } + + @Test + void limitsNestingAndExpansionWithoutStackOverflow() { + assertThrows(ConversionException.class, () -> translate("SNOWFLAKE", "(".repeat(200) + "1" + ")".repeat(200))); + assertThrows(ConversionException.class, () -> translate("SNOWFLAKE", "-".repeat(200) + "1")); + assertThrows(ConversionException.class, () -> translate("SNOWFLAKE", "1+".repeat(5000) + "1")); + assertThrows(ConversionException.class, () -> translate("SNOWFLAKE", "1".repeat(32769))); + String formula = "SUM(orders.amount)"; + for (int i = 0; i < 20; i++) formula = "NULLIF(" + formula + ", 0)"; + String expanded = formula; + assertThrows(ConversionException.class, () -> translate("SNOWFLAKE", expanded)); + } + + @Test + void redundantParenthesesStayWithinTheBoundedFastParsingPath() { + assertTimeoutPreemptively(Duration.ofSeconds(5), () -> { + for (String dialect : List.of("SNOWFLAKE", "ANSI_SQL")) { + for (int depth : List.of(20, 60, 126)) { + String grouped = "(".repeat(depth) + "orders.amount" + ")".repeat(depth); + assertEquals("SUM([orders].[amount])", translate(dialect, "SUM(" + grouped + ")")); + assertEquals("SUM([orders].[amount])", translate(dialect, + "(".repeat(depth) + "SUM(orders.amount)" + ")".repeat(depth))); + } + assertEquals("SUM([orders].[amount])", translate(dialect, "SUM(" + "- ".repeat(128) + "orders.amount)")); + assertEquals("SUM((IF (NOT (NOT [orders].[active])) THEN 1 ELSE 0 END))", + translate(dialect, "SUM(CASE WHEN " + "NOT ".repeat(128) + "orders.active THEN 1 ELSE 0 END)")); + } + }); + } + + @Test + void normalizationRetainsOriginalLimitsAndOperandTypes() { + for (String dialect : List.of("SNOWFLAKE", "ANSI_SQL")) { + for (String expression : List.of("SUM(" + "- ".repeat(129) + "orders.amount)", + "SUM(CASE WHEN " + "NOT ".repeat(129) + "orders.active THEN 1 ELSE 0 END)", + "SUM(" + "(".repeat(128) + "orders.amount" + ")".repeat(128) + ")", + "SUM(- -orders.status)", "COUNT(- -orders.amount)", + "SUM(CASE WHEN NOT NOT NOT NOT orders.amount THEN 1 ELSE 0 END)", + "SUM(CASE WHEN orders.amount IS NOT NOT NOT NULL THEN 1 ELSE 0 END)", + "SUM(CASE WHEN orders.amount IS NOT NOT NOT NOT NULL THEN 1 ELSE 0 END)", + "COUNT(((DISTINCT orders.amount)))", + "SUM(orders.amount)) + (1", "SUM(--orders.amount)")) { + assertTrue(assertThrows(ConversionException.class, () -> translate(dialect, expression), expression) + .getMessage().contains("Metric 'net_value':")); + } + } + } + + @Test + void redundantGroupingCannotTurnTuplesIntoFunctionArguments() { + for (String dialect : List.of("SNOWFLAKE", "ANSI_SQL")) { + for (String function : List.of("COALESCE", "ROUND")) { + for (int depth : List.of(1, 20)) { + String tuple = "(".repeat(depth) + "SUM(orders.amount), 2" + ")".repeat(depth); + assertTrue(assertThrows(ConversionException.class, + () -> translate(dialect, function + "(" + tuple + ")")) + .getMessage().contains("tuple-valued function arguments")); + } + } + assertEquals("IFNULL(SUM([orders].[amount]), 2)", + translate(dialect, "COALESCE((SUM(orders.amount)), 2)")); + } + } + + @Test + void temporalAggregatesCanBeUsedInNumericPredicates() { + assertEquals("(IF (MIN([orders].[ordered]) = MAX([orders].[ordered])) THEN 1 ELSE 0 END)", + translate("SNOWFLAKE", "CASE WHEN MIN(orders.ordered) = MAX(orders.ordered) THEN 1 ELSE 0 END")); + } + + @ParameterizedTest + @ValueSource(strings = {"SUM(ALL orders.amount)", "SUM(UNIQUE orders.amount)", + "SUM(orders.amount IGNORE NULLS)", "SUM(orders.amount RESPECT NULLS)", + "SUM(orders.amount) IGNORE NULLS", "SUM(orders.amount LIMIT 1)", + "SUM(orders.amount HAVING MAX orders.quantity)", "SUM(orders.amount ORDER BY orders.quantity)", + "SUM(orders.amount) KEEP (DENSE_RANK LAST ORDER BY orders.quantity)", + "SUM(orders.amount).attribute", "private_schema.SUM(orders.amount)", "\"SUM\"(orders.amount)", + "SUM(CASE orders.amount WHEN 1 THEN 2 ELSE 0 END)", "N'prefixed'", + "orders.status ISNULL", "orders.status NOTNULL", "PRIOR orders.amount = orders.quantity", + "!orders.active", "(SELECT amount FROM orders)", "orders.amount IN (1, 2)", + "SUM(orders.amount) AS alias", "orders.amount(+) = orders.quantity"}) + void parserAcceptanceNeverDiscardsUnsupportedSqlModifiers(String expression) { + assertThrows(ConversionException.class, () -> translate("SNOWFLAKE", expression), expression); + } + + @ParameterizedTest + @ValueSource(strings = {"CEIL(AVG(orders.amount))", "FLOOR(AVG(orders.amount))", + "ROUND(AVG(orders.amount))", "ROUND(AVG(orders.amount), 0)", "ROUND(AVG(orders.amount), -2)"}) + void integralRoundingSatisfiesAnIntegerMetricDeclaration(String expression) { + Map metric = metric("SNOWFLAKE", expression); + metric.put("datatype", "Integer"); + String output = MetricExpressionTranslator.translate(metric, source(), target()).expression(); + metric.put("expression", Map.of("dialects", List.of(Map.of("dialect", "TABLEAU", "expression", output)))); + assertEquals(output, MetricExpressionTranslator.translate(metric, source(), target()).expression()); + } + + @Test + void positiveRoundingPrecisionDoesNotClaimAnIntegralResult() { + Map metric = metric("SNOWFLAKE", "ROUND(AVG(orders.amount), 2)"); + metric.put("datatype", "Integer"); + assertThrows(ConversionException.class, () -> MetricExpressionTranslator.translate(metric, source(), target())); + assertEquals("0.12345678901234567890123456789", translate("SNOWFLAKE", "0.12345678901234567890123456789")); + } + + @Test + void quotedDotsAndEscapesPreserveFieldReferenceBoundaries() { + Map source = Map.of("datasets", List.of(Map.of("name", "ORDER.ITEMS", "fields", + List.of(Map.of("name", "NET REVENUE", "datatype", "Decimal"))))); + Map target = Map.of("semanticDataObjects", List.of(Map.of("apiName", "ORDER.ITEMS", + "semanticMeasurements", List.of(Map.of("apiName", "NET REVENUE", "dataType", "Number", + "dataObjectFieldName", "net__c"))))); + for (String dialect : List.of("SNOWFLAKE", "ANSI_SQL")) { + assertEquals("SUM([ORDER.ITEMS].[NET REVENUE])", MetricExpressionTranslator.translate( + metric(dialect, "SUM(\"ORDER.ITEMS\".\"NET REVENUE\")"), source, target).expression()); + var field = (MetricExpression.Field) SqlMetricExpressionParser.parse("\"a\"\"b.c\".\"d\"\"e\"", dialect); + assertEquals(List.of(new MetricFieldResolver.Identifier("a\"b.c", true), + new MetricFieldResolver.Identifier("d\"e", true)), field.parts()); + } + assertThrows(IllegalArgumentException.class, () -> SqlMetricExpressionParser.parse("[a.b].[c]]d]", "ANSI_SQL")); + } + + @Test + void separateAggregatesRequireConnectedDatasets() { + Map twoSources = Map.of("relationships", List.of(Map.of("name", "orders_returns", + "from", "orders", "to", "returns", "from_columns", List.of("amount"), "to_columns", List.of("amount"))), + "datasets", List.of( + Map.of("name", "orders", "fields", List.of(Map.of("name", "amount", "datatype", "Decimal"))), + Map.of("name", "returns", "fields", List.of(Map.of("name", "amount", "datatype", "Decimal"))))); + Map twoTargets = new LinkedHashMap<>(Map.of("semanticDataObjects", List.of( + Map.of("apiName", "orders", "semanticMeasurements", List.of(Map.of("apiName", "amount", "dataObjectFieldName", "amount__c", "dataType", "Number"))), + Map.of("apiName", "returns", "semanticMeasurements", List.of(Map.of("apiName", "amount", "dataObjectFieldName", "amount__c", "dataType", "Number")))))); + Map metric = metric("SNOWFLAKE", "SUM(orders.amount) - SUM(returns.amount)"); + assertThrows(ConversionException.class, () -> MetricExpressionTranslator.translate(metric, twoSources, twoTargets)); + twoTargets.put("semanticRelationships", List.of(Map.of("apiName", "orders_returns", "isEnabled", true, + "leftSemanticDefinitionApiName", "orders", "rightSemanticDefinitionApiName", "returns", "criteria", + List.of(Map.of("leftSemanticFieldApiName", "amount", "rightSemanticFieldApiName", "amount"))))); + assertEquals("(SUM([orders].[amount]) - SUM([returns].[amount]))", + MetricExpressionTranslator.translate(metric, twoSources, twoTargets).expression()); + assertThrows(ConversionException.class, () -> MetricExpressionTranslator.translate( + metric("SNOWFLAKE", "SUM(orders.amount - returns.amount)"), twoSources, twoTargets)); + } +} diff --git a/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricFieldResolverTest.java b/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricFieldResolverTest.java new file mode 100644 index 00000000..bb9d64fc --- /dev/null +++ b/converters/salesforce/src/test/java/org/apache/ossie/converter/MetricFieldResolverTest.java @@ -0,0 +1,331 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ossie.converter; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Set; +import org.apache.ossie.converter.MetricFieldResolver.Identifier; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +class MetricFieldResolverTest { + + @Test + void acceptsDatasetsConnectedByAnExportedRelationshipInEitherDirection() { + MetricFieldResolver resolver = graphResolver(List.of( + relationship("Orders", "Customers")), "Orders", "Customers"); + resolver.validateDatasets(Set.of("Orders", "Customers")); + resolver.validateDatasets(Set.of("Orders")); + resolver.validateDatasets(Set.of()); + } + + @Test + void acceptsFactDatasetsConnectedThroughAnIntermediateSharedDimension() { + MetricFieldResolver resolver = graphResolver(List.of( + relationship("Orders", "Customers"), relationship("Returns", "Customers")), + "Orders", "Returns", "Customers"); + resolver.validateDatasets(Set.of("Orders", "Returns")); + } + + @Test + void rejectsDisconnectedDatasetsAndRelationshipsThroughMissingTargets() { + MetricFieldResolver resolver = graphResolver(List.of( + relationship("Orders", "MissingCustomers"), relationship("Returns", "MissingCustomers")), + "Orders", "Returns"); + IllegalArgumentException error = assertThrows(IllegalArgumentException.class, + () -> resolver.validateDatasets(Set.of("Orders", "Returns"))); + assertTrue(error.getMessage().contains("disconnected datasets Orders, Returns"), error.getMessage()); + assertTrue(error.getMessage().contains("declare supported relationships"), error.getMessage()); + } + + @Test + void bindsQualifiedAndUniqueUnqualifiedSqlNamesToExportedApiNames() { + MetricFieldResolver resolver = resolver("Orders", "revenue", "Decimal", "Currency"); + assertEquals("[Orders].[revenue]", resolver.resolve(sql("oRders", "REVENUE"), false).expression()); + assertEquals("[Orders].[revenue]", resolver.resolve(sql("revenue"), false).expression()); + assertEquals("Orders", resolver.resolve(sql("revenue"), false).dataset()); + assertEquals("Decimal", resolver.resolve(sql("revenue"), false).datatype()); + } + + @Test + void normalizesQuotedSqlIdentifiersAsSpecified() { + MetricFieldResolver resolver = resolver("orders", "revenue", "Decimal", "Number"); + assertEquals("[orders].[revenue]", resolver.resolve( + List.of(new Identifier("ORDERS", true), new Identifier("REVENUE", true)), false).expression()); + assertError(resolver, List.of(new Identifier("orders", true), new Identifier("revenue", true)), + false, "Unknown dataset"); + } + + @Test + void resolvesQuotedDeclarationsWithoutRenamingExportedObjects() { + MetricFieldResolver resolver = resolver("\"Order Items\"", "\"Unit \"\"Price\"\"\"", "Decimal", "Number"); + assertEquals("[\"Order Items\"].[\"Unit \"\"Price\"\"\"]", resolver.resolve( + List.of(new Identifier("Order Items", true), new Identifier("Unit \"Price\"", true)), false) + .expression()); + assertError(resolver, sql("ORDER ITEMS", "UNIT PRICE"), false, "Unknown dataset"); + } + + @Test + void tableauReferencesUseExactApiNamesAndRequireDataset() { + MetricFieldResolver resolver = resolver("Orders", "revenue", "Integer", "Number"); + assertEquals("[Orders].[revenue]", resolver.resolve(sql("Orders", "revenue"), true).expression()); + assertEquals("[Orders].[revenue]", resolver.resolve(sql("orders", "revenue"), false).expression()); + assertError(resolver, sql("orders", "revenue"), true, "Unknown dataset"); + assertError(resolver, sql("Orders", "Revenue"), true, "Unknown field"); + assertError(resolver, sql("revenue"), true, "must use [dataset].[field]"); + } + + @Test + void rejectsPhysicalSourceNamesAndUndeclaredFields() { + MetricFieldResolver resolver = resolver("Orders", "revenue", "Decimal", "Number"); + assertError(resolver, sql("Orders__dll", "revenue"), false, "Unknown dataset"); + assertError(resolver, sql("Orders", "revenue__c"), false, "Unknown field"); + assertError(resolver, sql("warehouse", "Orders", "revenue"), false, "physical source paths"); + assertError(resolver, List.of(), false, "must name a declared field"); + } + + @Test + void rejectsAmbiguousUnqualifiedFieldsAcrossDatasets() { + Map orders = dataset("Orders", field("amount", "Decimal")); + Map returns = dataset("Returns", field("amount", "Decimal")); + MetricFieldResolver resolver = new MetricFieldResolver( + Map.of("datasets", List.of(orders, returns)), + Map.of("semanticDataObjects", List.of(targetDataset("Orders", targetField("amount", "Number")), + targetDataset("Returns", targetField("amount", "Number"))))); + assertError(resolver, sql("amount"), false, "Ambiguous field"); + assertEquals("[Orders].[amount]", resolver.resolve(sql("Orders", "amount"), false).expression()); + } + + @Test + void rejectsDuplicateDatasetDeclarationsEvenWhenOnlyOneHasTheField() { + MetricFieldResolver resolver = new MetricFieldResolver(Map.of("datasets", List.of( + dataset("Orders", field("amount", "Decimal")), dataset("ORDERS", field("other", "Decimal")))), + Map.of()); + assertError(resolver, sql("Orders", "amount"), false, "Ambiguous dataset"); + assertError(resolver, sql("amount"), false, "Ambiguous dataset"); + } + + @Test + void rejectsDuplicateFieldDeclarations() { + MetricFieldResolver resolver = new MetricFieldResolver(Map.of("datasets", List.of( + dataset("Orders", field("amount", "Decimal"), field("AMOUNT", "Decimal")))), Map.of()); + assertError(resolver, sql("Orders", "amount"), false, "Ambiguous field"); + } + + @Test + void checksThatDeclaredFieldsWereActuallyExportedAsDirectFields() { + Map source = Map.of("datasets", List.of(dataset("Orders", field("amount", "Decimal")))); + MetricFieldResolver missingDataset = new MetricFieldResolver(source, Map.of()); + assertError(missingDataset, sql("Orders", "amount"), false, "was not exported"); + + MetricFieldResolver calculatedField = new MetricFieldResolver(source, Map.of( + "semanticDataObjects", List.of(targetDataset("Orders")), + "semanticCalculatedDimensions", List.of(Map.of("apiName", "amount", "expression", "1 + 2")))); + assertError(calculatedField, sql("Orders", "amount"), false, "calculated or omitted fields"); + } + + @ParameterizedTest + @ValueSource(strings = {"profit+tax", "profit-tax", "profit=tax", "1", "TRUE", "(profit)", + "Orders.profit", "\"profit\"", "[profit]", "SUM(profit)", "profit;tax", ""}) + void rejectsExpressionsMisclassifiedAsPhysicalFields(String binding) { + Map target = new LinkedHashMap<>(targetField("revenue", "Number")); + if (binding.isEmpty()) target.remove("dataObjectFieldName"); + else target.put("dataObjectFieldName", binding); + MetricFieldResolver resolver = new MetricFieldResolver( + Map.of("datasets", List.of(dataset("Orders", field("revenue", "Decimal")))), + Map.of("semanticDataObjects", List.of(targetDataset("Orders", target)))); + assertError(resolver, sql("Orders", "revenue"), false, "revenue"); + } + + @Test + void rejectsDuplicateExportedObjectsAndFieldsAcrossKinds() { + Map source = Map.of("datasets", List.of(dataset("Orders", field("amount", "Decimal")))); + Map target = targetDataset("Orders", targetField("amount", "Number")); + MetricFieldResolver duplicateDatasets = new MetricFieldResolver(source, + Map.of("semanticDataObjects", List.of(target, target))); + assertError(duplicateDatasets, sql("amount"), false, "Ambiguous exported Salesforce dataset"); + + Map duplicateFields = Map.of("apiName", "Orders", + "semanticDimensions", List.of(targetField("amount", "Number")), + "semanticMeasurements", List.of(targetField("amount", "Number"))); + MetricFieldResolver resolver = new MetricFieldResolver(source, + Map.of("semanticDataObjects", List.of(duplicateFields))); + assertError(resolver, sql("amount"), false, "Ambiguous exported Salesforce field"); + } + + @Test + void infersMissingPortableTypeOnlyFromKnownExportedType() { + MetricFieldResolver resolver = new MetricFieldResolver( + Map.of("datasets", List.of(dataset("Orders", Map.of("name", "amount")))), + Map.of("semanticDataObjects", List.of(targetDataset("Orders", targetField("amount", "Currency"))))); + assertEquals("Decimal", resolver.resolve(sql("amount"), false).datatype()); + } + + @Test + void rejectsUnknownOrConflictingTypes() { + assertError(resolver("Orders", "amount", "String", "Number"), sql("amount"), false, + "use compatible field types"); + assertError(resolver("Orders", "amount", "Opaque", "Geo"), sql("amount"), false, + "no supported datatype"); + MetricFieldResolver resolver = new MetricFieldResolver( + Map.of("datasets", List.of(dataset("Orders", Map.of("name", "amount")))), + Map.of("semanticDataObjects", List.of(targetDataset("Orders", targetField("amount", "Geo"))))); + assertError(resolver, sql("amount"), false, "no supported datatype"); + } + + @ParameterizedTest + @ValueSource(strings = {"bad[field", "bad]field", "bad\nfield", "bad\tfield", "bad\u0000field"}) + void rejectsUnrepresentableExportedNames(String name) { + MetricFieldResolver resolver = resolver("Orders", name, "Decimal", "Number"); + assertError(resolver, sql("Orders", name), true, "cannot be represented safely"); + } + + private static MetricFieldResolver resolver(String dataset, String field, String sourceType, String targetType) { + return new MetricFieldResolver(Map.of("datasets", List.of(dataset(dataset, field(field, sourceType)))), + Map.of("semanticDataObjects", List.of(targetDataset(dataset, targetField(field, targetType))))); + } + + @Test + void disabledRelationshipsDoNotConnectDatasets() { + Map disabled = new LinkedHashMap<>(relationship("Orders", "Returns")); + disabled.put("isEnabled", false); + MetricFieldResolver resolver = graphResolver(List.of(disabled), "Orders", "Returns"); + assertThrows(IllegalArgumentException.class, + () -> resolver.validateDatasets(java.util.Set.of("Orders", "Returns"))); + } + + @ParameterizedTest + @ValueSource(strings = {"changed endpoint", "swapped keys", "empty criteria", "missing criterion", + "missing join field", "formula join", "missing enabled", "missing declaration", "duplicate declaration"}) + void unverifiedRelationshipsCannotEstablishConnectivity(String defect) { + Map valid = relationship("Orders", "Customers"); + Map changed = new LinkedHashMap<>(valid); + List> declarations = List.of(sourceRelationship(valid)); + switch (defect) { + case "changed endpoint" -> changed.put("rightSemanticDefinitionApiName", "Returns"); + case "swapped keys" -> changed.put("criteria", List.of(criterion("id", "tenant"), criterion("tenant", "id"))); + case "empty criteria" -> changed.put("criteria", List.of()); + case "missing criterion" -> changed.put("criteria", List.of(criterion("id", "id"))); + case "missing join field" -> { + changed.put("criteria", List.of(criterion("missing", "id"), criterion("tenant", "tenant"))); + Map source = new LinkedHashMap<>(sourceRelationship(valid)); + source.put("from_columns", List.of("missing", "tenant")); + declarations = List.of(source); + } + case "formula join" -> changed.put("criteria", List.of( + Map.of("leftSemanticFieldApiName", "id", "rightSemanticFieldApiName", "id", "leftFieldType", "Formula"), + criterion("tenant", "tenant"))); + case "missing enabled" -> changed.remove("isEnabled"); + case "missing declaration" -> declarations = List.of(); + case "duplicate declaration" -> declarations = List.of(sourceRelationship(valid), sourceRelationship(valid)); + default -> throw new AssertionError(defect); + } + MetricFieldResolver resolver = graphResolver(declarations, List.of(changed), "Orders", "Customers", "Returns"); + Set referenced = Set.of("Orders", defect.equals("changed endpoint") ? "Returns" : "Customers"); + IllegalArgumentException error = assertThrows(IllegalArgumentException.class, + () -> resolver.validateDatasets(referenced), defect); + assertTrue(error.getMessage().contains("disconnected"), error.getMessage()); + } + + @Test + void computedPhysicalJoinKeyCannotEstablishConnectivity() { + Map edge = relationship("Orders", "Customers"); + Map computed = new LinkedHashMap<>(targetField("id", "Number")); + computed.put("dataObjectFieldName", "profit+tax"); + MetricFieldResolver resolver = new MetricFieldResolver(Map.of( + "datasets", List.of(dataset("Orders", field("id", "Integer"), field("tenant", "Integer")), + dataset("Customers", field("id", "Integer"), field("tenant", "Integer"))), + "relationships", List.of(sourceRelationship(edge))), Map.of( + "semanticDataObjects", List.of(targetDataset("Orders", computed, targetField("tenant", "Number")), + targetDataset("Customers", targetField("id", "Number"), targetField("tenant", "Number"))), + "semanticRelationships", List.of(edge))); + assertThrows(IllegalArgumentException.class, + () -> resolver.validateDatasets(Set.of("Orders", "Customers"))); + } + + private static MetricFieldResolver graphResolver(List> relationships, String... datasets) { + return graphResolver(relationships.stream().map(MetricFieldResolverTest::sourceRelationship).toList(), + relationships, datasets); + } + + private static MetricFieldResolver graphResolver(List> declarations, + List> relationships, String... datasets) { + return new MetricFieldResolver(Map.of( + "datasets", java.util.Arrays.stream(datasets) + .map(name -> dataset(name, field("id", "Integer"), field("tenant", "Integer"))).toList(), + "relationships", declarations), Map.of( + "semanticDataObjects", java.util.Arrays.stream(datasets) + .map(name -> targetDataset(name, targetField("id", "Number"), targetField("tenant", "Number"))).toList(), + "semanticRelationships", relationships)); + } + + private static Map relationship(String left, String right) { + return Map.of("apiName", left + "_to_" + right, "leftSemanticDefinitionApiName", left, + "rightSemanticDefinitionApiName", right, "isEnabled", true, + "criteria", List.of(criterion("id", "id"), criterion("tenant", "tenant"))); + } + + private static Map criterion(String left, String right) { + return Map.of("leftSemanticFieldApiName", left, "rightSemanticFieldApiName", right); + } + + private static Map sourceRelationship(Map target) { + return Map.of("name", target.get("apiName"), "from", target.get("leftSemanticDefinitionApiName"), + "to", target.get("rightSemanticDefinitionApiName"), + "from_columns", List.of("id", "tenant"), "to_columns", List.of("id", "tenant")); + } + + private static List sql(String... parts) { + return java.util.Arrays.stream(parts).map(part -> new Identifier(part, false)).toList(); + } + + @SafeVarargs + private static Map dataset(String name, Map... fields) { + return Map.of("name", name, "source", name + "__dll", "fields", List.of(fields)); + } + + private static Map field(String name, String datatype) { + return Map.of("name", name, "datatype", datatype, "expression", Map.of("dialects", List.of( + Map.of("dialect", "ANSI_SQL", "expression", "physical_column__c")))); + } + + @SafeVarargs + private static Map targetDataset(String name, Map... fields) { + return Map.of("apiName", name, "semanticMeasurements", List.of(fields)); + } + + private static Map targetField(String name, String datatype) { + return Map.of("apiName", name, "dataType", datatype, "dataObjectFieldName", "physical_column__c"); + } + + private static void assertError(MetricFieldResolver resolver, List parts, + boolean tableau, String expected) { + IllegalArgumentException error = assertThrows(IllegalArgumentException.class, + () -> resolver.resolve(parts, tableau)); + assertTrue(error.getMessage().contains(expected), error.getMessage()); + } +}