[core-spec]: add FILTER (WHERE ...) aggregate modifier

[christianeu-db]

Summary

Adds the SQL-standard FILTER (WHERE <predicate>) modifier to aggregate expressions in Ossie_SQL_2026, extending the filtered-aggregation capability already marked REQUIRED by the expression language. A metric MAY attach a per-aggregate predicate, for example SUM(store_sales.ss_ext_sales_price) FILTER (WHERE store_sales.ss_ext_sales_price > 0).

The change is confined to core-spec/expression_language.md — specifically its Conditional Aggregations (REQUIRED) section. It adds no new schema node and no new field to Metric, Field, or Dataset. Filtered aggregation was already listed as REQUIRED in the expression language ("All aggregations should support filtered aggregation"), but the only spelled-out form was the CASE WHEN rewrite.

The change is additive and backward compatible. Every existing model remains valid, and the CASE WHEN form remains supported.

This PR was motivated by the "declare a filter on a metric, then layer filters on top" pain point from discussion #342. Query-time layering itself is enabled via MEASURE() and is defined here for single-aggregate metrics; see Composition.

Motivation

Many metrics include a filter as a core part of their definitions: "valid sales only", "domestic customers", "completed orders". Ossie metrics expose a single expression, so today filter logic has to be folded into the aggregate with CASE WHEN:

metrics:
  - name: valid_sales
    expression:
      dialects:
        - dialect: ANSI_SQL
          expression: "SUM(CASE WHEN store_sales.ss_ext_sales_price > 0 THEN store_sales.ss_ext_sales_price ELSE 0 END)"
  - name: valid_profit
    expression:
      dialects:
        - dialect: ANSI_SQL
          expression: "SUM(CASE WHEN store_sales.ss_ext_sales_price > 0 THEN store_sales.ss_net_profit ELSE 0 END)"

Three problems follow from this:

1. The predicate is buried inside the aggregate. A reader, a validator, or a consumer that wants to know "what does this metric filter on" has to parse the CASE body. The filter has no explicit syntactic role.
2. A consumer cannot layer an additional predicate onto an existing metric expression. A consumer that wants "valid sales, but only for the Toys category" has no way to add a predicate to an existing metric. It must author a new CASE expression that repeats both predicates.
3. The CASE ... ELSE 0 idiom is subtly wrong for some aggregates. It works for SUM, but AVG, COUNT, MIN, and MAX need ELSE NULL (and careful reasoning about what a substituted zero does to the result), which is a common source of incorrect metrics.

FILTER (WHERE ...) addresses these as a composable building block. On its own it fixes problems 1 and 3: the predicate has an explicit syntactic role rather than being encoded indirectly in a CASE expression, and its empty-input behavior is the aggregate's own well-defined behavior rather than a hand-chosen sentinel. Problem 2 — layering a context filter onto an existing metric — is solved not by FILTER alone but by MEASURE(m) FILTER (WHERE ...) where the Relational Query Interface (#354) is available; this iteration defines that layering only for single-aggregate metrics (see Composition).

Prior art

FILTER (WHERE ...) is defined in SQL:2003 as the <filter clause> on a set function — optional feature T612 — the same standard the Ossie expression language is based on. Engines split into those that support it natively and those that require a CASE rewrite:

Native FILTER (WHERE ...)	Requires CASE rewrite
PostgreSQL, DuckDB, Apache Spark	Snowflake, BigQuery

Engines in the second group have a mechanical, well-known CASE-rewrite, so a converter can always lower the clause to a supported form. This is the same portability posture the expression language already takes for constructs such as APPROX_COUNT_DISTINCT.

Why an expression-language extension rather than a first-class filters node. Discussion #342 opened with a proposal for a top-level filters object. This PR intentionally addresses aggregate-local filtering through the existing expression language: a predicate on an aggregate is already within the expression language's remit and needs no new node or resolution rule. Named / shared filters, cross-dataset filters, and parameterized filters require additional modeling and resolution semantics and remain out of scope.

Proposed change

Extend core-spec/expression_language.md to define the aggregate FILTER clause. The clause is a postfix modifier on an aggregate expression:

<aggregate_expression>(<args>) FILTER (WHERE <predicate>)

FILTER MAY modify an aggregate expression. This includes ordinary aggregate function calls (SUM, COUNT, ...) and, where the Relational Query Interface (#354) is supported, MEASURE(...).

The clause is added under the existing Conditional Aggregations (REQUIRED) section, alongside the DISTINCT modifier and the CASE-based form the section already describes, so that filtered aggregation has one explicit, portable spelling. The default dialect (Ossie_SQL_2026) supports it; dialect-specific expressions MAY continue to use their engine's native form.

Because the language's Not Supported table lists a bare WHERE clause as unsupported ("Use filter property instead"), the spec text will state explicitly that FILTER (WHERE ...) is a postfix aggregate modifier and is not the unsupported standalone WHERE clause, to avoid a reading conflict.

Aggregate FILTER semantics

The FILTER (WHERE <predicate>) clause has the semantics of the SQL <filter clause> (SQL:2003 optional feature T612). FILTER is applied to the aggregate's input, not as a query WHERE, so it never removes output groups.

The one behavior worth calling out against the CASE rewrite: when no row matches, the aggregate sees a genuinely empty input and returns its correct empty-input value automatically, whereas CASE ... ELSE <sentinel> forces the author to hand-pick that value.

Composition with metrics and MEASURE()

The following rules are normative and specific to Ossie. They apply only where the Relational Query Interface (#354) is available, since they concern layering a FILTER clause onto an existing metric via MEASURE(). Ossie has not yet defined general measure composition, so this iteration defines contextual filtering only for the single-aggregate case; the composed / multi-aggregate case is left to a future measure-composition proposal.

1. Intrinsic vs contextual filters. A FILTER clause written into a metric definition is its intrinsic filter. A FILTER clause layered onto that metric at query time via MEASURE() is a contextual filter. Both are ordinary FILTER (WHERE ...) clauses; the terms name where the clause was written, not two different constructs.
2. Single-aggregate composition by AND. This iteration defines contextual filtering only for a metric whose definition is a single aggregate expression that MAY carry an intrinsic FILTER. For such a metric m — say m = SUM(x) FILTER (WHERE p) — a reference MEASURE(m) FILTER (WHERE q) MUST evaluate exactly as SUM(x) FILTER (WHERE p AND q). When m has no intrinsic filter, the effective predicate is just q. A contextual filter MUST NOT weaken or replace an intrinsic filter. Because composition is by AND, a contextual predicate that contradicts the intrinsic one simply produces empty filtered input, which returns the aggregate's ordinary empty-input value — no special-casing is required.
3. MEASURE() interaction. MEASURE(m) behaves as an aggregate, so MEASURE(m) FILTER (WHERE q) MUST apply q per rule 2.

Scope and resolution constraints

The following is an Ossie modeling restriction for this iteration, not part of SQL FILTER semantics:

• Single dataset only. A FILTER (WHERE ...) predicate MUST reference only fields of the same dataset as the base aggregate. A predicate that references another dataset is out of scope for this iteration and MUST be rejected. This avoids coupling filtered aggregation to relationship-path resolution, which is being specified separately.

Examples

Intrinsic filter in a metric definition (uses FILTER alone; no MEASURE() required):

metrics:
  - name: northern_sales
    expression:
      dialects:
        - dialect: ANSI_SQL
          expression: "SUM(store_sales.ss_ext_sales_price) FILTER (WHERE store_sales.ss_store_region = 'North')"

Reusing a boolean field on the dataset as the predicate (reference an existing field instead of re-authoring the condition). Here store_sales.is_valid_sale is a boolean field on the store_sales dataset — the only kind of reuse this iteration supports. The same pattern should extend to a boolean dimension in the future, once Ossie defines how to build measures from dimensions:

metrics:
  - name: valid_sales
    expression:
      dialects:
        - dialect: ANSI_SQL
          expression: "SUM(store_sales.ss_ext_sales_price) FILTER (WHERE store_sales.is_valid_sale)"

Contextual filter layered at query time (requires the Relational Query Interface, #354). This is well-defined because northern_sales is a single aggregate; the effective predicate is ss_store_region = 'North' AND ss_category = 'Toys':

SELECT MEASURE(northern_sales) FILTER (WHERE ss_category = 'Toys')
FROM store_sales_metrics

Open questions

• REQUIRED vs RECOMMENDED (discussion point). This PR proposes that native FILTER (WHERE ...) support be REQUIRED of the default dialect. Whether it should instead be only RECOMMENDED — with the CASE-rewrite as the portable fallback for engines and converters that do not implement it natively — is open for discussion. Filtered aggregation is already REQUIRED; this question is specifically about whether the FILTER spelling itself must be supported.

Related Issues

Discussion: #342, #5. Related PRs: #354 (Relational Query Interface / MEASURE()), #246 (Foundational Semantics).

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure — core-spec/expression_language.md
• Spec changes have been discussed on the mailing list or in a linked issue — discussed in [IDEA & PROPOSAL] Ossie Extension: Shared Filters, Shared Dimensions & Metric References #342; a [DISCUSS] thread on dev@ossie.apache.org is still to come
• Breaking changes to the spec are clearly called out in the summary — N/A, purely additive; no schema change

Ontology

• Ontology changes in ontology/ are consistent with spec changes — N/A, no ontology change
• New or modified terms are defined and documented — N/A, no ontology changes

Converters

• Converter logic in converters/ is updated to reflect spec or ontology changes
• New converters include tests under the converter's test directory — N/A, no new converters

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes — the change is itself a spec document
• New features or behaviors are documented with examples where appropriate — worked examples included
• CONTRIBUTING.md is updated if the contribution process changed — N/A

Examples

• examples/ are added or updated for any new spec constructs or converter support — add a filtered-metric example to the TPC-DS model

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files — ASF header added at the top of ossie/filter-where-spec.md
• No third-party dependencies are added without PMC/IPMC approval

AI disclosure

Per the ASF Generative Tooling Guidance, this contribution was prepared with AI assistance. All specification decisions and design choices are mine. I have reviewed and verified every change.

[kayemkim]

Two things from running this against actual engines, plus one from the compiler side.

First, a confirmation. On PostgreSQL 18 and DuckDB 1.5, with a four-row sales table (three COMPLETED rows of 100, 50 and 70, one CANCELLED row of 200), AVG(CASE WHEN status='COMPLETED' THEN amount ELSE 0 END) returns 55.0 and AVG(amount) FILTER (WHERE status='COMPLETED') returns 73.33. That is problem 3 exactly, and it is the same divergence I posted about in discussion #29 with the CASE form, so +1 to giving the predicate its own spelling.

Second, something the text could state so that two consumers do not disagree. The FILTER clause acts inside the aggregate, so it never removes output groups. Grouping the same table by month:

WHERE status='COMPLETED' ... GROUP BY sale_month          -> 2024-01 150 | 2024-03 70
SUM(CASE WHEN ... ELSE 0 END) ... GROUP BY sale_month      -> 2024-01 150 | 2024-02 0    | 2024-03 70
SUM(amount) FILTER (WHERE status='COMPLETED') GROUP BY ... -> 2024-01 150 | 2024-02 NULL | 2024-03 70

Under #354 these are two different, both legitimate results: an intrinsic FILTER in the metric definition keeps the February row with NULL, while a contextual WHERE status='COMPLETED' on MEASURE(revenue) drops it, since 7.3 filters input rows before grouping. What the current text does not say is whether a consumer may lower an intrinsic FILTER to a WHERE when the filtered metric is the only aggregate in the query. That rewrite is tempting (it is what my compiler does today: it carries the predicate separately from the aggregate and emits it as WHERE), and it changes the row set. One sentence saying that an intrinsic FILTER does not eliminate groups, or equivalently that it must not be pushed into WHERE, would settle it. If the intent is to allow the pushdown, saying that is just as useful.

Third, on "Engines without native FILTER (WHERE ...) support MAY lower it to the equivalent CASE form": it is worth spelling the equivalent out. It is SUM(CASE WHEN p THEN x END) with the implicit ELSE NULL, and COUNT(*) FILTER (WHERE p) is COUNT(CASE WHEN p THEN 1 END). The SUM(... ELSE 0 END) example a few lines above in the same section is the form problem 3 is about, so a reader lowering FILTER by analogy to that example would reintroduce the bug. With the lowering written down, REQUIRED looks like the right answer to the open question: it costs a converter one mechanical rewrite.

[christianeu-db]

Thanks for the feedback! Replies below:

Two things from running this against actual engines, plus one from the compiler side.

First, a confirmation. On PostgreSQL 18 and DuckDB 1.5, with a four-row sales table (three COMPLETED rows of 100, 50 and 70, one CANCELLED row of 200), AVG(CASE WHEN status='COMPLETED' THEN amount ELSE 0 END) returns 55.0 and AVG(amount) FILTER (WHERE status='COMPLETED') returns 73.33. That is problem 3 exactly, and it is the same divergence I posted about in discussion #29 with the CASE form, so +1 to giving the predicate its own spelling.

Glad you flagged this divergence in #29 as well. Originally, this PR was motivated by filter composition but the subtle issues with case are another reason to push towards a standard solution.

Second, something the text could state so that two consumers do not disagree. The FILTER clause acts inside the aggregate, so it never removes output groups. Grouping the same table by month:

WHERE status='COMPLETED' ... GROUP BY sale_month          -> 2024-01 150 | 2024-03 70
SUM(CASE WHEN ... ELSE 0 END) ... GROUP BY sale_month      -> 2024-01 150 | 2024-02 0    | 2024-03 70
SUM(amount) FILTER (WHERE status='COMPLETED') GROUP BY ... -> 2024-01 150 | 2024-02 NULL | 2024-03 70

Under #354 these are two different, both legitimate results: an intrinsic FILTER in the metric definition keeps the February row with NULL, while a contextual WHERE status='COMPLETED' on MEASURE(revenue) drops it, since 7.3 filters input rows before grouping. What the current text does not say is whether a consumer may lower an intrinsic FILTER to a WHERE when the filtered metric is the only aggregate in the query. That rewrite is tempting (it is what my compiler does today: it carries the predicate separately from the aggregate and emits it as WHERE), and it changes the row set. One sentence saying that an intrinsic FILTER does not eliminate groups, or equivalently that it must not be pushed into WHERE, would settle it. If the intent is to allow the pushdown, saying that is just as useful.

I can update the description / documentation to cover that the filter acts inside the aggregate / doesn't remove aggregate groups. This was a bit of a tricky balance - we mostly wanted to defer to the standard rather than restate it (to avoid diverging).

In terms of lowering / optimization, that is up to the engine so long as it doesn't change the output. There are certainly some cases where it is safe to push the filter down into a standard WHERE clause but, to your point, that's not always the case.

Third, on "Engines without native FILTER (WHERE ...) support MAY lower it to the equivalent CASE form": it is worth spelling the equivalent out. It is SUM(CASE WHEN p THEN x END) with the implicit ELSE NULL, and COUNT(*) FILTER (WHERE p) is COUNT(CASE WHEN p THEN 1 END). The SUM(... ELSE 0 END) example a few lines above in the same section is the form problem 3 is about, so a reader lowering FILTER by analogy to that example would reintroduce the bug. With the lowering written down, REQUIRED looks like the right answer to the open question: it costs a converter one mechanical rewrite.

Good callout - it's worth putting one example of the different lowering to give a flavor to engines.